Cloudflare has revealed plans to block mixed-use crawlers from accessing ad-supported customer websites by default, a move aimed at enhancing publisher control over AI interactions. This decision could significantly affect how major tech companies like Apple, Google, and Microsoft operate their web crawlers.
New Default Settings for Crawlers
Starting September 15, 2026, new customers and existing customers adding new sites will have their settings adjusted to allow search crawling while blocking AI training and agent access to ad-supported pages. This change also applies to free-tier customers who have not modified their settings. Cloudflare emphasizes that this approach ensures that content generating revenue cannot be crawled without explicit permission from the content owners.
Impact on Major Tech Companies
The decision could affect crawlers operated by major tech firms, as both Googlebot and Bingbot serve dual purposes: search indexing and AI training. While these companies offer opt-out mechanisms for AI data harvesting, the mixed-use nature of their crawlers complicates the situation for publishers who fear losing visibility in search results.
Publisher Incentives and Partnerships
In addition to blocking crawlers, Cloudflare is rebranding its “Pay Per Crawl” model to “Pay Per Use,” aiming to reward publishers based on the value their content generates rather than mere access. This initiative includes partnerships with Ceramic.ai and You.com to facilitate payments to publishers whenever their content is utilized in AI-driven searches.
Enhanced Visibility for Publishers
Cloudflare is also introducing a new Business Insights Dashboard, which will provide publishers with detailed analytics on bot activity and traffic generated by AI models. This tool aims to enhance transparency and help publishers better understand the dynamics of their content’s interaction with AI agents.
As the internet increasingly accommodates non-human traffic, Cloudflare’s initiatives reflect a broader trend towards establishing a sustainable ecosystem for content monetization and AI engagement.
This article was produced by NeonPulse.today using human and AI-assisted editorial processes, based on publicly available information. Content may be edited for clarity and style.
