Cloudflare has introduced new classifications, analytics capabilities, and commercial partnerships aimed at creating a more transparent and sustainable ecosystem for the agentic Internet, bringing together website owners and AI companies to improve content discovery, efficiency, and monetization.
The company said the rapid growth of artificial intelligence is transforming how users access information and interact with online services. With automated agents and bots now accounting for more than half of all web requests, Cloudflare believes the Internet requires new standards that balance AI innovation with content ownership and creator compensation.
The new tools are designed to give website owners greater visibility and control over how AI systems access their content. Businesses will be able to determine whether their data can be used for AI training, AI-powered agents, or search discovery, helping them protect intellectual property while maintaining visibility across emerging AI platforms.
Cloudflare said many website owners want their content to be discoverable by AI systems but are concerned about AI models using their work without compensation. The company aims to address this challenge by enabling clearer classifications of AI crawlers, improving transparency, and creating opportunities for publishers and creators to monetize their content.
“Last year we provided site owners with transparency and control over what bots access their content, and we are thrilled with the benefits it has had to the ecosystem,” said Matthew Prince, co-founder and CEO of Cloudflare. “Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge.”
New Rules for AI Crawlers
Cloudflare is proposing new default classifications for AI crawlers to help separate different types of automated access, including search, AI agent activity, and model training.
The company said mixed-use crawlers that combine multiple purposes without providing clear controls create challenges for both content owners and transparent AI companies. According to Cloudflare, many businesses are forced to choose between remaining discoverable through AI systems and allowing unrestricted access to their content.
Over the next two months, Cloudflare will engage with website owners and AI companies to refine its approach. Beginning September 15, 2026, new customers and new websites will have defaults that allow search access while blocking AI training and agent usage for pages containing advertisements unless customers choose otherwise.
The same changes will apply to existing free customers who have not modified their settings by the deadline.
Analytics to Support AI-Era Content Strategies
Cloudflare is also introducing the Attribution Business Insights dashboard, designed to help businesses understand how AI systems consume their content and measure the value generated by AI-driven traffic.
The dashboard will provide insights into AI crawler activity, content usage, and referral traffic from AI platforms, helping publishers and businesses make more informed decisions around AI visibility and licensing opportunities.
As content discovery shifts from traditional search engines to AI-powered answer engines, Cloudflare highlighted the growing importance of Answer Engine Optimization (AEO). Similar to search engine optimization, AEO focuses on understanding how frequently content is referenced in AI-generated responses and how prominently it appears across different AI models.
Improving AI Search Efficiency and Creator Compensation
Cloudflare is also working to make AI crawling more efficient by reducing unnecessary requests for unchanged content. The company said more than 50% of AI crawler traffic is spent repeatedly fetching pages that have not changed, creating additional costs for publishers and AI providers.
Through its network visibility, Cloudflare is developing signals that can help AI companies identify when content has changed and needs to be retrieved, reducing unnecessary crawling while improving response accuracy.
The company is also evolving its Pay Per Crawl initiative into Pay Per Use, enabling publishers to receive compensation when their content directly contributes value to AI-generated results.
Cloudflare is working with AI companies including Ceramic.ai and You.com to develop models where publishers can be paid when their content appears in AI search results or is accessed by AI agents. The initiative aims to create a scalable compensation framework beyond individual licensing agreements between large publishers and AI companies.
Building the Foundation for the Agentic Economy
Cloudflare said the latest initiatives build on its broader efforts over the past year to create a more balanced AI ecosystem. These include AI Crawl Control, partnerships with creator platforms such as beehiiv, and the development of Web Bot Auth, an open framework designed to help AI agents verify their identity when interacting with websites and online services.
The company is also working with payment providers to establish secure transaction frameworks for AI-powered commerce, enabling autonomous shopping agents to interact safely with merchants without exposing sensitive credentials.
With AI agents expected to become a major interface for online discovery and commerce, Cloudflare said its goal is to create an Internet ecosystem based on transparency, discoverability, and fair compensation.
The company believes these new capabilities will help organizations, creators, and AI companies collaborate more effectively while ensuring the next generation of the Internet develops in a sustainable and trusted manner.
