Cloudflare, Inc., the leading connectivity cloud company, announced today new classifications, enhanced analytics, and industry-defining commercial partnerships that bring together site owners and transparent AI companies so that the agentic Internet can flourish. Cloudflare's new tools and integrations help site owners and AI companies optimize for discoverability, efficiency, and monetization. By establishing these new rails, Cloudflare is helping provide the foundation for a healthy, collaborative agentic economy.
Cloudflare introduces new classifications, analytics, and partnerships for agentic Internet.
Automated agents and bots drive more than half of all web requests today.
September 15, 2026 deadline for new default crawler classifications.
Attribution Business Insights dashboard provides visibility into AI bot content consumption.
Pay Per Use evolution pays publishers when content creates value.
Partnerships with Ceramic.ai and You.com for content compensation.
Over the next two months, Cloudflare will engage with the ecosystem, listen to feedback, and run tests to finalize new defaults and classifications with a deadline of September 15, 2026. For new customers and for new sites for existing customers, on September 15, 2026, the defaults will be set to allow for search but block training and agent use for pages with ads. Mixed crawlers that do not give site owners the ability to choose between search, agent use, and training, will be blocked on all pages with ads. These changes aim to level the playing field between AI companies and legacy search companies.
Cloudflare is introducing the new Attribution Business Insights dashboard, stripping away the data asymmetry that has long left content owners negotiating in the dark. For the first time, business stakeholders can clearly see how AI bots consume their content and track how much human traffic specific AI companies actually send back. As discovery shifts from search engines to answer engines, customers need to perfect Answer Engine Optimization (AEO), understanding not just who is crawling, but how your content is cited in AI-generated answers.
Cloudflare is working on two fronts: making AI search more efficient, and paying creators automatically when their work is used. Cloudflare data suggests that over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages. Cloudflare is testing signals with leading AI companies to improve freshness and reduce wasted crawling. Cloudflare is evolving Pay Per Crawl into Pay Per Use, so publishers are paid when their content actually creates value. The company is working with Ceramic.ai and You.com to turn that principle into a working market.
Matthew Prince, co-founder and CEO of Cloudflare, stated: "Now that the majority of traffic on the Internet is non-human, we must go further and act faster so that a sustainable ecosystem can emerge. Cloudflare's new tools and partnerships give website owners increased visibility and commercial opportunities and benefit AI companies that have bots with clear and transparent intent. We hope that our proposed default changes encourage mixed-use crawlers to separate out search from agent use and training."
Tyler Denk, CEO of beehiiv, stated: "We are entering a new era where content discoverability is being completely redefined and independent creators deserve to be at the forefront of that shift. The future is about giving creators control over their content and audience."
Anna Patterson, founder and CEO of Ceramic.ai, added: "To scale the future of AI search, we need a partner with massive reach and a shared commitment to transparency and fair compensation. By bringing our pay-per-query model to their network, we ensure millions of content owners can seamlessly opt in to be compensated every single time their content appears in our search results."
About Cloudflare
Cloudflare, Inc is the leading connectivity cloud company. It empowers organizations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare's connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools. Powered by one of the world's largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organizations globally.
