Cloudflare, a leading connectivity cloud company, has introduced a groundbreaking policy to block AI crawlers from accessing website content without explicit permission or compensation by default. This initiative, effective immediately, empowers website owners to control how AI companies use their content, marking a significant shift toward a sustainable internet ecosystem that prioritizes content creators’ rights.
Cloudflare blocks AI crawlers by default, requiring explicit permission for access.
Website owners can decide which AI crawlers to allow based on their purpose.
Introduces "Pay Per Crawl" model for publishers to monetize content access.
Over 1 million customers already use Cloudflare’s AI crawler blocking option.
Supported by major publishers like Condé Nast, Gannett, and The Atlantic.
Aims to protect creators while fostering responsible AI innovation.
Cloudflare’s new policy addresses the growing issue of AI crawlers scraping content without driving traffic back to original sources, disrupting the traditional internet model where search engines generate ad revenue for creators. By requiring AI companies to obtain permission and specify whether crawlers are used for training, inference, or search, Cloudflare gives website owners unprecedented control over their intellectual property. This move aims to preserve the incentive for creating high-quality content, crucial for a vibrant internet.
As part of this initiative, Cloudflare has launched the "Pay Per Crawl" feature, allowing select publishers to charge AI companies for content access. This model creates a transparent marketplace where creators can set terms and pricing, turning unmonetized data usage into new revenue streams. Currently available to a limited group of publishers, the feature is poised to redefine the economic dynamics of content usage in the AI era.
The policy has garnered support from over 30 major publishers and technology companies, including ADWEEK, The Associated Press, and Reddit. “Cloudflare’s innovative approach to block AI crawlers is a game-changer for publishers and sets a new standard for how content is respected online,” said Roger Lynch, CEO of Condé Nast. “This is a critical step toward creating a fair value exchange on the Internet that protects creators, supports quality journalism and holds AI companies accountable.” Similarly, Steve Huffman, Reddit’s co-founder and CEO, emphasized, “AI companies, search engines, researchers, and anyone else crawling sites have to be who they say they are. And any platform on the web should have a say in who is taking their content for what.”
Cloudflare, which manages 20% of web traffic and handles trillions of daily requests, leverages its advanced bot management solutions to distinguish between human users and AI crawlers. Since introducing a one-click AI crawler blocking option in September 2024, over one million customers have adopted it. The new default setting ensures that every new domain on Cloudflare starts with AI crawler restrictions, streamlining the process for content protection.
Cloudflare’s permission-based model, coupled with its development of a new protocol for AI bot authentication, sets a precedent for a fairer digital ecosystem. By balancing the needs of content creators and AI innovators, Cloudflare is paving the way for a sustainable internet where quality content thrives and creators are fairly compensated.
Cloudflare, Inc. is the leading connectivity cloud company on a mission to help build a better Internet. It empowers organizations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare’s connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools, so any organization can gain the control they need to work, develop, and accelerate their business.
Powered by one of the world’s largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organizations – from the largest brands to entrepreneurs and small businesses to nonprofits, humanitarian groups, and governments across the globe.
TechIntelPro empowers B2B technology decision-makers with the latest industry trends, deep insights, leadership perspectives and actionable intelligence to drive both organizational and professional milestones—and thrive on the tech pulse.