As AI-powered search continues to evolve, maintaining accurate and up-to-date information remains a critical challenge. Cloudflare and OpenAI have announced a joint research pilot aimed at improving how AI search engines discover, crawl, and index content across the open web by leveraging real-time network insights.
Cloudflare and OpenAI have partnered on a research initiative designed to improve how AI search systems discover and index information across the internet. The pilot will evaluate whether real-time network signals can help AI systems identify important content updates more efficiently, leading to more accurate and timely responses for users.
Rather than relying solely on traditional web crawling methods, the project investigates how infrastructure-level insights can improve the way AI models understand when and where content changes occur.
"By sharing our sophisticated network signals, we can find a better way to make AI search more efficient and help people get quality answers faster," said Matthew Prince, co-founder and CEO of Cloudflare.
Cloudflare's global connectivity network powers more than 20% of the web, providing visibility into website activity, content updates, and traffic behavior in real time.
The research pilot will examine how signals such as content freshness, page modifications, and traffic quality can improve AI search engines' crawling strategies. By identifying meaningful changes more effectively, AI systems may be able to prioritize relevant content while reducing unnecessary crawling activity.
This approach aims to help AI-powered search platforms deliver more current information without placing additional load on websites.
As part of the collaboration, OpenAI contributes its frontier AI models, large-scale search and answer systems, and real-world user search queries to evaluate the effectiveness of signal-driven indexing.
The companies will study whether infrastructure-based signals enable AI systems to discover updated content more quickly and improve the freshness of generated answers.
"Up-to-date information is important for delivering accurate answers to people using ChatGPT," said Nick Ryder, VP of Research at OpenAI. "Piloting this with Cloudflare will allow us to explore whether network-level insights can help us discover content more efficiently."
The research reflects the growing importance of infrastructure intelligence in supporting next-generation AI search experiences. As AI-generated answers increasingly rely on timely web content, improving content discovery and indexing has become a key area of development for search providers.
Through this pilot, Cloudflare and OpenAI will evaluate whether combining large-scale AI systems with real-time network intelligence can create more efficient crawling strategies and deliver higher-quality search experiences for users across the open web.
Cloudflare, Inc. is the leading connectivity cloud company. It empowers organizations to make their employees, applications and networks faster and more secure everywhere, while reducing complexity and cost. Cloudflare’s connectivity cloud delivers the most full-featured, unified platform of cloud-native products and developer tools, so any organization can gain the control they need to work, develop, and accelerate their business.
Powered by one of the world’s largest and most interconnected networks, Cloudflare blocks billions of threats online for its customers every day. It is trusted by millions of organizations – from the largest brands to entrepreneurs and small businesses to nonprofits, humanitarian groups, and governments across the globe.
Learn more about Cloudflare’s connectivity cloud at cloudflare.com/connectivity-cloud. Learn more about the latest Internet trends and insights at radar.cloudflare.com.
TechIntelPro empowers B2B technology decision-makers with the latest industry trends, deep insights, leadership perspectives and actionable intelligence to drive both organizational and professional milestones, and thrive on the tech pulse.