.jpg)
Fastly's latest Threat Insights Report reveals that automated bots now constitute 29% of all web traffic, with the majority classified as unwanted, driving organizations across key industries to intensify their bot management strategies. The analysis highlights the dual-edged nature of bots, from essential AI crawlers to malicious headless bots, and their significant impact on operations, revenue, and security.
Bots account for 29% of all web traffic, with about 25% of total traffic being unwanted bots.
Bot requests increased by 2% quarter-over-quarter, representing billions of additional requests.
AI crawlers from Meta and OpenAI dominate their categories, comprising 60% and 68% of traffic respectively.
Headless bots targeting Financial Services and Commerce made up 89% of such traffic in Q3.
Industries are blocking more "wanted" bots (like AI crawlers) over data usage and revenue concerns.
The report underscores the need for advanced detection to manage bot risks and opportunities.
The report delineates a complex bot ecosystem where even wanted automation presents challenges. While AI crawlers and fetchers are critical for indexing, organizations are increasingly blocking these requests due to concerns over data scraping and diminished referral traffic. Notably, only 1% of users click through to source websites after viewing an AI summary, directly impacting Media and Entertainment publishers.
A significant threat emerges from sophisticated headless bots, which simulate human interaction at machine speed. In Q3, 89% of headless bot traffic targeted the Financial Services and Commerce sectors, aiming to exploit transactions and data. This trend underscores the critical need for advanced detection capabilities that can differentiate between malicious automation and legitimate headless browser use without disrupting software delivery.
Bot traffic distribution reveals sector-specific pressures. The Commerce and Public Sector sectors received the highest volume of crawler traffic (88% and 96% respectively), driven by demand for real-time data like pricing and inventory. Meanwhile, Education saw the highest fetcher volume (77%). These patterns force organizations to balance the operational benefits of bots with the need to protect competitive data and infrastructure.
The findings advocate for a proactive and nuanced approach to bot management. As bots become a critical business issue impacting visibility, revenue, and security, organizations must implement sophisticated strategies to safeguard their infrastructure, optimize legitimate automation, and navigate the competitive challenges of the AI era.
About Fastly
Fastly’s powerful and programmable edge cloud platform helps the world’s top brands deliver online experiences that are fast, safe, and engaging through edge compute, delivery, security, and observability offerings that improve site performance, enhance security, and empower innovation at global scale. Compared to other providers, Fastly’s powerful, high-performance, and modern platform architecture empowers developers to deliver secure websites and apps with rapid time-to-market and demonstrated, industry-leading cost savings. Organizations around the world trust Fastly to help them upgrade the internet experience, including Reddit, Neiman Marcus, Universal Music Group, and SeatGeek.
TechIntelPro empowers B2B technology decision-makers with the latest industry trends, deep insights, leadership perspectives and actionable intelligence to drive both organizational and professional milestones—and thrive on the tech pulse.