AI bots are scanning websites not for indexing, but for training models and providing answers without visiting the source. Automated traffic has already surpassed human traffic, so website owners need access control and, if possible, content payment.
⚡ TL;DR
- ✅ Scale: According to Cloudflare Radar, as of early June 2026, bots generate 57.5% of HTML requests compared to 42.5% from humans. The main driver is agentic AI (Tom's Hardware).
- ✅ Purpose: Nearly 80% of AI crawling is for model training, not search. Referrals are minimal: in July 2025, Anthropic made ~38,000 crawls per one referral (Cloudflare).
- ✅ Control: Starting September 15, 2026, Cloudflare will block "mixed-use" crawlers by default on ad-supported pages (TechCrunch), and Pay Per Crawl is transitioning to Pay Per Use (Technology.org).
🎯 Section 1: Introduction to Crawling in the AI Era
Crawling is the automated scanning of websites by bots. Search bots (Googlebot, Bingbot) index pages and send visitors back. AI bots (GPTBot, ClaudeBot, Meta-ExternalAgent) primarily collect data for training LLMs, as well as for RAG and real-time responses. According to Cloudflare, in 2025, training accounted for ~80% of AI crawling, search for 18%, and user actions for 2% (Cloudflare).
The consequence for a website: content ends up in AI responses, but the visitor and ad revenue do not. This is compounded by infrastructure strain: automated traffic already constitutes 57.5% of HTML requests (Tom's Hardware).
The rules of the game are also changing. Starting September 15, 2026, Cloudflare will block "mixed-use" crawlers (those combining search, agents, and training) on ad-supported pages. This applies to new clients, new websites, and free plans (TechCrunch). The payment model is also shifting: instead of paying per page load, publishers will be paid when content is actually used in an AI response (Technology.org).
📚 Section 2: Traditional Crawling - The Basics
Traditional crawling has been the foundation of search indexing since the 1990s. A bot starts from a seed URL (known addresses, sitemap), follows links, extracts text, meta tags, and HTML structure, renders JavaScript, and builds an index. The bot gets access rules from robots.txt and indexes the mobile version of pages (mobile-first).
Example user agents:
- Googlebot:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
- Bingbot:
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)
The key difference from AI bots is the referral traffic. In 2025, Google made between 3.8 and 22.5 crawls per referral to a site (January–July), meaning in the units and tens, not thousands (Cloudflare). The "search visibility for content access" exchange worked. In AI crawling, this exchange is nearly broken.
🔄 Section 3: Evolution to AI Crawlers
Until 2023, crawlers primarily indexed pages for search. After the generative AI boom, bots emerged for training models, for searching in AI assistants, and for real-time user queries. Meta, OpenAI, and Anthropic use separate bots for this, while Google and Microsoft use unified crawlers (Cloudflare). It's these "mixed" bots without separated objectives that Cloudflare is now blocking by default.
Share among AI bots, May 2024 → May 2025 (Cloudflare):
| Bot | 2024 | 2025 |
| GPTBot (OpenAI) | 5% | 30% |
| ClaudeBot (Anthropic) | 27% | 21% |
| Meta-ExternalAgent | — | 19% |
| Amazonbot | 21% | 11% |
| Bytespider (ByteDance) | 42% | 7.2% |
Shares change monthly; check the latest snapshot at Cloudflare Radar AI Insights.
Crawls per referral, July 2025 (Cloudflare):
| Platform | Crawls per 1 Referral |
| Anthropic | ~38,000 (in January it was ~287,000) |
| OpenAI | ~1,100 |
| Perplexity | ~195 |
| Microsoft | ~41 |
| Google | ~5 |
📖 Read also
⚙️ Section 4: How AI Crawlers Work
Technically, an AI crawler functions like any other bot: request, HTML, processing. The difference lies in their purpose and behavior. Cloudflare categorizes bots by their intent: model training, search, or user actions (Cloudflare). By 2026, Cloudflare settings will feature these categories as Search, Agent, and Training (Search Engine Journal).
- Discovery. URLs are sourced from sitemaps, links, and previous crawls. URL selection is less precise than Google's: around 34% of requests from ChatGPT and Claude hit 404s, compared to 8% for Googlebot (Vercel, Dec 2024 data).
- Fetching. An HTTP request with the bot's user-agent. According to Vercel's research, none of the major AI crawlers (OpenAI, Anthropic, Meta, ByteDance, Perplexity) executed JavaScript. Exceptions include Gemini (via Googlebot infrastructure) and Applebot. Content that appears only client-side is not visible to them.
- Bot Type. Approximately 80% of AI bot traffic comes from crawlers that collect data proactively. The remaining 20% are fetchers, which retrieve pages in real-time based on user requests (Fastly, April–July 2025).
- Scale. Fetcher peaks reached 39,000 requests per minute (Fastly). Over 50% of AI bot crawling traffic involves re-fetching unchanged pages (Cloudflare data via TechCrunch). For comparison, on the Vercel network over a month, GPTBot made 569 million requests, Claude 370 million, and Googlebot 4.5 billion (Vercel, Dec 2024).
📊 Section 5: Statistics and Traffic Impact (2025–2026)
- Bot Share: 57.5% of HTML requests versus 42.5% from humans (Cloudflare, June 2026).
- Crawler Growth (May 2024 → May 2025): GPTBot +305% requests, Googlebot +96% (Cloudflare).
- Who's Crawling: According to Fastly (April–July 2025, Fastly network), Meta accounts for 52% of AI crawler traffic, Google 23%, and OpenAI 20%. At Cloudflare, Meta-ExternalAgent's share is 19% (May 2025). Methodologies differ, hence the varying figures.
- AI Visitors: In early 2025, there was 1 AI visit for every 200 human visits; by Q2, it was 1 for every 50 (TollBit). By the end of 2025, it was 1 for every 31 (TechRadar).
- Shift from Training to Responses: From Q2 to Q4 2025, training crawls decreased by approximately 15%, RAG bots increased by ~33%, and AI indexers by ~59% (Digiday via TollBit).
- Referrals: Click-through rates from AI tools to publisher sites dropped from 0.8% (Q2) to 0.27% (Q4 2025) (Digiday via TollBit). Google referrals to news sites in March 2025 were ~9% lower than in January (Cloudflare).
- AI Overviews and CTR: Organic CTR for informational queries with AIO dropped by 61%—from 1.76% (June 2024) to 0.61% (September 2025) (Seer Interactive). However, a subsequent study showed a reversal: 1.3% in December 2025 to 2.4% in February 2026 (Seer Interactive, 2026). Extrapolating 2025 trends to 2026 should be done cautiously.
- Geography: Over 90% of Meta and Google's crawler traffic is directed at North American sites (Fastly). Consequently, data for models is skewed towards English-language content.
⚖️ Section 6: Ethical Challenges
The main issue is asymmetry: bots consume content, but return few visitors (see table in Section 3). Voluntary guidelines haven't resolved this.
- robots.txt Ignored. The percentage of AI requests bypassing robots.txt rose from 3.3% (Q4 2024) to 13.26% (Q2 2025) (TollBit), reaching about 30% by Q4 2025, with ChatGPT-User at 42% (Digiday). According to a TollBit report, updated terms from major AI companies permit bots acting on user requests to disregard robots.txt (Digiday).
- Lawsuits. On December 5, 2025, The New York Times filed a lawsuit against Perplexity (TechCrunch). The suit alleges, among other things, that Perplexity's crawlers intentionally circumvented technical protection measures, including robots.txt. Earlier, on June 11, 2025, Disney and NBCUniversal sued Midjourney (AfroTech).
- Standards and Payment. RSL 1.0 (official specification released December 10, 2025) extends robots.txt: it allows specifying conditions and payment separately for training and AI search (RSL). It doesn't stop bots on its own but is supported by Cloudflare, Akamai, and Fastly (The Verge). Cloudflare is evolving Pay Per Crawl into Pay Per Use (Section 7).
- llms.txt is Not a Permission Mechanism. This is a proposed index file for models. Google has stated it does not support it (Stan Ventures), and site access is managed by robots.txt.
Practical takeaway: signals like robots.txt and RSL are requests; enforcement requires CDN/WAF level implementation.
🛡️ Section 7: Technical Issues and Solutions
Problems. Fetcher peaks of up to 39,000 requests/min can resemble a non-malicious DDoS attack (Fastly). Over half of crawling traffic involves re-fetching unchanged pages (TechCrunch). Some bots ignore robots.txt (Section 6).
1. robots.txt by Bot Purpose
OpenAI and Anthropic have categorized their bots by purpose (OpenAI, Anthropic):
| Company | Training | Search | User Request |
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Google | Google-Extended (token) | Googlebot | — |
To remain visible in AI responses but prevent content from being used for training, block only the training tokens:
# Model Training - Block
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Search and User Requests - Allow
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Google-Extended is a control token in robots.txt, not a separate crawler (Cloudflare). It prohibits usage for Gemini but doesn't affect Search. Googlebot crawls for AI Overviews and AI Mode as well (TechCrunch). For "user request" bots, robots.txt is not a guarantee (Section 6).
2. Cloudflare AI Crawl Control
Cloudflare categorizes bots into Search, Agent, and Training, allowing separate control for each category, even on the free plan (Search Engine Journal). Starting September 15, 2026, for new domains, new sites, and free accounts that haven't altered settings, Training and Agent will be blocked on pages with ads, while Search will be allowed (TechCrunch).
Trap: Mixed crawlers (Googlebot, Applebot, Bingbot) are evaluated under the strictest rule. Any Training block, including the old "Block AI bots" toggle, will affect them too (Search Engine Journal). Check your settings and logs: Googlebot should not receive a 403.
3. Payment: Pay Per Use and RSL
Pay Per Crawl (a marketplace where sites charge AI bots) is evolving into Pay Per Use: payment is based on when content provides value in an AI response, not just for being fetched. Initial partners include Ceramic.ai and You.com (TechCrunch). An alternative not tied to a CDN is RSL 1.0: machine-readable licensing terms that integrate with robots.txt (The Register).
4. If You Want AI to See Your Content
- Serve important content in the initial HTML (SSR/SSG), as most AI crawlers do not execute JavaScript (Vercel, 2024 data—verify with your logs).
- Keep redirects and sitemaps up-to-date to prevent bots from hitting 404s (Vercel).
5. Monitoring
Count requests by user-agent in your logs and cross-reference IPs with published ranges (OpenAI publishes gptbot.json). Separately verify that Googlebot and Bingbot are not receiving 403s or challenges.
🔮 Section 8: The Future of Crawling
Below are not hypotheses, but changes that have already been announced or are in effect.
- Bots are being separated by purpose. Cloudflare has given AI companies until September 15, 2026, to separate search crawlers from training and agent crawlers, and is blocking mixed ones on pages with ads (TechCrunch).
- Pay-per-use. Pay Per Crawl is turning into Pay Per Use (TechCrunch). RSL 1.0 is developing alongside it – an open standard for licensing terms (The Register). Publishers are already licensing content directly: according to the WSJ, Amazon pays the New York Times $20–25 million per year (eWeek). In the first quarter of 2026, USA Today Co. for the first time called its AI licensing revenue "notable" (Digiday).
- Regulation in the EU. The AI Act obligations for general-purpose models have been in effect since August 2, 2025. The voluntary Code of Practice directly names robots.txt as a protocol to be respected (Jones Day). The Commission has been studying which other opt-out protocols to recognize (CEPIC), so the list may expand.
- Agents are going beyond robots.txt. Bots already account for 57.5% of HTML requests, and the driver is agent traffic (Tom's Hardware). OpenAI notes that robots.txt rules may not apply to ChatGPT-User (OpenAI). Control is shifting to CDN/WAF and authentication.
- Search. Semrush predicted in 2025 that for digital marketing and SEO topics, visitors from AI search might outnumber those from traditional search by early 2028 (Semrush). This is a forecast, and actual CTRs are already fluctuating (Section 5).
What to do now: separate bots by purpose in robots.txt, ensure Googlebot doesn't receive 403 errors, monitor logs and Cloudflare Radar, and decide whether to enable payment (Pay Per Use, RSL) (see Section 7).
❓ Frequently Asked Questions (FAQ)
- Does GPTBot respect robots.txt?
GPTBot and OAI-SearchBot do; they can be managed separately in robots.txt. ChatGPT-User is not guaranteed: OpenAI states that since actions are initiated by the user, robots.txt rules may not apply (OpenAI documentation). To completely block access, a server-level or WAF rule is needed.
- How do AI crawlers affect a website?
Load: fetcher peaks reached 39,000 requests per minute (Fastly), and over 50% of AI bot crawling traffic is re-downloading unchanged pages (TechCrunch). Returns: CTRs from AI tools to publisher sites in Q4 2025 were 0.27% (Digiday via TollBit). The actual share of bots on your site will be shown in the logs.
- How can I monetize my content?
Direct deals currently involve major publishers (e.g., NYT and Amazon). For others, there's Cloudflare's Pay Per Use (publishers enable it voluntarily; early partners include Ceramic.ai and You.com) and RSL 1.0 licensing terms (RSL).
- Who are the leaders among AI crawlers?
It depends on the source and period. Cloudflare (May 2025, among AI bots): GPTBot — 30%, ClaudeBot — 21%, Meta-ExternalAgent — 19% (Cloudflare). Fastly (April–July 2025, Fastly network): Meta — 52% of AI crawler traffic, Google — 23%, OpenAI — 20%. Check Cloudflare Radar for current data.
- Should I block all AI bots?
Not necessarily. Block training tokens (GPTBot, ClaudeBot, Google-Extended), but leave search and "on user request" bots if you want to appear in AI responses (see Section 7). There's a risk: Cloudflare's blocking of the Training category, including the old "Block AI bots" toggle, also affects mixed crawlers like Googlebot (Search Engine Journal).
✅ Conclusions
AI crawling differs from search crawling in its purpose and return: primarily training and responses without site visits. Bots already account for 57.5% of HTML requests, and robots.txt is merely a request, not a mandate. Therefore, it's beneficial to follow a simple scheme:
- separate bots by purpose in robots.txt (training, search, user request);
- check CDN settings: Googlebot and Bingbot should not receive 403 errors;
- for visibility in AI, serve important content in the initial HTML;
- monitor logs and Cloudflare Radar, and address payment issues through Pay Per Use or RSL.
Read about structured data for your site in the materials on Schema.org and markup testing.
🔥 Read also