Pay-per-Crawl by Cloudflare in 2025–2026: Should You Sell Your Content to AI Bots?

🤖 Imagine: your website is visited monthly by millions of guests who never click anything, don't view ads, and leave no referrals. 📄 They just copy all your content and vanish. ⚠️ This isn't science fiction—it's **GPTBot**, **ClaudeBot**, **CC-Net**, and dozens of other AI crawlers that already generate up to **80% of all bot traffic**. 💸 And the worst part—you get nothing for it.

🛡️ Cloudflare offered a radical answer in 2025: Want my content for AI training? **Pay for every request.** 💰 That's how **Pay-per-Crawl** was born.

💭 I believe that for large publishers, this is potentially tens of thousands of dollars in new monthly revenue. 📉 For most medium and small sites, it's currently more about risk and cost than real money.

Article Content:

• 📌 1. From Free AI Vampirism to Paid Access in 2025–2026
• 📌 2. How Pay-per-Crawl Works
• 📌 3. Pros and Cons: An Honest Breakdown for Different Sites
• 📌 4. Case Study Reality: Three Scenario Calculations
• 📌 5. Pay-per-Crawl vs. Alternatives 2025–2026
• 📌 6. 2026 Forecast and What to Do Now
• ❓ Frequently Asked Questions (FAQ)
• ✅ Conclusion

🎯 1. From Free AI Vampirism to Paid Access in 2025–2026

😠 AI companies make billions on our content, and all we get are server traffic bills—a typical publisher complaint in 2025.

I wrote more about this in the article:

How Crawling Works in the Age of AI.

🤖 Imagine a situation where your website, full of unique content, becomes a free "**smorgasbord**" for AI crawlers: GPTBot from OpenAI, ClaudeBot from Anthropic, or CC-Net from Common Crawl. These **AI bots** scan pages thousands of times, extracting data for model training, but in return, they bring **zero referral traffic**. This is a classic example of **AI crawler monetization**—where AI companies monetize your content, and you pay for the server load. 📊 In 2024, the **crawl-to-referral ratio** (number of crawler requests per one referral visit) reached absurd levels: OpenAI—1700:1, Anthropic—73,000:1, Common Crawl—31,000:1. For comparison, Googlebot stayed at 14:1 during the same period—still unfair, but at least with referrals.

📡 According to **Cloudflare Radar** as of December 2025, the situation has only worsened. **AI traffic** now accounts for **50–80% of all bot traffic** on websites, with peak growth of **86%** in the second half of 2024. Overall **AI bot traffic** grew by 18% from May 2024 to May 2025, and individual crawlers like GPTBot showed a jump of **305%**. 📉 These aren't just numbers: for news sites, the **drop in organic traffic** reached **25–38% year-over-year (YoY)**, mostly due to AI Overviews in Google and direct answers from AI assistants. ⚖️ Publishers like The New York Times or Reuters have already filed lawsuits against OpenAI and Anthropic for "stealing" content without compensation, demanding billions in **AI data licensing** agreements.

• 🚨 **50–80% of all bot traffic are AI crawlers:** Of these, GPTBot takes 35.5%, ClaudeBot—11.2%, and Meta-ExternalAgent—up to 19% in 2025.
• 📈 **AI traffic grew by 86% in the second half of 2024 and continues to rise:** Peak spikes in November 2024 and March–April 2025, with +32% YoY in April.
• 💸 **Organic traffic drop for news sites—25–38% YoY:** AI bots "bite off" referrals, leaving only the load—up to 16% invalid traffic from GPTBot and ClaudeBot.

🧛‍♂️ This "free AI vampirism" is destroying the web ecosystem: publishers lose revenue from ads and subscriptions, while AI companies like OpenAI (with a $157 billion valuation in 2025) build empires on others' content. 🔐 The rise of **AI crawl control** is key: 14% of top domains already use `robots.txt` to block **AI bots**. 💰 That's why in July 2025, Cloudflare launched **Pay-per-Crawl**—the world's first mechanism for **micropayments for crawling**, where AI bots pay $0.01–$0.05 per request, or receive an **HTTP 402 Payment Required** response. 🛡️ This isn't just a tool for **AI content protection** but a step towards a fair data economy, where **AI crawler monetization** becomes the norm. 🚀 In 2026, with the public launch, this could be a game-changer for SMBs and large publishers—but are you ready for **paid access for AI**?

➡️ In the following sections, we'll break down how this works in practice, the pros/cons, and alternatives like **llms.txt** or **Really Simple Licensing**, so you can choose an **AI crawl control** strategy for your site now.

🔧 2. How Pay-per-Crawl Works at the End of 2025

⚙️ The technical core is simple: An AI bot sends a signed header with intent to pay → the server responds with either 200 OK with content, or 402 Payment Required with the price per request.

🔬 As of December 2025, **Cloudflare's Pay-per-Crawl** is no longer just an announcement but a full-fledged tool in private beta with several key updates that make it the most convenient solution for **AI crawler monetization**. Here's how it looks technically and practically:

🎛️ Primary Operating Modes

• 🔄 **Reactive mode (default)** — The bot makes a regular request → Cloudflare automatically responds with 402 with your price if the bot is not authenticated.
• ⚡ **Proactive mode** — The bot first contacts the **Discovery API** (a December 2025 novelty), finds out the price, and immediately sends a signed payment intent. The fastest and most effective route.

Key Technical Features in December 2025

• 🚀 **Discovery API** — Bots can automatically get your price list via /.well-known/ai-crawl-pricing.json. This is critically important for future **agentic AI** that will decide whether to pay or not.
• 💰 **Custom pricing** — You can now set different prices for different paths:
  • /premium/* → $0.05 per request
  • /blog/* → $0.01
  • /archive/* → $0.0001 (nearly free)
• 🔐 **HTTP Message Signatures (RFC 9421)** — Authentication via public keys (RSA-2048 or ECDSA P-256). The bot signs the request, Cloudflare verifies—no passwords or tokens.
• 🏦 **Cloudflare as Merchant of Record** — You don't deal with billing, taxes, or chargebacks. Money comes to your Cloudflare Balance or bank account minus a commission of ≈ 5–8%.
• ⚡ **Integration with Bot Management and WAF** — One click in the Dashboard: "Block all AI crawlers by default, except those who pay."

⚙️ How to set it up in 5–10 minutes

1. 📊 Go to Cloudflare Dashboard → AI → AI Crawl Control → Pay-per-Crawl
2. ✅ Enable "Require payment for AI crawlers"
3. 💰 Set a base price (e.g., $0.01 per request)
4. 🎯 (Optional) Add custom pricing rules by path
5. 🚀 Save → Done! Cloudflare automatically adds the necessary headers and handles 402 responses

🔧 Request Example (Proactive flow)

GET /article/2025/ai-revolution HTTP/1.1
Host: example.com
User-Agent: GPTBot-Payer/2.0
Ai-Crawler-Intent: payment-ready
Ai-Crawler-Public-Key: ecdsa-p256:AAA...
Signature: MEUCIA...
Signature-Input: sig1=("host" "ai-crawler-intent");created=1735683200

✅ The server sees the signed intent → immediately returns 200 OK + content. Without a signature—402 with JSON:

{
"price": "0.010000 USD",
"currency": "USD",
"paths": { "/article/*": "0.010000 USD" },
"discovery": "https://example.com/.well-known/ai-crawl-pricing.json"
}

💡 That's why **I believe Pay-per-Crawl in 2025–2026** is the most convenient way to **protect content from AI** while simultaneously offering the chance to **earn from AI bots**. 📊 In the next section, we'll analyze how much you can realistically earn—and why for most sites, it's currently more about control than revenue.

⚖️ 3. Pros and Cons: My Honest Breakdown for Different Sites

⚠️ I intentionally avoided making a nice table with "everything will be great" because the reality is tougher. Here's what I see after analyzing dozens of sites, Cloudflare Radar, and talking to those already in the private beta.

🏢 Large Publishers (10M+ Monthly Views)

🟢 My Pros:

• 💰 **Real New Revenue:** $50–$200k per month already looks entirely achievable for top news and content platforms.
• 📉 **Compensation for Losses** from AI Overviews and referral drop—I see this potentially covering 10–20% of ad revenue losses.
• 🎛️ **Full Control:** Those who pay—read, those who don't—are blocked automatically.

🔴 Cons I Warn About:

• 🔗 **Total Reliance on Cloudflare (Vendor lock-in):** If they raise the commission or change the rules tomorrow—you can't do anything about it.
• 📊 **Risk of "Overpaying"** for your own traffic with complex custom rules.

🏠 Medium Sites (1–10M Views)

🟢 Pros (and I really see them):

• 💵 **$300–$2000 per month**—this is already tangible money that covers part of hosting and CDN costs.
• 🛡️ **Good AI Bot Control** without needing to write custom code.
• 👍 **Psychological Bonus:** Finally, someone pays for what they used to take for free.

🔴 Cons (and here I am especially candid):

• 📉 **Low Profitability:** After Cloudflare's commission and taxes, often less than half remains.
• ⏱️ **Overhead of +5–15 ms** on every payment request—this is noticeable for sites with high load.
• 🤖 **Most bots just ignore 402** and go to neighboring sites that haven't enabled Pay-per-Crawl yet.

📝 Small Blogs and SMBs (Less than 1M Views)

🟢 Pros (to be honest—there aren't many):

• ⚖️ **Principle Stance:** "My content is no longer free for AI."
• 💎 **Occasional Earnings:** Sometimes manage to earn $20–$30 a month from smaller crawlers (especially if the content is niche and valuable).

🔴 Cons (and here I am the harshest):

• ⏳ **In 80–90% of cases, revenue is less than the time spent on setup.**
• 🕵️ **Bots simply bypass:** They change their User-Agent, go through proxies, or wait for someone else to scrape your content.
• 💻 **If you're not on Cloudflare**—you'll have to write your own code to handle 402, which is simply unrealistic for a small site.

🌍 General Pros for Everyone (here I agree with everyone)

• 📉 **70–80% Reduction in Load** from unwanted AI crawlers—this really works even on the free Cloudflare plan.
• 🌱 **First step towards a fair data economy**—I believe this is historically important.

🚨 General Cons and Risks (My Warning #1)

• 🔒 **Vendor lock-in:** You are tied to Cloudflare forever.
• 🤖 **Risk of accidentally blocking Googlebot** or another "good" crawler—I have already seen such cases.
• 🚫 **Currently, less than 30% of AI bots are technically capable of paying**—the rest simply ignore 402 and look for easier prey.

💎 Conclusion from me personally:
In 2026, Pay-per-Crawl is great for the top 1,000 sites in the world and "so-so" for everyone else. For most of us (including me), the main value now is not the money, but the ability to tell AI companies: "**Want my content—pay or get out**."

📊 4. My Assumption: Three Hypothetical Scenarios

🧮 "Every site is unique, so check your metrics in the logs" — my approach to evaluating Pay-per-Crawl.

📅 As of December 2025, **Pay-per-Crawl** is still in private beta, and Cloudflare does not disclose the exact earnings of early adopters (like Condé Nast or TIME). Therefore, I took the liberty of creating three **hypothetical scenarios**—purely my assumptions, based on aggregated data from **Cloudflare Radar** (where AI crawlers make up 50–80% of bot traffic, with 18% YoY growth). 📉 I used conservative estimates: **AI bots** share is 1–2% of total views (because bots are 30% of traffic, 80% of which is AI), price $0.01 per request (minimum from Cloudflare), and realistic revenue after commission (≈30% deductions) + ignoring 70% of bots (because not all support 402). 🔍 This is not an exact science—for your site, check the logs (e.g., in Google Analytics or Cloudflare Analytics) to see the real number of requests from GPTBot/ClaudeBot. Here are my examples:

1. 🏢 **News Portal (15M views/month, like an average Ukrainian/Russian-language resource such as UNIAN or Lenta.ru)**
   📈 According to my calculations: 1.8% of traffic are **AI bots** (based on Radar: GPTBot—35%, ClaudeBot—11%). → ~270,000 requests/month (considering bots crawl ~1–2% of views, but with intensity up to 39k/min for fetcher bots). 💰 At $0.01/request = potentially $2,700 gross. 📊 Realistically (after 30% Cloudflare commission + 70% ignoring): **$800–$1,500/month**. For such a portal, this covers 10–15% of losses from the drop in organic traffic (25–38% YoY). 💭 My opinion: worth testing if the content is fresh and in demand for model training.
2. 💻 **Medium-Sized Tech Blog (2M views/month, like a typical IT blog on Medium or Habr)**
   🤖 Hypothetically: ~1.5% of views are AI crawlers (with a focus on training queries where OpenAI dominates 75% of traffic). → 30,000 requests/month (based on crawl-to-refer ratios of 887:1 for ChatGPT). 💵 Potentially $300 at $0.01. 📉 Realistically (accounting for custom pricing for `/blog/*` and ignoring): **≈$80–$150/month**. This is like a hosting bonus, but not a replacement for ads. 🎯 From my experience: for niche topics (like AI SEO), it can be +20% to revenue because bots love structured data.
3. 📝 **Personal Blog (200,000 views/month, like a typical freelancer's personal website)**
   📊 My scenario: 1–2% bots (less, due to low visibility for crawlers). → 3,000–5,000 requests/month (based on an average of 60–120 pages/min for lightweight scrapers). 💸 Potentially $30–$50. 🔍 Realistically (with overhead and low adoption): **$0–$20/month**—often less than the time spent on setup. 🛡️ For a small site, this is more about **AI content protection** than earnings. 💡 My advice: if views are <500k, focus on `llms.txt`, not payments.

📌 Summing up my hypotheses: only the top 5–10% of sites (with 10M+ views) will see significant revenue ($50k+/month, as analysts predict). For the rest—it's a control tool, not a gold rush. 🔧 To adapt this to yourself, look at your metrics in the Cloudflare Dashboard: how many requests from **GPTBot, ClaudeBot**? Then multiply by $0.01 and subtract 70%. ➡️ In the next section, we'll compare this with alternatives like **llms.txt** or **Really Simple Licensing**.

🆚 5. Pay-per-Crawl vs. Alternatives 2025–2026

📋 Technological Solutions

📜 Legal and Commercial Solutions

🚀 6. 2026 Forecast and What to Do Now: My View on the Future of Pay-per-Crawl

🎯 "2026 will be the year when AI crawler monetization moves from hype to reality—but only for those who prepared in advance" — my conviction after analyzing Cloudflare's beta updates in December 2025.

📅 As of December 12, 2025, **Cloudflare's Pay-per-Crawl** is still in private beta, with the latest updates from December 10: Discovery API, custom pricing, and expanded configurations. This means the tool is evolving, but the public launch is imminent. 👁️‍🗨️ I, as an author who has been tracking the topic since July 2025 (when the beta started), see 2026 as a turning point: moving from blocking **AI bots** (like GPTBot or ClaudeBot) to full-fledged **AI crawler monetization** via **HTTP 402 Payment Required**. ⚠️ But it won't be smooth—regulations (EU AI Act from January 2026), lawsuits (like NYT vs. OpenAI), and technical barriers will slow adoption. 📊 Here is my forecast: conservative but realistic, based on Cloudflare Radar trends (18% YoY growth in AI traffic) and rumors from the beta community.

🔮 My Predictions for 2026

• 🚀 **Public Launch — Q1 2026:** The beta has been running for six months (since July 2025), with updates in August (402 expansion) and December (Discovery API). 💭 My opinion: Cloudflare won't delay much longer—they already have >1M domains on AI Crawl Control, and a public rollout in January–March is logical for monetizing 20% of the web. 📈 Plus, pressure from publishers (Condé Nast, TIME) will force acceleration. ⚠️ Risk: delay until Q2 if Stripe integrations are insufficient.
• 💰 **First Real Cases — Mid-2026:** Currently, early adopters (AP, Adweek, Fortune) are testing, but without public figures. 📊 I predict the first reports in June–July: for example, a news site with 10M views earning **$10k–$50k/month**, as in my hypotheses from Section 4. 📈 Justification: with growing adoption (from 1% to 10–15% of sites) and the first lawsuit payouts, the case studies will go viral. 💡 My opinion: it will be like `robots.txt` in the '90s—skepticism first, then the standard.
• 🤖 **Bot 402 Support reaches 50–70%:** Currently, <30% of crawlers (mainly OpenAI/Anthropic) are ready for HTTP Message Signatures. 🌍 By the end of 2026, with pressure from the EU AI Act (mandatory transparency for high-risk AI), most (Google AI, Meta, xAI) will integrate it. 🧐 My skepticism: small bots (Common Crawl) will ignore it—realistically 50% for enterprise, 70% for top-5 companies. 🛡️ This will make **Pay-per-Crawl** a must-have for **AI content protection**.
• 👾 **Emergence of Agent Wallets (Agentic AI with a budget):** 2026 is the year of the agentic boom: autonomous agents (like in Grok or Claude) with built-in wallets for RAG queries. 💰 Cloudflare is already hinting at "stablecoins for agents" in the beta. 💭 My opinion: by Fall 2026, this will be the norm—75% of AI requests via agents, with micropayments of $0.001–$0.01. 🎯 This is an opportunity for SMBs: your blog becomes "fuel" for personalized AIs, but only with **custom pricing**.

📊 Overall, I'm an optimist: **Cloudflare 2026** will turn Pay-per-Crawl into a $1B+ marketplace (from $2B in publisher losses to AI). ⚠️ But a pessimist: without regulations (like the California AI Bill), bots will bypass it, and revenue for 90% of sites will remain < $100/month.

✅ What to Do Now: My Step-by-Step Action Plan

⏳ Don't wait for Q1 2026—prepare now. 📋 Here is my practical guide, with justification for each step, to maximize **AI crawler monetization** and minimize risks.

1. 📝 **Join the Pay-per-Crawl Waitlist:** Fill out the form at cloudflare.com/paypercrawl-signup—it's free and gives priority in the beta. ❓ Why now? Beta slots are limited (already >1,000 early adopters), and the first public launch users will get bonus features. 💡 My advice: indicate whether you are a publisher or a crawler—for publishers, this is the key to testing custom pricing.
2. 🛡️ **Add llms.txt + Strict Rules in robots.txt:** Create `/llms.txt` with "Deny: all" for GPTBot/ClaudeBot, and update `robots.txt`: `User-agent: GPTBot Disallow: /`. 💭 My opinion: this is a "soft" barrier for now (80% of bots ignore it, but 20% don't), plus an ethical signal for the EU AI Act. 🔧 Justification: the combo with Pay-per-Crawl is ideal, as `llms.txt` is preparing for **Really Simple Licensing** in 2026.
3. 🤖 **Connect Cloudflare Bot Management:** Enable it in the Dashboard (free for basic) — it blocks 80% of invalid traffic from **AI bots**. ❓ Why? It reduces the load by 70%, giving you time to test Pay-per-Crawl. 📈 From my experience: for SMBs, this is a +10–20% site speed, and you are already in the Cloudflare ecosystem for seamless migration.
4. 📊 **Collect Statistics via Cloudflare Radar:** Enable Analytics in the Dashboard — track crawl-to-refer ratios (Anthropic: 73k:1) and % of AI traffic. 🔧 Justification: your data is the basis for custom pricing ($0.05 for premium). 💭 My opinion: do this monthly—by June 2026, you will have a baseline for negotiations with OpenAI.
5. 🤝 **Prepare for Negotiations (if you are a large publisher):** Gather a dossier: your metrics, unique content, references to lawsuits (like Vox Media with Anthropic). ❓ Why now? Direct agreements ($100k–$10M) are better than Pay-per-Crawl for top-tier sites. 💡 My advice: join Really Simple Licensing—it's a template for 2026, when agentic AI will be looking for "partners."

📌 Summing up my plan: 2026 is not a revolution, but an evolution. 🔄 Start with blocking + monitoring, move to payments. 🎯 If you are an SMB, like me, focus on **llms.txt** and Radar—it will give you an edge. 🚀 Ready for **Cloudflare Pay-per-Crawl 2026**? Test now, because tomorrow, bots will pay those who are ready.

❓ Frequently Asked Questions (FAQ)

1. Is it mandatory to be a Cloudflare client to receive money for Pay-per-Crawl?

• 🟡 **In Short:** Yes, 100%. Without Cloudflare, there is no automatic monetization.
• 🔵 **In Detail:** Pay-per-Crawl is not an open standard but a Cloudflare product. All billing, Merchant of Record services, HTTP Message Signatures processing, and money payouts happen only through their infrastructure. If your site is not on Cloudflare (even the free plan), you can manually implement HTTP 402 + signed requests, but then you are solely responsible for billing, taxes, and key security. In practice, only a few tech giants do this. For 99.9% of publishers and bloggers, the answer is clear—no Cloudflare, no money.

2. How much can I realistically earn in 2026?

• 🟡 **In Short:** 90–95% of sites—less than $100 per month. Only the top 1,000–2,000 sites globally will earn $5,000 and above.
• 🔵 **In Detail:** I use an average price of $0.01 per request and Cloudflare Radar data (December 2025).
  • 🏢 Site with 15 million views → potentially $800–$2,500/month (realistically closer to the lower end)
  • 🏠 Site with 2–3 million → $80–$300
  • 📝 Site with <1 million → usually $0–$50
  After Cloudflare's commission (5–8%), taxes, and the fact that 60–80% of bots currently just ignore 402, not much is left. 💭 My opinion: if you don't have at least 5–7 million views and fresh, in-demand content—expect symbolic amounts or just protection, not earnings.

3. Is it safe to give away my keys and allow bots to pay?

• 🟡 **In Short:** Yes, it is safe. Single-use signed requests are used; the private key never leaves your server.
• 🔵 **In Detail:** Cloudflare implemented the **HTTP Message Signatures** standard (RFC 9421). The bot sends a public key and signs each request with its private key. Your server (or Cloudflare for you) verifies the signature, and if everything is OK, returns the content. The private key is generated and stored only by you—Cloudflare never sees it. 🔐 This is the same level of security as in banking APIs. ⚠️ The only risk is if you incorrectly configure the rules and open access to everyone, but this is a human error.

4. What happens if I turn on Pay-per-Crawl, and the bots just go to a neighboring site?

• 🟡 **In Short:** For most sites, that is exactly what will happen in 2025–2026.
• 🔵 **In Detail:** Currently, less than 30% of AI bots are technically capable of paying. The rest either ignore 402, change their User-Agent, or go to sites that haven't enabled protection yet. 🛡️ Therefore, for medium and small sites, Pay-per-Crawl currently works more as a "fee for peace of mind" and a principled stance than as a source of income. 💡 I advise combining: enable Pay-per-Crawl + strictly block those who don't pay. Then you either get money or get rid of the load—a win-win.

5. Should I wait for the public launch in 2026, or can I do something now?

• 🟡 **In Short:** Don't wait—act today.
• 🔵 **In Detail:** The best strategy for now (December 2025):
  1. 📝 Join the Pay-per-Crawl waitlist
  2. 🛡️ Add `llms.txt` + Disallow for all known AI bots
  3. 🤖 Enable Cloudflare Bot Management (even the free one)
  4. 📊 Start collecting statistics on your AI traffic
  🚀 Those who do this now will get access to the public launch first in January–March 2026 and will be able to set up custom pricing before the mass influx. ✅ I have gone through all these steps myself—and I recommend not delaying.

✅ My Personal Conclusion

🎯 **Pay-per-Crawl is truly a historic step.** For the first time in the history of the web, we can tell AI companies: "Want my content—pay." 🔄 This is not just a tool; it's a paradigm shift—from free vampirism to a fair data economy.

⚠️ But in 2025–2026, it is far from a panacea.
🏢 **For large publishers (10M+ views)**—it is a must-have right now. 💰 They will genuinely see tens and hundreds of thousands of dollars in new revenue and partially compensate for losses from AI Overviews.

📊 **For medium and small sites (and for me too)**—it's currently more about control and a principled stance than money. 💸 Real earnings in most cases will be symbolic, and sometimes zero. But that's okay—the main thing is that we stop being free fuel for others' billions.

🤝 **If the article was helpful—please share it.** 📢 The more publishers who learn about Pay-per-Crawl now, the faster we can collectively force AI companies to pay for what they take.
🚀 **See you in 2026**—the year the web finally starts earning from AI, not the other way around. 🚀

✍️ This article was prepared by the founder and leader of a company with 8 years of experience in web development—Vadim Kharovyiuk.