For two decades the web ran on an implied bargain: crawlers could read anything not disallowed, and publishers accepted the scraping because search sent traffic back. AI broke that bargain — and the 2026 numbers are unambiguous about how badly. At the launch of Cloudflare's pay-per-crawl framework, the company's own measurements put the ratio of pages crawled to humans referred back at 14:1 for Google, roughly 1,700:1 for OpenAI, and 73,000:1 for Anthropic (Cloudflare, July 2025).
The direct answer to "should my site allow AI crawlers?" is now: the market has repriced access, the defaults have flipped, and "allow everything" is no longer a neutral setting — it is a decision, and most sites are making it by accident.
The Defaults Flipped
Two infrastructure events define the new era:
July 1, 2025: Cloudflare turns opt-out into opt-in. Cloudflare — which routes roughly 20% of web traffic — began blocking AI crawlers by default on new domains, at the network level, before the site's robots.txt is even consulted. Alongside the default block, it launched Pay Per Crawl, letting site owners charge crawlers per request via an HTTP 402 response. By 2026, Cloudflare reported customers sending over a billion 402 responses on an average day (Cloudflare, 2026).
July 1, 2026: the system matures — with a catch. Year two brought a crawler classification system (search vs. training vs. agent), an attribution dashboard, and Pay Per Use. From September 15, 2026, "mixed-use" crawlers — bots that blend search indexing, AI training, and agent activity in one pass — are blocked by default on ad-carrying pages for new customers. The catch: as of the announcement, only two AI companies (Ceramic.ai and You.com) had signed on as payment partners. OpenAI, Google, Anthropic, and Perplexity had not (Cloudflare via The Register and CJR reporting, 2026). Blocking a mixed crawler today mostly means disappearing from it — payment requires a counterparty willing to pay.
The Licensing Market: Nine Figures for the Few
While per-crawl payment remains nascent, direct licensing has become a real market with real prices — for the few who can command them:
03M in aggregate data-licensing contract value disclosed in its S-1; combined Google and OpenAI API licensing near
30M a year (LLM Pulse tracking, 2026)
News Corp–OpenAI: reported **50M+ over five years** (WSJ, 2024); News Corp–Meta added up to $50M/year over three years (March 2026)
The New York Times–Amazon: reported 0-25M/year (2025) — even as the Times' suit against OpenAI and Microsoft remains the defining fair-use test in court
Anthropic's .5B settlement over pirated training books (2025) set the downside price of getting access wrong
Market tracking (LLM Pulse, August 2026) counts roughly two dozen OpenAI publisher deals at an average of 24M each**, within **2.92B in committed spend across the big four buyers (OpenAI 53%, Google 12%, Microsoft 9%, Meta 6%)
The structure of that market is the uncomfortable part: it pays the brand-name corpus and strands the long tail. The Associated Press, Time, The Atlantic, Reddit, and Getty sign nine-figure deals; the mid-sized trade blog with excellent reporting signs nothing, because nothing forces an AI company to come to the table.
The Enforcement Gap: robots.txt Is a Sign on an Unlocked Door
The technical layer has its own asymmetry. robots.txt remains genuinely useful — reputable AI companies honor it — but it has no enforcement power, and 2026 produced two canonical violations: ByteDance's Bytespider has been repeatedly documented fetching robots.txt and crawling disallowed paths anyway, and Perplexity was caught in August 2025 by Cloudflare, which published evidence that when Perplexity's declared crawler hit a Disallow, a stealth crawler took over impersonating a normal Chrome browser and rotating IPs. Cloudflare de-listed Perplexity as a verified bot over it.
Actual enforcement lives at the edge — CDN-level fingerprinting and challenges — which is precisely why the Cloudflare default matters more than any robots.txt guidance. Meanwhile the standards layer is improvising: RSL (Really Simple Licensing), launched September 2025 and backed by Reddit, Yahoo, and Medium, embeds machine-readable license terms in robots.txt and lets sites pool into ASCAP-style collectives. It is promising and it is voluntary — a stated license, honored only if the AI company chooses to or a court compels.
The block statistics show how the wind is blowing: GPTBot is the most-blocked AI crawler, appearing in 5.52% of all robots.txt DISALLOW rules network-wide in Q1 2026; 25% of the top 1,000 sites block GPTBot outright, and among 107 prominent sites checked in June 2026, 44.9% block at least one major AI crawler (TechnologyChecker and multiple cross-referenced sources, 2026).
What This Means for Sites That Are Not Media Giants
Three practical positions, ordered by what the evidence actually supports:
1. Block training, allow retrieval — the evidence-backed middle path. The bot taxonomy matters more than the blanket decision: blocking training bots (GPTBot, CCBot, Google-Extended, ClaudeBot) keeps content out of model training while leaving retrieval bots (OAI-SearchBot, PerplexityBot, Bingbot) — the ones that feed cited answers — running. Block retrieval and you delete yourself from the answer layer this entire series is about. One warning: blocking CCBot also drops you from Common Crawl, which many third-party models reuse — a blast radius far wider than one vendor.
2. Check your defaults — especially on Cloudflare. Because the default flipped, a site that never made a decision may already be blocking GPTBot, ClaudeBot, and PerplexityBot at the network edge. If AI citations matter to your business, verify your actual crawl-control settings rather than assuming your robots.txt reflects reality. (This series covered the edge-blocking failure mode in detail; the 2026 default changes make that failure mode far more common.)
3. Price your content honestly. If you are a recognized brand with a large archive, licensing conversations are real — the market data above is your comp set. If you are not, "charge" is not yet a real option (two payment partners), and the honest choices are allow-and-benefit-from-citations, block-and-accept-invisibility, or the training-block/retrieval-allow split. What is no longer honest is the old default: unlimited everything, decided by no one.
Limitations
Cloudflare's crawl-to-referral ratios are the company's own measurements, published to justify its product line — directionally credible, not independently audited. Pay Per Use partner counts and the September 15 enforcement date are as of the July 2026 announcement and may change. Licensing figures are from press reports and market trackers with undisclosed methodologies. The NYT fair-use case remains unresolved, and its outcome could restructure everything above. None of this is legal advice.
Frequently Asked Questions
Should I block AI crawlers from my site?
Split the decision by bot type. Blocking training bots (GPTBot, CCBot, Google-Extended, ClaudeBot) keeps your content out of model training while preserving most AI-answer visibility. Blocking retrieval bots (OAI-SearchBot, PerplexityBot, Bingbot) removes you from AI answers entirely. A blanket "block all AI bots" is the one option the evidence actively discourages for anyone who wants citations.
Does Cloudflare block AI crawlers automatically now?
Yes, for new domains since July 1, 2025 — GPTBot, ClaudeBot, and PerplexityBot are blocked at the network level by default. From September 15, 2026, mixed-use crawlers are also blocked by default on ad-carrying pages. If your site sits behind Cloudflare and you have never touched AI Crawl Control, check the dashboard before assuming you know what your site allows.
How much do AI companies pay for content?
At the top of the market: Reddit ~$60M/year from Google plus OpenAI access (near 30M/year combined), News Corp 50M+ over five years from OpenAI, the NYT 0-25M/year from Amazon, within roughly .92B in committed licensing spend across the big four AI buyers (LLM Pulse, 2026). The market pays the brand-name corpus; the long tail is largely stranded.
What is Pay Per Crawl, and can I use it?
Cloudflare's Pay Per Crawl lets site owners charge AI crawlers per request (HTTP 402). It remains in private beta as of August 2026, and only two AI companies have signed on as payment partners — so for most sites it is a framework to watch, not revenue to book. Cloudflare reports over a billion 402 responses per day across its network, but volume of responses is not volume of payments.
What is RSL and will it replace robots.txt?
RSL (Really Simple Licensing) adds machine-readable license terms to robots.txt — including per-use charges and collective licensing pools — and launched in September 2025 with Reddit, Yahoo, and Medium behind it. Like robots.txt itself, it is a stated license, not technical enforcement; it works only if AI companies honor it or courts compel them. Treat it as a developing standard, not a solved one.
Last updated: September 9, 2026 Sources and method note: Cloudflare crawl-to-referral ratios, default-block dates, Pay Per Crawl/Use details, and 402 volumes from Cloudflare announcements and The Register/CJR reporting (2025–2026); licensing figures from Bloomberg (2024), WSJ (2024), SEC S-1 filings, and LLM Pulse market tracking (August 2026); Perplexity stealth-crawl evidence from Cloudflare (August 2025); blocking statistics from TechnologyChecker Q1 2026 robots.txt analysis and cross-referenced June 2026 site checks; RSL details from RSL Collective announcements (2025). Cloudflare figures are first-party and product-adjacent; treat all as dated snapshots.
