The crawler rulebook finally has an enforcer

On September 15, Cloudflare starts blocking a category of crawler it didn’t used to touch. Any new domain on its network will, by default, allow crawlers that do search indexing and block the ones doing AI training or agent fetching. The part that matters: it also blocks anything that mixes the two, on any page that carries ads. Cloudflare’s own post frames this as giving site owners a real choice for the first time. Some crawlers do both jobs. A crawler that indexes for search and also feeds AI training gets judged on the sum of what it does, not the part it wants credit for. Alongside the block, Cloudflare is widening its crawl-payment marketplace into “Pay Per Use.” A publisher gets paid when their content actually surfaces in an AI answer, rather than merely when a bot requests the page.

The question worth sitting with is who ends up holding the enforcement mechanism once a voluntary system stops being voluntary.

For most of the crawler web, access control has run on declaration. A site states its preference, a crawler is expected to honor it, and nothing forces the second half beyond reputation and the threat of a lawsuit. That’s a fine system right up until someone has an incentive to lie about who’s asking. In July, News Corp sued Brave, alleging the company disguises its crawlers so publishers can’t detect or block them, then resells near-verbatim summaries of news content to AI companies. Brave’s answer is that it’s just indexing, the way any search engine must, and that News Corp is trying to muscle a smaller competitor out on behalf of Google. That claim has its own plausibility — Brave had already sued News Corp first, after a cease-and-desist. The suit will take years to resolve. I won’t referee it here. What matters for this argument is the shape of the allegation, not its verdict. A declared boundary is only as good as the honesty of whoever’s crossing it — and once money is on both sides, somebody eventually has a reason to lie about which bot they’re running.

Cloudflare’s answer to that problem is Web Bot Auth, an IETF draft built on cryptographic signatures. A crawler operator publishes a public key, signs every request with the matching private key, and the receiving server checks the signature instead of trusting a User-Agent string anyone can type. Identity stops being asserted and starts being verified, cheaply, per request. Combine that with a default that blocks anything unverified or mixed-purpose, and the crawl-to-referral math Cloudflare had already published gets an enforcement mechanism to go with it. A year earlier, over one week in June 2025, its own data put Anthropic’s crawler at close to 71,000 page requests for every referral sent back. That ratio was always an argument. Now there’s a lever attached to it.

The precedent for what happens next is the ad-blocking wars. Eyeo built Acceptable Ads on the same move — give the smaller side (users, then publishers) leverage the incumbents didn’t want them to have, then monetize the leverage by selling passage back to whoever pays. Large ad networks pay for a place on that whitelist, on terms Eyeo has never published. Publishers have called the arrangement a protection racket for over a decade, and a publisher sued Eyeo over exactly this — the suit was dismissed. Blocking itself isn’t the problem. Whoever operates the chokepoint tends to end up as the toll authority, whether or not that was the founding pitch.

There’s a real counter here. Publishers currently get nothing from AI crawling, full stop, and one-off licensing suits take years each — the NYT-OpenAI case is still grinding through discovery two years in. A marketplace that pays out per surfaced answer, even one company’s marketplace, might just be scaling what would otherwise happen deal by deal, lawsuit by lawsuit. And Cloudflare, unlike Eyeo, publishes its methodology in public rather than negotiating opaque bilateral fees.

The cryptographic layer will win fast, because it settles a dispute already sitting in court. Nobody wants to relitigate whether a crawler lied about its own identity when a signature can just prove it. But the rulemaking doesn’t move to a standards body once that layer ships. It concentrates in the handful of infrastructure companies that already sit in front of enough of the web to set the default, with Cloudflare fronting more than 20% of domains as the clearest case. That’s a quieter story than “publishers versus AI,” and it’s the one that actually decides who gets to reach whom this fall.