Nobody is reading your llms.txt
The agent-ready web has an evidence problem, and this summer the numbers arrived. Ahrefs analyzed llms.txt files across 137,000 sites: 97% of them never get read. Of the bots that do fetch the file, 77% aren’t AI tools at all. Google’s John Mueller has been saying the quiet part for a year: no AI system currently uses llms.txt. The detail that elevates it from data point to comedy is that some of Google’s own properties serve the file anyway — an internal CMS added support, and nobody bothered removing it. The company whose search team calls the format speculative is also publishing it, by accident, at scale.
None of this is really about one file format. It’s about a pattern the web keeps repeating.
We’ve run this experiment before
The idea of a sidecar file that describes a site to machines is old, and its track record is strangely bimodal.
The failures: meta keywords, killed by its own incentives — a self-description no consumer
could trust was gamed into meaninglessness within years. The semantic web’s RDF era, killed
by author burden — publishers were asked to do real ongoing work for consumers who never
materialized. Microformats survived only in niches where a specific consumer actually parsed
them.
The successes: robots.txt, sitemap.xml, RSS. What separates them isn’t design quality —
robots.txt is a famously crude format. It’s that a consumer with skin in the game committed
to exercising the file. Crawlers fetch robots.txt because ignoring it has consequences.
Search engines poll sitemaps because they committed to. Feed readers poll RSS because that’s
the product. In every surviving case, the file is load-bearing for someone’s pipeline — which
means lying in it, or letting it rot, breaks something visible.
That’s the test, and it has nothing to do with adoption curves: who fetches this, and what breaks if it lies? If the answer is “nobody, and nothing,” the file is decoration. Decoration is fine. It just shouldn’t be sold as infrastructure.
Applying the test to the agent stack
llms.txt today has publisher adoption without a committed consumer — the exact profile of
meta keywords in its optimistic phase. Mueller’s practical advice is the honest version:
create the file when a platform that actually sends you traffic asks for it, not before. The
cost of shipping one is near zero, which is precisely why adoption numbers mean nothing —
artifacts that cost nothing to ship always ship ahead of the evidence.
The more interesting bet is the opposite shape: not a description but an interface. WebMCP — browser-native tool registration a visiting agent can call directly — cannot become decoration, because it does nothing until it’s exercised. An unused interface is just dead code; a used one is load-bearing by definition, which puts it on the robots.txt side of the ledger. The caveats are real: it’s a Chrome origin trial, it doesn’t run headless, and the API has renamed itself three times already — if an example you find doesn’t work, check which name it’s calling. Young, churning, unproven. But structurally it’s the kind of thing that survives, because the only way for it to matter is for something to actually call it.
The likely near future is unglamorous: the descriptive files persist as cheap options — kept because they cost a build step, not because they work — while the operable interfaces slowly accumulate the consumers that decide everything. If you’re deciding where to spend effort, history’s advice is consistent: bet on the layer that breaks when it lies.
For the curious: this site ships the whole stack, llms.txt through WebMCP — generated from one registry, so it costs a build step — and /for-agents/playground shows which layers an agent can actually exercise.