# Nobody is reading your llms.txt

> 97% of llms.txt files never get read, and we've run this experiment before. What separates the sidecar files that shaped the web from the ones that became decoration.

Published: 2026-07-03
Tags: agents, llms-txt, web-history
Canonical: https://aaronchartier.com/blog/nobody-is-reading-your-llms-txt

---
The agent-ready web has an evidence problem, and this summer the numbers arrived. Ahrefs
[analyzed llms.txt files across 137,000 sites](https://ahrefs.com/blog/llmstxt-study/): 97%
of them never get read. Of the bots that do fetch the file, 77% aren't AI tools at all.
Google's John Mueller has been
[saying the quiet part for a year](https://www.seroundtable.com/google-does-not-endorse-llms-txt-40789.html):
no AI system currently uses llms.txt. The detail that elevates it from data point to comedy
is that some of Google's own properties serve the file anyway — an internal CMS added
support, and nobody bothered removing it. The company whose search team calls the format
speculative is also publishing it, by accident, at scale.

None of this is really about one file format. It's about a pattern the web keeps repeating.

## We've run this experiment before

The idea of a sidecar file that describes a site to machines is old, and its track record is
strangely bimodal.

The failures: `meta keywords`, killed by its own incentives — a self-description no consumer
could trust was gamed into meaninglessness within years. The semantic web's RDF era, killed
by author burden — publishers were asked to do real ongoing work for consumers who never
materialized. Microformats survived only in niches where a specific consumer actually parsed
them.

The successes: `robots.txt`, `sitemap.xml`, RSS. What separates them isn't design quality —
robots.txt is a famously crude format. It's that **a consumer with skin in the game committed
to exercising the file**. Crawlers fetch robots.txt because ignoring it has consequences.
Search engines poll sitemaps because they committed to. Feed readers poll RSS because that's
the product. In every surviving case, the file is load-bearing for someone's pipeline — which
means lying in it, or letting it rot, breaks something visible.

That's the test, and it has nothing to do with adoption curves: **who fetches this, and what
breaks if it lies?** If the answer is "nobody, and nothing," the file is decoration.
Decoration is fine. It just shouldn't be sold as infrastructure.

## Applying the test to the agent stack

llms.txt today has publisher adoption without a committed consumer — the exact profile of
`meta keywords` in its optimistic phase. Mueller's practical advice is the honest version:
create the file when a platform that actually sends you traffic asks for it, not before. The
cost of shipping one is near zero, which is precisely why adoption numbers mean nothing —
artifacts that cost nothing to ship always ship ahead of the evidence.

The more interesting bet is the opposite shape: not a description but an interface. WebMCP —
browser-native tool registration a visiting agent can call directly — cannot become
decoration, because it does nothing until it's exercised. An unused interface is just dead
code; a *used* one is load-bearing by definition, which puts it on the robots.txt side of the
ledger. The caveats are real: it's a Chrome origin trial, it doesn't run headless, and the
API has renamed itself three times already — if an example you find doesn't work, check
which name it's calling. Young, churning, unproven. But structurally it's the kind of
thing that survives, because the only way for it to matter is for something to actually call
it.

The likely near future is unglamorous: the descriptive files persist as cheap options — kept
because they cost a build step, not because they work — while the operable interfaces slowly
accumulate the consumers that decide everything. If you're deciding where to spend effort,
history's advice is consistent: bet on the layer that breaks when it lies.

---

*For the curious: this site ships the whole stack, llms.txt through WebMCP — generated from
one registry, so it costs a build step — and [/for-agents/playground](/for-agents/playground)
shows which layers an agent can actually exercise.*