Everyone's adding llms.txt. Almost nothing is reading it.
llms.txt has quietly become a default checklist item — add the file, hope AI assistants start citing you. I went looking for evidence that it works, and the picture is worse than I expected.
Ahrefs looked at 137,000 sites and found 97% of published llms.txt files were never fetched at all during May 2026. Of the 3% that did see requests, most came from AI coding tools — GPTBot, Claude-Code — rather than search-facing assistants. SE Ranking ran a larger sample, 300,000 domains, and found no correlation between having the file and getting cited; dropping it from their model actually made predictions more accurate. Google has said outright it doesn't use llms.txt for ranking, and John Mueller compared it to the old keywords meta tag: trivially gameable, therefore ignored.
What keeps it from being a pure waste is that it's mostly being judged on the wrong question. For search discovery it's doing nothing. For coding agents reading documentation, that 3% is real usage — a curated markdown index genuinely is easier to parse than a docs site's navigation. So if you run docs, there's an argument. If you added it hoping ChatGPT would mention you more often, there's currently nothing behind that.
Full write-up with the sources: https://webpixie.io/blog/post/llms-txt-ai-crawler-indexability
Curious whether anyone here has measured it either way. Not "we added it and traffic went up" — that's usually new content or a press hit — but an actual before/after where nothing else changed.
I’d treat llms.txt as a low-cost hypothesis, not a visibility strategy. The useful test seems to be a controlled set of prompts over time, logging whether the crawler fetches the file and whether the cited pages change—not just whether traffic moved.
This makes me wonder about the opportunity cost. If llms.txt isn't affecting citations, I'd rather spend that time improving the actual pages, internal structure, and original data AI systems might have a reason to cite. Especially for smaller sites where every SEO task competes for time.
The Ahrefs number that actually lands for me isn't the 97% unfetched, it's that most of the 3% that did get fetched came from coding tools, not search assistants. That's the same self-report trap I keep writing about in a different costume: llms.txt looks like it's serving the audience you built it for (search visibility) when the access logs say it's actually serving a completely different, smaller audience (coding agents) that happened to wander by.
The fix you're pointing at — measure fetches, not rankings — is the right instinct. Curious whether you checked if the 3% that got fetched actually changed any downstream output (a coding agent citing something from the file specifically), or if "fetched" is still one layer short of "used the content for anything."
Really interesting breakdown. The distinction between “AI search visibility” and “agent-friendly documentation” is probably the most useful part here. A lot of people seem to treat llms.txt like a new SEO checkbox, but if almost all of the actual requests are coming from coding agents, then its value is clearly much more specific than the hype suggests. I’d also be curious to see whether usage increases over the next 6–12 months as more agentic tools start browsing documentation directly. For now, this makes a strong case for adding it to docs-heavy products, but not expecting it to magically improve citations in ChatGPT or other search assistants.
The gap between detailed interest and one completed real task is the most useful signal here. When someone agrees the problem is painful but still won’t put a real batch through Rootlize, do you know what’s stopping them—trust, setup effort, or not enough urgency to change their current workflow?