The "block AI crawlers" robots.txt snippet you pasted probably turned off a rule you already had
Title: The "block AI crawlers" robots.txt snippet you pasted probably turned off a rule you already had
I've spent a few weeks reading other people's robots.txt files while building a checker, and the same two mistakes keep coming back. Both come from pasting one of the "block AI crawlers" snippets that are everywhere right now. Both are silent: the file still parses, nothing errors, and the rule you lost is one you wrote yourself.
- A named group REPLACES the * group. It does not add to it.
Say your file has had this for years:
User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Then you paste a snippet that names bots explicitly — including the ones you want to allow:
User-agent: GPTBot
Disallow:
Per RFC 9309 a crawler obeys the single most specific group matching its token, and falls back to * only when no named group matches at all. GPTBot now has exactly one rule: nothing disallowed. /wp-admin/ and /checkout/ are open to it. You didn't unblock them — naming the bot at all did.
This bites hardest with the snippets that write a group for every AI bot, the allowed ones included. Every allow entry costs you your * rules for that bot.
- Adding a second AI section instead of editing the first one.
Plenty of sites added an AI block a year ago, forgot, and now paste a fresh one at the bottom. You end up with two groups naming the same agent with different rules, and what happens next depends on whose parser you ask — some merge duplicate groups, some take the first one they hit. Either way you no longer know what your own file says, and the one that tends to win is the older, stricter one. Your new "please crawl me" did nothing.
The fix for both is boring: don't paste. Read your live file and change only the lines you mean to change.
Two things I built while digging into this, free, no signup:
-
passcite.com/tools/ai-crawler-checker — give it a URL, it reads the live file and tells you which AI crawlers can reach it right now, and which rule decided each verdict
-
passcite.com/tools/ai-robots-txt-generator — reads your live robots.txt, rewrites only the AI groups, leaves every other byte untouched, and prints what it changed vs what it carried over
This is useful. How are you finding your first users so far?
Google publishes the parser Googlebot uses as open source: https://github.com/google/robotstxt. Its robots_main binary takes a robots.txt file, a user agent and a URL and returns allowed or disallowed, so you can run your checker's verdicts through it for Googlebot and Google-Extended, including the duplicate group and * shadow cases
This is a great catch — a lot of people copy-paste robots.txt snippets without checking what they're overriding. Did you find a safe way to layer the new rule without losing the old one?
Good write-up. What would you do differently if you started again?
Nice work shipping it. What has been the biggest challenge since launch?
One clarification for the duplicate-group case: RFC 9309 section 2.2.1 requires the rules from groups matching the same user-agent token to be combined. For a conforming parser, it is not simply the first or oldest group winning. Section 2.2.2 then applies the most specific matching path rule.
For example, two ExampleBot groups containing Disallow: /private/ and Allow: /private/help/ should combine: /private/data stays disallowed, while /private/help/ is allowed. That would make a useful regression test for your checker. If a particular crawler behaves differently, showing that as a documented implementation difference would help separate it from the RFC result.
Source: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1
AI-assisted response.
The "named group replaces the wildcard" behavior is the one that gets people silently every time. It is consistent with RFC 9309 but completely unintuitive if you have been writing robots.txt by copy-paste. Most people assume named entries are additive to the wildcard.
The duplicate section issue is messier in practice. Older files accumulate cruft — a block from 2023, another from early 2024, a fresh one someone pasted last week. Which rule wins is crawler-defined, not spec-defined, so you end up with a file that says different things to different bots with no reliable way to audit it without a tool like yours.
One edge case worth adding to the generator: sites using Allow paths within a disallow context. Those are wildcard-dependent too and can silently open up when you add a named group. Has your checker caught any sites where an Allow override was dropped this way?
Silent failures are the expensive ones, because nothing in the stack tells you and you find out a quarter later. You have already built the thing that sells this: the before-and-after diff showing which of a person's own rules the snippet just switched off. I would make that the landing page rather than a tool sitting under /tools, because fear of what you already lost is a far stronger reason to click than curiosity about crawler policy.
How did you decide this was worth building in the first place?
This is such a good catch. The named group shadowing the wildcard is exactly the kind of silent failure that makes robots.txt changes scary. Showing the effective rules for each bot before deploy would make this genuinely useful.
Both halves exist now, though only because ylynbuilds asked for the same
thing a few hours ago. The checker prints the group each bot matched —
including falling back to * — and when a named group claims a bot it now
lists the * Disallow rules that stopped applying to it.
The before-deploy half is the generator, and writing it turned up the trap:
the obvious fix is
User-agent: GPTBot+Allow: /, which is the exact bugin the post — it hands that bot /wp-admin/ as well. So it repeats the *
group's Disallow lines inside the new group, minus a blanket
Disallow: /.Longest match keeps them winning.
There's a second failure mode pasted snippets cause that's arguably worse: people use one blunt block for three different jobs. GPTBot is a training crawler, Google-Extended opts you out of Gemini training but not AI Overviews, and answer engines like Perplexity often reply from third-party search indexes rather than fetching your site at all. So a blanket Disallow can cost you citations in AI answers while doing nothing about content that was already trained on. Before touching robots.txt I'd decide which of the three you're actually objecting to — training, retrieval, or live fetch — because the levers are different, and most snippets don't distinguish them at all. On the fix itself: Google follows the most-specific-group rule the post describes, but not every bot parses the same way, so testing per bot token beats trusting any single interpretation.
Agreed, and that split is what the tool is built around — every token is
tagged training or answers, because "keep me out of the training set" and
"keep citing me" are opposite requests and one Disallow can't express both.
I cut the three slightly differently. Retrieval and live fetch I keep on one
side, because they pair up per vendor and losing either costs the same
thing: OAI-SearchBot / ChatGPT-User, PerplexityBot / Perplexity-User,
Claude-SearchBot / Claude-User. The third one I break out is the resold
corpora — CCBot, Diffbot — the only blocks whose effect you can never check
afterwards, since nobody publishes which dataset drew from which crawl.
Google-Extended is the case that breaks my own clean split, and you're right
about it: it covers grounding as well as training, and there is no token for
AI Overviews at all — those come off the regular Googlebot index, so opting
out means leaving Google Search. Two more worth naming:
meta-externalfetcher is documented as possibly ignoring robots.txt because a
person asked for the fetch, so a rule there is a request, not a lock. And
Applebot-Extended isn't a crawler at all — it's a flag Apple reads about
pages Applebot already fetched.
On testing per token rather than trusting one interpretation: agreed,
that's why the checker evaluates each token separately. And your first point
has no robots.txt answer at all — already-trained content is a deletion
request, not a crawl rule.
The single most specific group rule is the easy-to-miss part. A useful checker could show the effective group for each bot token, including wildcard fallback, and flag when a named group shadows the old * group. That makes the regression obvious before deployment.
Both halves are in the tool, but only the first one: it prints the group each bot matched, including the * fallback. The shadow flag wasn't there — and worse, it couldn't be: the checker evaluates the path /, and the rules you lose are the ones on /wp-admin/ and /checkout/. So the exact regression this post is about was the one thing my own checker couldn't show you.
Fixed: when a named group claims a bot, it now lists the * rules that stopped applying to it. Thanks — that was the useful kind of comment.
Blocking crawlers feels decisive until you notice you also blocked the ones that help you get cited. I treat robots rules like a changelog now: one intentional allow or deny per change, then I ask the same buying question in ChatGPT and Perplexity a week later to see if anything moved.
The changelog framing is the right one, and "one intentional allow or deny per change" is the part most people skip — batch three changes and you've spent a week's wait to learn nothing.
On the week: I've been assuming seven days too, but someone told me yesterday they've seen articles picked up in Copilot within a couple of days, and his read was that Microsoft owns both the index and the assistant so there's no third party in between. If that's right, the right wait is per-assistant, not one number. Have you seen any of the three move faster than the others?
Interesting take. Would you still recommend this approach to someone starting today?
The advice in the post is "read your own file instead of pasting someone else's" — that one doesn't really age. If you mean something narrower, say which part and I'll give you a straight answer.
The first line of this post says "Title:". That's the label from my own draft, pasted in along with everything else.
A post about pasted text quietly carrying things you didn't mean to carry.
I'll take it.
(If there's an edit button on IH, I haven't found it — genuinely asking.) 😂
This comment was deleted 20 hours ago