The "block AI crawlers" robots.txt snippet you pasted probably turned off a rule you already had
Title: The "block AI crawlers" robots.txt snippet you pasted probably turned off a rule you already had
I've spent a few weeks reading other people's robots.txt files while building a checker, and the same two mistakes keep coming back. Both come from pasting one of the "block AI crawlers" snippets that are everywhere right now. Both are silent: the file still parses, nothing errors, and the rule you lost is one you wrote yourself.
- A named group REPLACES the * group. It does not add to it.
Say your file has had this for years:
User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Then you paste a snippet that names bots explicitly — including the ones you want to allow:
User-agent: GPTBot
Disallow:
Per RFC 9309 a crawler obeys the single most specific group matching its token, and falls back to * only when no named group matches at all. GPTBot now has exactly one rule: nothing disallowed. /wp-admin/ and /checkout/ are open to it. You didn't unblock them — naming the bot at all did.
This bites hardest with the snippets that write a group for every AI bot, the allowed ones included. Every allow entry costs you your * rules for that bot.
- Adding a second AI section instead of editing the first one.
Plenty of sites added an AI block a year ago, forgot, and now paste a fresh one at the bottom. You end up with two groups naming the same agent with different rules, and what happens next depends on whose parser you ask — some merge duplicate groups, some take the first one they hit. Either way you no longer know what your own file says, and the one that tends to win is the older, stricter one. Your new "please crawl me" did nothing.
The fix for both is boring: don't paste. Read your live file and change only the lines you mean to change.
Two things I built while digging into this, free, no signup:
-
passcite.com/tools/ai-crawler-checker — give it a URL, it reads the live file and tells you which AI crawlers can reach it right now, and which rule decided each verdict
-
passcite.com/tools/ai-robots-txt-generator — reads your live robots.txt, rewrites only the AI groups, leaves every other byte untouched, and prints what it changed vs what it carried over
Thanks for sharing the numbers, that makes it much easier to follow.
Correction to the post: the second mistake isn't "up to whose parser you ask". RFC 9309 §2.2.1 says groups that name the same crawler are combined. So last year's Disallow: / plus this year's empty Disallow: still blocks, while last year's Disallow: / plus this year's Allow: / allows — the two tie, and Allow wins a tie. It comes down to how the new snippet spells "allow".
My own checker was taking the first group and ignoring the rest. Fixed today, thanks to m_montazeri and nortise below.
This is useful. How are you finding your first users so far?
Google publishes the parser Googlebot uses as open source: https://github.com/google/robotstxt. Its robots_main binary takes a robots.txt file, a user agent and a URL and returns allowed or disallowed, so you can run your checker's verdicts through it for Googlebot and Google-Extended, including the duplicate group and * shadow cases
Did it — thanks, exactly the right test.
Built robots_main and ran both parsers over the same files. First pass: 17 disagreements on 61 hand-written cases, from three bugs. Duplicate groups weren't combined (m_montazeri's point above). User-agent names were matched as substrings, so a group for Applebot also caught Applebot-Extended. And a Crawl-delay line ended a group, which the RFC says it mustn't. On the live robots.txt of 30 large sites, the old checker got a verdict wrong on 5: the Applebot case on booking.com and pinterest.com, a group named Fetch read as meta-externalfetcher's on Wikipedia, omgili read as Omgilibot on Amazon, and TripAdvisor's duplicate groups.
Now: 72 recorded cases in the test suite, plus 21,000 generated files and those 30 live files, with zero disagreements. One difference kept on purpose: Google reads Allow: /index.html as also allowing /. The RFC doesn't, and I'd rather not tell a site its homepage is open to crawlers that may not share that rule.
It also turned up a trap I hadn't seen. User-agent: Googlebot, then only a Crawl-delay line, a blank line, then User-agent: GPTBot and Disallow: / — that's one group per the RFC and robots_main, so Googlebot is blocked from the whole site. Python's parser splits on the blank line and says the opposite. The validator now flags the line where the groups join.
This is a great catch — a lot of people copy-paste robots.txt snippets without checking what they're overriding. Did you find a safe way to layer the new rule without losing the old one?
Yes, with one catch: robots.txt has no inheritance. A named group can't add to the wildcard group, only replace it. So the only way to layer is to repeat the wildcard rules inside the new group — its Disallow lines, plus any Allow carve-outs inside them — and put Allow: / on top. Longest match keeps the repeated rules winning where they apply.
Or, if the wildcard group already lets the bot in, don't name it at all. That's the whole trap.
The generator does the repeating for you. A correction to my reply to brianainews above: when I wrote it, only the version inside the app did this — the free generator didn't until today, and it was also dropping the carve-outs. Both fixed.
Good write-up. What would you do differently if you started again?
Test against Google's own parser from day one, instead of after a reader in this thread told me to.
Nice work shipping it. What has been the biggest challenge since launch?
One clarification for the duplicate-group case: RFC 9309 section 2.2.1 requires the rules from groups matching the same user-agent token to be combined. For a conforming parser, it is not simply the first or oldest group winning. Section 2.2.2 then applies the most specific matching path rule.
For example, two ExampleBot groups containing Disallow: /private/ and Allow: /private/help/ should combine: /private/data stays disallowed, while /private/help/ is allowed. That would make a useful regression test for your checker. If a particular crawler behaves differently, showing that as a documented implementation difference would help separate it from the RFC result.
Source: https://www.rfc-editor.org/rfc/rfc9309.html#section-2.2.1
AI-assisted response.
You're right, and it was worse than a wording problem: my checker was one of the parsers that takes the first group. It read the first GPTBot section and never saw the second.
Fixed, and your ExampleBot case is in the test suite as written. It matters on live files too: tripadvisor.com names Google-Extended in two groups, and the second is Disallow: / with an Allow for a single page. The old checker read only the first group and said allowed. Combined, it's blocked from almost the whole site.
"Depends on the parser" is still true in the wild — Python's urllib.robotparser takes the first matching group, and the first matching rule rather than the longest — but I wrote it as if the spec left it open, and it doesn't. Thanks for the section number; it made this quick to check.
The "named group replaces the wildcard" behavior is the one that gets people silently every time. It is consistent with RFC 9309 but completely unintuitive if you have been writing robots.txt by copy-paste. Most people assume named entries are additive to the wildcard.
The duplicate section issue is messier in practice. Older files accumulate cruft — a block from 2023, another from early 2024, a fresh one someone pasted last week. Which rule wins is crawler-defined, not spec-defined, so you end up with a file that says different things to different bots with no reliable way to audit it without a tool like yours.
One edge case worth adding to the generator: sites using Allow paths within a disallow context. Those are wildcard-dependent too and can silently open up when you add a named group. Has your checker caught any sites where an Allow override was dropped this way?
Two things, and the first is my fault. "Which rule wins is crawler-defined, not spec-defined" is what my post implied, and it's wrong: RFC 9309 says groups naming the same crawler are combined (more in the reply to m_montazeri). Some parsers still take the first group — Python's standard library does — but the spec is clear.
Your edge case is real, and my own fix had it. The safe way to let one bot in is to repeat the wildcard rules inside its group, and I was repeating only the Disallow lines. On WordPress's default file (Disallow: /wp-admin/ with Allow: /wp-admin/admin-ajax.php) that closed admin-ajax.php for the one bot being let in. It carries the carve-outs now, and the checker reports both directions: rules a named group opened, and carve-outs it closed.
Has it caught one? I don't store the files people check, so I can't give you a count. But of the 30 large sites I tested today, two have exactly your shape. github.com's wildcard group blocks ?tab= URLs but carves out the achievements tab, and the group it gives OAI-SearchBot, ClaudeBot and PerplexityBot copies the block without the carve-out, so Google's own parser says those three can't fetch an achievements page every other crawler can. nytimes.com does the same to Amazonbot on Wirecutter links with a utm_source. Possibly deliberate in both cases, but that's the shape.
Silent failures are the expensive ones, because nothing in the stack tells you and you find out a quarter later. You have already built the thing that sells this: the before-and-after diff showing which of a person's own rules the snippet just switched off. I would make that the landing page rather than a tool sitting under /tools, because fear of what you already lost is a far stronger reason to click than curiosity about crawler policy.
Agree on the framing — "what you already lost" is a better reason to click than crawler policy, and the tool page still leads with the second one. Worth changing.
Not the homepage, though: the product is something else (why AI assistants recommend a competitor instead of your client), and the tools sit next to it. As of today the diff shows both halves of the loss — rules a pasted snippet opened, and carve-outs it closed — so there's more to lead with.
How did you decide this was worth building in the first place?
It fell out of the main product. Passcite looks at why AI assistants recommend someone else over a site, and the first thing that has to be true is that their crawlers can read it at all. Reading real robots.txt files for that is where both mistakes in the post kept turning up.
This is such a good catch. The named group shadowing the wildcard is exactly the kind of silent failure that makes robots.txt changes scary. Showing the effective rules for each bot before deploy would make this genuinely useful.
Both halves exist now, though only because ylynbuilds asked for the same
thing a few hours ago. The checker prints the group each bot matched —
including falling back to * — and when a named group claims a bot it now
lists the * Disallow rules that stopped applying to it.
The before-deploy half is the generator, and writing it turned up the trap:
the obvious fix is
User-agent: GPTBot+Allow: /, which is the exact bugin the post — it hands that bot /wp-admin/ as well. So it repeats the *
group's Disallow lines inside the new group, minus a blanket
Disallow: /.Longest match keeps them winning.
There's a second failure mode pasted snippets cause that's arguably worse: people use one blunt block for three different jobs. GPTBot is a training crawler, Google-Extended opts you out of Gemini training but not AI Overviews, and answer engines like Perplexity often reply from third-party search indexes rather than fetching your site at all. So a blanket Disallow can cost you citations in AI answers while doing nothing about content that was already trained on. Before touching robots.txt I'd decide which of the three you're actually objecting to — training, retrieval, or live fetch — because the levers are different, and most snippets don't distinguish them at all. On the fix itself: Google follows the most-specific-group rule the post describes, but not every bot parses the same way, so testing per bot token beats trusting any single interpretation.
Agreed, and that split is what the tool is built around — every token is
tagged training or answers, because "keep me out of the training set" and
"keep citing me" are opposite requests and one Disallow can't express both.
I cut the three slightly differently. Retrieval and live fetch I keep on one
side, because they pair up per vendor and losing either costs the same
thing: OAI-SearchBot / ChatGPT-User, PerplexityBot / Perplexity-User,
Claude-SearchBot / Claude-User. The third one I break out is the resold
corpora — CCBot, Diffbot — the only blocks whose effect you can never check
afterwards, since nobody publishes which dataset drew from which crawl.
Google-Extended is the case that breaks my own clean split, and you're right
about it: it covers grounding as well as training, and there is no token for
AI Overviews at all — those come off the regular Googlebot index, so opting
out means leaving Google Search. Two more worth naming:
meta-externalfetcher is documented as possibly ignoring robots.txt because a
person asked for the fetch, so a rule there is a request, not a lock. And
Applebot-Extended isn't a crawler at all — it's a flag Apple reads about
pages Applebot already fetched.
On testing per token rather than trusting one interpretation: agreed,
that's why the checker evaluates each token separately. And your first point
has no robots.txt answer at all — already-trained content is a deletion
request, not a crawl rule.
The single most specific group rule is the easy-to-miss part. A useful checker could show the effective group for each bot token, including wildcard fallback, and flag when a named group shadows the old * group. That makes the regression obvious before deployment.
Both halves are in the tool, but only the first one: it prints the group each bot matched, including the * fallback. The shadow flag wasn't there — and worse, it couldn't be: the checker evaluates the path /, and the rules you lose are the ones on /wp-admin/ and /checkout/. So the exact regression this post is about was the one thing my own checker couldn't show you.
Fixed: when a named group claims a bot, it now lists the * rules that stopped applying to it. Thanks — that was the useful kind of comment.
Blocking crawlers feels decisive until you notice you also blocked the ones that help you get cited. I treat robots rules like a changelog now: one intentional allow or deny per change, then I ask the same buying question in ChatGPT and Perplexity a week later to see if anything moved.
The changelog framing is the right one, and "one intentional allow or deny per change" is the part most people skip — batch three changes and you've spent a week's wait to learn nothing.
On the week: I've been assuming seven days too, but someone told me yesterday they've seen articles picked up in Copilot within a couple of days, and his read was that Microsoft owns both the index and the assistant so there's no third party in between. If that's right, the right wait is per-assistant, not one number. Have you seen any of the three move faster than the others?
Interesting take. Would you still recommend this approach to someone starting today?
The advice in the post is "read your own file instead of pasting someone else's" — that one doesn't really age. If you mean something narrower, say which part and I'll give you a straight answer.
The first line of this post says "Title:". That's the label from my own draft, pasted in along with everything else.
A post about pasted text quietly carrying things you didn't mean to carry.
I'll take it.
(If there's an edit button on IH, I haven't found it — genuinely asking.) 😂
This comment was deleted a day ago