Did you block ChatGPT from your site by mistake?

mehdizare's release-artifact frame is the right one, and RemNavi's 429 just showed the layer it does not cover: robots.txt allowing all six while the CDN refuses one of them on any uncached page. I want to add the failure one step further along, because I had it earlier this year and my own monitoring hid it.

My scheduled poster started getting 403s from my own site's API. I had written an alert for exactly that case and it told me the token had expired, so I reissued it. Still 403. The cause was a toggle on the hosting side, "block API access from overseas IPs", which is what my cloud scheduler is. The host was the one answering, in a different character set, with the host's copyright line in the body. My alert had hardcoded 403 equals expired credential because that was the only 403 I had ever seen. Right status code, wrong story, told with complete confidence.

My repair was to stop letting the alert interpret. It quotes the response body now and branches on who answered. A detector that explains can lie to you. A detector that quotes cannot.

That is the part of RemNavi's finding I would keep: the first manual check warms the cache, the retry comes back 200, and you tick it off. The check itself produced the clean result. Same shape as my alert. Reading robots.txt tells you your stated policy. Fetching once tells you what one request got. Neither tells you which layer answered, and the layer is what separates a policy problem from an edge problem.

So the decision I would put in front of anyone who just posted a domain above: whatever check you rely on, make it record who answered, app or edge, and keep the first request separate from the retry. A check that passes on the second try is a different fact from one that passes on the first. That is the test that would have caught mine, and the one a warm cache cannot fool.

The half of this nobody audits is rendering. Even with all six bots allowed, the retrieval crawlers handle client-side JavaScript badly, so a React app can be wide open in robots.txt and still hand back an empty shell with nothing to cite. Worth fetching your own pages the way the bot sees them, because I have watched sites fix the robots file, declare victory, and stay invisible for exactly that reason.

The floor-vs-lever framing matches what we're seeing from the infra side too: unblocking only guarantees crawlability, but whether an engine cites you depends on whether it wants a second source at all. The utilityseo.com data point is instructive — fully open robots.txt, GPT mentioned them 5/5 while citing mostly their own pages, but Kimi answered 4/5 with zero sources. That's not a robots.txt problem, it's a retrieval-policy difference between engines. Worth treating AI visibility as two separate pipelines: access control (robots.txt, one-time fix) and citation eligibility (do you have third-party material retrieval bots can corroborate). The 848→233 crawls-per-referral number suggests the second pipeline is where everyone is actually losing.

This is a really good point. I think a lot of people assume robots.txt is the whole story, when it’s really just the first step. The cache issue is especially interesting because it can make a quick manual test look completely fine. Checking uncached pages and watching for 429s seems like a much better way to catch these hidden blocks.

Just ran this on my own site expecting a clean bill and didn't get one. robots.txt allows all six. GPTBot still gets 429 on any page that isn't already in the CDN cache, while a browser UA, Googlebot, ClaudeBot, OAI-SearchBot and PerplexityBot all get 200 on the same cache-busted URL seconds apart. So not rate limiting, and nothing you'd ever see by reading robots.txt.

What makes it nasty is how it hides. Your first manual check warms the cache, so the retry comes back 200 and you tick it off. The homepage always looks healthy because it's cached anyway. It's the deep pages nobody has requested lately that get refused, and on a large site that's most of them.

So testing the bot names isn't quite enough by itself. Test them against a URL that isn't cached yet, and watch for 429 as well as 403.

Worth separating two things that get discussed as one: whether AI crawlers
can reach you, and whether AI assistants actually refer anyone to you. They
need different checks, and passing the first tells you nothing about the
second.

For the second one I ended up tagging AI referrers explicitly in GA4 —
chatgpt.com, perplexity.ai, claude.ai and friends get flagged at ingest, so
"did an assistant send anyone" becomes a column I can read every morning
instead of a thing I guess about. Across four sites the honest answer so far
is zero referrals, on sites that are days old. That's useful to know as a
baseline rather than as a disappointment.

One trap if you're verifying with a visibility dashboard: those panels
generally need a JS-capable browser to register the page at all. Checking
with curl and seeing nothing is the expected result, not evidence you've
blocked something.

Useful split. I had lumping "block OpenAI" in my head as one decision, not two.

Take you up on the offer: postboxmap.co.uk. Curious which of the six we are actually allowing.

So robots.txt is basically becoming part of your AI SEO strategy now. Weird timeline

The training-vs-retrieval split is the right mental model, and it maps to a measurement gap I keep running into: even when GPT cites you, that visit lands in analytics as direct, so you never see which answers actually send people. We built amami.dev to split AI-referrer sources for exactly this reason.

Taking you up on it: utilityseo.com. Our robots.txt allows everything except /api/ and /_next/, so I expect a clean bill, which is why I can offer a data point on the downstream question in the comment above.

We had our domain run through six engines this week, fully open to all six of the bots you list. GPT named us in 5 of 5 answers while citing almost nothing but our own pages. Perplexity 4 of 5. Kimi 4 of 5 with no sources at all. Qwen 1. GLM 0.

So being crawlable did its job and then stopped mattering. The constraint was not access, it was that we have no third party material for a retrieval bot to find, so the engines wanting a second source either fell back on a directory listing or refused us outright on the best tools question.

Worth saying because unblocking is a floor rather than a lever. It stops you being invisible. It does not make you citable.

The distinction between training and search crawlers is useful, but the bigger test seems downstream. Have you seen sites that changed their robots.txt actually gain measurable AI-referred traffic or visibility, rather than just fixing a technically incorrect block?

The training-vs-answer distinction is the one that gets lost in copy-paste policies. For client sites I treat robots.txt as a release artifact: diff it in review and test the six bot names after every CMS or CDN change. Otherwise a template update can quietly remove you from answers.