Did you block ChatGPT from your site by mistake?
Last year a lot of sites copied a robots.txt that blocks every OpenAI and Anthropic bot by company name. That blocks two different things at once.
GPTBot and ClaudeBot only collect training data. OAI-SearchBot, ChatGPT-User, Claude-SearchBot and Claude-User are the ones that fetch your page when a real person asks the AI about you right now. Block the first group if you want. Keep the second, or you are simply not in the answer.
84.2% of sites checked in Q3 2026 had no AI crawler policy at all, so most of us are running whatever we copied.
Reply with your domain and I will check your robots.txt and tell you which of these six bots you are blocking. No signup.
I build scout7.ai, that is where this comes from: https://scout7.ai?utm_source=indiehackers&utm_medium=post_sep_7
The floor-vs-lever framing matches what we're seeing from the infra side too: unblocking only guarantees crawlability, but whether an engine cites you depends on whether it wants a second source at all. The utilityseo.com data point is instructive — fully open robots.txt, GPT mentioned them 5/5 while citing mostly their own pages, but Kimi answered 4/5 with zero sources. That's not a robots.txt problem, it's a retrieval-policy difference between engines. Worth treating AI visibility as two separate pipelines: access control (robots.txt, one-time fix) and citation eligibility (do you have third-party material retrieval bots can corroborate). The 848→233 crawls-per-referral number suggests the second pipeline is where everyone is actually losing.
Useful split. I had lumping "block OpenAI" in my head as one decision, not two.
Take you up on the offer: postboxmap.co.uk. Curious which of the six we are actually allowing.
So robots.txt is basically becoming part of your AI SEO strategy now. Weird timeline
The training-vs-retrieval split is the right mental model, and it maps to a measurement gap I keep running into: even when GPT cites you, that visit lands in analytics as direct, so you never see which answers actually send people. We built amami.dev to split AI-referrer sources for exactly this reason.
Taking you up on it: utilityseo.com. Our robots.txt allows everything except /api/ and /_next/, so I expect a clean bill, which is why I can offer a data point on the downstream question in the comment above.
We had our domain run through six engines this week, fully open to all six of the bots you list. GPT named us in 5 of 5 answers while citing almost nothing but our own pages. Perplexity 4 of 5. Kimi 4 of 5 with no sources at all. Qwen 1. GLM 0.
So being crawlable did its job and then stopped mattering. The constraint was not access, it was that we have no third party material for a retrieval bot to find, so the engines wanting a second source either fell back on a directory listing or refused us outright on the best tools question.
Worth saying because unblocking is a floor rather than a lever. It stops you being invisible. It does not make you citable.
ran your actual numbers through scout7.
908 posts on your site. 0 of the 20 articles that feed ai answers name you. se ranking sits in 18 of those 20 instead.
not sure what questions you tested, but here are 3 a real buyer types before they know your name:
— best seo tools for small businesses, affordable site audit and rank tracking
— best ahrefs alternatives
— best seo platform for small agencies with client reporting
typed your company name into google though, and the top 7 results were all your own pages. so crawling was never the problem. marketing was.
exactly what u said, just with the receipts. full report here: https://scout7.ai/sharedFS/utilityseo.com/aeo_reports/2026-09-08-v2.html
The distinction between training and search crawlers is useful, but the bigger test seems downstream. Have you seen sites that changed their robots.txt actually gain measurable AI-referred traffic or visibility, rather than just fixing a technically incorrect block?
tbh no clean single-site before and after yet. the only hard number i have is ecosystem level, openai crawls per referral went 848 to 233 in two months, so unblocking is table stakes not a growth lever
That distinction between table stakes and a growth lever is interesting. I’d be curious to dig into what you’re seeing behind that. If you’re open to it, what’s the best email to reach you on?
dropping my email here in public is not a good idea.
the growth lever you're asking about is basically what scout7 works on, whether ai actually names your brand, and helping build that up. here is the link, try it out.
https://scout7.ai?utm_source=indiehackers&utm_medium=reply_to_comments
The training-vs-answer distinction is the one that gets lost in copy-paste policies. For client sites I treat robots.txt as a release artifact: diff it in review and test the six bot names after every CMS or CDN change. Otherwise a template update can quietly remove you from answers.
release artifact is the right frame. testing the six names after every cdn change beats my one time check, stealing that for the next one
Glad it’s useful. The post-CDN check is the part that catches silent template overwrites. One-time audits miss those.