Made a complete AI crawler cheat sheet — turns out most "block AI" advice is backwards

While digging into why some client sites weren't showing up in ChatGPT/Perplexity

answers, I kept running into the same confusion: people either block every AI

bot out of general unease, or leave everything wide open without knowing what

each one actually does.

Turns out there are really only 3 kinds of AI crawlers:

1. Training crawlers (collect data for future models) — blocking these is a

privacy choice, doesn't affect visibility today

2. Search/retrieval crawlers (the ones that actually feed live AI answers) —

block these and you're just invisible, full stop

3. User-triggered fetchers (fire when someone in a chat asks the AI to check

your page right now)

The mix-up between #1 and #2 is apparently the most common reason a business

quietly disappears from AI answers with zero explanation.

Put together a full reference list (GPTBot, ClaudeBot, PerplexityBot,

Google-Extended, and a bunch of lesser-known ones) with a block-or-allow call

for each, plus a ready-to-copy robots.txt template:

Curious if anyone here has checked their own robots.txt for this recently —

I was honestly surprised how often this gets set wrong by accident (old

plugin, template default, etc.) rather than on purpose.