Anthropic sends you one visitor for every 1,782 pages it crawls. Do you still let it in?
Cloudflare measures how many pages each AI company crawls for every one visitor it sends back to your site.
On 1 August 2026, over the last 28 days: Anthropic crawled 1,782 pages per visitor. OpenAI 233. Perplexity 303. Google 4.79.
Two months earlier, Anthropic was at 4,580 and OpenAI at 848. So the deal is getting better fast. It is still hundreds of pages read for one visit.
Say why in the comments. I want to hear from people who allow everything and from people who block it all.
Built at Scout7: https://scout7.ai?utm_source=indiehackers&utm_medium=post_sep_9
One thing worth adding from the data/scraping side: there's actually a way to verify a request is really from the bot it claims to be (Anthropic/OpenAI), not a spoofed user-agent - checking IP ranges/reverse DNS, a fairly standard traffic-verification technique. So "allow/block" doesn't have to be blind - you can verify the crawler is legit, and in the meantime manually track brand mentions in AI answers once a month or so as a rough proxy metric, until real "crawl-to-mention" dashboards exist.
The crawl-to-visitor ratio is measuring the wrong thing, and it matters because you're building a product on it. "1,782 pages per visitor" assumes the value of an AI crawler is referral traffic — visits it sends back. But in an answer-engine world, the value isn't the click back, it's the citation: when ChatGPT or Claude recommends your product inside an answer, the buyer converts having never visited your site at all. The visit isn't the unit anymore.
So the ratio looks catastrophic by a metric that's quietly obsolete. If those 1,782 crawled pages are how the model learned to name you when a buyer asks "what should I use for X," then blocking the crawler to protect your visit-ratio is trading a dead metric (referral clicks) for an invisible one (being recommended). You'd be optimizing exactly the wrong variable, and you can't even see the one that matters in your traffic dashboard, because a recommendation that converts leaves no referral trail.
It's the web1 reflex applied to a world that broke it: "a crawler should return traffic" was true when crawling fed a search results page you clicked through. Now crawling feeds an answer the buyer never leaves. The question isn't "how many visitors does it send," it's "does it recommend me when someone asks the buying question" — and that's not a traffic number.
So for Scout7's angle: are you measuring crawl-to-visit, or crawl-to-mention? Because the second is the one that decides whether letting Anthropic in was worth it, and it's the one nobody has a dashboard for yet.
At 1,782 pages crawled per visitor, the economic trade-off seems more important than the raw crawler volume. Are site owners actually seeing enough downstream AI referrals or conversions to justify keeping Anthropic access open?
I would choose 2, then judge it by outcomes rather than crawler volume. Separate search crawls from training crawls, and log AI referrals plus whether the landing pages are actually useful to those visitors. If allowing a crawler produces no discoverability or visits after a reasonable window, blocking it is a product decision, not a moral one. The ratio alone is not enough to decide.
The four options I meant to put in a poll:
Reply with a number, or say why.