ChatGPT crawling fell 69% on my site—while Claude rose 142%
I run AIToolsRecap.com, an AI tools discovery and comparison site.
Instead of relying only on Search Console, I started logging crawler requests by user agent. After 36 days, the data showed a sharp shift—even though I made no relevant technical or content changes.
I indexed each crawler against the first week of August, with Aug 1–7 set to 100:
WeekChatGPTClaudeBingGoogleAug 1–7100100100100Aug 8–14793358792Aug 15–21571818090Aug 22–283124280110
The control group is what makes this interesting.
If the site had suffered a broad technical problem—such as broken pages, blocked crawlers, or a robots.txt change—I would expect multiple crawlers to decline together.
That didn’t happen.
Google stayed within 10 points of its baseline. Bing dropped by around 20% and then remained stable. ChatGPT alone declined consistently, falling to just 31% of its original level. Meanwhile, Claude finished at 242%.
The shift began during Aug 8–14—the same period when others independently observed ChatGPT Search relying more heavily on domain-scoped site: queries.
That timing is interesting, but I want to be precise: one website’s server logs can show correlation, not prove causation.
My main takeaway is that AI crawler activity behaves more like a live signal of retrieval strategy than a backlink profile.
Backlink profiles usually change gradually and largely reflect actions taken by the site owner. Crawler traffic can change dramatically within weeks because an AI vendor adjusts a retrieval setting that publishers never see.
There is another important detail: total crawler activity didn’t decline proportionally—the composition changed.
If I watched ChatGPT alone, this would look like a crisis. Looking at the entire crawler mix, it looks more like traffic being reallocated between AI platforms.
The limitations are clear:
This is one website in one vertical.
The observation period is only 36 days.
User agents can be spoofed.
Correlation does not prove causation.
Crawl volume is not the same as citation volume.
The direction of the change is more transferable than its exact magnitude.
If you publish content, consider logging AI crawler requests by user agent and indexing them against a baseline week. Search Console won’t show this activity, and crawler changes may become visible before their effect on AI citations or referral traffic.
Full study, methodology, data and sources:
https://aitoolsrecap.com/Blog/ai-crawler-behaviour-study-36-days-2026
Is anyone else tracking AI crawler activity in their server logs? I’d be interested to know whether you saw a similar shift in August.
