AIToolsRecap

I run AIToolsRecap.com, an AI tools discovery and comparison site.

Instead of relying only on Search Console, I started logging crawler requests by user agent. After 36 days, the data showed a sharp shift—even though I made no relevant technical or content changes.

I indexed each crawler against the first week of August, with Aug 1–7 set to 100:

WeekChatGPTClaudeBingGoogleAug 1–7100100100100Aug 8–14793358792Aug 15–21571818090Aug 22–283124280110

The control group is what makes this interesting.

If the site had suffered a broad technical problem—such as broken pages, blocked crawlers, or a robots.txt change—I would expect multiple crawlers to decline together.

That didn’t happen.

Google stayed within 10 points of its baseline. Bing dropped by around 20% and then remained stable. ChatGPT alone declined consistently, falling to just 31% of its original level. Meanwhile, Claude finished at 242%.

The shift began during Aug 8–14—the same period when others independently observed ChatGPT Search relying more heavily on domain-scoped site: queries.

That timing is interesting, but I want to be precise: one website’s server logs can show correlation, not prove causation.

My main takeaway is that AI crawler activity behaves more like a live signal of retrieval strategy than a backlink profile.

Backlink profiles usually change gradually and largely reflect actions taken by the site owner. Crawler traffic can change dramatically within weeks because an AI vendor adjusts a retrieval setting that publishers never see.

There is another important detail: total crawler activity didn’t decline proportionally—the composition changed.

If I watched ChatGPT alone, this would look like a crisis. Looking at the entire crawler mix, it looks more like traffic being reallocated between AI platforms.

The limitations are clear:

• This is one website in one vertical.

• The observation period is only 36 days.

• User agents can be spoofed.

• Correlation does not prove causation.

• Crawl volume is not the same as citation volume.

• The direction of the change is more transferable than its exact magnitude.

If you publish content, consider logging AI crawler requests by user agent and indexing them against a baseline week. Search Console won’t show this activity, and crawler changes may become visible before their effect on AI citations or referral traffic.

Full study, methodology, data and sources:

https://aitoolsrecap.com/Blog/ai-crawler-behaviour-study-36-days-2026

Is anyone else tracking AI crawler activity in their server logs? I’d be interested to know whether you saw a similar shift in August.

I run an AI tools directory and I've never seen anyone publish what a placement actually delivers. So here's mine, with real numbers.

WHAT ONE PLACEMENT PRODUCED

110 clicks in the first 6 days. 106 of them from the homepage banner alone. Not impressions, clicks — people who landed on the tool's own site.

WHAT THE TRAFFIC UNDERNEATH IT LOOKS LIKE

July: 1.26M Google impressions, 3,230 clicks. That single month produced about 71% of every click the site has recorded since launching in March.

314 ChatGPT citations, 127 Copilot, 468 total across six AI engines. DR 30, 799 referring domains, $0 ever spent on ads or backlinks.

THE PART THAT MATTERS FOR YOUR TOOL

I log crawler traffic by user agent. Over six days: ChatGPT hit the site 28,842 times. Googlebot came 1,744 times. That's 16 to 1.

When someone asks ChatGPT "what's the best AI tool for X," it's pulling from pages like these. A listing isn't just a backlink anymore — it's presence in the retrieval layer that's increasingly deciding what gets recommended.

And new listings get indexed by Google in about 24 hours.

WHICH PAGES ACTUALLY CONVERT

Not the ones you'd expect. One of my explainer pages ranks at position 5.7 with 398,000 impressions and a click rate that rounds to zero. A comparison page ranks at 4.7 and converts at 6%. One position apart, 150× the click rate.

So if you list, the thing worth having is a spot on a comparison page against your competitors — not a directory row nobody scrolls to. Someone searching "your tool vs their tool" is thirty seconds from a decision.

THE OFFER

Listing is free: https://aitoolsrecap.com/JoinUs.aspx

You get a public listing page, eligibility for community reviews, and indexing usually inside a day. No card, no trial, no catch. Paid tiers exist if you want comparison placement or editorial coverage, but the free one is genuinely free and most tools here are on it.

If you're building something in AI and want a comparison page pointing at you rather than at a competitor, send me the URL.

Happy to answer anything about the numbers above. I publish these monthly whether they're good or not.

Site launched March 18. No ads, no outreach budget.

This month something shifted.

Companies I didn't contact started emailing me. Not indie founders — established brands with real marketing budgets. Found our articles through search, asked about placement.

The numbers behind why it's happening:

5.15M Google impressions since launch. 385 ChatGPT citations. Copilot 183 times. DR 32, 850 referring domains. MSN and Bing News syndicating our content.

None of that was bought.

The comparison pages did it. Verdict-first structure, specific use case, no hedging. That's what ranks and that's what AI systems cite. Everything else is noise.

First paid placement closed in May. More in progress.

Pat @ AIToolsRecap

Google sent the monthly summary. July: 1.26M impressions, 3,230 clicks.

Site total since launching in March: 4.95M impressions, 4,540 clicks.

So one month out of four and a half produced about 71% of all the clicks, off roughly a quarter of the impressions. July's CTR was 0.26%. Lifetime is 0.09%.

Nothing about my ranking changed much — average position has sat around 7 the whole time. What changed is which pages get the impressions. The high-volume explainer pages that rank for questions Google answers inside the search results are fading, and comparison pages are taking their share. One explainer sits at position 5.7 with 398,000 impressions and a click rate that rounds to zero. One comparison page sits at position 4.7 and converts at 6%. One position apart, about 150× the click rate.

Other number from the same email: 383 pages picked up their first impressions in July. Meanwhile 14 pages carry all 314 of my ChatGPT citations. Indexing spreads, value concentrates.

Four months, $0 on ads or backlinks. DR 30, 799 referring domains, 468 citations across six AI engines.

Still working out what those 14 pages have in common. Will post it when I know.

I started logging bot traffic by user agent this week. Six days of data:

ChatGPT 28,842 retrievals. Bing 6,430. Perplexity 2,286. Googlebot 1,744. Claude 623.

AI crawlers hit the site about 4× more than search crawlers, and ChatGPT alone outpaces Googlebot more than 16 to 1.

The part I didn't expect: Bing crawls me 3.7× more than Google, and Copilot is my second-biggest citation source at 127. My two heaviest crawlers are my two biggest citation sources, and Googlebot — the one everyone optimizes for — is the lightest of the five.

If you're chasing AI citations, Bing Webmaster Tools is probably worth more of your attention than it's getting.

Four months in: 4.89M Google impressions, 4,260 clicks, 468 citations across six AI engines, DR 30, $0 on ads.

Curious whether anyone else is logging by user agent, and whether the ChatGPT-to-Googlebot ratio looks similar on other sites.

314 ChatGPT citations, spread across 14 pages. 127 Copilot citations, across 14. Out of a couple hundred pages published since March.

I'd assumed AI citations scaled with volume — publish more, get cited more. They don't. A small handful of pages does all the work and the rest sits in the index doing nothing.

Same pattern I found last week with clicks: one page at 395K impressions and 161 clicks, another at 1,370 impressions and 88 clicks. Concentration, not spread.

Current numbers: 4.89M Google impressions, 4,260 clicks, 468 citations across six AI engines, DR 30, 799 referring domains. Four months, $0 on ads or backlinks.

Going through those 14 now to work out what they have in common. If anyone's done this exercise on their own site I'd like to know what you found.

Ahrefs currently estimates AIToolsRecap's organic traffic at 121 visits a month. Google Search Console says 2,630 clicks in the last 28 days. It's off by more than 20×.

I mention it because before I built this I was checking incumbent directories in Ahrefs to work out whether the niche was worth entering. If I'd trusted those numbers I wouldn't have started. Those tools badly undercount newer sites, and if you're pre-launch and sizing up a market that way, you're probably reading a number that's wrong by an order of magnitude.

Real figures as of today: 4.89M Google impressions, 4,260 clicks, 314 ChatGPT citations, 127 Copilot, DR 30, 799 referring domains. Four months, $0 on ads or backlinks.

Most AI tool directories focus on listings.

What I noticed while building is that users usually decide much later — when they search things like:

• “ChatGPT vs Claude”

• “best AI coding assistant”

• “is Cursor better than Copilot?”

• “which AI tool should I use?”

That’s the problem I’m focusing on with AIToolsRecap.

Instead of only listing tools, I’m building around:

• comparison pages

• editorial reviews

• user ratings

• AI-search visibility

• decision-stage discovery

Still very early, but already seeing:

• 100+ AI tools listed

• daily new reviews coming in

• growing search impressions from comparison keywords

I’m a solo founder building this full-time and learning as I go.

Curious how other founders think about discovery now that AI search is changing how people browse products.