AEO Radar

I've been watching AI-search referrals climb in our analytics for a year and finally got curious how prepared other sites actually are. So I picked a defined set — the top 100 WebApp landing pages from BuiltWith / G2 / pick a list with a public methodology — and ran the same checks against each.

What I looked at, per site:

• llms.txt at the root (the emerging AI-equivalent of robots.txt for content guidance)

• AI crawler stance in robots.txt — explicit rules for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, OAI-SearchBot, CCBot

• ai.txt (Spawning consent format)

• JSON-LD presence — Organization, WebApplication/WebSite, FAQPage, Author/Person

• The classic SEO hygiene basics — single H1, canonical, OG tags, mobile viewport, sitemap, robots

• Security headers (CSP, HSTS, XFO, XCTO, Referrer-Policy, Permissions-Policy)

• Live response health — status, TTFB

A few things I didn't expect:

1. The big-name SaaS players were worse than the indie tools on AEO signals specifically. Indie tools shipping llms.txt looked like a generational shift — the enterprise sites are still optimizing for 2018 SEO.

2. AI crawler stance is split into three camps: explicit allow, explicit block, and "we have no idea this is a thing." Camp 3 was the biggest.

3. TTFB doesn't correlate with revenue. Some sites pushing $XXM ARR were serving the homepage in 1.5s+. Cloudflare's not free, but it's not optional anymore either.

If I were starting fresh and actually cared about AI search visibility, here's what I'd ship in an afternoon:

• A real llms.txt describing the site, contact, and content guidance — this is the cheapest legitimate AEO signal

• Explicit robots.txt rules for the major AI crawlers (allow or disallow, just be explicit)

• Organization + WebSite JSON-LD on every page (not just home)

• FAQPage schema where appropriate — AI Overviews/Perplexity quote these heavily

• An ai.txt if you have an opinion on training

What I'm still not sure about:

• Whether llms.txt actually moves the needle or it's just signaling. I haven't seen first-party data either way.

• Whether AI Overviews preference for sites with extensive JSON-LD vs. ones with great prose. My gut says "great prose still wins" but the schema seems to be a tiebreaker.

Curious what others are seeing. Has anyone here tracked AI-search referrer growth and tied it back to a specific change?

(If anyone wants to run this scan against their own site, I built a free thing that runs all the checks above — happy to share in DMs or comments if it's not against the rules.)