I spent 10 minutes squinting at robots.txt so you never have to
A few months ago I was updating one of my sites and had a dumb little realization. I genuinely had no idea whether ChatGPT or Claude could read it. Google I knew how to check. But GPTBot? ClaudeBot? Perplexity? No clue. They crawl differently, they follow different rules, and a lot of them do not run JavaScript at all.
So I did the sensible thing. I opened robots.txt, squinted at it, opened devtools, squinted some more, and gave up ten minutes later knowing nothing.
That little annoyance turned into BotLens. You click it on any page and it gives you a plain 0 to 100 score for how readable that page is to AI. It reads the robots rules, the meta tags (including the sneaky noai one that some CMSes quietly add for you), the HTML structure, and whether the actual content is in the page or gets painted in later by JavaScript.
The best part was testing it. I pointed it at a pile of slick modern sites, a few of them quite well known, and a surprising number serve crawlers what is basically a blank wall. Everything shows up only after JavaScript runs, which plenty of AI bots never bother doing. Looks perfect to you and me. Invisible to a bot.
It is free, it runs entirely in your browser so nothing gets sent anywhere, and the code is open on GitHub.
I would love to know what you point it at and what you get. Bonus points if you find a big name site that scores embarrassingly low.
What stood out to me is that you're translating technical crawlability into something people can actually act on.
Most site owners don't care whether a bot can parse their HTML. They care whether AI can understand and recommend their business. Turning that uncertainty into a simple, actionable signal feels much more valuable than another SEO diagnostic.
Exactly, that is the gap I kept bumping into. "Can a bot parse my HTML" is a means to an end. What people actually want to know is "will AI understand my business well enough to recommend it."
I tried to make the score reflect that intent rather than just spitting out technical flags. Still figuring out how far to push it toward plain language versus keeping the raw signals for the devs who want them. Curious which side you would lean.
Thanks for the thoughtful read.
Interesting question.
Your reply made me realize the part I find most interesting isn't really the balance between plain language and raw signals.
It's a consequence of choosing one direction over the other that I don't think I can explain properly in a thread without oversimplifying it.
If you're interested, what's the best email to reach you on?