CrawlerCheck

Hey Indie Hackers,

I've been lurking and learning from this community for a while, and I'm finally taking the plunge and sharing a project.

My name is Attila, and I'm a web developer and SEO expert from the depths of Transylvania, Romania. For years, I've run my own little web services business, and in that time, I've seen the same painful (and expensive) mistake happen more than once: a client accidentally blocks Google in their robots.txt or a stray noindex tag. Their organic traffic just disappears overnight.

Manually debugging this is a pain. You have to check the robots.txt file, then dig through the source code for meta tags, then inspect the HTTP headers. It's tedious and easy to miss something.

So, I built the tool I always wished I had: CrawlerCheck.

Link: https://crawlercheck.com

It’s a dead-simple app that does just one thing: you give it a URL, and it instantly tells you if it’s crawlable by checking the robots.txt, meta tags, and X-Robots-Tag all at once. No fluff.

I built the backend in Go and the frontend in Svelte because I wanted it to be ridiculously fast.

The tool is brand new, and I'd genuinely love to get some feedback from this community. Please give it a try and let me know what you think. Is it useful? Does it break? Is there anything you'd add?

Happy to be here and looking forward to hearing your thoughts!

Cheers.