robots.txt
API to check crawl rules from robots.txt files
We are live in HackerNews, check it out here: https://news.ycombinator.com/item?id=20548264
We are very proud of the welcome that the service is having in the community, and all the feedback we are receiving. If you have any question, feedback or suggestion, let us know!
It took 6 weeks, but I'm finally done building the minimum viable product for robots.txt. I worked mostly nights and weekends, and I coded the whole thing in Java and Kotlin since that's that I'm the most comfortable with.
I'm excited to start talking to customers and see what they think about the product so far!
-
2
Have you found this to be a major pain point for you?
Google search console and related services are pretty good (mobile friendly test, structured data testing tool etc.) at analyzing your website "crawlability" and they even send me messages for things I wasn't even aware of. I have found that is all I need to make sure my page is accessible.
Looks like google also has some experimental API for using their tools programmatically: https://developers.google.com/webmaster-tools/
-
1
Hi @MF706!
Thanks for your comment. This project born because I have multiple web crawlers and I wanted to centralize queries to robots.txt files and check these rules in one system with high throughput. All with an easy integration using an API.
Seems that the Google experimental URL testing tools impose a usage limit of 1 QPS, this can be good for SEO tools for personal use, but for a set of web crawlers, it is not good because normally web crawlers exceed x100 these number of requests per second.
This service (mainly) must be used by people/business who code web crawlers and want to externalize the robots.txt rule checking.
-
Comment