1
0 Comments

Development Update 12

Today's release makes our broken link checks better internet citizens.

Previously our crawler would crawl 100 pages on a site, making two simultaneous connections every 1/4 second, but we noticed that we were sometimes getting 429 responses ("too many requests") from some web sites as we were hitting their rate limits,.

With today's release we've added per-site settings to limit the number of pages we crawl, the number of concurrent requests we make and the time between each request.

Hopefully this should improve the reliability of our crawler results and avoid placing too much load on our customer's websites.

Trending on Indie Hackers
I launched 12 companies from $0 to $2MM ARR. Let me answer your launch questions. AMA! 40 comments Why is Building in Public a good thing to do? 11 comments My website translation app just hit $18k MRR 🤑 9 comments [12 months free] for honest FEEDBACK on my product! ~ PhotoATM 6 comments I've asked 750+ startup founders. "How did you come up with your idea?" Here are their answers 3 comments 9 Ways to Validate Your Startup Idea 2 comments