My first real stress test: 20,000+ URLs in one sitemap

July 9, 2025

Last week, I got humbled.

Someone dropped a URL into TrackMySitemap.com — my lightweight SEO scanner — and hit "Scan".
It was just another day in indie SaaS land… until it wasn’t.

Boom: 20,000+ URLs in One Sitemap

Turns out the site was bilibili.com, a massive Chinese video platform.

Their sitemap contained over 20,000 links.

Here’s the kicker:
I had a hardcoded limit: 500 URLs per scan.
I thought, “This should keep things safe.”

But guess what?

The limit didn’t work.

How It Was Supposed to Work

The logic was simple:

if (urls.length > 500) {

urls = urls.slice(0, 500);

}

Clean. Simple. Buggy as hell.

In edge cases, the slice was skipped entirely.
Maybe malformed XML, maybe async race condition — maybe I just messed up.
Result: the server tried to scan 20,000 pages in one go.

And it died.

I Had Two Options

Blame the user for pushing my system beyond its limits
Fix the damn thing and make my tool better

I chose option 2.

Fixing the Mess (and Making It 10x Better)

Over the next 24 hours, I:

Added enforced link caps, not just in the parser but across the whole flow
Rewrote the sitemap processor to handle large jobs in chunks
Added parallel scanning threads with batched timeouts
Improved memory handling

And… it worked. Like really worked.

I re-ran the scan for the user. It took longer, but didn’t crash.
All 20,000+ links were handled gracefully.

What I Did Next

I emailed the user and said sorry
Sent them their scan link
Gave them a 25% discount coupon

They appreciated it.

Lessons for Indie Hackers

Your users will do things you didn’t plan for.
Don’t assume they’ll play nice.
“Limits” aren’t real unless they’re enforced end-to-end.
I had a limit — but it wasn’t backed by code in the right place.
The first "fail" is your best QA moment.
If no one's broken your app yet, you're not live enough.
Own the mistake. Fix it. Thank the user.
Early adopters are your best critics and biggest fans — if you treat them well.