SEO Firewall - TrackMySitemap

Let’s go back to our LEGO city.

You’ve already built a cool map for Google (your sitemap.xml), showing all the buildings and streets it can visit.

But now you realize…

"Wait! Some places are private! I don’t want Google going into my secret lab or unfinished buildings."

So what do you do?

You put up a sign at the front gate that says:

"Google, you can go here, but not there."

That sign is called robots.txt.

🪧 What Does Robots.txt Do?

It’s a set of instructions for search engine robots (like Googlebot). It tells them:

• Which parts of your site they can crawl

• Which parts they shouldn’t

But here’s the twist:

Robots.txt doesn’t block people — only polite robots that follow the rules.

Google listens.
Other bots might ignore it. But if your goal is SEO, robots.txt is the polite way to set boundaries.

🔍 How Robots.txt and Sitemap.xml Work Together

If sitemap.xml is your map,
robots.txt is your rulebook.

They go together like this:

• sitemap.xml: “Here’s everything I want you to find.”

• robots.txt: “But don’t look behind these doors.”

In fact, you can tell Google where to find your sitemap right inside robots.txt.

User-agent: *

Disallow: /private/

Allow: /

Sitemap: https://example.com/sitemap.xml

Let’s explain what this means…

🧾 Breaking Down a Sample robots.txt

User-agent: *

This means: "The rule below is for all robots."

Disallow: /private/

This says: "Don’t go into the /private/ folder."

Allow: /

This says: "You’re allowed to visit everything else."

Sitemap: https://example.com/sitemap.xml

This tells the robot: "Oh, and here’s the map of my city."

🛠️ How Do You Make a Robots.txt File?

It’s just a plain text file, like a note you write in Notepad.

1. Open a blank text file

2. Write your rules (like the example above)

3. Save it as robots.txt

4. Put it in the root of your website
   So it’s available at:

   https://yourwebsite.com/robots.txt

🤔 Should You Always Block Things?

Not always.

But here are a few smart reasons to use robots.txt:

✅ You want to block:

• Admin pages (/admin)

• Test pages (/staging)

• Search results pages (/search)

• Duplicate content

⚠️ But don’t block pages if you want them to appear in Google Search!
Google can’t index what it can’t see.

🚫 Warning: Blocking ≠ Hiding

Blocking a page in robots.txt tells Google:

“Don’t crawl this page.”

But if another site links to it, Google might still index the URL — just without the content.

If you want to completely hide something from search, use meta tags like:

<meta name="robots" content="noindex">

✅ Summary (Like You’re 5)

• robots.txt is a rule sheet at the front door of your website

• It tells robots: “Come in here, but stay out of there”

• It’s often the first file Googlebot looks at

• You can add your sitemap.xml link inside it

• Don’t block pages unless you have a good reason

Imagine you built a big LEGO city with lots of buildings, streets, and tiny signs. Now you invite Google — a smart robot — to come look at it.

But there’s a problem:
Google has never been here before. It doesn’t know where your buildings are. Some streets don’t even have signs. It might get lost.

So what do you do?
You give Google a map.

That map is called sitemap.xml.

🧭 Why Does Google Need a Sitemap?

Google is like a robot explorer. It visits websites to understand what’s there, what pages exist, and which ones to show in search results.

But websites can be confusing:

• Some pages don’t have links pointing to them.

• Some are hidden behind menus.

• Some are hard to reach.

If Google can’t see a page, it might never know it exists.
That’s bad if it’s an important page — like your homepage or blog!

A sitemap is your way of helping Google by saying:

"Hey Google, here’s a list of all the pages on my website. You don’t have to guess!"

🧾 What’s Inside a Sitemap.xml File?

A sitemap is just a text file written in XML (which stands for "eXtensible Markup Language" — but don’t worry, it’s just a format computers like).

Here’s what a super simple sitemap looks like:

<?xml version="1.0" encoding="UTF-8"?>

<urlset xmlns="https://www.sitemaps.org/schemas/sitemap/0.9">

<url>

<loc>https://example.com/</loc>

</url>

<url>

<loc>https://example.com/about</loc>

</url>

</urlset>

Let’s break that down:

• <urlset> is like opening the map.

• <url> means "one page on the website."

• <loc> is the actual link (called a URL).

So this file tells Google:

“My site has two pages: the homepage, and the about page.”

You can add as many <url> blocks as you want.

🛠️ How Do You Make a Sitemap?

There are 3 easy ways:

1. Use a generator tool

There are websites like xml-sitemaps.com or tools like Screaming Frog that will make one for you.

2. Let your CMS handle it

If you use WordPress, Shopify, Wix, or other tools — they usually create a sitemap automatically at:

https://your-website.com/sitemap.xml

3. Write it yourself

You can just copy the example above and change the links. Save it as a file named sitemap.xml and upload it to the root of your site (that’s like the homepage folder).

🚀 Bonus: Tell Google About Your Sitemap

Even if your sitemap is perfect, Google needs to know about it. Here’s how:

1. Go to Google Search Console

2. Add your site (if you haven’t already)

3. Go to Sitemaps

4. Enter the path to your sitemap
   (for example: sitemap.xml)

5. Click Submit

Now Google knows exactly where your pages are. You’ve just made its job a lot easier.

✅ Why It Matters

Without a sitemap, Google has to guess where your pages are.
With a sitemap, it knows exactly what to crawl.

That means:

• Your pages get indexed faster

• Fewer pages are missed

• Google understands your site structure better

So if your website is like a LEGO city, then sitemap.xml is the blueprint that helps Google explore every building.

Be First on Google by Fixing What Blocks You First

Most SEO tools alert you when it’s already too late.

After Google skipped your pages.

After your traffic dropped. 

After rankings fell.

TrackMySitemap finds what’s blocking you, before Google does.

It scans your sitemap, robots.txt, and meta tags to catch invisible issues that kill indexing.

No guessing.

No delays.

We compare what you want indexed to what actually can be.

Every mismatch is a lost ranking, unless you fix it first.

TrackMySitemap shows you how.

Right now.

Most SEO tools warn you too late.

After Google already ignored your pages. After rankings dropped. After traffic vanished.

TrackMySitemap protects you before it happens.

Like a firewall for your SEO — catching invisible blockers in your sitemap, robots.txt, and meta tags.

We don't guess. We audit your intent vs. reality.

And every mismatch? That's lost traffic you’ll never get back — unless you catch it first.

TrackMySitemap does. Before Google does.

Hello IndieHackers!

It's been a while since I've been here, but I bring you an interesting article about SEO.

Most people think SEO = keywords.

They use tools like Semrush, Ahrefs, or Surfer to find the right terms, optimize content, build backlinks.

But here’s the problem: Even the best content won’t rank if Google can’t see it.

That’s where Technical SEO comes in.

What does that mean?

It’s all about making sure:

• Your page returns a valid 200 OK, not a broken 404

• It's not blocked by robots.txt

• It’s actually listed in your sitemap.xml

• There’s no noindex tag silently excluding it

• Canonical tags don’t point away from it

• There aren’t 10 conflicting versions of the same page

• The page loads fast, has lang, viewport, and alt texts

In short: Technical SEO makes your content available to Google.

If your site has technical issues, Google either:

• skips it entirely

• or ranks it lower because it looks “broken”

SEO Tools:

• Semrush / Ahrefs → Keywords & links

• Surfer SEO → Content optimization

• TrackMySitemap → Technical SEO & indexability

• Screaming Frog → Full-site crawls

• GSC → Visibility diagnostics

Great SEO is not just about what you say. It’s also about whether Google can hear you at all.

A while ago, I launched a SaaS product. I did everything “right”: content, landing pages, internal links. But traffic just didn’t grow. Worse — pages weren’t even being indexed by Google.

The scary part? I had no idea.

Turns out, technical SEO is often silent. Your sitemap.xml says one thing. Robots.txt says another. Your meta tags block pages. Or worse — you have a canonical tag pointing away from your own domain.

No alert. No red flag. Just zero visibility.

So I built something for myself: a tool that runs 30+ technical SEO checks to scan your sitemap, analyze your pages in real time, and spot conflicts before they destroy your search presence.

I call it: TrackMySitemap – your SEO Firewall.

⚡ Why “SEO Firewall”?

Most SEO tools are reactive. You get a report after your traffic drops.

TrackMySitemap is different. It audits your sitemap and pages like a firewall: constantly checking for indexation blockers, broken links, and metadata issues — before they hurt you.

It’s not about chasing rank. It’s about protecting visibility.

🛠 How it works

• Paste your domain

• We find and parse your sitemap(s)

• We run 30+ tests per page, like:

  • noindex tags

  • robots.txt rules

  • missing canonical tags

  • duplicate metadata

  • broken links

  • thin content

• You get a clear report: what passed, what failed, and what’s blocking you.

🧪 Why I built it

Because SEO issues shouldn’t be discovered after they hurt you. I wanted something to act before the damage is done. Something to give me peace of mind that Google sees what I intend it to.

That’s what a firewall does.

👉 Try it free

If you’ve ever launched a page and found it missing in Google… or shipped content that didn’t rank… it might not be your keywords. It might be a silent technical issue.

Check your site now

Hey Indie Hackers 👋

I recently launched TrackMySitemap, a small tool that checks your sitemap for technical SEO issues.

The idea came from personal frustration, I’d spend time building sites, only to find some pages weren’t indexed because of silent issues like:

• Broken links in sitemap.xml

• Robots.txt blocking URLs listed in sitemap

• Core pages missing entirely

• Invalid XML structure

So I built a focused tool to answer one question:

🧠 Is your site being indexed properly?

No guessing, no bloat — just clarity.

After scanning over 100 websites, here’s what I found:

• 42% of sitemaps contain broken or redirected URLs

• 1 in 4 sites block their own sitemap URLs in robots.txt

• 18% fail basic XML validation

• 35% are missing high-priority pages

I’d love to hear how you handle technical SEO for your projects and if you want to give it a try: https://trackmysitemap.com

Happy to answer questions or share what I learned along the way 🙌

Hey Indie Hackers,

Just a quick update from my solo project TrackMySitemap, which I launched a week ago.

The idea behind it is pretty simple:
Most SEO tools give you surface-level reports. I wanted to build something that goes deeper, a tool that scans your sitemap, checks every URL, and uncovers hidden SEO issues that can quietly kill your traffic.

Last week I released the MVP with 12 automated SEO tests, and the response has been great, people are already finding broken links, misused meta tags, indexation issues, etc., that other tools weren’t showing.

But I realized... it wasn’t enough.

A few users gave feedback like:

“I fixed what you found, but is there anything else I might be missing?”

The answer is almost always: yes.

So this week, I’m rolling out 20+ new tests, things like:

• Orphaned pages

• Canonical + hreflang conflicts

• Pages blocked by robots.txt but still in sitemap

• Soft 404s

• JavaScript-blocked content

• Thin vs. duplicate ratio

• Internal linking gaps

• Sitemap vs. crawl mismatch
  ...and more.

These are the real SEO killers and most tools don’t check for them at all.

Why this matters

After this update (in a few days), TrackMySitemap will run 30+ checks per site link.
I don’t know any other tool that goes this deep by default, especially not in this format (fast, focused, and without bloated dashboards).

If you run a content site, SaaS product, or just want to make sure your SEO isn’t leaking traffic — give it a try:

👉 trackmysitemap.com

Would love to hear what you think and what else you'd want it to check.

Thanks for the early support! It really means a lot 🙌

Last week, I got humbled.

Someone dropped a URL into TrackMySitemap.com — my lightweight SEO scanner — and hit "Scan".
It was just another day in indie SaaS land… until it wasn’t.

Boom: 20,000+ URLs in One Sitemap

Turns out the site was bilibili.com, a massive Chinese video platform.

Their sitemap contained over 20,000 links.

Here’s the kicker:
I had a hardcoded limit: 500 URLs per scan.
I thought, “This should keep things safe.”

But guess what?

The limit didn’t work.

How It Was Supposed to Work

The logic was simple:

if (urls.length > 500) {

urls = urls.slice(0, 500);

}

Clean. Simple. Buggy as hell.

In edge cases, the slice was skipped entirely.
Maybe malformed XML, maybe async race condition — maybe I just messed up.
Result: the server tried to scan 20,000 pages in one go.

And it died.

I Had Two Options

1. Blame the user for pushing my system beyond its limits

2. Fix the damn thing and make my tool better

I chose option 2.

Fixing the Mess (and Making It 10x Better)

Over the next 24 hours, I:

• Added enforced link caps, not just in the parser but across the whole flow

• Rewrote the sitemap processor to handle large jobs in chunks

• Added parallel scanning threads with batched timeouts

• Improved memory handling

And… it worked. Like really worked.

I re-ran the scan for the user. It took longer, but didn’t crash.
All 20,000+ links were handled gracefully.

What I Did Next

• I emailed the user and said sorry

• Sent them their scan link

• Gave them a 25% discount coupon

They appreciated it.

Lessons for Indie Hackers

1. Your users will do things you didn’t plan for.
   Don’t assume they’ll play nice.

2. “Limits” aren’t real unless they’re enforced end-to-end.
   I had a limit — but it wasn’t backed by code in the right place.

3. The first "fail" is your best QA moment.
   If no one's broken your app yet, you're not live enough.

4. Own the mistake. Fix it. Thank the user.
   Early adopters are your best critics and biggest fans — if you treat them well.

What’s Next?

I'm shipping new features this month:

• Scan history for each site

• More accurate indexing analysis

• Performance tracking

• ...and yes, better alerts when something breaks 😅

If you want to check your own site (and hopefully not break my app 😅) try it here: TrackMySitemap.com

Your Turn

What’s the biggest stress test your product went through?
Ever had a user teach you something painful but valuable?

Let’s swap stories below.

Hey Indie Hackers 👋

I’m building TrackMySitemap — a tool that scans and analyzes your website's sitemap for SEO issues (like pages blocked in robots.txt, missing from sitemap.xml, poor structure, etc).

The problem I ran into:

Originally, scans ran on a cron-based system, depending on subscription level (e.g., weekly for Starter, daily for Pro). It was simple but had two issues:

1. Users wanted more control — not everyone wants automatic scans.

2. It felt abstract — users weren’t sure what they were paying for or what would happen next.

So I switched to a Scan Credits model

Now users can launch scans manually, and each scan costs a credit. This made things:

• Clearer: "1 scan = 1 credit"

• More flexible: You use credits when you need

• Easier to explain in pricing

Next step: pricing model

Right now I’m thinking about:

• Monthly Plan – $15/mo for active users

• Lifetime Plan – $100 one-time for early adopters (includes a certain number of scan credits)

• Free Tier – for trial/lead gen (limited scans & features)

My questions for you:

• Have you tried credit-based models in your SaaS? Did it help retention or clarity?

• What do you think about offering both monthly and lifetime pricing?
  Does this make the offer clearer or more confusing?

• Any examples of tools you love that nailed this balance?

Would love to hear your feedback — and if you're curious, the app is live:
https://trackmysitemap.com