🤖 What Is Robots.txt?

Let’s go back to our LEGO city.

You’ve already built a cool map for Google (your sitemap.xml), showing all the buildings and streets it can visit.

But now you realize…

"Wait! Some places are private! I don’t want Google going into my secret lab or unfinished buildings."

So what do you do?

You put up a sign at the front gate that says:

"Google, you can go here, but not there."

That sign is called robots.txt.

🪧 What Does Robots.txt Do?

It’s a set of instructions for search engine robots (like Googlebot). It tells them:

• Which parts of your site they can crawl

• Which parts they shouldn’t

But here’s the twist:

Robots.txt doesn’t block people — only polite robots that follow the rules.

Google listens.
Other bots might ignore it. But if your goal is SEO, robots.txt is the polite way to set boundaries.

🔍 How Robots.txt and Sitemap.xml Work Together

If sitemap.xml is your map,
robots.txt is your rulebook.

They go together like this:

• sitemap.xml: “Here’s everything I want you to find.”

• robots.txt: “But don’t look behind these doors.”

In fact, you can tell Google where to find your sitemap right inside robots.txt.

User-agent: *

Disallow: /private/

Allow: /

Sitemap: https://example.com/sitemap.xml

Let’s explain what this means…

🧾 Breaking Down a Sample robots.txt

User-agent: *

This means: "The rule below is for all robots."

Disallow: /private/

This says: "Don’t go into the /private/ folder."

Allow: /

This says: "You’re allowed to visit everything else."

Sitemap: https://example.com/sitemap.xml

This tells the robot: "Oh, and here’s the map of my city."

🛠️ How Do You Make a Robots.txt File?

It’s just a plain text file, like a note you write in Notepad.

1. Open a blank text file

2. Write your rules (like the example above)

3. Save it as robots.txt

4. Put it in the root of your website
   So it’s available at:

   https://yourwebsite.com/robots.txt

🤔 Should You Always Block Things?

Not always.

But here are a few smart reasons to use robots.txt:

✅ You want to block:

• Admin pages (/admin)

• Test pages (/staging)

• Search results pages (/search)

• Duplicate content

⚠️ But don’t block pages if you want them to appear in Google Search!
Google can’t index what it can’t see.

🚫 Warning: Blocking ≠ Hiding

Blocking a page in robots.txt tells Google:

“Don’t crawl this page.”

But if another site links to it, Google might still index the URL — just without the content.

If you want to completely hide something from search, use meta tags like:

<meta name="robots" content="noindex">

✅ Summary (Like You’re 5)

• robots.txt is a rule sheet at the front door of your website

• It tells robots: “Come in here, but stay out of there”

• It’s often the first file Googlebot looks at

• You can add your sitemap.xml link inside it

• Don’t block pages unless you have a good reason