One file silently replaced my robots.txt in production — there was no build warning
This is an easy Next.js issue to miss in code review, but it can directly affect search visibility: when robots.ts exists, it can silently take over the robots output even if the project also contains a perfectly valid-looking robots.txt.
I used to treat robots.txt as the single source of truth. The file was correct, the deployment succeeded, the build produced no warnings, and the site loaded normally. The problem was that the App Router also contained an robots.ts file.
Next.js did not flag the two files as a potentially conflicting configuration. The generated metadata route took precedence, so I thought I was changing the robots file while production was actually serving a different version.
The frustrating part is that every local check passes:
robots.txt exists;
TypeScript passes;
next build passes;
the deployment command exits with code 0;
the homepage and application still work normally.You only catch it by requesting robots.txt from production and comparing the actual HTTP response with the file you think is authoritative.
sitemap.ts has the same class of risk. When a project keeps both a static file and an App Router metadata route, someone can easily edit the wrong source. That matters even more in an SEO or GEO workflow: you can publish a large batch of content, submit your sitemap, and wait for search engines or AI crawlers to discover it, only to find that they are receiving an outdated set of rules or URLs.
My current rule is simple: every project gets one authoritative source for each generated endpoint. I also add robots.txt and sitemap.xml to the post-deployment smoke tests and check the production response, not just whether a file changed in the repository.
A successful build only proves that the application can build. It does not prove that the crawler-facing endpoints are serving the content you intended.
Do you use robots.txt or robots.ts in your Next.js projects? Have you added these metadata routes to CI or deployment checks?
It's great to see awareness around the importance of managing the robots.txt file, especially during batch content publishing. I ran into a similar issue early on with my project, where a minor oversight in version control led to an incorrect robots.txt being deployed in production.
The impact was significant our site traffic dropped by about 30% in a week because search engines were blocked from indexing key content. To ensure something like that doesn't happen again, I implemented a few critical checks in our deployment process. Here are some steps that worked for us:
Version Control: We added a checklist to our deployment procedures to track any changes to core SEO files, including the robots.txt. Every change must be reviewed by at least one other team member.
Automated Testing: We set up a staging environment where we can preview the robots.txt before it goes live. This way, we can see the impact of any changes in a controlled setting.
Monitoring Tools: After making changes, we actively monitor search console alerts and crawl stats to catch any issues early. Establishing a routine to check these stats weekly has helped us stay ahead of potential problems.
Backups: Keeping several versions of our robots.txt on hand has been crucial. This way, we can quickly roll back any changes that might disrupt our SEO strategy.
Choosing to automate parts of the content publishing process certainly helps in managing these workflows more effectively, but it's essential to maintain control over core SEO elements like the robots.txt. It’s all about balancing efficiency with quality management, and sometimes, a little caution goes a long way in preserving your hard-earned SEO gains.