How I Reduced AI Bot Traffic and Improved Site Performance Using LLM.txt
I learned an interesting lesson about AI crawlers this month.
I run a small SaaS that depends heavily on organic traffic. After noticing some long-tail rankings slipping, I dug into my server logs and found AI crawlers hitting the site almost continuously. Besides consuming bandwidth, the extra requests were increasing page response times during busy periods.
I started reading about LLM.txt, which works similarly to robots.txt but is designed to provide guidance for AI crawlers. After creating one and setting rules for the models I wanted to manage, my server load dropped noticeably and overall performance became much more stable.
I'm still monitoring the long-term SEO impact, but it made me realize that AI crawler management is becoming another part of technical SEO that site owners should understand.
Has anyone else started using LLM.txt or monitoring AI crawler traffic? I'd be interested to hear what changes you've noticed.
Interesting approach. I've seen similar issues with aggressive crawlers, but setting custom rate limits felt like a pain. Does the generator handle custom user-agent strings for less common crawlers, or just the big ones like GPTBot and Claude?
Nice find on the LLM.txt approach. I've been dealing with similar crawler issues and just blocking GPTBot at the server level, but your method sounds more surgical. Does the generator handle custom user-agent strings for newer models like Gemini or Perplexity?
Interesting approach. Did you notice any change in your search rankings after implementing the LLM.txt file, or was the benefit purely server-side? I'm curious if restricting AI crawlers might indirectly affect how your content is referenced by AI-generated summaries.
Appreciate you checking in! The LLM.txt generator was a game-changer for my server load — hope it helps you too.
Interesting. I've been seeing more discussion around llm.txt , but I am still unsure how much of the improvement comes from the file itself versus blocking or rate - limiting specific crawlers. Have you noticed any measurable impact on AI referral traffic or search rankings yet ?