How to: keeping vulnerability-scanning bots at bay from indie servers

Today I glanced at my server's CPU graph and saw this odd bump! For several hours the CPU was higher than usual. Hm!

I scrolled down and saw that the bandwidth was elevated during this period too!

Bandwidth graph

So I figured something was continually hitting my server. Looking at /var/log/nginx/access.log I saw a lot of entries like this:

access server logs

Over 240k attempts from the same IP scanning for commonly-available paths in web applications resulting in 404 responses. Well that's no good! It didn't seem to be causing major problems but I don't want that happening on my server.

But there is good news! There's a wonderful piece of software called fail2ban that can ban IPs that repeatedly try things and fail on your server. The most common is bots that try to log in to your server via SSH (check /var/log/auth.log if you want to see it yourself). By default, fail2ban bans these SSH bots after several failed attempts in a certain time window (all these settings are configurable so you don't accidentally ban yourself if you mess up a password or something).

If you don't have fail2ban installed on your server, install it right now. Your server will instantly get just that much more secure just from the SSH protection alone.

But I wanted to ban bots that created too many 404s on my server. So I made a /etc/fail2ban/jail.local file with the following rules:

enabled = true
maxretry = 5
bantime = 10m

enabled = true
filter = nginx-web-vulnerability-bot
action = iptables-multiport[name=HTTP, port="http,https", protocol=tcp]
logpath = /var/log/nginx/access.log
findtime = 1m
maxretry = 10
bantime = 1d

And I also made a /etc/fail2ban/filter.d/nginx-web-vulnerability-bot file:


failregex = ^<HOST> -.*\s404\s.*

ignoreregex = ^<HOST> -.*favicon\.ico.*

Together, these two files will ban IPs for a day that make more than ten 404 requests in the last minute. Just run fail2ban-client reload after you've created the files and they will be automatically discovered.

(The ignoreregex part is to make sure I don't ban actual users whose browsers automatically request favicon.ico on every page load since I don't have one at the moment.)

So far no bots have been banned on my server because I just did this but I did test it out on the existing logs.

I thought I'd share this little tidbit for your to copy and paste to your own server since there's a lot to think about as an indie hacker. I hope you find this useful! And if you have any other solutions or ways to improve mine, please let me know in the comments!

I got some of my information about fail2ban configuration from this article.

  1. 2

    I feel like an easier and cheaper solution is to use a CDN.

    1. 2

      You mean easier and cheaper than using a dedicated server? Because it's definitely easier and cheaper to copy and paste the lines above than switch to a CDN :)

      But yeah, "serverless" applications (or Heroku for that matter) can definitely help prevent some of these issues but they come with trade-offs — they're relatively new and less supported than the classic server. And a digital ocean instance is $5/month and I can put anything I want on my machine: my nodejs app, a database, etc. Is there a particular CDN you like for dynamic, stateful apps?

      1. 2

        they're relatively new and less supported than the classic server.

        That statement might have been true 30 years ago,

        We always use Cloudflare or AWS CloudFront.

        1. 1

          Cool, glad it works for you.

  2. 1

    You'll notice that these requests tend to be hitting php endpoints.

    If your site is NOT using php, you could add something like this to your nginx file (before all your valid locations):

    # block php scans http://goo.gl/pqrtgY
    location ~ (\.php|myadmin) {
    return 444;
    access_log off;

    it also prevents this bot requests filling up your logs.

    1. 1

      Nice! Unfortunately the bots hit a lot more than just PHP endpoints and I also don't want them using my server resources :)

  3. 1

    Caught a bot that was spamming me with 93 requests in a second :)

    screenshot of jail catching a bot
    screenshot of log file

Trending on Indie Hackers
How do you decide what idea to work on? 42 comments Design & UI/UX takes me so long- am I doing everything wrong? 32 comments Looking for feedback on a note-taking tool focused on your personal interests. 7 comments My new self destructing notes app is on product hunt today. Would love some support. 6 comments Here's how I'm going to try to scale my tech newsletter from 0 subscribers to 10 000 subscribers 5 comments Bad news for Indie Hackers... 1 comment