Incident management as an indie hacker?

When running a SaaS or any kind of service that depends on something centrally is working (which is pretty much any software today) your users expect it to work 24/7. Some services may be more critical than others, potentially financially impacting your users, making a swift fix and immediate remedy crucial for your users and your business.

How do you deal with this as an indie hacker and solo-entrepreneur? Not having a team with an on-call schedule does mean that you are the single person able to deal with an incident.

What about longer flights? Family holidays? Vacations? Or even something as primitive as getting sleep at night.

One big learning for me personally was more than 10 years ago when I was running a client-server/online-based PC game[1] with many thousands of active, paying users.

Back then the server was co-located (cloud wasn't the standard yet) and the data center people accidentally swapped out the wrong hard drive in a RAID-5 failure, rendering the whole game unplayable and unable to restore the data. Hence everything, including the huge database, had to be restored from backup and that took about 10-15 hours.

Once the restoration was set in motion it was way past midnight after a very stressful evening. It was a pretty sleepless night. All the possible worst-case scenarios circled around in my head; when was the last time the backup was verified? What if it is old? What if it even backed up a corrupted database? What else do is needed to get the server up running again? What if it CAN'T be restored? Must the subscription fee be returned? What about the micro-payments? Users will riot losing their progress! It will be the end - of everything!!!

Thankfully restoration worked, and the game was up and running - but I was a complete wreck. I said to myself; "never again!", and set out on a plan to be able to sleep at night the next time we had an outage.

The most important insight was to ensure I was in a state where I KNEW the status of everything, so I didn't have to worry about it being the worst-case scenario.

Among other things, I did the following:

• Monitoring and alerts: never ever had to wonder if something was working or about to fail.
• Redundancy: multiple backups as well as (where financially possible) redundant infrastructure
• Restore verification: automated verification of backups and restoration
• Emergency plans: checklists of exactly what to do in case of an incident, including how to communicate with users

A year or two later, we had another incident. That time we followed the protocol, and then went out for a beer while waiting for restoration to complete.

The key learning for me was to prepare for mental health - if something goes wrong, how should I've had prepared to be able to cope with the situation in the best possible way.

However - even if small, we were a team working on the game. So we could make sure if one person was unavailable then someone else could step in.

But how do you reach peace of mind as an indie hacker when there is no one but yourself in case something goes wrong?

[1]: the game is still around actually, thehunter.com