I slept through 6 hours of downtime during a launch 🤦♀️🚀💥
I wanted to share a major failure of mine.
As the title says, my app's backend went down while i slept for about 6 hours. It was pretty devastating because it happened to be on the day we launched on Product Hunt. As soon as I woke up and saw that there was an issue I was able to fix the issue and bring the app back online within 5 minutes... but the damage was already done.
In total we dropped between 20-50 signups (which is a lot for us as we're just starting out) and we got two 1 star reviews as a result. Needless to say, I felt extremely stupid.
To prevent this from happening again, i'm using a low end Android phone that's only job is to receive Heroku Alert emails and I sleep with it next to my head so I wake up if/when there's another issue.
What do you use to notify you of issues like this while you sleep? Does anyone else even experience downtime?
Also please share your biggest fuck up story, so i don't feel so bad. 😿
The graph shows signups per hour. It's easy to spot when we went offline
UPDATE:
Turns out there was a bug in my app that was causing database connections to not be properly released in some cases. Huge thanks to @BetaPeak who just helped me debug and fix the issue. This community is so awesome.
But have you tried dropping your entire database 6 months after launch, with no backup? Sht happens. Welcome to the club and good luck with your product!
Oh my fucking god. No way that happened! I would die. Literally though cause my cofounder would kill me.
Oh, yeah way! I built the product very quickly, and when it caught fire, I hired external help. They didn't know, and I forgot, that
db:seedwas attached to the production key, there was no staging. Lucky for us, they ran it during the day. We watched the data disappear, live. I called them and they stopped the script, meanwhile we started salvaging cache files. When it was all over, I had to email all the affected users and ask them to reupload. Took a few weeks, but people were understanding.I am doing an AMA on here this Thursday, feel free to ask more questions ;)
This is why we have built an additional check in db:seed where it ensures that it does NOT run in live/production environment. If live environment, it will error out :).
I've done similar things with businesses in the past, including Indie Hackers. I think the second time IH was #1 on Hacker News, the site went down for multiple hours and I was away doing who knows what. Not a great feeling to come back and discover that. 🤦🏿♂️
Just keep pushing!
Oh wow... well it looks like you guys survived 😅. That makes me feel better. Thanks for sharing
Hi @desmondmc, there is a nice tool I use on all my projects called: https://uptimerobot.com/, it has a free plan and checks continually the availability of your service and notifies you by email, SMS or voice call ...
I hope this helps you
I've been using uptime robot for a couple of years. Works well.
You had me at voice call! This is exactly what i need. Thanks, this is super helpful! I was actually thinking of building a service like this if it didn't exist.
I once pushed an iOS update that wasn't fully QA'd and crashed if you tried to make a new account. Users got very mad, some left bad reviews AND I had to wait a full day for Apple to review my version with fixes. 😱
Try reaching out to the 1-star reviews and explain the situation + fix. More often than not, they'll be willing to change the review and maybe even become your biggest fans 😄
Oh man that's rough! I remember when apple reviews could take weeks sometimes. It's a good thing that wasn't the case here 😂. Yea it's good advice though, we've reached out to the users and are hoping they'll change their review.
What was the cause of the downtime?
Good question!
TLDR: I still don't know exactly.
I'll tell you what i do know, maybe someone here can help 🙏. My backend is running is node so it's obviously single threaded. Something seems to happen during login and signup requests where the JS thread just locks up and stops handling other incoming requests. It's happened a couple times before and it always seems to happen right after a login or signup request. Once i restart the heroku dyno the problem goes away.
The thing is i have no idea what could be causing it. I'm not doing anything complicated during these requests just hashing the password using
crypto.pbkdf2and doing some routine database calls.If anyone thinks they could help i'd be happy to share the source and my logs of the incident.
Total guess: an Express route isn't calling
next, or a similar type of callback isn't being called to advance the stack. So it just hangs.Updated the post but it turns out i wasn't properly releasing db connections in some cases. So once they were all used the app just stopped responding to requests because it would wait forever for a connection from the pool to free up. Rookie mistake.
This comment was deleted 6 years ago
Do you have any error tracking set up? I use sentry.io.
At the moment i get by with just https://papertrailapp.com/ logs. If an error gets thrown it shows up in the logs there and it's usually enough to debug. This probably won't be enough as we scale but for now it's ok 🤷♂️.
But in this case no error gets thrown. It behaves like the code just ran into a
while(1). Heroku router keeps sending the app requests but they just timeout because the app doesn't handle them.Yeah that’s a lot more tricky to debug, but you might still receive an error, or at the very least a timeout error, which may give you a clue as to what might be happening.
Perhaps you can alleviate things by setting up clustering so that your app doesn’t go down entirely. Might help you get better sleep at least!
I think we've all been there at some point, not long after I joined my current job at a Telco software company I misread a log file and thought it was ok to do a restart. Accidentally dropped 10000 calls of the network 😭
If you want an affordable crash reporting tool I've developed CritiMon.com. There's a free account of you want to give it a try. As you say it wouldn't have identified this specific use case but it will automatically detect unhandled errors.
Yikes! Hope you didn’t get in too much trouble for that. Thanks I’ll check out Critimon :)
Thankfully not. It was one of the more understanding customers that knew it was a mistake. If it was another customer may have been in a bit more trouble lol
Have you managed to monitor memory usage for the node process? It may have some sort of a memory leak and ultimately hanging as a result from it. Does restarting the process (not the dyno) solve the problem as well?
Yea Heroku monitors memory usage it stayed flat throughout the outage. You can see in this image:
https://ibb.co/WcbQLHg
It's interesting how requests drop to 0 for a few minutes before the responses start timing out at 30s. Any ideas why that may be?
Yea I saw that too. That's actually a 30 min period where heroku is reporting no traffic but the thing is the logs show that we were handling requests fine before the timeouts were happening. You can see that in the logs here we sent a response 9 seconds before the timeouts started happening:
https://ibb.co/mcvYTRb
Not sure if worth noting, but your last request was 409 Conflict... depending on your app logic and what you do in that Exception handling, you may have done something to either the app state or the db state to break all subsequent requests. Just guessing though, no way to say without the source :)
Shit i didn't even notice that 409. Yea you might be onto something. If nothing else maybe i can reproduce the issue by recreating that collision.
If you're interested I'll gladly give you access to the source.
Sure man, I'd be happy to have a look. On most major git repos - yasen on the betapeak.com domain (that's email obfuscation level 999)
hahaha ok i'll add you now. Please don't judge too hard as this is my first major node app. Maybe let's move this conversation to twitter. Message me @desmondmc89
Many (many) years ago I soft launched a web service that created on the fly screenshots of websites. Some steady traffic then a couple of weeks later a blog wrote a nice article about it, which was shared around by others. Completely overwhelmed the small server.
At the same time my wife went in to labour. I didn't see all the emails and snarky comments for nearly 48 hours and didn't have the time to fix it all quickly.
You take the good with the bad. Your product is now better for the downtime.
if you share your story on /r/tifu the same interesting way you did here you can count on redeeming lost users. Some of the stories on this subreddit are going really viral.
Ha that's a not a bad idea. I'm thinking maybe people outside this community might not think it's such a big deal. Worth a try though
Yep they might not. However, if you sell your story with a little more details from a perspective of non-founder person it may sound a bit more like a big deal:
Don't forget to add details of your app somewhere
We just implemented OpsGenie for this type of alerting. We just had some downtime recently related to assets incorrectly being referenced.
I used to work for a downtime communication product so very familiar with this. We had an internal thread where people would talk about their mess ups -- It happens to everyone at all levels! Transparency is often the best way to handle it if it impacted users and owning the mistake/talking about it helps internally :)
There's ups and downs.
Can feel your pain @desmondmc 😓
🤗
Ah no! I feel for you man!
I was actually just thinking about "up-time monitors" today, will do some research,
otherwise I'll be building one to monitor front and back-ends.
Like your spirit. You jumped right out of the guilt. Super cool.
The website looks awesome. :)
Sorry to hear about this. I am really paranoid about downtime, so I have set up a plethora of alarms in CloudWatch (we host on AWS) with email and SMS alerts should anything affect our server performance. I also link the SNS alerts to Notify17 which can send me phone notifications of problems.
On top of that I also use active external monitoring using UpTimeRobot.com to constantly check all our servers that they are responding normally to requests. I also run New Relic, and soon, AppSignal monitoring on the production servers so I can check logs and detect performance issues with the DB etc. I also write a custom dashboard for our tech team to use using CLoudWatch graphs to monitor CPU, Memory and Traffic performance (referenced in this tweet: https://twitter.com/dsabar/status/1193709219834257408)
In any case, rest assured - As long as you use this incident as a learning exercise and put in place steps to prevent it happening again, your startup will be the better for it in the long run.
This comment was deleted 6 years ago
Yea you're probably right. I was pretty down about it yesterday but i'm already starting to laugh at my own stupidity.
Thanks!