💀 Taking down production..
We did this at work recently and the stories that came out of it were pretty incredible to hear.
What is your worst "I just took down production.." horror story? I'll start.
One time, during YC ( S15) - a couple days before demo day I accidentally deleted most of the production data.. All of our users (~50 company's) data, gone. The mistake I made was thinking I was connected to my local dev instance, but was really connected to production.
I remember feeling like I just got punched in the gut, just horrible.
After some deep breathing, got down to work trying to restore the data. We managed to resolve it with a mixture of logs, a db backup and some difficult conversations with our customers.
I still get a hard time about it from my co-founders. 😂
Here's my similar story.
I was on my way to a massive client interview. I had the slides and the code on my computer. The meeting was at 7am so I had to wake up at 5 to make sure I got there on time. I managed to just make the bus and as soon I got on it I went to sleep, ready to wake up when I got closer to the building.
When I got to my stop I woke up and my laptop was gone. I didn't have any time. I went into the building and was going to explain the situation when I get a message that the person I was meeting with today had an emergency personal issue and would need to reschedule.
We met a week later and I was all fine, but the world is weird. Things have a way of working out.
Someone stole your laptop while you were sleeping? That's brutal sorry to hear, but glad it didn't ruin your meeting.
Oh wow. Stressful! Out of curiosity, how did you notify your customers about what had happened? And did you lose any customers from this? I assume there must have been at least some complaints or concerns - how did you handle these conversations?
This has never happened to me before, thankfully, but I bet it's more common than most people would think. I did work in a company a long time ago and I accidentally send a ton of confidential information to the wrong client, but it was my first ever job, I was 17 and my boss just took care of it. So, I still have no idea how I'd approach a situation like the one you describe - I can only imagine the stress levels!!
Luckily we were able to recover most of the data within a couple hours. There was some recently updated profile data that was lost and we had to get the users to make the changes again.
At this point in the company we were not handling as many critical pieces as we are today so there was no financial or massive impact to the companies. For the most part they were hearing about it from us first- which helps a lot 😅
Oh fair enough - close call! I guess in a way it's a good thing it happened when it did; at least now you're more prepared and managed to mitigate for an eventuality like this happening.
Another engineer and I updated a piece of credit card processing code and deleted about $1 million in credit card transactions. Unfortunately, we needed the data to reconcile and prove we were entitled to the funds, so we spent a few days recovering the information through logs and other tables. It was a nightmare and we still never found about $26,000 of it. That day I wanted to crawl into a hole and never be seen again!