Ever wiped out all your users' data? I did and this is what happened.

It was gone in the blink of an eye.

(Ryeboard)[http://ryeboard.com/] is a heavily content focused product so it has always done a great job with security and redundancy. The team took steps to make sure that our users' things were safe and trauma proof. Everything was looking great... until it wasn't.

During an overhaul to our file management system plugged into AWS S3, I ended up wiping out our entire system.I forgot to comment out the part in the code where the actual deletion happened. To make matters worse, our S3 service was the only part of our stack that was... not backed up.

Our users lost all their files in an instant.

Here's what I did shortly after that:

1. Had a mental break-down.
2. Slapped myself to attention.
3. Backed up our empty S3 stack (Soul crunching)
4. Apologized to my team.
5. Emailed all our users an apology.
6. Waited for the backlash

...but it never came.

In fact we received heart-warming emails telling us to not worry and just to make sure it doesn't happen again in the future. Some even upgraded to our premium plans with my "I'm-sorry-I-was-dumb-please-take this-coupon-as-my-pathetic-yet-only-way-of-apology" coupon. It was incredibly humbling.

This happened just a few weeks ago and I learned several lessons that day:

1. I am incredibly dumb.
2. Make sure to backup EVERYTHING. (For S3 buckets, make sure to version them.)
3. Honesty and communication is super important to your users. Especially when you're in an early stage and you make a mistake. Don't try to cover it up!
4. Users can be quite forgiving but don't abuse this trust!

I now have the clarity and courage to write this story out. I hope this story helps out those of you in a similar rut!