September 29, 2020

Accidentally erased user data for the first time.

Alexander Isora 🦄 @alexanderisora

I did this. I deleted a user website. Permanently and irreversibly. It was published and accessible for visitors from the Internet and I killed it.

The story begins in the morning. I woke up and noticed that server was getting too many requests in the past few hours. I checked the database and discovered a population of fake users and websites. Someone passed our protection and was able to programmatically create hundreds of new records.
I spent some time improving the filter and clearing the database from bots.

Even though I was as focused as a SpaceX Dragon pilot while deleting those rows, I did a mistake and accidentally erased a real website made by a real user.

The reaction from the owner came a little later:

Alt text

Me:

Alt text

Thank goodness, we make daily and weekly backups and regularly test the restoring functionality. So I was able to get the website back in a few minutes after we got the message.

Unicorn Platform customer reaction

Lesson learned: the less manual action you do, the fewer risks you have. We constantly add improvements in our processes to do as little manual operations as possible.

What is up

We are working hard to ship the most wanted update ever: the blogs.
You can become the first user by joining this waitlist https://unicornplatform.com/create-blog/.
Or by following our company Telegram https://t.me/unicornplatform or Twitter https://twitter.com/unicornplatform/

  1. 13

    Happy ending story, I felt a bit deceived by:

    I deleted a user website. Permanently and irreversibly.

    and

    Thank goodness, we make daily and weekly backups and regularly test the restoring functionality. So I was able to get the website back in a few minutes

    1. 2

      +1, typical click bait
      buy hey, guys were prepared for that having backup system in place!

    2. 1

      Hey. Sorry, I did not want to trick you. I mean the deleted data was actually erased from the disk not marked as 'deleted'. And there are no options to restore it.

      I had an outdated copy of the launched website on another disk so I was able to create a new website that looks the same as the erased one (except the editions made during the last 24 hours). This is not an act of data restoration in my opinion.

      Have a good day!

  2. 10

    Sh*t happens.
    This is my lesson from this post: do regular backups.

    1. 8

      And regularly try to restore them!

      1. 1

        So this was just another successful test! 😂

        1. 1

          Each "test" makes me jump out of my skin. NOT recommended 😂

  3. 3

    I'm forever backing up databases (just in case)! The importance cannot be understated!
    Even when I am running some code that cannot possibly make anything go wrong in the database, I will first do a backup! Extra bit of time is worth the extra piece of mind.
    Automating these processes early on in the development of systems is also a pretty important step I'd say!

  4. 3

    I know the feeling. When I was working as freelancer in the initial days of my programming, in my very second project, I reset the client's WordPress site, thinking that it'll fix the issues. Everything broke.
    Luckily the guy had backed up the site and I immediately restored it.

    Lesson Learned: Check everything once and once more before you are doing anything in production. Otherwise Murphy's law kicks in.

    1. 1

      I can imagine your feelings because I still remember my first projects. I was deadly afraid of failure because I might not be able to fix it :P

      Sometimes even triple check doesn't give a safe result :) That is why we have backups.

  5. 2

    Thanks for sharing! Great recovery, it's sad that bots target sites like this though

    1. 1

      It is fine. I'm thankful for this extra opportunity to test the backups :)

      1. 1

        I need to get daily backups going 😅

  6. 2

    Amazing lesson! CTRL + S forever

  7. 1

    Reads more like a marketing Trojan horse. Zero substance of interest.

  8. 1

    Ouch I can only imagine the feeling! :D I am glad you sorted it out :)

  9. 1

    Definitely speaks to the benefit of having separate DEV and PROD environments! Write your SQL, execute it in DEV, validate the outcome, then run the same in PROD.

  10. 1

    What do you use for your backups @alexanderisora? Is it something you built yourself?

    1. 1

      We use https://www.autobus.io/ which pairs pretty well with Heroku.

  11. 1

    Props to you for handling this stressful situation! I used to work in a company which hosted hundreds of databases for clients, and we had a nightly schedule which restored random backups on to a side server to test their validity. Was super helpful to us.

    1. 2

      Wow, man this is big. I can not even imagine the amount of responsibility, fun, stress such work can give.

  12. 1

    I do twice-daily remote database backups for Zlappo and even weekly snapshots of my production AND staging droplets every week.

    I keep my backups for at least a week, so that if I notice a problem I have sufficient backups and time to work with. If something happened over a long weekend, I can go back to the backup 4 days ago and restore that.

    You never know if you're going to get hacked, attacked, data will get corrupted, user mistake, your mistake, etc. etc.

    This is also why I don't give users the power to bulk-delete anything. It just seems like a recipe for disaster. I have cronjobs that periodically clean out old, unused, orphaned, or otherwise unnecessary data, so users won't have to worry about it.

    Whenever possible, always back up everything before you do anything

  13. 1

    Nice! Was it a raw SQL or did you manually delete the rows (with DBeaver or something)?

    1. 2

      I used the default Django admin dashboard to delete the fakes.

      1. 2

        I see Django, I give the person a 🍪

  14. 1

    It’s hard to explain to some people that database don’t have “undo”.

    I feel your pain.

    1. 1

      "Undo" would be a cool option! :D

  15. 1

    A similar thing happened to me last week on Bear Blog. We had a surge of spam blogs being opened and while I was very careful when deleting and blocking users, I managed to delete one actual user site.

    I completely understand your feeling!

  16. 1

    This comment was deleted 25 days ago.

Today's Top Milestones
  • New library - ICONZ
    I am so happy to announce that we created a new premium 3D library ICONZ 🎉 🎉 After months of work and preparation, it is finally here. I believe ICO
  • First production use
    Yesterday I received a message from one company that they use https://github.com/chatscope/chat-ui-kit-react to build their commercial product. The be
  • Crossed 500$ in book sales
    The book I wrote in May/June just crossed 500$ in sales. I put the book on Gumroad first as this was the easiest and cheapest way to share my work wit