Today at around 10:45pm, after a couple of glasses of red wine, I deleted the production database by accident ๐จ.
Thankfully my database is a managed database from DigitalOcean, which means that DigitalOcean automatically do backups once a day. After 5 minutes of hand-wringing and panic, We went into maintenance mode and were able to restore a backup. At around 11:15pm, 30 minutes after the disaster, we went back online, however 7 hours of scoreboard data was gone forever
oops... :D
Yup
lmao
It's sad, it happens, more frequent than you'd head aloud
With all the protections in place things still happen.
Did you separate a DDL vs DML user? (app shouldn't be allowed to change schema, only a deployment user)
You'r production doesn't use a server on
localhost
? are you sure it's that part of the code than? unlikely..config.DevelopmentConfig
a bit weird, don't you have someconfig.Env
to check?..Did you have someone connecting from their dev env to a production DB server? (hopefully this is blocked)
Or do you run things on the production nodes manually?
Were you trying to debug something in production?
...
These are all great suggestions, do you have more written out (maybe w/ more detail)?
A pre-deploy checklist or todo/readme to refer to would be great
mmm... I don't know where all of these come from, surely there is some rules of thumb security list, I kinda know them part from schools bigger part from work exp..
It mostly comes from the base "Principle of Least Privilege" - if it doesn't absolutely need a privilege/permission, it shouldn't have it
Never cross the streams
Production env. should be as locked down as possible in terms of who can access it and how it can be accessed. (beware due of locking yourself out accidentally, and you might block your own app 1st time if you do it badly ^^)
This might be partial but it's the big tickets of access lock down to prevent issues with production and production data.
This is great! I will try and implement some of these once I've figured out what went wrong
I've faced such situations too ๐
Which is why now I created this docker image using RCLONE and crontab that syncs up Mysql database and Minio storage to Google Drive (for free!)
One can easily customize the bash scripts to fit specific needs
Now all my servers have 30 min backups that I can quickly restore :D .. + backup service is free!
link to the repo ๐๐ผ https://github.com/negati-ve/docker-rclone-google-drive-cron-backup,
In case any one needs it
I've come close to it, but for a lesson I learned from a Hacker News thread a while back. I think it was when that one employee at Gitlab accidentally deleted a production database which caused 24 hours of chaos. There was one commenter on there who said that they have mandated within their own organisation, that before any database query is run, the operator has to pause and verbally explain (even if to themselves sitting at the console) in plain English what the command is going to do.
So, if you type in:
DELETE FROM dbproduction.user WHERE last_login < DATE(2020, 06, 01)
Then before hitting the 'Execute' button, you pause and say "I am about to delete users from the production database where their last login date was before 1st of June this year".
It is amazing how that focuses you, and any distractions by co-workers or emails or phone notifications get pushed aside as you listen to your own words. It really works to bring your mind back to what you are about to do.
Another we've done here is to have a separate db login for general viewing and maintenance on our database that will not allow a DELETE or DROP command to be run. For that, we have to use a separate login where the password has to be looked up in our password management system, and that password is never remembers, and rotated frequently.
Very interesting! Thanks.
The thing is, I wasn't writing raw SQL.
I know that pain. I once lost a production database by stopping an EC2 instance. The MySQL DB was on that same box. I didn't realize that stopping the instance wipes all of the data ("restarting" does not).
I was trying to save costs by not using a managed database (RDS), and had to learn the hard way.
We've all been there - we've all done that. Growing by mistakes also means that those things hopefully only happen once during a career :)
How you did that? could you tell us the story?
Sobered up pretty quickly I imagine.
That's why the backup plays a very important role... We often think that, Whats the point to pay for automatic backup or infrastructure security. But this couple of bucks saved the day.. Cheers!
It happens man..!
One of my colleagues at last startup accidently rhn the command
Sudo rm -rf /
He deleted the whole server.
Are you using environment variables for database connexion uri/username/password? Or did you put them in a config file?
You should use environment variables, so developers have localhost variables configured, but production variables are only used on the production server.
If anyone needs access to the production database, just give them read-only access and force them to use a GUI. It should be forbidden to have production access in your local env variables.
Yes, the problem is I have production ENV variables accessible on my dev machine. I need a change of policy.
That sucks , but things like that happen and your users should understand!
I backup my production database hourly , I wonder why digital ocean doesnโt offer hourly backups on managed databases?
This comment was deleted 3 years ago.