Hacked
Got up this morning to log into Centori as our intern had saved a blog post that she wanted me to review and I could not log in... odd.
I figured perhaps lastpass had the wrong password as I recently changed it, so I tried a password reset email and there was no record of the user found... heart beating harder now.
Logged into my hosting admin and saw that a ton of data had been deleted, then hopped into my usage tracking app to see who logged in and what they did. It turns out one of our APIs was not secured properly and they managed to delete a bunch of users while keeping a select few.
I've taken the app completely offline, but honestly I do not know what to say. It's times like this, as a self-taught coder, that I feel like such an idiot who is in way over my head and it's painfully obvious now. What's even worse is I have paying customers who can be hurt by my incredible mistakes (thank God we don't store any payment info, thanks Stripe for managing that completely).
Has anyone every come back from something like this? Or am I done for? I've notified all our customers and have paused payment on their billing until this is fixed and we can talk about them staying on, though honestly I would not blame them at all for leaving... I would.
Update
It appears someone created an account and was able to expose a vulnerability in one of our APIs that allowed for deletion/updating of users was exposed so they were able to delete me and a bunch of other accounts as well... Fortunately password info is not available in the API so that should be safe, but man this is absolutely crushing me.
One final edit
I just wanted to add to this post that I am amazed and humbled by the response it got, thanks so much to everyone who commented/reached out via email it really means a lot. I made a mistake that came back to bite me, but it's comforting to know that it doesn't have to be the end of me. As long as I learn my lesson of course, and I've definitely learned this one.
As a fellow self-taught coder, I can tell you that this happens to people with degrees as well. You live and you learn. Next time you'll be super vigilant about security.
Just seconding this comment. This happens to huge organizations all the time (unfortunately). Security can be a hard problem, and regardless of how you learned to code, unless you have specifically worked to learn cybersecurity you likely have gaps in the area (I've worked with dozens of software engineers professionally and I'd say MOST of them don't have a keen eye for security).
It's great you protected the most sensitive data in this case. If you've got to get hacked, inconveniently losing "non-sensitive" data is definitely better than losing things like passwords, SSNs, CC#'s, etc.
I would be curious to know what the API vulnerability was so others can learn from this? Maybe in a future post after its fixed!
Thanks @allison and @devenjarvis! That helps put things in perspective, this has been a great wakeup call to how things can be abused and the need to focus on that earlier rather than later. Just glad this happened before I started scaling or thought about quitting my full time job.
Def can do a post on it in the future too in more detail- it was a pretty simple backdoor I left open. I built the app on Django and use the Django Rest Framework for our APIs, the issue was that objects were accessible to users via our API that should not have been accessible. This allowed the bad actor to delete users he/she should not have been able to access (including me) as well as other data.
Huge oversight, but fortunately easily fixable as DRF allows for customizable permissions.
ah damn,! That's really unfortunate to hear - did you have any backups? Was the issue these routes were unauthenticated? Or they were authenticated but didn't have proper authorisation?
Unfortunately we haven't backed up in a while... another thing to work on as regular backups would have been a huge help here.
We just had a check for whether the user was authenticated but not whether they had object permissions, so anyone could create an account and access data that belonged to other users. I figured 'we're small, this won't be an issue for a while' and sadly it bit us.
+100
I am surprised that you look up to people with degrees as if they have more knowledge compared to self-taught coders, curious if you can share why do you think so?
Eh I guess it's just that (I think) they have more formal training, though from what my developer friends say they rely far more on documentation/forums than they do their education.
I would suggest getting your website/API audited by a security specialist or a security firm so that ur other vulnerable endpoints gets patched before a hacker finds them.
Thanks! Are there any you'd recommend/have worked with?
I can recommend https://appsecure.security/
have you worked with them directly?
nope I haven't worked with them but they are quite famous in web security community.
I get hacked almost on the regular and it's actual money that hackers drain from a hot wallet (bitcoin) it hurts less and less as I patch things up. And I've had no incidents for months. So you live, you learn, you start thinking the way they do. It's all part of it.
You may not be able to prevent hacking as there's always a way but you can make sure backups are in place.
SQL Injection? Or too many rights on the API that was able to delete most of the data?
The latter - some of our APIs only checked for whether a user is authenticated but there were no object-level permissions.
For ex, I can access my user info, but also other user info (just first/last name and email) and delete the user.... not good at all and bit us hard.
Happened to me once but it wasn't this bad. I've been into hacking for a while, so I know ways to get hacked and also to secure websites. Feel free to reach out to me in case you need help securing your app.
Thanks I'll shoot you an email!
Sadly this is something that happens from time to time. Sorry you had to go through it, it really does suck.
Your best course of action in terms of 'coming back' would be to restore from your daily or hourly backup but keep the app offline. Then fix the issues you know about, and if you can afford it, hire a pen tester to help find more issues.
One other thing you can do is open a bug bounty program, offering cash for reporting vulnerabilities. Ethical hackers do this for a living and it's better that they find a flaw than an unethical one.
You're certainly not 'done' - you can recover from this, but you've then got a lot to do to rebuild trust with your cystomers, especially when you're a small operation.
Personally in terms of customers I'd be totally open and honest. Create a blog post as a post-mortem explaining what happened, who and what were affected, and what steps you have taken to make sure it doesn't happen again. Allow your customers to comment - some will rage and that's to be expected. The best thing you can do is keep calm, apologetic and honest.
As a sidenote, before you even consider opening your product back up get an SSL certificate. Theres really no excuses for not having one in this day and age, and as I'm sure you'll be aware Google will significantly lower your ranking. Letsencrypt is free and takes just a few minutes to set up.
Good luck with the rebound :)
Thanks! Got the SSL cert set up and already working on closing existing holes before taking everything back online.
You're right, being communicative and honest is key in rebuilding that trust.
I wouldn't worry too much, it happens to everyone, even big known companies have vulnerabilities, even more stupid issues like leaving an unsecured database out on the Internet at times!
In some ways you've probably gained more trust from your existing customers as you were open and honest about what happened instead of trying to hide the fact that you had a vulnerability.
I could name a couple of reasons why that would happen in big companies. For example it can happen when hiring or delegating work to people that don't know basic security theory. In my company we are developing a micro-service for authentication/authorization which is basically secure by obscurity, like the one used in OP's API. However, as a junior dev, I don't have enough power nor responsibility to jump in and suggest to rework the architecture in such a way to become more secure, even though I know how to do it and why. I know this is going to bite us sometime in the future (hoping to be in a better company by then). I'm sure stuff like this happens all the time everywhere.
Oh yea definitely, that is very likely the reason for big companies to have security issues. My point was for the OP that these vulnerabilities happen to everyone, no matter if you're in an indie developer or a massive well known enterprise company. Its unfortunately just something that happens from time to time, its unfortunately impossible to think of every potential issue and its how the company handles such issues is how customers perceive your business.
Hello Tyler,
Not a cool thing, as it was clearly meant to harm. After all, the person who found the exploit could simply have reported it, potentially getting paid a small reward in the process.
As others have said, securing a platform is difficult, being self-taught or not does not really make any difference: you don't really get taught security anyway).
Large companies are actually often doing worse due to the complexity of their infrastructures as they juggle with multiple departments and spreading responsibilities. Developers don't bother thinking about security since there are dedicated security teams. And dedicated security teams are too far removed from the actual development process to really be efficient and simply end up setting global rules and defining best practices (and then those best practices get pushed at the very end of the backlog because let's face it, it won't add any visible value to the product).
One thing I haven't seen mentioned that is also important, next to security: did you have a backup, how old was it, and can you use it to restore your platform?
I'm sure you will recover from this, just be transparent with your customers and they will understand. And if they don't, you will quickly find new ones and just have a little more churn :-).
Thanks I really appreciate it, unfortunately our latest backup is pretty old as I also haven't taken the time to automate them... but that's also at thee top of the list to ensure we can weather these storms a lot better in the future.
I had similar happen to me. Except, it was a fatal event for my startup because I was bootstrapping and outsourcing development but could not afford to pay an expert to patch vulnerabilities. Ended up shutting everything down and wasting a lot of money.
Oh wow I'm really sorry to hear that, that's really tough.
Personally i would never use a website that is not served on a secure connection (https), especially when it is a product that claims to improve your ranking on Google.
Argh I keep forgetting to set it up, bought the cert and setting it up now.
Hey don't stress, everyone makes mistakes! FYI for the future to save yourself some money, most people now a days are using letsencrypt which provide free SSL certs and free updates. They also provide a lot of CLI tools for easy setup
You really don't have to buy anything these days:
Thanks for the tip! I use let's encrypt for my web app but the website is a managed wordpress site with godaddy and let's encrypt advises against trying to connect their cert to godaddy.
I bought the ssl cert ($70/year) though will look into some other hosting options that can give a bit more flexibility there
I'm pretty sure everyone advises against anything that has to do with Godaddy :-)
Ha yeah I've been looking for options as I've been thinking about getting the website off of Wordpress as well... but that'll come later :)
It's stories like these why motivated me to build a verification system project. Too much security hacks these days.
Damn, thats really awful! It's a good practice to keep back up of your data.
Also another "good" way to look at this is you have created something successful and with it you got haters. So just keep growing and leave them behind
Ha that is a nice spin on it, thanks!
Phew, that must be scary. Don't blame yourself too much. A good excuse to nail web security through this incident.
Thanks -definitely. Already got our APIs secured to close the backdoor they found but going to give a lot more thought into any other ways we're exposed.
Man, that really sucks to hear! That's so weird - I think of small IH SaaS as such a low target for hacking.
I hope you had backups!
The good thing is, pretty much every successful IHer has said they had moments where their app crashed or did something very wrong, they thought the company was dead, but they came out stronger than ever after being open, communicative, and working hard to fix the underlying issues.
Thanks that's good to know, definitely not what I was hoping for on a Saturday morning but at the very least I've got some good lessons from this and so far my customers are understanding.