Got Hacked. How do you guys handle this situation?
Hello guys,
as said in the title I got attacked by some unknown person that created more than 5.000 accounts in just the span of 7 minutes!
The only thing that I could do was shut-down my backend and wait it out ... and I hated it. Now I've added some basic ip-filtering, but I'd like to know how you guys handle this.
Basically, how do you protect your front facing APIs (signup, signin, etc.) from just being spammed?
Thank you for your help.
P.S.
Since others may be interested, I also added a delay to my public facing APIs, so instead of responding to a request in 500ms it actually takes a couple of seconds, which an actual user wouldn't bother, but a bot will definitely be slowed down.
Check out CSRF token, and rate-limiting. What framework do you use? There should be something ready to use, without getting into more complicated setups like API gateways, WAF, etc... If I were you I would start with googling "<framework name> CSRF token".
That's not good advice, CSRF won't change anything, you need to make step changes here, the best solution is three fold:
on signup - require a federated account. Make it someone else's problem, don't allow username + password. Not only is that a terrible user experience, it also is a ton of work to get right. If you federate to google sign in (or fakebook, ewww), they will handle it for you. Then in your service, limit users to one account per federated login (or however many you want)
on api requests - every cloud solution has something to help prevent attacks at the application level, you can rate limit on ip address, or have different rate limiting levels based on subscription plans.
care less - why do you care that 5000 accounts were made, if you can't handle 5000 accounts, which is a measly 12 api calls a second, you have some critical issues with your infrastructure. Why is it a problem that someone even created 5000 accounts?
Well the problem with those accounts are:
But thank you for your suggestions. Regarding federate accounts, I don't see many B2B SaaS doing it, is it just me?
We only use SaaS products if they provide federated login. Otherwise it is a non-starter. Obviously they also need to support SSO on the cheapest plan as well. As a matter of fact, our product: https://authress.io does not offer username/password.
Yeah, that's another reason to never let the user specify their own email. It should always be generated and verified by another provider.
Reporting should never drive architecture/security/performance/etc... That being said, # of accounts probably isn't the metric that provides meaningful value for the business, sure it is interesting, but not valuable.
CSRF does not prevent this from happening, it just makes it slightly more difficult
Making it more difficult is exactly the point of this exercise. It cannot be completely prevented as far as I know. We make it harder to break into and hope that bad guys will go hunt for less protected prey.
Very important here to not make it so hard that regular users will start dropping at onboarding.
+1 for @drikerf suggestions:
Also interestingly no one mentioned Cloudflare, they have "Under attack mode" which can be activated manually. When active it will start showing captcha to new users. This will help to keep the app online while complicating attackers life.
That was a really good question. Seems like many people have this problem.
I am DevOps myself. I have to deal with this kind of stuff all the time. The use of bots grows every year, they are not going away anytime soon.
Can you please explain in a couple of sentences what a CSRF token does? And how it helps to stop this kind of behaviour?
Sorry, but I'm not knowledgeable in security very much and would really appreciate your help.
CSRF (Cross-site request forgery) token is a one-time secret generated server-side. It is random and unique for each page load. It is added to the form (e.g. Sign Up form) as a hidden field like
<input type="hidden" name="csrf-token" value="CIwNZNlR4XbisJF39I8yWnWX9wX4WFoz" />
When the form submitted this token is validated and expired. It also expires over time. The server should reject the form if the token is invalid or has expired.
The main purpose of it to prevent attackers from constructing a fully valid HTTP request suitable for feeding to a victim user.
For example, if Indiehackers wouldn't have CSRF protection I could have tricked you to open my website, with a helpful security blog post for example.
In real life without a CSRF token, this request will fail, and there is no way for me to retrieve your CSRF token.
That was a simplified scenario, but I hope it explains it.
In our case, as a side-effect, it allows submitting forms only once. To submit the form again you have to load it from the form page with the new CSRF token. That makes the process more complex, cuts off simple form submitters.
Oh I see, thank you you have explained this very well.
Now my question is, who generates the CSRF? If it is the backend, then he could just request new tokens and then consume them?
It depends on how that happened. Did they start to do user registration requests directly in the backend? If they all came from the same IP, you can add some rate limiting logic to handle this exact test case. Or you can add email verification for all accounts, and restrict some usage until they validate their email.
Or you can prevent spammy emails to register by having a forbidden email domain list (including popular disposable email providers)
Or, when registering, you can issue a MX DNS request to actually see if there is an email server responsible for the submitted email address, and block registration requests that fail this check.
If the frontend was abused, you can add a captcha when registering to prevent that.
Honestly I don't know if it was done via frontend or directly in the backend.
The problem is that all the emails were from real people so they have stolen them in some way (probably email leak).
I managed to get the IP and banned it completely.
Anything you do is probably not enough to eliminate 100% spam. Combine 2-3 approaches. Limit signups based on IP within some time-frame. Do email verification. Do manual verification if you need to.
The only thing that will really work is paywall :)
The problem was that of all those emails were actually real!
I meant requiring they click a link in the email.
Okay sure, but that wouldn't solve the problem right? The account would still be created and I would still have to send an email to an unknown person.
You would automatically delete unverified accounts.
Security/Spam protection can be a tricky topic. Some things you can consider for the specific case of signups are:
Thank you for the suggestions!
Sometech stack like Firebase have concept called auth tokens, you can check at serverside if it is from authorized/logged in user. If you like to discuss further feel free to contact me.
Your website is looking great. I advise you don't mention about this attack anywhere on your site,blog or official twitter, these things may cause trust issues in financial products.
Hey, thank you.
The problem is that the attack was pointed at the signup form, and since I don't yet have an account for that part of the process, I don't understand how I can implement any authorization.
You are right, fortunately customer data is a top priority and no problem has been made for them, this was more of a nuisance for us, since a lot of spam accounts were created and a lot of welcome emails were sent.
Got it, its difficult if the attack is on sigup, as user is not yet loggedin.
CAPTCHA filter is the solution for these kind of problems, only humans allowed in.
I would guess the guy did it through automation. So one strategy is to place ReCaptcha before your key transactions to block all automation traffic. Then the attacker is essentially fighting against Google.
IP filtering typically works for elementary attackers. Advanced attackers can easily bypass IP-level defenses.
Overall anti-spam & abuse is a very challenging task (I worked in this space for more than 4 years).
Thanks for the help. A couple of questions.
How can they bypass IP filtering if it is done on the backend?
And for the captcha, doesn't it just prevent the form from submitting? If they get the endpoint, can't they just send a request to it in another way?
They can just use different IPs.
With captcha, the attacker needs to attach the right solution in each request to your signup endpoint.
If you use Supabse, firebase or hosted APIs via AWs or Google cloud, all the rate limitations, banning of IPs and such should be handled for you. I always say, its not worth the headache to build user accounts from scratch. Better to use available services that have been battle tested.
I unfortunately do not have any advice besides review your security practices and hope they give up soon.
Yeah thank you. We are on Azure and have added the IP filtering there.
Do you have any other security suggestions?
I use a ruby gem called rack-attack. Basically it can ban IPs when they spam certain routes too many times within a timeframe. So it could have mitigated this scenario heavily
Yeah definitely, we have added this now.
All of my APIs require authorization.
How are you making sure that it is your front end calling your api?
If your APIs have authorization checks, then why you want to ensure that it is your front end?
Front end attestation is a problem that will likely never have a good solution. One common idea is to obfuscate your front-end code and then send whitebox encrypted payload for server-side check. But apparently this requires huge engineering effort.
The authorization tells me who (in theory - in reality it tells me which credentials) is using my API. At the outset, there is only 1 authorized user - the front end I built.
But you missed the point. How do you control the use of your API?
Adding authorization is not difficult, nor is it unusual. I personally would never expose an API to the internet that didn't use some kind of authn/authz solution. That's practically an invitation for someone to test your API and see what they can get it to do - which is what it sounds like happened.
"One common idea is to obfuscate your front-end code and then send whitebox encrypted payload for server-side check."
I can't see how that is easier to implement any other mainstream authorization/authentication solution.
@alexrm suggestions to check out CSRF and rate-limiting are also very good things to consider.
ELI5: how can you require authorization for a "signup" API?
You have no user created yet, and it must be easily available from a public/easily reachable part of your website, so how can you tell if it is a real person or a bot?
The part I’m specifically referring to is that your website (not a user of your website) should be the first (and probably only) authorized user of your API.
This assumes the website is some variation of a single page app, and not a traditional server side rendered, full post back website.
You should spend some time learning about how one goes about protecting their APIs. Your comment To me indicates that you are lacking some fundamental knowledge on the subject. Securing your APIs is a common task, but is not trivial.
https://dev.to/luke_redroot/api-design---securing-user-registration-api-endpoint-with-multiple-clients-cgj
https://security.stackexchange.com/questions/119034/how-can-we-protect-signup-apis-from-brute-force-registration
https://stackoverflow.com/questions/26038191/protecting-user-sign-up-api
https://stackoverflow.com/q/52606843
Thank you for the resources! I'll definitely read on them.
P.S. luckily no real damage has been caused
The best solution is three fold:
on signup - require a federated account. Make it someone else's problem, don't allow username + password. Not only is that a terrible user experience, it also is a ton of work to get right. If you federate to google sign in (or fakebook, ewww), they will handle it for you. Then in your service, limit users to one account per federated login (or however many you want)
on api requests - every cloud solution has something to help prevent attacks at the application level, you can rate limit on ip address, or have different rate limiting levels based on subscription plans.
care less - why do you care that 5000 accounts were made, if you can't handle 5000 accounts, which is a measly 12 api calls a second, you have some critical issues with your infrastructure. Why is it a problem that someone even created 5000 accounts?
We are using 3 solutions :
3)Signup by social media (it's harder to signup with 5k of gmail/facebook account). When the email signup is spammed, we remove the email signup and let only the social media signup.
But since we are using google recaptach, we are not spammed anymore.
How the guy can signup 5k account ? : with automated tool like zenoposter
Thank you, a captcha definitely works.
I was wondering though, what prevents someone from directly sending a request to your endpoint, ignoring the captcha?
Basically bypassing the frontend and going directly to the backend.
ReCaptcha is working with 2 key : one public (for the front that you can use with html/js/...), and a secret key that is using in the server side (php/....).
So you just need to check in the form process on the service side that the 2 key are matching, and that's it