October 17, 2019

We've detected suspicious users abusing our API

Kevin Sahin @KevinSahin

Today we found some suspicious sign-ups with strange email domain names.

In ScrapingBee API we offer 1000 free API calls on sign up, and those people are abusing this by registering multiple times. 😡

It seems those emails are not coming from traditional disposable email services so I wasn't able to find them on open databases on Github.

There are many things that could be done to solve this problem, like additional fields on Sign up, phone number, or requiring a Credit card ...

The problem is, most of these solutions would lower our conversion rate.

How do you solve this problem?
I've heard about two APIs to check the "reputation" and score an email address:

Any feedback about this?

  1. 9

    What about a solution similar to Firebase, where there is a free limit, then a better free limit if you add a credit card (pay as you go).

    So for instance, 100 free api calls for no Credit Card accounts, but 1000 free api calls if an authorized card is added.

    Could try an AB test to see if it affects conversions?

  2. 5

    I would not block them right away because maybe there's some information hidden in this behaviour. I would try to talk to them to understand their pain point and the use case.

    Then offer them a special price pack.

    If this does not convert and it continues then automate the ban system for this type of users.

    1. 1

      Maybe that's just people who can't afford it I don't know.

      But you're right, we should try to talk to them!

  3. 3

    The problem you have is very common.

    The best things you can do is to block multiple sign ups from the same-IP, require email account verification, and possibly add credit card requirements. Unfortunately, all of that is fairly easy to navigate around. You all have rotating proxies for your service, there are many temporary email services, and adding credit cards will either drop sign ups or force them to use virtual CC. A heartier solution is going to involve pretty heavy tracking solutions similar to fraud prevention. It'll cost you money out of pocket. If it's not costing you too much money you might want to just roll with it and hope they convert to paid. Another option is to just manually watch it until it becomes too big of an issue.

    I'm assuming you aren't a native English speaker because there are a few grammatical issues. All in all it looks good and I wish you luck.

    1. 1

      Thanks for your detailed answer and feedback, I think you're right.

      Yes, we're a French company, always looking to improve our English. You could tell me which grammatical issues?

  4. 2

    Let me share our experience how to filter sign-up abuse. They usually using disposable email and free email to generate massive number of free sign-ups.

    We signed up for MailboxValidator API to detect the email type during sign-up. Reject them if it is disposable email domain. We decided to accept the free email domain. It has reduced abuse significantly.

    I hope it helps.

  5. 2

    Kevin, check this list out, seems to be updated frequently https://gist.github.com/michenriksen/8710649 and there is also this free API https://block-temporary-email.com/

    It might not completely solve the issue, but it can help mitigate.

  6. 2

    I'm guessing they will be maxing out each one of their API keys which normal customers wouldn't typically be doing?

    Perhaps set up a daily monitor for people using their max allocation and reach out to them personally... the good thing is they obviously love your service. They just have to fork out the cash for it now! 😄

    1. 1

      Yes, they are maxing out, but like 80% of our paying customer.
      As we said in a previous milestone, we have a system to give 10k free calls for people that are willing to schedule a call with us.

      Most of our paying customers for the highest plans ($99+ per month) comes from this.

      We have an automated email once people are reaching 90% or 100% of their quota, but you're right we should probably reach out to them manually!

      1. 1

        That is an insight and nice trick. Thank you

      2. 1

        Awesome - I love the idea of free API calls for having a call with you!

        1. 1

          Thanks ! It works really well for us :)

  7. 1

    Keep it simple and not scalable until it becomes a major pain.
    I.e. flag multiple signups from the same IP for manual review.

  8. 1

    This is identity verification problem. You probably want to verify an identity source for which Users can’t get multiple accounts, like a phone number. Or social accounts to some extent(because they themselves prevent abusive accounts), emails are too easy to get multiple.

  9. 1

    I would recommend a combination of automated checks + SMS based verification for suspicious logins.

    You could use clearbit scores and anyone with below 95 directly gets in. Others are forced to enter sms otp

  10. 1

    I would avoid checking IPs and cookies... due to the nature of your service, it's likely your audience is aware and experienced in getting past IP address restrictions.

    I'd use twilio to send a confirmation code by SMS. While you're setting that up, use the twilio lookup API and store country_code, carrier.name & carrier.type. If you are still seeing fraudulent account creation, look for patterns in those values and push those to manual verification.

    As for avoiding the hit on sign up / activation, you could push this down funnel. It's not a step in the sign up flow but rather a step you need to complete before the API key is activated. This way a user has already invested their time setting up a request in postman or written some code already. They are not going to abandon because they have to spending 30 seconds verifying their account.

    The verification process doesn't have to catch 100% of the fraudulent accounts. It just has to be more of a hassle than the alternative.

  11. 1

    @KevinSahin, we are working on SaaStraQ - a one-stop platform helping users to discover & try different software products in OneClick.

    So why "OneClick" & how it works?
    To try any products available on SaaStraQ, users need to signup with SaaStraQ first. Then they click on "OneClick" trial button.

    To offer users direct access to trial without any extra effort to signup on the partner's page, APIs between two platforms i.e. SaaStraQ & partner exchange data.

    So the user doesn't have to signup again and again on different platforms.

    Now, if the user's trial period ended for even a single product and need to game the system with new credentials, they will have to signup with SaaStraQ once again. They may lose access to other trial accounts and data and documents saved on different platforms during trial would be no more accessible.

  12. 1

    If your customers are primarily B2B maybe you could require a "work" email address upon registration and website URL.

  13. 1

    Hi Kevin ... just upgraded to a paying plan this morning.

    We had the same problem at myPresences and were getting a lot of fake accounts (generally from pakistan). Our problem is we use a lot of resources (API calls and scraping calls) to onboard a new business and this was costing us.

    I started banning certain free email providers from some countries but they just started getting more and more sophisticated with Proxies and appearing to come from the US.

    In the end I am am using clearbit risk for anyone who signs up and am denying signup to anyone with a score of 100. I am also blocking and signups using free email accounts from certain countries, they must use a business domain.

    This seems to have worked but I am worried about false positive and check every denial .. it seems to be working well.

    Can definitely recommend clearbit risk.

  14. 1

    I think you should introduce some extra checks on the email confirmation page. Those who signed up likely will not hesitate to pass these extra steps while confirming their email.

  15. 1

    I normally log in the IP address and check for its multiple instance and reject the second one. Although there are plenty who can bypass it, as I've always got "free users" who just won't pay for shit. The other thing I do is require email verification every single time. Again, they have gotten around this too, even with my "fake email detection system" (Github has a list of all fake email hosts). For my latest API, I'm actually requiring a credit card to be on file to even use it, even though the first 500 API calls are free.

  16. 1

    The usual advice I've read on this is – if it's just a few duplicate accounts, it is probably not worth your time to deal with that. You would be investing time to automate the handling of these accounts, and there might be better uses of that time.

    1. 1

      Yes you're probably right!

  17. 1

    Got the same problem.
    We did few things to try to remove those issues :

    1. track the user with the cookie (in the case they are not changing their browser. You can save many data in the cookie : user-agent, language... each computer have his own fingerprint, so you can check if the user already signup with the same fingerprint.
    2. save idea, but without a cookie, just insert in a database all the fingerprint and check before eachs signup
    3. about the free quota, give only 10 api call each as daily limit (in addition to 1000 monthly), and if they want to remove this limit, they have to send you a message to the support. This way, you'll probably have many email supports, but at least, those people will be lazy to send an email with each account
    4. to get the free quota, people will need to signup (or confirm their account) with a social account (fb > google > twitter). It's much more difficult to create a social network account and use it than create an account in your plateform with a email/password. Fb account is harder to create compare to google and twitter. You can even try linkedin account.

    Note : i'm french, so if's not clear, I'll translate :)

  18. 1

    I would collect geoinformation, to see if they can be pinpointed to a single location. If it is used in bulk, there is a large possibility that it will be localized somewhere. If that doesn't help, then some captcha can help.

    On the ethical and legal side, it is a bit ironic as a lot of websites also have legal clauses that consider web-scraping a violation of their ToS. I would be careful with the pressure that is put on websites by those people who violate your API, but as much by your own ethical users, to make sure your API won't get blacklisted.

    In general, it is a big problem on the web, there is always something poking around, scraping, automating, running penetration tests, etc. Good luck Kevin.

    1. 1

      Thanks for you answer!

      About the ethical and legal side, I'm not a lawyer so don't take what I say for legal advice:

      Web scraping can be a grey-area thing. It can be legal, or not, depending on many factors. As you said, degrading a web site's performance by sending too many requests is not only unethical but also illegal.
      In our case, we limit the concurrency for free trials and paid accounts.

      On the other hand, scraping public information responsibly isn't illegal in most European countries and in the US despite the fact that it can be a violation of ToS.

      Linkedin just lost a lawsuit about this in the US.

      If we receive a complaint from a website we would comply!

      Remember that the biggest web scraper/crawler in the world is Google.

      1. 1

        When it is labelled "scrapping" it sounds sketchy. When it is called "indexing" by search engine it is in fact desirable :D I've never thought about this in those terms but makes perfect sense now.

  19. 1

    That sucks!
    I had the same issue. Unlike your product, I handle my request through RapidAPI.
    The thing is that even though RapidAPI requires to enter a credit card information, I still got abusive users. I guess their abusive detection isn't the best.

    About checking for email reputation,
    How this stops an abuser from opening multiple Gmail addresses?

    Although it may affect your conversion rate:

    • Require a credit card
    • Limit Free plan (e.g instead of 1000, make it 100).
    • Additional Email or even better IP address reputation check.
    • Block usage from certain countries

    These could work to minimize abuse.

    P.S I run an anti-spam service. I can help you with IP address reputation checks and blocking usage for certain countries.

  20. 1

    You could limit the calls by IP, or otherwise add recaptcha on the page. The new version does not require most users to click on images and that.

    1. 1

      Thanks for this idea!

  21. 1

    Depending on the number of sign-ups you get, the easiest way would be some sort of manual fraud detection mechanism. One way I've seen something similar implemented is through Slack interactive message buttons e.g. you send a notification with the user details to a #signups channel that your whole team listens to, and if anyone notices something funny you can click a block button on the notification message. It's not a perfect solution but sometimes it's enough to convince whoever is signing up those accounts to stop using your service.

    1. 1

      Thanks for your suggestion!

      Actually that's kind of what we do at the moment. We have a slack channel for signups and that's how we discovered this (several sign-ups with 1mn interval with similar email pattern).

      The button to block it inside slack is a good short term solution, but an automated system would be better long term.

      1. 1

        Thing is, fraud detection is one of those open-ended problems that are really hard to tackle without a human in the loop. Apart from some simple measures like a captcha or IP blocking, it might not be worth to invest in a more sophisticated system or external services at this point.

        In the end, this isn't a core part of your product and spending time and focus on solving this problem right now might have a higher cost than the overall cost of the abuse.

  22. 0

    Hi Kevin,
    I'm not sure if this will work for you . . . . create additional fields on Sign up and hide them for only bots to see. Once they fill in those hidden fields, redirect them to any external website (We mostly use Google) or redirect them to home without creating an account.

    1. 1

      It's not bots, it's real users using multiple email addresses to take advantage of the free 1000 API calls :)

      1. 1

        Oh! Maybe let users register only with their social accounts, have moderators block users with new social handles or something . (they need to check users SM profiles to confirm if its new or not). Unfortunately, I've not seen any tool out there that can outsmart humans with regards to stuff like these. I'd prefer to sign up with my SM account than filing in more fields or adding my card details.

        1. 1

          Or make your freemium less attractive.