We've been Hacked - "Notice of reported brute-force activities originating from your Droplet StagingServer"
Hi guys,
I hope you're all having a nice summer!
We just received this from Digital Ocean and immediately destroyed our staging server. I thought since I always get such amazing help from this community I wanted to hear if anyone has experienced the same issue and has any additional actions we should take to avoid this happening again or to prevent damage on our "real" server?
Thank you so much in advance!
Please see the full message here (any sensitive information has been replaced with XXXX):
Hi there,
We’ve received a report from a 3rd party that your Droplet StagingServer at XXXX is participating in a brute-force password dictionary attack (https://en.wikipedia.org/wiki/Brute-force_attack) against a remote server via SSH.
Please review the report at XXXX and take any needed actions to resolve the abuse originating from your Droplet.
To prevent any further spread of the malware responsible for this activity, we’ve placed a block on outbound port 22 (SSH) connections originating from your Droplet. This block will prevent the malware from attacking any other systems on the Internet, but will still allow you access to your Droplet to work towards recovering your data. No other services on the Droplet are affected at this time.
Unfortunately, this activity indicates that there is a severe compromise of your Droplet, and is not an incident that can be recovered from. At this time, we are going to require that you backup your data and migrate to a new droplet.
Please note that failure to migrate to a new Droplet may result in us taking escalated action on this Droplet should we receive notice of further abuse originating from it, up to and including full suspension of network access.
Please let us know if you have any questions, or require assistance to take needed actions on your Droplet.
Regards,
Security Operations
DigitalOcean Security
Holymoly! How did they get in to your droplet?
Very good question 😬
A lot of the time, hackers get in through the third-party addons and extensions people install into their CMS, forums and other self-hosted software.
While any software can have flaws that can be exploited, most core software like CMS packages go through a more rigorous security review before they are released.
However, smaller third-party addons and extensions tend to get developed quickly by fewer people and updates get pushed out without that review process.
In any case, once your system has been breached, you have to assume that everything has been compromised. Just moving to a new server/droplet is not likely going to solve the problem.
Even if the breach was done via an automated process, it could have installed additional access points and methods that would allow it back in even if you found and fixed the exploit that was originally used.
You need to either hire an experience admin to help you migrate to a new server or consider moving your content to an SaaS-based (service) solution.
Thank you!!!
I would try following this tutorial https://www.youtube.com/watch?v=kR06NoSzAXY it helps you setup and secure your droplet. Good Luck
thank you so much!
Security person here. You should probably lock down network access to your server and investigate what happened before fully redeploying. Good places to start would be looking at bash history and application logs (assuming it's Linux).
Taking general security precautions goes a long way: things like using key-based SSH and restricting network access to only what is necessary. Here is a good guide https://www.digitalocean.com/community/tutorials/recommended-security-measures-to-protect-your-servers
Brilliant, thanks a lot!
Something similar has happened to me. I had a server running CouchDB and apparently CouchDB had a combination of a privilege escalation and remote code execution bug which allowed someone to gain access. Fortunately, all they did was mine crypto currency, though.
Had I kept CouchDB up-to-date, it wouldn't have been a problem. The issue had been patched long before my server got hacked.
So definitely make sure you keep your services like that up-to-date.
Thank you!
Do you use a common CMS? Like Wordpress or Drupal?
Hi Thomas! Nah, coded in Angular and Yii2 :/ We have a WP blog connected but that one should effect the security on our staging server I imagine.
Ah, I wrote CMS but I thought PHP frameworks ...
Make sure you use the latest version. Bots can exploit security breaches very easily on those things.
Maybe you can find in the logs (Apache or Nginx) what was the entry point.
Probably an exploit in one of the services running in your droplet.
It's common for scammers to scan the net for exposed servers with known vulnerabilities and install backdoors.
There are 2 ways to avoid this:
No need for a security expert but a dev that follows good practices.
Most servers get exploited because they run outdated software with known vulnerabilities. Get the latest security updates.
I have always wondered how these massive companies get hacked. Uber, Canva etc.
Small sized companied get hacked through Employee devices. They infect a careless employee with a mail or pirated s/w and and get access to their account. But, I would not guess it happening for such a massive organisations. Very scary.
The bigger you are, the bigger the attack surface.
yeah a bit of a shock really. I feel like I receive an email about every other day from BIG companies having had "security breaches" and leaked data.
Tack! We work with an outsourced dev team so will def make sure they step up their game on security. Luckily it was the staging server and not our main one.