AWS, or you'll be sorry?
Do you use AWS because it does things for you that you couldn't do yourself on Linode or Digital Ocean?
I'm not asking if there are conveniences you like about AWS, but whether there are things it does for you (scaling? server/data security? GDPR compliance? perhaps things you haven't even thought about?) that even if you wanted to do them "manually" on Linode/DO, you fear you'd get them wrong (or that you'd simply be unaware that they were potential issues in the first place).
(I'm a competent but non-expert sysadmin using Linode and DO for various projects, some with low user count or low stakes for security/data loss, some with higher use/stakes. I've avoided AWS because the cost calculator is mind boggling and because I prefer using/learning underlying standard technologies instead of magic tools that mask the standards. I realize these might not be fair characterizations of AWS, but regardless, my question still stands.)
Edit based on comment: I did read early on that trying to run your own mail server is a bad idea, so I have been using mxroute.com for that, and it's been working fine.
Hi Michael;
I'm a DevOps consultant by trade and spend a fair bit of time either advising or actively working with startups to get their infrastructure right. The main question is, do you have the skills, time or resources either yourself or on hand to 'manually' manage things like Postgres etc? If not, then AWS is more of a one-stop shopping, allowing you to easily spin up infrastructure. If you are confident in your ability to run these then I tend to go with DO for cost simplicity. The key thing is not so much scale (Most startups don't wrestle that to start) but more stability and security.
For about half my clients, DO has been the right choice and has allowed them to have the service they need at a predictable cost. For others, particularly those who need certain AWS derived API's, AWS made sense. My personal preference is to invest time in automation, either Packer and Terraform or Ansible and containers to make it easier to migrate from DO to another provider later on if needs be.
Good answer, that's actually the bunch of very good reasons not to choose 'small' services.
And, by the way, there's also Google Cloud Platform, which has everything a startup might need, and also very generous free tiers...
Yup, GCP has some really nice features (Global network is very shiny). And if you have a need for it, they are of course the gold standard for Kubernetes hosting.
That being said it just doesn't seem to have the mindshare. When I sit down and chat with folks and lay out the various options, GCP seems to be skipped over quickly. In terms of interest, it seems to go AWS, Azure and GCP as the 'ones that come to mind'. That could be a reflection of my geographic area/customer base though so I'd take my experience as severely anecdotal.
True, but maybe this all depends on the area you are. I can tell you that the managed version of the features (Appengine Standard environment) is absolutely a ready to go solution, that gives you the chance to focus on the business logic. Of course, the more you need to go deep down in the details/needs, the more you can add advanced features. And, at the very worse, you can still go multi cloud (but in my opinion this is no more a startup thing ;) )
Thanks for the thoughts. Yes, I've had no trouble setting up Apache, NGINX, Node.js, Postgres, Let's Encrypt, and doing the various "best practices" (at least based on the tutorials I've read) to secure them and make them restart automatically on crash or system reboot. As for fancy things like Terraform or Ansible, I guess I imagine that as things grow (if they grow, which I know is statistically unlikely, but still) I may decide to look into those tools, or into, say, the load balancing tools that Linode/DO offer.
So regarding what you say about stability and security being the issues, do you have examples of "horror stories" that competent but non-expert sysadmins have run into with Linode/DO that would have been avoided had they used AWS?
EDIT: I realize I missed your point about setting up containers ahead of time to make migration easy later. I'll ponder that...
I think my favourite one is a client who set up their internal AWS network as a class 16 subnet on the 11.0.0.0 block. The upshot is that their 'internal' network was an external network that belonged to the DOD (And others). They also had issues with cost shock when certain machines were over-allocated and never examined for use leading to more than 80% of their AWS spend being wasted. To be honest, I could write an awful lot of blogs on horror stories!
It sounds like you are in a good place to use DO. I'd encourage the use of at least Ansible to provision things, it's really simple to setup and can pay dividends if you need to provision another instance. But I'd avoid the complexity of EC2 until your at either the scale or requirements that need it.
Wait, are you saying the five page google doc I have with my notes on all the steps I took to set up my server is not a convenient way to set up another instance? ;)
Ansible, Chef, Puppet, Salt, etc... or even just a big script that you gives you a one command setup from a fresh install would all give a decent time savings and make your setup more reliable
To be fair, that's five pages more documentation then most places I've been to :)
With something like Ansible, you can just rewrite your five page document into YAML files and you wouldn't need to copy-paste them anymore :). If something happens, you can easily setup another server with one command and it doesn't matter if it would be EC2, DO or Linode. (Speaking from my own experience.)
Ansible is a paid product? Is it worth it, or is a free alternative just as good?
It's free and Open Source. To give you an example of how easy it is, you can check out (the excellent) docs: https://docs.ansible.com/ansible/latest/index.html.
Generally speaking, I recommend doing it a chunk at a time rather then doing it all at once. So for example, start by automating user creation, then once that's done and working, package installation etc.
Ah, that Red Hat site sure makes it look like a paid product. I see that they say Ansible Engine (a paid product) is "built on the simple, powerful, and agentless foundational capabilities derived from the Ansible project." I guess that quoted part is marketer speak for "free version that will likely meet your needs."
Thanks for all of your advice.
If you want to use it as a "script", a sequence of commands that you manually run on your servers, than it's free (and open source).
If you're a small company and don't have the resources to do things manually, you have two choices better than AWS: Docker and Heroku.
I find AWS cost much higher than DO. No amount of playing with the cost calculator will tell you what the final bill will be. You just get a surprise.
I have a friend who is a student and had to use AWS in one of her classes. It was all very controlled and there was no traffic on the site she set up, but still she ended up getting charged for a bunch of stuff she didn't realize she was using. Granted, it might have been user error, but she's not an idiot and it sounded like it was easy to make a mistake and get a surprise bill.
It's shockingly easy to do. Elements such as Elastic IP's that are left unattached can easily be missed, and wrack up quite a hefty bill quickly.
I agree with the general idea that AWS pricing can be hard to see through, but an unattached elastic IP costs $3.60 per month, let's not exaggerate for dramatic effect.
That's a fair point - I've seen it around botched automation where an EIP was allocated and never destroyed. Although an individual EIP is not disastrous it racks up. That being said, thankfully AWS has a relatively low limit on allocating those out on a new account.
I guess the point is that it's easy to pick up $3.60 here, another there, and by the end of the month have a shock. Billing alerts are your friend in that regard.
I just read this today https://dev.to/juanmanuelramallo/i-was-billed-for-14k-usd-on-amazon-web-services-17fn
lol that dude stored his AWS account credentials in a public repo and was surprised when his account got compromised? Rookie mistake.
Well our first bill for https://seyoh.com was like $150. And that is basically just two environments (staging, production) both consisting of an EC2 plus a postgres RDS. Bearing in mind that technically one whole environment is covered under the free tier. I nearly fell off my chair. Really wish we just went DO.
Yikes. What made you go with RDS and what kind of traffic load was it under?
Thanks for the validation.
I think this by itself is an opportunity - reselling AWS services where you take on the risk of running up a huge bill in return for giving peace of mind to entrepreneurs. Something like bill insurance along the lines of health insurance.
I can see someone happily paying $10/month to use all manner of AWS services with the knowledge that if their bill hits say $500, bill insurance will step in to fill the breach. The numbers might need to be tweaked but I can see it work.
But isn't that what Heroku essentially does? If I recall correctly their stuff runs in AWS
I dropped Linode (literally) on July 1st because I moved my entire infra over to AWS. I had exactly the same worry as you - their billing seemed incomprehensible.
Before I tell you what tipped me over the edge to AWS, here is what I used Linode for .
I had always avoided setting up own mail server because I wasn't sure I would be able to safely maintain my server reputation. I don't even know for sure what this means but I think it means I could very easily get my domain name tagged as a spammer.
Then I tried AWS SES and couldn't be happier. I even offer it as a service to clients to get them their own custom email addresses.
My costs are down 75% which is an added bonus.
My understanding of AWS billing is that the major money sink is "always-on" infra like EC2. Here too, you can use their Linode/DO competitor called LightSail. If you use mostly serverless lambda functions, your cost savings could be pretty high.
AWS also allows you to sell your services and products on the AWS Marketplace which, imho, is a trick that Linode and DO should somehow emulate.
So are you saying that you switched to AWS mainly because of the email issue but now you are using it for everything and no longer use Linode? Were there any other things that AWS "solved" for you (i.e., an actual problem you were having with Linode) besides email?
When I first started fooling around on AWS in late May, my sole aim was to try out lambda functions because I had heard so much about it. I was convinced AWS would rip my wallet in half. The email setup was what convinced me that I was ready to move to AWS.
The more I played with Lambda functions, the more I realized how expressive I could be with AWS. With Linode, not only did I have to write my function, I also had to keep it alive, wire it to nginx (by giving it a domain or /path, assign a port, write the conf scripts, etc.), worry about alerting if the process went down and so on. The outcome of all this extra stuff was that I was never confident my function was production ready. Consequently, I was never confident telling someone I could help them with nginx/firewall/Ubuntu 16.04/certificats, etc. I was a script kiddie who knew how to google. With AWS, I simply connect lambda to API gateway in 5 minutes and I am a lot more confident that I have a robust solution I can call prod ready. The underlying complexity may be abstracted from me but that's ok - I don't want to operate at that level anyway.
So, my actual problem with Linode was that I wasn't expressing my ideas at all. I was merely tinkering, creating a solution from glue and hope. I'd have to become an nginx expert, an Ubuntu 16.04 expert, ACM certificates expert, a nodejs expert, firewall expert, a network security expert and more before I could deploy a system I felt was ready for production.
Right. Your list of worries is the one that made me add "or you'll be sorry" to the title of this post. I am in a bit of a different position because I want to learn all of the underlying technology. So I don't care about the inconvenience. But I do care about some killer issue that will inevitably leave me feeling like my sites are not truly up to production standards. This is especially true around things like keeping processes alive, and security, as you list. However, whenever I look into any of those issues, it's always like, actually this is not so hard, you just follow this security best practice or run this process monitoring tool. So that's why so far I haven't jumped to AWS. As I said, with email read it was a bad idea to host it, so I got mxroute.com. Otherwise, I made this post trying to see if someone with security or other sysadmin expertise would say, "don't try to do all of this yourself on DO." So far, that isn't the response I'm seeing.
Without looking at your specific setup, no one will be able to tell you that it is possibly broken. You're basically saying that someone at arm's length should be able to predict what pitfalls you may have overlooked and warn you about it. That's not a realistic request, imho.
We are talking about technology whose main USP is standardized, repeatable instructions for a repeatable outcome (aka algorithms). 95% of the stuff that could derail you are things you overlooked, not things which are impossible to do correctly on your own.
Take something as trivial as nginx/https/domain/upstream. You could get everything right but forget to setup a cron job for cert-bot auto-renew every 90 days. Everything is doable and if you're in that mindspace where you want to learn the basics yourself, you're ok with DO/Linode.
Right, thanks. I understand that it's a bit ill-defined and didn't mean to sound like I was asking for a validation of my stack without the details given. I am in fact trying to put together a site where I detail all the steps in setting up my DO instance. The hope is in small part to get feedback/advice/corrections, but in larger part to serve as a checklist for others trying to set up the same kind of thing. Kind of like a mega-tutorial, but kind of different.
Yes, it's definitely a goal to learn how manage a site from DO/Linode, so I'm encouraged that the basic response to this thread has been "you're okay doing it."
(And yes I do have a cron job to renew my HTTPS certificates. :) )
I use SES too, but that doesn't mean moving everything to AWS.
Lightsail is basically a more expensive DO that also gets throttled more.
Could you expand on the Lightsail throttling? I just moved my project over to Lightsail from DO since I received a bunch of credits from AWS but I'm wondering if I made a mistake. A quick Google search didn't turn anything up.
I've just seen a lot of comments like https://news.ycombinator.com/item?id=13126066 over the past couple of years. Beyond this, the listed stats are worse for Lightsail.
The $5 droplet has 2x the memory of the $5 lightsail.
A $15 droplet has 2 CPUs, 2 GB of RAM, a 60 GB SSD and 3TB of transfer. In contrast a $20 lightsail has 1 CPU, 2 GB of RAM, a 40 GB SSD, and 3 TB of transfer—equal or worse across the board and 33% more expensive.
If you have free credits, though, then you'd might as well use them. Just understand that Amazon gave those to you with the hope of getting you hooked into their ecosystem and profiting in the future!
More context @ https://www.indiehackers.com/forum/3eeb2d8ed7?commentId=-LGVz63BtTHi2AH7z5Wd
This comment was deleted 8 years ago
Didn't know about LightSail, thanks for mentioning it!
Thanks for reminding me to put a note in about email. As I now say above, I have been using mxroute.com for email, and it's been working fine. I got a deal of unlimited domains/space/bandwidth for 10 years for $90, so I've got email covered for a long time. :)
I'd use AWS and Google Cloud only once your product is at scale and the specific features of AWS make sense. If you can make $1000 or even $1MM per month with a product running casually on a few servers, I'd stay on Linux and regular servers. Docker now provides a nice middle ground that can add a lot of productivity.
I for one, have been slowly learning AWS by force from job to job, and I hate it ever step of the way.
The product makes sense and fills a need, but the GUI is a pure piece of $#*@. The first time I encountered AWS was in 2010 and once I read a bit of the docs and saw what it was UX-wise, I quickly found another engineer to pass the task to. Eight years later, not much has changed. The older services have been polished somewhat, the complexity is as high as always, and the newer services are crappy. Case in point, I just started using Systems Manager Parameter Store. It's supposed to be a key/value store for strings (think environment variables). Get this: you can't store an empty string. I'm not even asking to store a null. Just an empty string!
Just store "null". ;)
Thanks for the info. I will look closer at Docker. But from what I read it's mostly just what I'm calling a "convenience" tool, right? I.e., it's not something that inherently will increase my security or site uptime?
There are various ways in which it provides faster development, security, reliability, fewer bugs, faster deployments, etc.
Try to google for "benefits of docker" or "benefits of software containers". For example:
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/7.0_release_notes/sect-red_hat_enterprise_linux-7.0_release_notes-linux_containers_with_docker_format-advantages_of_using_docker
https://dzone.com/articles/top-10-benefits-of-using-docker
Thanks. I'm not seeing significant arguments against using Docker. But I'm also not seeing key arguments in favor of using it just in terms of security, especially on a VPS that is only running one application (i.e., apps on the VPS don't need to be protected from each other). I do understand that more reliable deployments and easier patching are part of security, and for those reasons alone it seems worth using.
One thing I didn't quite understand is the performance implications. It read articles comparing the performance cost of Docker vs. the performance cost of VPS. But if I'm using a Docker container on a Linode or DO VPS, I then have two layers between me and the OS and I pay the cost of both?
I'm a lead of a team of web developers and our main focus is building products from scratch for startups. We've used AWS almost exclusively, mainly due to the fact that they provide full control over your environments. You can delegate to a mid-level developer to use something like ElasticBeanstalk to spin up some not so fussy projects, but you can also build entirely customizable stacks using Terraform.
In addition, their product portfolio is extensive and it often allows us to move much quicker by utilizing the tight integration between their services - spin up a queue, worker instances, ES clusters, Elasticache nodes, then just fix up their security groups and you're ready to go, while also allowing you to completely customize the stack with additional "plug and play" services, should the product requires it in the mid to long terms.
Bottom line - I've found having full control over the infra makes it easy to build products quickly, but also plan for the scaling phases down the line.
Thanks for sharing. All of that sounds like it falls into the category of "conveniences" as I mentioned in the OP. Do you have specific examples where Linode or DO failed you (as opposed to being less convenient)? You mention being able to trust a non-expert developer with ElasticBeanstalk. What would be your worries/experiences if that developer was working on a DO machine?
I assume you're talking about using EC2 instances. I host on Heroku (which is hosted on AWS, surprise!).
I use AWS for the tooling. S3, SQS, and SNS are fantastic tools that require little to no setup. S3 hooks into their CDN Cloudfront brilliantly. They are also insanely cheap. Why would I want to set up and manage any of those things myself?
There's a million other tools that make life easier with AWS, and it's all billed at once, which is great.
Thanks for the feedback. How many users per month do you support? And what is your monthly bill from Amazon?
My bill runs less than a dollar per month. My current project doesn't have a ton of users, but previous projects have, and it's similar pricing. SQS gives you 1 million messages per month for free. S3 is similarly cheap.
Why focus only on AWS vs DO/Linode? With both of them you are responsible for all the operations especially when something goes wrong. As a small indie hacker why take on the burden? Especially in conjunctoin with this other current thread "How do yo ulearn to run your service?" https://www.indiehackers.com/forum/how-do-you-learn-to-run-your-service-3386fb2c14
I wonder why not look at managed hosting companies? Yes, they probably limit your tech stack, but if you are in the LAMP faction with the occasional ES, Redis usage then it is a really good choice to keep focussing on software.
They employ experienced system operators that can help you setup and maintain your application, often for a really good price if you add up AWS/DO + doing or hiring a sysadmin yourself.
Thanks. It's a good question. The way it started is that I just wanted to make several mostly static sites (some minor dynamic elements like a "contact us" form, or other simple CGI processing) for both myself and some friends. In the past I had used managed hosting for things like that. I got tired of paying $10/month for each site. I've been very happy to be able to host several low-traffic sites on one $5 Linode.
And many of the projects I have in mind are half-way between these simple static sites and a full-blown SaaS. So for those also, I'm learning a lot by figuring things out on Linode/DO, in a pretty low-stakes situation.
As I start to work on more serious things, I'll reassess whether something managed for me makes sense.
The miracle of AWS / Google Cloud / Azure has been their functions as a service (FaaS) stuff. It is starting to take over the industry because we you get straight to business logic and not worry about anything else.
Absolutely. We use Azure Functions and swear by it. Decoupled, event-driven applications that scale through a serverless architecture is just awesome. We love the pay only for what you use (consumption model) for backend workloads, and use the App Service plans for API level functions. It takes what is good about microservices and pushes the whole thing to FaaS. Easier to build and manage.
Plus, what you build at your desktop runs just as well in the cloud.
Can you give a sense of how many users you support and what you pay per month?
With FaaS, you pay per usage.
I do IoT and we calculate based on a device's average requests/day by any User.
We have priced out that each device should roughly cost $2.5 for a 5 year period of average active usage. That should cover the cost for that one device for that period.
Hope that makes sense.
Thanks.
This comment was deleted 4 years ago
I'm still trying to separate mere "convenience" from something that actually takes care of things it would be very hard for me to do myself without years of experience, especially around the area of server and data security. Does Heroku do magic for you in that realm?
I feel like we might be on a similar level in terms of sysadmin skills. One thing that I never really feel good about is managing my own database. I'm always worried I might misconfigure something or not handle backups properly. That's why I ended up with Heroku for several of my projects. They provide managed Postgres instances, I set up scheduled backups and the starting price is doable (USD 16 for "server" + DB with up to 10 million rows).
On a side note, I really miss a DBaaS offer from DigitalOcean. In the end, most people still rely on a relational DB and it's likely the most sensitive/difficult to manage part of the application.
Thanks for the feedback. I didn't see the $16 plan. Are you paying for "Hobby" or "Standard" level for the Heroku Postgres? Would using a tool like Barman make you feel better about DO Postgres? https://www.digitalocean.com/community/tutorials/how-to-back-up-restore-and-migrate-postgresql-databases-with-barman-on-centos-7
I haven't had any issues with just a Hobby dyno ($7) and a Hobby Postgres instance ($9 for the Basic option, which supports up to 10M rows).
Barman seems nice and I would have probably used it, but for now my scripts seem to do the job. Still, nothing compares to just Heroku automatically setting up the DB automatically based on my Django dependencies, handling passwords, and (simple logical) backups :)
Since starting Rails development I've been using heroku and I love it. Just so freakin easy.
What Heroku services do you use and how much do you pay per month?
As I recall at the moment it's about $7 a month. Just the lowest paid tier Dyno and a hobby tier postgres instance.
Their cli makes it so nice to work with and the fact it just works out of the box. Super easy.
I tend to host landing page + blog as a jeykll site for free on GitHub pages and then applications are just on a subdomain.