Buy your own servers

This is a long story. I'd initially started writing a tweet about this when I realized it deserved to be a much longer post. It's a case for buying servers and using commercial data-centers vs using the public cloud (Azure, Google Cloud, Digital Ocean, AWS).

Before turning to buying my own servers, I was squarely in the public cloud camp. They are cheap to spin up, are across the globe and it's easy to scale infinitely. This is all true until the costs start to add up. Being an indie hacker without any external investors, these costs are real.

When I started out I used Google's App Engine and it worked great. I was able to spin up servers that scaled elastically and self-healed. I wouldn't need to worry about being popular and blowing up (even though I was in a high touch B2B market facepalm). It also allowed me to focus on building the software and not worry about server configuration and all that. In this sense, the PaaS that Google was offering was great. Another thing I got was Google Cloud's High Availability Postgres DB. This ensured that the DB was resilient and, being Google Cloud, could scale with us. I could also schedule daily backups and all that. For all this, I barely paid anything at the start. My bills started off at zero (thanks to Google Cloud Credits) and slowly grew to a baseline of around $150 - which I was more than happy to pay because the business was growing.

Things then took a turn when my front-end servers couldn't respond to traffic fast enough. I started optimizing DB queries and had to start caching (I delayed setting up caches because the architecture was changing rapidly and I din't want it to slow me down). Being a distributed system, the cache had to live in a separate machine. I opted for Redis Cloud servers on Google Cloud and encrypted the traffic between my cache and the servers to make sure no user data leaked. With all these efforts, response times were greatly improved. Unfortunately, before long, the servers were OOMing a lot and I had to scale up to the next tier. Then the next tier etc. (See https://cloud.google.com/appengine/docs/standard#instance_classes) By the end of it all, I was sending Google checks that made the $5 or so I started out with look like child's play. This was split between growing storage costs, ridiculous compute costs and even worse DB costs.

This is the public cloud's bait and switch model. It's great for small loads (Thanks Digital Ocean for running some of my instances for almost 10 years!) but they get you once you're too invested to leave.

It didn't take long for me to realize that I was paying Google more than it cost my ThinkPad every month for less than a tenth of the resources. I get there there's data replication, internet costs and all that but still, Moore's law anyone?

I started looking at data center costs and quickly realized I was drinking the marketing Kool-Aid of giant software companies. I had already written off hosting my own servers before even counting the cost. Sometimes, tech can be an echo-chamber where alternatives are shut down before even been considered. Marketing works.

I ended up buying a used dell PowerEdge server for about $700 with 120GB RAM, 32 cores (2 socket 16 core each), redundant power supplies, redundant network ports etc. I also bought a number of SAS drives. (I didn't even know there was such a thing as SAS drives when I started down this path).

As to where I placed it, the problem with many data centers is that you need to buy an entire cabinet (They sell them in Us). Luckily, I managed to get a reseller who bought space in bulk and sold it to smaller customers for a profit. For 2Us, 2 way power redundancy, network redundancy, being next to a major City's main internet switch and /28 IP block, I only pay $200 a month! The data-center is well secured and even requires fingerprint access. I migrated all my servers (I run my own analytics, schedulers, dbs, front-end, cache from VMs from the same server) and spend an additional $10 on Digital Ocean for external Zabbix monitoring. I still keep all my files and backups on Google Cloud Storage because it's actually cheaper. This is my cover for a worst-case scenario. I should be able to get all services back up in a few hours which is acceptable per my SLA. Maybe someday I'll dockerize everything.

The entire migration took 2 weeks and involved learning about hypervisors, server monitoring tools and maintaining multiple replicas. Services that were compute-only were really easy to migrate and mostly involved DNS changes. DB Migration was the hardest.

I've been running with the new hardware since November, 2019 and I've only had one failed drive (Writing this towards the end of Apr, 2020). Let's face it, there's nothing wrong with a 5 year old server. Server-grade processors from 5 years ago work just fine. Error-correcting RAM too. All I did was get new drives. Unfortunately a new drive failed within the first two weeks of installation but the seller sent a replacement. Thankfully, I was running a RAID 10 setup so everything kept humming just fine. Swapping drives on server hardware is ridiculously simple and doesn't even require a restart! If you're considering this, get actual blade servers for your workload. I've neither regretted buying a used machine nor have I found it hard to run my own infrastructure. The open-source community has really matured and there are a lot of good alternatives out there for hypervisors or even container management. I use proxmox and it's been great.

Surprisingly, running my own hardware has actually been more reliable than the cloud. I had about three Google Cloud outages (US-East1) in 2019 alone. At first, all I could tell my customers was that there was nothing I could do. I ended up setting up a mirror in Google's US-Central for use during outages. There were also days with a ton of network congestion on Google Cloud and customers would complain. Since I moved, there have been zero such complaints. This is partly because all services are co-located within the same machine but also because most of my customers are in the same Geographic region where the server is hosted.

Public clouds are great to start projects with but beyond a certain size, I believe the cost of setting up your own infrastructure is totally worth it. Hardware costs have really dropped and there are a ton of used servers being sold for dirt cheap after their lifetime. New servers aren't that expensive either. There's a high initial cost to configuring everything though and YMMV depending on how much you know about server administration and security. TBH, the public cloud come with a high initial cognitive cost too.

If we manage to grow further and increase our presence in other states, I'm looking forward to setting up a second or series of servers as mirrors of each other.

I hope my experience will help inspire anyone at the cross-roads. As always, think for yourself. My solution may not be right for your company.

Otherwise, stay safe out there and pray we don't die from this virus. If you read to the end, you're a bona fide hero :)