Product Development May 29, 2020

I could have run my app for 18 years with the time I wasted on Google Cloud

Mark Mossberg @offlinemark

Lesson: Your time is the most valuable resource, so get comfortable spending money to save it.

Or in other words, Don't Cheap Out.

When I started, I really had it stuck in my head that I wanted timestamps.fm to run for free. Other than the domain, I wanted to run on some cloud free tier and be a widely used service that didn't need to charge, because it had no costs.

This led me to my first big mistake.

I initially set up on Heroku's basic hobby plan ($7 /month).

"This isn't sustainable" I thought (still in "free web app" mindset), so I made plans to port over to Google Cloud (I was already using their S3 equivalent) because they had a nice looking free tier. The I could run my app for free, "sustainably"!

What I thought would be a couple hours of rewriting config files and porting the app evolved in 50-60 hours of constant pain and debugging.

I first investigated Cloud Run - Google's serverless container platform. You get millions of invocations for free, so this was potentially a good fit for my simple, stateless utility app. It ended up being a lot of overhead to generate container images, and deploy them, and inexplicably, logging to stdout seemed broken.

I then moved to App Engine, which seemed like a better fit, and said it's a good fit for applications "Intended to run for free or at very low cost, where you pay only for what you need and when you need it" [1]. Did you say free? Sign me up.

Here's what happened next that eventually drove me off the platform:

  • Everything related to managing App Engine autoscaling instances. Their "autoscaling" behavior is confusing and often inconsistent with the docs.
  • Instances are not shut down automatically when you push a new version. I had to manually write a script in my CI to shut old versions down, and only realized this once I burned through my $300 of credit in 2 weeks and was on track to receive a $1k bill by the end of the month.
  • Deploying from github was extremely laborious to set up and required writing custom Github Actions. It worked out of the box with Heroku.
  • The billing reporting. Confusing interface that hides the fact that credits are being spent in the default view.
  • The charting. It's really difficult to do simple log data visualization tasks, and their whole Metrics system makes it hard to visualize historical data. Alerting was equally complicated.
  • Exporting logs for proper querying and analysis was difficult, and I never really figured out how to do it with BigQuery.
  • Setting up DNS was much more complicated than Heroku.
  • Confusing caching behavior. There are internal caches in Google's infra that you cannot flush, which manifested as css and static images not updating properly. Ultimately this is what broke the camel's back, where my site incurred ~10 additional seconds of latency when processing large files, entirely due to what I can only attribute to internal resource caches.

All of this was time spent purely on backend infrastructure tasks that did absolutely nothing to drive the product forward.

If I conservatively value my time at $30/hr, I spent 50hr * $30 = $1500 on this. With Heroku hobby tier being $7, I could have run my app for 214 months, or 18 years, with the time I wasted on Google Cloud.

For much of that debugging, I was convinced I was always just around the corner from a nice, free infrastructure, but things just kept coming up.
I ultimately decided to move back to Heroku and am now gladly paying the $7 / month for infrastructure that just works. This move has paid off time and time again, as I've done more infrastructure tasks like setting up a staging environments, which I've been able to do quickly so I can get back to driving the product forward.

I have definitely learned my lesson. Your time is the most valuable resources as an early founder, and it's worth every penny to invest in saving it.

[1] https://cloud.google.com/appengine/docs/the-appengine-environments

  1. 8

    As a happy GCP user - I cannot disagree more on all of the above. It may be you are too used to Heroku.

    This is me telling after using most of the major cloud platforms out there in various gigs over the last decade.

    1. 2

      same here, love GCP. Have tried them all.

      1. 2

        Why GCP when you can use AWS? What I hate about GCP is that when creating a VM they force you to use minimum a 350gb hard drive...

        1. 2

          I’ve used aws for years. Just prefer gcp now. ESP for kubernetes stuff. That’s also not true. There’s no mandatory 350GB disks. Have you tried recently?

        2. 1

          I spun up one with 10gb yesterday :)

          1. 1

            Really? thats new I guess, when I used it they forced minimum 350b hard drives making it really expensive.

            1. 1

              You must have seen "GCP Cost Calculator" which doesnt allow you to set the size for storage. And it defaults to 375GB.

              However, if u try to spin up a new compute instance - it defaults to 10gb

              1. 1

                That seems nice tbh, is GCP cheaper than AWS? The only thing that I remember right now is that AWS has a lot more availability zones than GCP.

    2. 1

      Same so far cost me almost zero to have my staging env on GCP.
      Also everything is just working fine.
      Will see what the cost will be when I go production.

      1. 1

        Cool to hear, glad it's working for you

    3. 1

      Thanks for weighing in Samblr! :) I'm truly curious about your experience, and for example what your log analysis pipeline looks like, and if you've ever run into issues with App Engine instances not scaling down.

      1. 2

        App engine logs are available on stackdriver with zero setup.

        If you want to save costs on app engine - leaving few links below

        https://medium.com/google-cloud/three-simple-steps-to-save-costs-when-prototyping-with-app-engine-flexible-environment-104fc6736495

        https://medium.com/@dai_shi/running-a-small-node-js-app-for-free-on-google-app-engine-b921735c2d5b

        And Im just just overwriting existing instances while deploying with flags --version PRODUCTION - so it doesnt spin up new instances.

        1. 2

          Thanks for the links!

          I'm aware that logs very easily enter stackdriver, the part I struggled with is the analysis part (especially long term), since Stackdriver only maintains logs for a few weeks or a month maybe. Re: analysis, I found the charting/metrics/visualization tools really hard to work with, but to each his own.

          That's interesting about the --version flag, I could have sworn I tried everything.

        2. 1

          Ah, I tried using --stop-previous-version, which doesn't work for App Engine standard environments, but I missed the documentation about --version right beneath, so this was my fault for not reading closely enough. Thanks.

          https://cloud.google.com/sdk/gcloud/reference/app/deploy

  2. 6

    I run all of my indie hacker experiments on a single low cost VPS using Piku. I was running them on the lowest tier $5 per month Digital Ocean droplet but I recently hit 21 apps and started to see load averages above 0, so I upgraded to the $10 per month box.

    $10/m with 20+ apps and git-push deploys, you can't beat that for value. Very flexible for experimenting.

    1. 2

      +1 for DigitalOcean, I love their UI but also their pricing and performance are really competitive.

    2. 1

      You were running multiple apps all one one droplet? That's a really interesting idea. I imagine you need to deal with some port forwarding for that kind of setup? Thanks for the piku rec, looks really interesting.

      1. 4

        From my side I can recommend Dokku - it seems that Piku offers pretty much the same features, but Dokku is more mature. DigitalOcean also has Dokku as a one-click app, so it simplifies things even more. IMO way to go if you want a Heroku-like setup with git and no-downtime deploys at a fraction of a cost later :)

        1. 1

          Nice, I didn't know about all these diy PaaS solutions!

  3. 2

    Interesting read!
    My experience with GCP is the exact opposite from what you've experienced, I love Cloud Run and it has been very easy for me to deploy my containers there. When reading through your issues it feels like you've tried to enforce methods you used in previous infrastructure setups.

    I've also done this at times, and it always end up bad, and with a lot of sweat put into it. 😤

    1. 1

      Thanks for reading! Can I ask what kind of containers you deploy?

      1. 1

        The containers that I deploy there are either static asset serving containers, or API serving containers built in Go. I went over to Cloud Run from running these containers in Google Kubernetes Engine, which removed a lot of time and stress in building configuration.

  4. 2

    I get your point, especially when all you're trying to do is build a product, but that's only one way to look at it.

    Another one is "I spent $1500 and got a ton of hands-on experience with cloud infrastructure".

    1. 1

      That is a great point! I did learn a lot, and it increased my breadth of experience with cloud platforms.

  5. 2

    Nice story and valuable lesson. I particularly like how you quantify your time "wasted" into something you can easily judge/understand.

    Once heard a co-worker say "you can't make money if you don't invest money".
    I'll never forget that saying.

    1. 1

      Nice, that's a good one

  6. 1

    You changed tech stack to save 7$ a month without understanding the new stack. Premature optimization is the root of all evil in engineer led indie projects. Speaking from my own experience , here :)

    I have been using Google App Engine (GAE) for last 7 years and still think it is an amazing platform with a very generous free tier. I have used it for projects ranging from social networks to small SaaS products. GAE allowed to scale cloud costs with usage which is very important when bootstrapping.

    GAE enforces certain limitations which probably don't make sense on a small scale, but once you dig deeper you understand that "this is the way" at internet scale. If you don't like/understand why those constraints are in place, simply spin up a VM and pay for it.

    1. 1

      Thanks for sharing. I'm interested in how GAE is economical for you- do you use standard auto-scaling instances? The Python standard environment seemed pretty expensive at $0.05 per instance hour (and indeed burned my credit quickly).

      Also super curious if you use the log analysis/visualization/alerting tools (Cloud Logging) and if that works well for you.

  7. 1

    I couldn’t read the whole thing just now but this really jumped out at me as a fundamental impossibility

    “ I wanted to run on some cloud free tier and be a widely used service that didn't need to charge, because it had no costs.”

    A widely used service that has no running costs?

    1. 1

      For some small definition of "widely used", and if you are only looking at the financial picture, you could plausibly build an app that runs on free tier, and doesn't really have costs. But I agree, I think this is a naive way to look at it, and I've learned a lot.

      1. 1

        Yeah man stick with Heroku. I swear by it and I’ve used them all.

        You can get a LOT out of the $7 tier.

  8. 1

    I can't understand how some people can willingly choose google cloud platform over AWS, but there are a lot of weird people soooo .. :))

    1. 1

      How about a more intuitive user experience? 😁 AWS is quite messy.

      1. 1

        I did like GCP's UI a lot, which was one of the main draws for trying it

  9. 1

    I agree with the phrase "Your time is the most valuable resource, so get comfortable spending money to save it", I've been more productive by paying for good services. You'd want to pay for services that are already close to what you'll use for your production environment. So yeah...even if you want your app to be free or inexpensive you'd still need to put some bucks here and there to make it work.

    1. 1

      Yup, definitely

  10. 1

    Sorry to hear that, man. I've been through similar experiences multiple times whenever I think reinventing the wheel will end up being better. Last example was setting up a landing page, which I thought I could do complete in less than 5h and ended up taking more than 40h, no joke.

    I think the focus should not be on Google Cloud, though, but on the fact that you changed something that was already working for the unknown that seemed easier / cheaper.

    This kind of experience always reminds me of my friend Joe. He's cheap as hell, they say. He never wants to spend any money on anything at all. But he'll spend hours of his time to save $5, "solving" it by himself. He won't bring a six pack to a BBQ. However, he's the first one to show up at my door and spend a whole day helping me fixing my car on a Saturday. My wife says he's cheap. I say he's not. He just pays with his time what other people pay with money.

    You have limited time and money. Decide what you use to pay for what you need.

    1. 2

      Thanks for sharing your experience :) "My wife says he's cheap. I say he's not. He just pays with his time what other people pay with money" I like this way of looking at it.

  11. 1

    Honestly, I believe that unless you really know your way around AWS/Azure/GCP, Heroku (or Digital Ocean, which I haven't really used) is where we should be as makers. Deploying to Heroku with a simple push to a remote repo is not only magical — it lets you spend all of your time on the product.

    And you can scale Heroku as much as you want. Granted, it can become expensive if you hit bigger numbers, but I think that by the time, we, as makers, are at that stage, we have more resources/money to make the move to a less-managed solution.

    Not a Heroku fanboy, just my two cents!

    1. 2

      Agreed! I definitely think there is a time and place for moving to the "real" cloud platforms, but I don't think the early stage is the time for that, unless you're already an expert (as you said).

      1. 1

        I actually think you should use cloud platforms in early stages, because it's cheap, once you grow you should move out because it gets expensive fast.

  12. 1

    Haha I am definitely guilty of this as well. And have spent way more time than I should have trying to make things work for free (or close to free) by keeping things serverless / static. I chart a lot of the time not as wasted but as learnings but still need to be better at drawing a line and just coughing up some cash to save time.

    1. 1

      It's a good point that time is not necessarily wasted if you're learning! Definitely offsets some of the costs. Just depends on how much you learn, how valuable that knowledge is, and what you're missing out on in the meantime :)

  13. 1

    Personally I run everything I can on Cloud Run. It's literally my favorite service in the world. I don't think this was a GCP issue. I've deployed dozens of projects on Cloud Run, and moved my company completely over to GCP and it's been awesome.

    1. 1

      Thanks for weighing in - my hat's off to you :) To be fair, I didn't spend as much time with Cloud Run as I did App Engine, so maybe I should have spent more time with it, but I'm not sure it would have changed my experience all that much. Would be very curious to hear how you make log based metrics work in particular, since I really struggled with that.

      1. 1

        Not sure what a log based metric is. Do you have an example?

        1. 1

          For example, if your backend logs when a user performs a certain action, and you'd like to graph how many of those actions were performed daily over the last week.

          1. 1

            These containers should be stateless, and are scale to zero meaning a log in a container goes away as soon as the container goes away. You'd have to push the logs elsewhere like to promethius/grafana to get what you're after.

            1. 1

              Google Cloud Logging was successfully collecting my App Engine logs, my main problem was that the analysis/visualization/alerting tools were really hard to use imo, and the log storage was only a few weeks/a month. There were sinks to export to GCS/BigQuery, but GCS isn't for analysis, and I also found BigQuery difficult to set up. That's the extent of my efforts, since I assumed GCP provided an end to end pipeline like AWS does with their hosted elk stack🤷‍♂️

  14. 0

    You cannot compare Heroku (a PaaS) with GCP. Heroku has abstracted away the nuts and bolts which is why running anything serious on Heroku costs an arm and the leg. GCP is the nuts and bolts.

    As a counterpoint, I run all my projects on GCP and am a huge fan.

    1. 1

      In the post I talked about using App Engine, which is GCP's PaaS solution, so I would say they can be compared.

      1. 1

        Oh my apologies

  15. 0

    Curious, why don't you just start with GCE?

    (People wants to try the new shiny abstract product and then say its abstracted) !

    1. 0

      If I understand correctly GCE is cloud virtual machines, and I'm not really interested in doing sysadmin work to maintain that atm

  16. -2

    This comment has been voted down. Click to show.

    1. 2

      Yup, I'm using Firebase now, but not as a replacement for GCP, since I need infrastructure to run my python web app, which I don't believe Firebase offers.

      1. 1

        You can use python with GCP functions.. they are very similar to firebase functions. I am already using python for some of the GCP functions.. They are also easy to test locally.

        1. 1

          I need more than just Cloud Functions, though, I have a whole web app?

          1. 1

            Without ever having done it personally, I am 100% certain you can do arbitrary work in response to an HTTP request inside a Cloud Function, which is all you need to run a web app. I know someone who runs everything in AWS Lambda.

      2. 1

        They do not. You could probably integrate GCP into Firebase pretty well tho.