6
19 Comments

Share your development workflow

Share your development workflow

I'm a self taught programmer who started to learn programming 2 years ago during the nights after work. I've built a few websites and web tools for a few users for fun, but I'm not confident about my workflow.

What is a good workflow for a solo hacker? Be sure to mention:

  • how you use git
  • which cloud provider and the platform (e.g. GCP on Compute Engine, or GCP on Google App Engine, heroku...)
  • how do you manage development, staging and production environments
  • how do you deploy
  • do you write tests

Also, would love to have an experienced programmer look at my setup and get some feedback, can buy you a coffee in return :D

#tech-questions

on March 15, 2019
  1. 3

    I use Zeit's Now for hosting and deploying. It's the simplest way to deploy node or static websites. And it's almost free. You don't need to worry about infrastructure.

  2. 3

    My answer for this as an indiehacker is very different from my answer as an employee. The last thing I want to do is make things overly complex and slow myself down.

    • I usually make feature branches now, but for the first 3 months, I just worked on master
    • Other than Amazon's Simple Email Service, I use Digital Ocean for everything. It's cheaper, it's faster and I don't need the complexity and layers of indirection of GCP, AWS or even Heroku.
    • I do dev on localhost, don't use staging and production is on DO.
    • I just pulled from git on production for the first couple of months, then set up deployment via Edeliver (an Elixir-based tool) and now I use Gitlab CI/CD. CI runs tests on every push to any branch and CD deploys to my server via Edeliver. Interestingly, though Gitlab uses docker, I don't have to use it on my server (which sacrifices some of Erlang VM's more powerful features).
    • Yes, I write tests but I'm not dogmatic about it!
    1. 1

      Thanks for this.
      By the way, why do you find GCP more complex than Digital Ocean? I've always wondered why DO is popular among developers, as I would assume if at any point a project becomes successful and needs to scale the de facto platforms are AWS or GCP; never heard of a company with traction running on DO. Also, with GCP you can fire up one of their services so easily (storage, queueing, hosted db, etc..), and also learning AWS / GCP is a marketable skill.
      Only reason I would personally use DO is to save an extra 10$ / month or so (GCP smallest instance is 14$ / month).

      1. 3

        Google and Amazon both support some fairly complex enterprisey features that lead to more complexity in the UI and they both offer a lot of niche services that encourage, but of course don't force, companies on the platform to create overly complex infrastructure that's dependent on GCP or AWS.

        I would assume if at any point a project becomes successful and needs to scale the de facto platforms are AWS or GCP

        This is just depressing.

        On nearly every benchmark I've seen EC2 underperforms a similar DO droplet. They come up regularly on sites where programmers hang out.

        I've done benchmarking myself and found the same to be true. On a $5/month droplet, running a live phoenix app that hits the database on page loads, I've benchmarked at well clear of 1,000 requests/second. That's more than enough to withstand being #1 on HN. It works out to many millions of uniques per month on a typical app with pretty spikey usage patterns. That's more than the series A funded startup I used to work for got ever.

        Of course before getting to a million uniques a month in actual traffic, I'd upgrade to a $20 or $40 server.

        Horizontal scaling is relatively easy for me due to my tech stack, but even if it weren't, just about any app that hit the limits of vertical scaling would be raking in buckets of cash. This is even true for ad-supported mobile games aimed at a populous market like China. My friend I mentioned above who scaled a frameworkless PHP site (which he updated via dragging files into a Filezilla FTP tool), was on a random WP host and scaled to 2MM uniques/month and 50k MRR before moving to a more "pro" PHP hosting site!

        FWIW, Digital Ocean is the third largest host in the world and while their usage skews towards medium sized sites, they've increased marketshare greatly in the past year, gettting net movement from Linode, Amazon and MS. If you're truly worried DO can't handle your site's scaling needs, then you're basically concerned that you'll grow far faster than World of Warcraft or even Uber did. That's basically the topic of my most recent podcast, Scalability Arrogance.

        also learning AWS / GCP is a marketable skill

        Yes. This reason I 100% agree with. If you're doing projects to help get yourself a job, learn AWS.

        1. 2

          Thanks for this, very insightful. What do you do for storage? You store on the VM? My stack uses S3 for static and user generated files, it's very cheap and again might as well make it part of the stack for learning purposes.

          1. 1

            For a new, exploratory project? The filesystem. I've also used S3 and DO Spaces in cases where I expected to need a large and unpredictable amount of space.

  3. 2
    • I use Gitlab for repos and Gitlab CI/CD to deploy to Zeit Now.
    • Since it's just me I just work on one branch and have that deploying automatically on every commit to my dev environment. Then when I want to shift code into production I do a manual deploy. To lock down dev I'm using basic auth on the node server as Now doesn't have any built in functionality for securing deployments.
    • For tests I've only really written E2E tests (using Cypress.io)
  4. 1

    Following 😊

  5. 1

    Certainly glad to help out. I've been a software engineer for almost 10 years now.

    • Git: depends on the project. For side projects (i.e., projects of just me), git is basically my backup. I will commit constantly (micro commits) so that HEAD always has a working bleeding edge. Stuff that doesn't work gets nuked (git reset --head). For large projects, i'll do the same locally but the main difference is that i'll use branches to work on features. Plus github reviews.
    • at work, I've used everything from AWS to heroku. For small projects, I've had success with heroku (ease of use). But cost-wise, a docker-based platform is most likely more cost effective. (And it's not really that more complex) I haven't really liked AWS because of its documentation.
    • in my previous job, our product was a linux binary. We used docker for everything. Development was in local docker containers. For staging (for us QA), we published the same docker containers to AWS. "Production" meant building binaries in docker and publishing to AWS.
    • tests. I'm gonna start by saying that 100% coverage is a pipe dream. The reason is because tests are code coupled to the production code and it needs someone to maintain it. The ROI is often prohibitive. Instead I tend to prefer "risk" based testing---test important features, features that are complex, etc. I often prefer the inverted testing pyramid (lots of unit tests relative to integration tests). But for recent API work, I threw that rule out the window and had lots of integration tests relative to unit tests (I guess because the APIs were simple and it was more important to test "workflows" involving multiple APIs).
    1. 1

      Thanks for this!
      Though what are github reviews that you've mentioned?

      1. 2

        I meant that in solo projects, I will always commit/merge to master. But in team projects, everyone goes thru code review process before merging their feature branch to master. The github/gitlab UIs for pull-request reviews have been adequate for me.

  6. 1

    I use git with a master and WIP branch. I push to a remote everynight. You never know when gremlins will steal your laptop

    I am aws man myself with terraform for structure. Though Im looking to do more static work with gridsome and netlify to save on costs.

    Managing environments is usually something too advanced for my solo projects so I have local and prod.

    I usually just have a script that runs my tests and does deploys locally with terraform. Doing a complete throwaway and rebuild of everything except data in a db.

    I write tests in jest as most my work is in JS. Should probably do more cypress testing of e2e features but I just click through things most of the time.

  7. 1

    Here's mine:

    • git: I try to follow this workflow:
      https://nvie.com/posts/a-successful-git-branching-model/
      but as a solo developer I'm not sure whether it's useful, seems more suited for team work. I've never really had cases where I thought "thank goodness I'm using this workflow", so not sure I should continue on this route, and in fact I never felt i really need git O.o (!)
    • I use GCP on Compute Engine, never tried Google App Engine, briefly tried Heroku when I started to learning programming but found it very opaque and couldn't understand what I was doing. I decided I should learn setting up a server from scratch so moved to GCP. I've only ever needed 1 instance since the apps I've built have served a small number of users.
    • My dev workflow is as follows:
    1. Build docker images locally of each service I use: Django app on a uWSGI server, postgres, redis, celery, rabbitmq, nginx
    2. push images to Google Cloud Registry using gcloud cli
    3. push a few simple deployment scripts to the GCP compute instance using gcloud cli
    4. ssh into GCP server, pull the images and deploy with docker-compose up. Static files get uploaded into S3 with one of the deployment scripts.

    For staging, nginx is configured to route requests to staging.mydomain.com to another uWSGI docker container which is configured to have staging environment variables, alongside yet another docker container for a staging postgres db. As I said, everything is on one VM instance.

    I've read about tools like Travis, CI/CD etc but wasn't sure if it's worth the effort for the scope of apps / scale I'm at. Also, I don't write any code for testing.

    Based on the above how can I improve my workflow?

    1. 2

      This is a very sophisticated setup for someone with only two years experience. Your setup will easily allow you to scale up to 5 devs and probably 2 teams of 4-7 devs. Since you are a solo dev you are not seeing the benefits of what you have done already. Keep doing git you could probably simplify to just a master branch https://softwareengineering.stackexchange.com/questions/312022/the-trend-of-the-develop-branch-going-away

      Not sure about your application but sounds like you got too many services. Lean on cloud providers and let them host all the services you are using images for. There could be savings there and easier to scale.

      However this sounds like it is working. So I wouldn't change a thing unless you start to have trouble keeping up with releases or it costs too much.

      Automate your steps 1-4 in a shell script is only thing I think should be done. A small mistake will happen at some point if that process is manual.

      1. 1

        Thanks for this! Indeed steps 1-4 are already automated through shell scripts.
        You mentioned I could move some services to cloud providers, but wouldn't that incur more costs? E.g. the db service is managed by myself (installed on the same VM instance as the web server and everything else). If I go for a hosted db it will have an extra monthly cost. Same for async / messaging services that I'm managing like celery / rabbitmq.

        1. 2

          This depends on your usage. Are you doing anywork to keep your queues and db scaled/up/and backed up? If so you may want to put a $ amount on your time and see if switching over is cost effective. Keep in mind learning all the services will cost you time/money. If you are having no problems then again dont do anything. Only fix things that are broken now. Dont dev for the future.

    2. 1

      This looks like a reasonable setup if you're doing it as a demonstration project in job-seeker's portfolio, but I think it's way overengineered for an indie (assuming you don't already have millions of uniques/month or some really intense usage patterns).

      For a comparison, Uber's initial dispatcher was just a page of PHP for its entire first year. No docker, redis, celery or rabbitmq was needed. Servers were a lot slower 9 years ago, too!

      Teespring was also incredibly barebones for a long time. Here's Walker Williams explaining why it was crucial for them. Groupon was that way too, they were a WordPress app until they had a crushing number of users.

      Don't worry about scalability now. The far harder thing is getting enough users that it's even an issue.

      1. 1

        Everything I did because I actually needed to improve the speed of deployment because I would spend too much time on it, not because I was building for scalability. Also, celery and rabbit are for async / background jobs, I don't know what else or simpler there is for a solo dev? Docker turned out to be really sweet and very neat to deploy your project on new instances.

        I guess I should have asked this question 2 years ago... at least now I can get this stack up on any new project pretty quickly.

        1. 1

          Also, celery and rabbit are for async / background jobs, I don't know what else or simpler there is for a solo dev?

          Most programming languages have mechanisms for this. In my case, it's just firing up a process in the EVM (which is the same thing RabbitMQ does, come to think of it).