Share your development workflow
I'm a self taught programmer who started to learn programming 2 years ago during the nights after work. I've built a few websites and web tools for a few users for fun, but I'm not confident about my workflow.
What is a good workflow for a solo hacker? Be sure to mention:
Also, would love to have an experienced programmer look at my setup and get some feedback, can buy you a coffee in return :D
#tech-questions
I use Zeit's Now for hosting and deploying. It's the simplest way to deploy node or static websites. And it's almost free. You don't need to worry about infrastructure.
My answer for this as an indiehacker is very different from my answer as an employee. The last thing I want to do is make things overly complex and slow myself down.
Thanks for this.
By the way, why do you find GCP more complex than Digital Ocean? I've always wondered why DO is popular among developers, as I would assume if at any point a project becomes successful and needs to scale the de facto platforms are AWS or GCP; never heard of a company with traction running on DO. Also, with GCP you can fire up one of their services so easily (storage, queueing, hosted db, etc..), and also learning AWS / GCP is a marketable skill.
Only reason I would personally use DO is to save an extra 10$ / month or so (GCP smallest instance is 14$ / month).
Google and Amazon both support some fairly complex enterprisey features that lead to more complexity in the UI and they both offer a lot of niche services that encourage, but of course don't force, companies on the platform to create overly complex infrastructure that's dependent on GCP or AWS.
This is just depressing.
On nearly every benchmark I've seen EC2 underperforms a similar DO droplet. They come up regularly on sites where programmers hang out.
I've done benchmarking myself and found the same to be true. On a $5/month droplet, running a live phoenix app that hits the database on page loads, I've benchmarked at well clear of 1,000 requests/second. That's more than enough to withstand being #1 on HN. It works out to many millions of uniques per month on a typical app with pretty spikey usage patterns. That's more than the series A funded startup I used to work for got ever.
Of course before getting to a million uniques a month in actual traffic, I'd upgrade to a $20 or $40 server.
Horizontal scaling is relatively easy for me due to my tech stack, but even if it weren't, just about any app that hit the limits of vertical scaling would be raking in buckets of cash. This is even true for ad-supported mobile games aimed at a populous market like China. My friend I mentioned above who scaled a frameworkless PHP site (which he updated via dragging files into a Filezilla FTP tool), was on a random WP host and scaled to 2MM uniques/month and 50k MRR before moving to a more "pro" PHP hosting site!
FWIW, Digital Ocean is the third largest host in the world and while their usage skews towards medium sized sites, they've increased marketshare greatly in the past year, gettting net movement from Linode, Amazon and MS. If you're truly worried DO can't handle your site's scaling needs, then you're basically concerned that you'll grow far faster than World of Warcraft or even Uber did. That's basically the topic of my most recent podcast, Scalability Arrogance.
Yes. This reason I 100% agree with. If you're doing projects to help get yourself a job, learn AWS.
Thanks for this, very insightful. What do you do for storage? You store on the VM? My stack uses S3 for static and user generated files, it's very cheap and again might as well make it part of the stack for learning purposes.
For a new, exploratory project? The filesystem. I've also used S3 and DO Spaces in cases where I expected to need a large and unpredictable amount of space.
Following 😊
Certainly glad to help out. I've been a software engineer for almost 10 years now.
git reset --head). For large projects, i'll do the same locally but the main difference is that i'll use branches to work on features. Plus github reviews.Thanks for this!
Though what are
github reviewsthat you've mentioned?I meant that in solo projects, I will always commit/merge to master. But in team projects, everyone goes thru code review process before merging their feature branch to master. The github/gitlab UIs for pull-request reviews have been adequate for me.
I use git with a master and WIP branch. I push to a remote everynight. You never know when gremlins will steal your laptop
I am aws man myself with terraform for structure. Though Im looking to do more static work with gridsome and netlify to save on costs.
Managing environments is usually something too advanced for my solo projects so I have local and prod.
I usually just have a script that runs my tests and does deploys locally with terraform. Doing a complete throwaway and rebuild of everything except data in a db.
I write tests in jest as most my work is in JS. Should probably do more cypress testing of e2e features but I just click through things most of the time.
Here's mine:
https://nvie.com/posts/a-successful-git-branching-model/
but as a solo developer I'm not sure whether it's useful, seems more suited for team work. I've never really had cases where I thought "thank goodness I'm using this workflow", so not sure I should continue on this route, and in fact I never felt i really need git O.o (!)
For staging, nginx is configured to route requests to staging.mydomain.com to another uWSGI docker container which is configured to have staging environment variables, alongside yet another docker container for a staging postgres db. As I said, everything is on one VM instance.
I've read about tools like Travis, CI/CD etc but wasn't sure if it's worth the effort for the scope of apps / scale I'm at. Also, I don't write any code for testing.
Based on the above how can I improve my workflow?
This is a very sophisticated setup for someone with only two years experience. Your setup will easily allow you to scale up to 5 devs and probably 2 teams of 4-7 devs. Since you are a solo dev you are not seeing the benefits of what you have done already. Keep doing git you could probably simplify to just a master branch https://softwareengineering.stackexchange.com/questions/312022/the-trend-of-the-develop-branch-going-away
Not sure about your application but sounds like you got too many services. Lean on cloud providers and let them host all the services you are using images for. There could be savings there and easier to scale.
However this sounds like it is working. So I wouldn't change a thing unless you start to have trouble keeping up with releases or it costs too much.
Automate your steps 1-4 in a shell script is only thing I think should be done. A small mistake will happen at some point if that process is manual.
Thanks for this! Indeed steps 1-4 are already automated through shell scripts.
You mentioned I could move some services to cloud providers, but wouldn't that incur more costs? E.g. the db service is managed by myself (installed on the same VM instance as the web server and everything else). If I go for a hosted db it will have an extra monthly cost. Same for async / messaging services that I'm managing like celery / rabbitmq.
This depends on your usage. Are you doing anywork to keep your queues and db scaled/up/and backed up? If so you may want to put a $ amount on your time and see if switching over is cost effective. Keep in mind learning all the services will cost you time/money. If you are having no problems then again dont do anything. Only fix things that are broken now. Dont dev for the future.
This looks like a reasonable setup if you're doing it as a demonstration project in job-seeker's portfolio, but I think it's way overengineered for an indie (assuming you don't already have millions of uniques/month or some really intense usage patterns).
For a comparison, Uber's initial dispatcher was just a page of PHP for its entire first year. No docker, redis, celery or rabbitmq was needed. Servers were a lot slower 9 years ago, too!
Teespring was also incredibly barebones for a long time. Here's Walker Williams explaining why it was crucial for them. Groupon was that way too, they were a WordPress app until they had a crushing number of users.
Don't worry about scalability now. The far harder thing is getting enough users that it's even an issue.
Everything I did because I actually needed to improve the speed of deployment because I would spend too much time on it, not because I was building for scalability. Also, celery and rabbit are for async / background jobs, I don't know what else or simpler there is for a solo dev? Docker turned out to be really sweet and very neat to deploy your project on new instances.
I guess I should have asked this question 2 years ago... at least now I can get this stack up on any new project pretty quickly.
Most programming languages have mechanisms for this. In my case, it's just firing up a process in the EVM (which is the same thing RabbitMQ does, come to think of it).