Do Indie Hackers test their code?
Software devs tend to agree that automated tests are a good idea in general. Opinions divide on the type of tests and whether they’re worth it at a certain level of maturity of a product.
I’m curious, how much test coverage does your current product have? Did you develop it from the start or added it later on? And having to start over, would you do anything differently?
Trending on Indie Hackers
Indie Hackers has almost no tests, although there have been a couple periods where I wrote lots of tests for parts of the site/code that no longer exist.
There are lots of problems we see when looking at bigger companies: regrets they have from the early days, technical debt they've built up, etc. But that's part of how they got to where they are — prioritizing well, knowing what to focus on and what can slip.
As an indie hacker, you have to look at every part of your business (not just the product), weigh their relative importance, and focus on the highest leverage thing. When you're small and looking for customers, that's rarely code. When you're bigger and established it's more likely be code, but not necessarily. Factors that come into play include: (1) whether you're hiring and onboarding lots of developers; (2) how big your existing customer base is, and how valuable it is to keep them happy and retained vs finding new customers; (3) your future product roadmap and how much it depends on existing code; etc.
If you have to spend a non-trivial amount of time to run your first test case, then that negative momentum can prevent you from prioritizing it early on.
It helps to start with a project template that already has a few tests written. Then adding additional tests for each sub feature you add becomes easier. You might still throw away the code later (based on reasons you mentioned), but decent test coverage might reduce the likelihood of that happening.
That makes a lot of sense. Perhaps tests should not be prioritised as a thing on its own but as a I need this part of the product to break less right now which may or may not include automated test coverage.
Testing? What's testing?
It's the action of writing code that doesn't rely on faith to work.
I think 100% test coverage of a non-trivial app is never worth it (except in Mars rovers and such) and it's less important still when building a MVP. I don't have any numbers at hand but I tend to have just the critical components tested. Unit tests when there's a complex algorithm and integration tests for the rest.
Part of my app is a JS embed which is critical and therefore 100% tested. Stripe integrations do also benefit from tests :) Having some broad integration tests help develop the app without fear of completely breaking. But I also get that writing tests for a UI constantly in flux is not easy and it's something I'd leave for later.
Starting over I'd probably add a bit more tests to my backend, hook all tests in my build scripts from the start, and maybe have a lightweight UI test set that could be run in IE :)
I don't know about just "testing the critical parts". Why do you have unimportant parts in the first place? Isn't it supposed to be a MVP, a version that's stripped of all the unnecessary parts?
I guess strictly speaking I'm over the MVP limit already, but I meant critical as in this piece of code has to work, otherwise there is no business. Sending transactional emails to customers is important but it being broken does not render your app useless. So whatever the core feature is, test that, and also the mechanism that brings you money.
(There's also critical but uncomplicated code, like code rendering the static HTML front page. I'd leave testing these for later as well.)
Testing just the critical components seems like a good tradeoff.
I have almost no testing right now. I started by TDDing everything but quickly gave it up so that I could move faster.
I realize that what I'm building now is of lesser quality and won't scale easily, but my immediate goals are to validate my ideas and make sure I'm building the right thing. My view is that there isn't much value in a high-code-quality product if it's not a viable product.
If the app takes off and I start feeling the pain of scaling? Good problem to have.
Makes sense! It's vital to find what to build first.
I’ve spent a few years as a tester in a funded start-up. I can’t speak to what Indie Hackers do, but because things change frequently and fundamentally trying to maintain a high test coverage target is wasteful. However, I don’t think all effort is lost, planning up front for tests can help you write better code. (?) Anyway, below is what I wish would happen before I arrived as a tester.
Building an automated build and deployment system before any tests. Even if its basic, just to force some forethought into the structure and design. While that might seem off topic, I think it goes a long way toward consistent releases and gaining rapid feedback.
There’s no point having tests in code that no one is using, but as soon as they do; and you want to extend the code, it would help to have it test ready. Think about testability and having some kind of plan for how you might cover the core features. Then, when you’ve got some traction and start adding new features, developers, start adding some tests.
I read something like this on Ministry of Testing recently: “If it’s not something you’d drop everything to fix, then you probably don’t need an automated test.” ...might help guide some decision.
I do not extensively test my applications but I do create automated tests for all critical features, just to ensure that its safe to deploy with peace of mind. I also use real time alerts, error monitoring tools, self healing instances and blue-green deployment. All these took quite some time to setup and get everything right.
I spent eight years on my application, and while it started with maybe 20% test coverage, it reached 87% before I sold it. These days, I wouldn't have it any other way. 100% test coverage is overkill, but now that I've so completely embraced testing, I find I'm able to write new code and ship so much faster than if the code wasn't tested.
The best way I've found to explain it is to time yourself manually testing your application end-to-end. Every critical detail. Now imagine you do that once per day or once per week, that time adds up. Of course, nobody regularly tests an application end-to-end manually that frequently.
With automated tests, though, upgrades to your language, framework, and dependencies are faster. The confidence you can have in new features or bug fixes not introducing regressions increases your development speed. And your ability to track down nasty obscure bugs increases drastically.
And, one of the best benefits is that when you have to make tests pass, you end up really learning about the intimate obscure details of your language and framework to understand how they really work. And that makes you a better developer.
It may feel like tests slow you down up front, but the long-term speed gains and improvements in quality are unquestionably worth it.
Brooklet does not have tests.
Instead, every time I deploy, I go through a checklist of manual tests.
How long does that routine usually take you?
Testing takes, maybe, 10-15 minutes per platform.
To me that's a point in favor of automated testing!
I also do the manual tests when there is a test suite set up. I find the value of tests comes more from catching regressions than from preventing manual tests before a deploy.
I'm a fan of testing. Someday Brooklet will have tests as well.
I've been on teams that write tests for everything and I've been on teams that have very few tests. In all honestly, while there's value in having tests there's also a considerable cost as well.
“There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.”
― C. A. R. Hoare
My philosophy these days is to write tests for the complex bits and just do some smoke testing on all the rest.
I'll play devils advocate in favor of testing here.
First, people assume that testing takes time. From my experience, when you don't have tests, you tend to check things manually a lot. I'd rather take time to write a test than constantly alt tab to check it works. Sometime writing a 5 lines test case can be a huge time saver.
Now for TDD, writing a test before your actual implementation can help you thinking about what you want to achieve. Again, taking time thinking "what do I really want to achieve with thik? Do I really need this feature? What is the bare minimum I need in order to make it work?" can be a huge time saver. Nothing is worth than sinking time into something that could have been done quicker or isn't what you actually needed.
That's a good point. If you're used to writing tests all the time, it might not be as big a time investment as for other devs.
If you practice TDD then you eventually get used to it. It might seems daunting at first when you ask yourself: "how can I possibly test that?" But when you have practiced it enough it's a huge time saver, even for just one developer doing a solo product.
I wouldn't go back to "refresh browser debugging" anytime soon.
+100 on people not taking into account manual testing time vs the time taken to write tests. Also on thinking about what you want before evening opening your editor. A couple of hours with a whiteboard or paper can save weeks of coding! :) I don’t go anywhere near my editor until I’ve a reasonable idea of what it is I want. If there’s some tech stuff I think might work but I haven’t used before I’ll think about doing a separate spike project to validate it quickly and figure out how to use it best in the main codebase.
It's always a bet upfront – I guess that's the hard part. You may decide to ignore tests and end up regretting that when you're processing hundreds of refund request from annoyed customers. You may also write full coverage for a feature that you end up scrapping after a while, as @csallen mentioned earlier.
We didn't wrote a single test for our current product. However to make this sustainable we need to spend more time to make the system components simple, in the sense of "not intertwined" (https://www.infoq.com/presentations/Simple-Made-Easy/). Otherwise you sooner or later reach a point where every change of the system is very risky, since everything is intertwined (like spaghetti code). In this situation you need tests to reduce the risk.
I've been in that position before (not with an indie project though), but we never seemed to get back around to writing tests. When the project got traction, there was also more pressure to add more features. It's hard to take maybe a month or two to focus on testing when all those other exciting things are going on as well. But I guess it can be done when quality becomes a major issue.
I worked on a partial rewrite of the application layer of a startup that started with no tests and got some scale. Development ended up stalling because fixing one bug would introduce two others. We ended up spending ~9 months in total, and managed to “replace the fuselage while the plane was still flying”. It was a good decision from management in the long term but it did mean no real new features for about 6 months. Not all places would go for that but once it’s done they had a much better base to keep expanding from.
What are automated test really for?
I'd also add
If it reduces your stress and /or makes you more efficient - go for it.
I just think most self solo builders feel just as good without it.
But let your work match yourself.
I have no plans to create tests for the initial launch of the MVP, but if things go well (i.e., I get paying customers and/or good feedback) then I'll find value in adding tests at that point. I'm a huge fan of using Python + Selenium for automated testing of web apps so that's probably the approach I'll take, along with basic unit tests.
Indie hackers need to move fast. Most of us really don't write tests as it increases the time to ship.
I write some tests as I go, but will likely harden with a lot more tests after seeing significant traction. I'd rather write new features that give me a chance at product-market fit then spend time getting 80% test coverage.
To start off the discussion, my current project (pre-launch) has full functional coverage on the back end and no coverage of the front end. It slows me down for sure but it also helped me catch a number of issues around my Stripe integration that could have backfired spectacularly in production.
The thinking behind the lack of coverage on the frontend is that the UI will change a lot more early on. But I’m wondering whether I’m just being lazy...
To my mind just aiming for a huge ~100% LOC coverage is nonsense in every project, but I definitely tend to cover most of the business logic, if I have some calculations and similar with tests, since it makes me more confident for future changes and it eases up the onboarding of people to a project as it acts as documentation in which you can believe :D
This comment was deleted 2 years ago.
Same here. For me, at least part of it is that testing an API is pretty easy compared to testing UI flows.