🚀 How many tests do you really need?

is there such a thing as enough tests?
For unit tests there is 100% coverage
And complete input validation of all types.

Usually what is common is to identify core, mission critical components, and test the heck out of them.
But not do so much for edge things.
Like you can think of a risk matrix, of what happens if this component fails... then categorise customer impact... if it's just doing to be a display issue it's not as big as an alarm issue...
Also the more users you have, the more common it may be to do transaction testing like sign-ups...
Another way to 'test' indirectly is to have good stats and alerts on them, for example, you track logins and alarms, and you know that during "normal days" it should be within a range.. (there are anomaly detection systems for better stuff), so if suddenly something is out of range you know there is an issue, due not from what... like no signups today where normal is 5-15, something is up, it could be stuff out of code as well such as servers, config and such, so it's a good view that isn't usually called 'testing' but it is protection of your systems which is very similar

Zero, users will test on production 😈

For reference, I'm developing Nodewood, a JavaScript SaaS boilerplate for people looking to build a SaaS from a decent starting point, and save time over personally developing the common subset of features every SaaS has. Given that, my point of view is a little skewed, since bugs in my code ripple out to everyone using it. Like library authors, I have to be a little stricter and less-comfortable with taking on technical debt than a SaaS founder can be.

I aim for 100% integration tests of the API, and unit tests only for things that make sense. i.e. validation can be cumbersome to test with integration tests, especially once you've set up a decent validation pattern. Unit testing those patterns means that anywhere else you use that validation, you don't need to be so strenuous when testing all possible permutations of that validation.

That said, I use tests to develop my API in the first place, so it's not like it's a hardship on me to add that coverage. I start off by defining all the important actions and considerations for my endpoint with a bunch of it.todo('should do xyz'), and then I write each test one at a time, turning it from red to green. When all tests are written and green, I can start using the API from my UI directly, since I know exactly how it's supposed to behave.

In the end, this development pattern saves me time, not just for the "less broken code to worry about later" standard reason, but because I don't have to worry about reloading pages, setting up state, sending requests, wondering whether the bug is in the UI or the API, etc.

It's great that your team is making testing a priority.

The question is a great one, very hard to answer, and to make matters more complicated; it can vary depending on the product you're building. Striving for high code coverage is a good idea for critical parts of your codebase that aren't likely to change much.

Manual tests may also be another option as a final sanity check before pushing a release, this can be a great way to catch issues before they reach production and can help steer what automated tests are worth prioritising.

Good luck!

For Writing Analytics, I have a high functional coverage for the backend – to ensure customers will be charged correctly, won't be sent 2,000 emails, and that the data that goes to the db is in a good shape. Basically, anything where the cleanup would be messy.

The UI is the exact opposite – I make lots more changes there, and the potential damage isn't that high. Even if there would be a severe issue like people not being able to log in or sign up (which hasn't happened yet, knock on wood), I could investigate and deploy a fix quickly.

When writing tests, I ask myself, "is it possible that this thing can be broken in the future? (is this an area that we change often?)", "if we break this, will we notice before we ship the bug?" if we can break it and we might miss it then the question is "what is the test I need to write so we can be safe from the embarrassment?" In other words, I try to be very pragmatic in terms of tests, and write just enough tests to make sure we don't ship bugs.

Ideally 100% (*) unit tests coverage, and of course more functional tests, integration tests possible.

There is a saying, "untested code is broken code", and this rule is valid everywhere except for machine learning stuff (which cannot be properly tested). Sometimes property based testing can help you discovering corner cases that are pretty hard to imagine.

• = the 100% coverage is opinionated, it's definitely not a silver bullet; but in my experience, not being able to take your code at 100% coverage can mean that you can improve code design.

IMO there are really two questions here.

1. How important is quality?
2. Which specific techniques are the best tradeoff for a given situation?

It’s really important to get beyond platitudes here. Everybody theoretically wants a higher-quality product. The trick is redirecting development time that could have gone into features, Product/market fit, etc., into improving quality instead. Or customers choosing to pass over cheap low-quality options for an expensive higher-quality one. These definitely happen, but are not universal. You need a clear vision of what level of quality your customers expect, or alternatively how much % you’re willing to spend, to focus the decisionmaking.

Once you have that, you know where you want to be on the cost/benefit curve. So you start with the tactics that are great cost/benefit and move down the list of less benefit until you’ve achieved your goal.

Indicators of excellent cost/benefit include:

• testing for problems that would harm your reputation if they occurred
• tests that can be written quickly
• tests that will run quickly. A slow test suite won’t be run
• tests for requirements people forget or are likely to break accidentally. Or did break.
• tests that encode information people would need to communicate about anyway, such as what steps reproduce a bug
• tests for things that aren’t obvious from the code, and the code can’t be changed to make them obvious

Indicators of poor cost/benefit include:

• We’ll probably change the expected results due to customer feedback, new learning etc.
• Testing the implementation details of code we plan to rewrite
• Behavior is nondeterministic due to time of day, network conditions, etc.

Keep in mind, automated testing is just one strategy for improving quality. There are many other ways to find/fix issues: logging, analytics, playgrounds, user testing, fuzzing, etc. Consider which tool will be best for the problem you’re solving.

It's a continuous function, not boolean. I feel for Stock Alarm, there are so many math conversions so you do want to make sure test coverage for those is 100%.

I've also seen devs religious about unit tests trying to have 100% test coverage on UI, even when it's just a test feature that may get thrown out in a month. To me, that should be negotiated because it can really slow down your speed of business innovation. Many devs don't understand that and insist to spend 3X time to cover an experimental UI. It's especially a problem when your company only has 3 - 6 months of runway.

On the other hand, for a company like Google, even for an experiment, there should be some automated coverage because it can afford the extra time that engineers have to spend on them.

I know I may freak out some people here but the best devs I work with write code that are logically sound so there are very few bugs. Usually they just need to write a few test cases to cover the corner cases to prevent future updates breaking the behavior. To me that's honestly the ideal state.

Unit tests cannot replace clear logical understanding of the problem. There are devs who mechanically write tests and tweak their code until all the tests pass without really understanding the problem. That tend to be the case in many companies where a good testing practice have not been established so people try to compensate by writing redundant tests.