Your product's complexity + testing
What's up everyone!
A lot of my work at my job recently was about testing, so wanted to talk to you all about it and see what you think:
Testing is an essential part of developing and maintaining a software product. It ensures quality (hence it's part of the umbrella term of Quality Assurance or QA for short).
But testing isn't just a matter of quality, it also about velocity and maintainability. If you want to move fast and make changes, one needs to have the confidence that they are not destroying existing functionality that users are already using.
Automating testing ensures that you are able to repeat tests as much as you want, every day or hour, with and without changes.
The importance of testing grows as your product grows in complexity and features, as well as with the number of users that might be affected by changes you make.
So I wanted to ask you all, how complex is your app and how do you test it? What's your take on the various types of testing (unit, integration, UI)?
Getting my first 100 users with $0: what actually worked
What's the point of AI generated comments?
Why good products are often hard to understand at first glance
Why I’m building an AI marketplace instead of another SaaS
Why can't your target customers always find your product? - Experience sharing
The exact prompt that creates a clear, convincing sales deck
Interesting discussion! The startup I'm a part of has also considered testing a lot.
Our application has some very complex parts, e.g. parsing of social media data in various formats (JSON, HTML, CSV, and more). But as with any other app, it also has a lot of boilerplate code e.g. authentication.
The goal for us was to get a working MVP ASAP, so with this in mind, we valued velocity over maintainability.
We quickly realised while developing the parser, it was "easy" to do it the TDD way, and write unit tests as part of its implementation. UI and integration testing was a bit harder. One due to constant UI changes and second because of difficulties getting an automated integration test flow up and running with e.g. GitHub Actions. This lead to a lot of maintenance and refactoring of the tests, which cost a lot of time!
It ended up with only the parser being covered with unit test, and all other aspects were left for future work.
I do though fear what implication this will have once we go live and if we should have prioritised differently, as you put it:
"The importance of testing grows as your product grows in complexity and features, as well as with the number of users that might be affected by changes you make."
I do look forward to others' input, as I've struggled a lot with UI and integration testing, while at the same time seeing the tremendous benefits it can provide!
I really appreciate reading about your experience. It shows just one way, out of many, that businesses can reach a position where they would benefit from something like what I'm currently building.
I imagine it's vital for companies to be able to go fast, even when that incurs some technical or maintenance debt (e.g. putting testing aside for a while), but it's also important to be able to regain control and reduce that debt in the future.
When you say you "struggled a lot with UI and integration testing," it does resonate with me. I imagine tests needing a lot of maintenance and failing when they shouldn't, because that's something I experienced a lot. Does that describe your experience?
For even complex apps and changing apps, there should be a way to easily detect changes and verify functionality, in a way that doesn't involve a lot of overhead. What do you imagine something that would help with this to look like?
Interesting. I would love to hear more about your project. Anywhere I can find it?
Yes it definitely describes my experience, and I'll give my quick thoughts as to why, but I will also talk with my team and try to collect some more considerations from them.
A unique problem with integration testing was the "time to result", as creating e.g. a Mac instance, downloading flutter, downloading the repository, building the application, and running the integration test took such a long time. Testing one small change to the UI could take >10 minutes. This led to a lot of context switching, as you would work on another task while the test run.
A common problem for both was mocking. While there's no specific issue with mocking it just took too long, and our problem might have been with the specific mocking framework we used.
As to something that would help us, that's a bit trickier. Two things come to mind:
Saving time should be a primary goal of any solution built for this. I think automating this process is a very doable first step to shorten the time to result (and take care of repeating tests).
One challenge I faced with integration tests was the effort required to setup the case to be tested, sometimes multiple processes are required to be invoked before getting to the actual test. That adds to the time it takes to code the test, as opposed to users who just click here and there to invoke those processes. That's a +1 for UI tests over integration tests.
I imagine that an always-ready test instance can help speed things up. It's an assumption for what I'm building actually. The QA system would access a test instance that is already up and running. There are CI/CD tools that automate the process of automatically deploying after making a commit (merging a PR).
I agree with the helpful features on your list and I am very excited at the prospect of building these features one day. For now though, I'm building a very minimal MVP.
The goal for me now is to build the basis on which I will continue building more things as I get more validation on the idea and let people like you tell me what to build next.
So I imagine that the first version would just allow the user to specify what to do (and how) and what to check for. For example saying something like:
This simple test would be an example of a UI/E2E test for the accounts search page. It would ensure that the pages involve are running, contain the UI elements expected, and also exercise the backend logic for search.
Generating something like that automatically, is an exciting future feature, definitely on the list, but not yet for the initial phases! I imagine people would still be interested in the continual automatic testing and notifications despite needing to tell the system what to do (once). Do you think that's a reasonable thing to expect? Would it be valuable enough for people to be interested in subscribing to a SAAS that does this?
That's more than enough, and we would be interested in at least trying it, to see if it meets our needs!
What's the ETA of an initial test release?
If I had a sales director I'd be fired now for only getting back to you after 3 months, hah... One day I'll hire a sales director 🙏
I'm still working on it. My full time job just consumed me for all this time. Thinking about it, the fact that my job distracted me from EndTests.com, is the very reason I need to focus on getting it done successfully.
Now that I'm back on it, I'm working on an "Exploration" feature, that allows it to "know" a system so it can do some initial setup automatically and not have the users need to manually set all the test cases from scratch on their own.
This feature might also be useful for other things as well, such as change detection...
I posted the same thing on Reddit and got 1 reply... Here, a whopping zero.