What strategies, tools, systems are you all using to test and quality assure your products before you ship?
Really interested in upping my game on this.
I'll give you a quick walk through of what my testing and QA process is like for development. This stuff is only going to apply for tech/software based products.
In dev, everything should be unit tested. All key functionality has unit tests written, and these are automatically run whenever code is committed to our development servers. This both ensures a reasonable level safety for the initial launch, but also helps prevent issues introduced by changes down the pipeline.
A lot of our stuff is web based, so we also run tools like selenium for browser testing.
Once a change has gone through all the tests, it goes into our dev environment. Here it gets manually tested against its intended functionality. It helps here to have user stories as it defines the exact functionality to test for with a desired end result. It's important though to try and think outside of the box and really try to break it, if you can break it in dev then it's a sure thing a user can too. It helps to have an outside set of eyes do the testing, as they can help find things you wouldn't have even thought of.
It's also important to note that our dev environment mirrors our production one in terms of setup. It would do us no good to do all this testing only to have issues crop up due to inconsistencies in environment.
Another thing we try to do is keep functionality at an absolute minimum. The smaller the change the easier it is to test. We've also been working a lot in more of a functional style as it makes things easier to test.
After the change itself is tested we also like to go back and just do a sweep over key functionality of the application that may have been changed or affected. If the unit tests are good any errors should hopefully be caught there, but you can never be too careful.
Hopefully that helps you a bit!
Awesome, thanks! A couple of questions...
Can you give an example of a set of manual tests against a user story?
What kind of thought patterns do you use to be "creative" in how you break it?
How formal are you in recording your acceptance tests?
What systems or processes do you use to manage your testing?
When you use "outside eyes", where do you find them?
How rigorous would you be testing the functions across browsers?
Usually if we have user stories we have them written up before development. So something like "The user clicks a button that opens a modal so they can edit their info".
Generally, just trying to think of non-standard ways to do things. If we expected someone to go 1-2-3 what happens when they try to go 1-3-2? It differs depending on the functionality of what's being tested.
We're not super formal. We have usually have a set of functionality, and if it passes that along with any practical edge-cases it's good.
We integrate with a handful of unit test libraries depending on the stack and then run a jenkins jobs that automates them before deploys. For more manual stuff we keep it in our project management software.
These can be anyone. If you have friends and family ask them to test stuff. If you're on a team having others, especially non-devs, try out functionality. If you have the budget, you can also hire dedicated testers. All this is great for UX testing too.
We generally test on the latest version of modern browsers, I.E 9+, and a few mobile devices. You really have to think about your user base here as you could waste a ton of time here checking for weird edge cases in a fringe browser.
One thing I forgot to mention, and it's brought up in the comments, is that this might be overkill for you. I've worked in finance where a lot of this is absolutely essential, but for a startup getting out the door might be more important. It's up to you to decide what a reasonable level of risk is, and develop accordingly.
Thanks, really clear.
We work in health-tech for the main business. Rigorous, governance heavy, slow... But we're pushing on a side-project for fun. I'm keen to try and adopt a more agile process for it, so felt a good question to ask IH community.
Is your product mission-critical? Is your customer going to sue you if things break?
The things being recommended here sound like overkill pre-launch.
Focus on finding customers. Make it clear it's a public beta and set expectations accordingly.
I don't disagree with the idea of proportional effort. But are you advocating a zero testing strategy?
What you do as a software engineer in a team in a major corporation looks quite different than what you do as a self-funded solo founder. Is the product just you? Are you outsourcing the software engineering?
Are you shipping a small and simple CRUD app? Then you can absolutely wing it while you acquire your first few customers. Your approach should be tailored to your particular product and target customer.
I completely agree with you. Proportional testing is the best way. But I don't agree with no testing.
Anyone can boot up few different sizes of screen/devices/browsers, check links work and run through the basic processes of their service before launch. I'd argue it's disrespectful to customers to not do a basic level of testing.
I'm really interested in other peoples processes and mental models, so I can improve our own approach. Not looking to judge, just curious.
On that note, to my orignial post... What's your testing process?