Developers June 1, 2020

How to test apps before releasing to production?

Misir Jafarov @misir

We are small startup/company. We don't want to ship broken versions to our customers. For now we don't have enough budget to spend on creating a QA & Testing team. How does new startups & small companies doing testing before shipping apps? I'm trying to test 2-3 times after building every feature. But that's not enough sometimes for detecting business critical bugs.

Simply I'm looking for an advice: how should we invest on testing? Or should we?

  1. 6
    1. Test as much as possible for all the edge cases you can think of.

    2. Be the first user of your application and do all possible things on the client side like uploading wrong input files, trying different browsers, different devices, os etc, invalid field inputs, make payments, cancel payments in between. I've even tried turning my wifi off while a payment is in process. Likewise, be mischievous with the end product.

    3. Get some beta users. They will find you ton of bugs to fix. It's okay. Fix them.

    Once you have done all the above steps and all the tests mentioned in other comments, release to production and your actual users will find a hundred other bugs for you. Trust me, it's okay as long as you fix these quickly and give them good customer support👍👍 I have done this before.

    1. 1

      Best response

  2. 5
    • Unit and UI Tests
    • Manual testing and reviewing your app before release (it’s extremely boring but helps)
    • Peer Code Reviews if you have at least 2 devs
    • Beta version where you ship first to a couple of users before it goes to everybody
    1. 2

      +1 for unit testing

      How I do this is to try and keep my clients stupid by putting most logic on the backend where I find it easier to test.

    2. 1

      I peer review my own code 😅

      1. 1

        Me too 😬

  3. 3

    Thanks everyone who replied to this post 😊

  4. 2

    As far as I know you can just release the app as beta. At least that way people will understand that the app might be a bit buggy. They might also be more inclined to give feedback about bugs that occur in a beta app

  5. 1

    I run a website https://testingbot.com which provides Automated Testing for both websites and mobile apps. The idea is to record a test, then run the same test on multiple combinations after each build/commit.

  6. 1

    I'm 100% on with automated testing. But not unit testing. I write feature tests (sometimes also called integration tests or other names). The idea is to imitate user actions to make sure your app responds as one would expect. They never failed me.

  7. 1

    Not sure if anyone has said this yet, but in addition to all the other useful suggestions, testing in production is actually a real thing.

    You add feature flags that let you test your code in production. You can even use this to roll out features to a subset of your users to see if anything breaks.

    For reference, just google "testing in production with feature flags" for great results.

  8. 1

    I’m pivoting my startup to exactly work on this idea. Easy to write tests, automatically asset unhandled exceptions, visual regression testing and have them run on the cloud in seconds with 1000s of shards.

    Would love to talk to you if you’re interested.

  9. 1

    What kind of app? You probably would want a test build if mobile or test enviornment on web dev and you can have a QA team go through some smoke tests and check the changes as well, also using a tool to catch errors in productions is very effective (Bugsnag, Honeybadger are the ones that I've used and both work great)

  10. 1

    Do as much testing as possible but then launch on Heroku so everyone on all devices can play with your app and give you feedback. Don't try to be pixel perfect on the first go around. You can do that later.

  11. 1

    I'm TDD'ing my apps: write the test first, then make it pass. This also helps me in figuring out, how I want the feature to work before I implement it. That leads for me to cleaner code and less bugs. And testing the rest is much more relaxed. Benefit: you can test your app on each commit per CI (Travis, Selfhosted DroneCI, …)

  12. 1

    My product Pull Dog (https://dogger.io) helps with that. And it's free.

  13. 1

    Unit testing, and also a separate staging infrastructure.

  14. 1

    Well with that setup, you can't do much. What you can do you can create a todo list what needs to be tested and to isolate your main workflows. Make sure that your main workflow is always tested. You may not have time to test everything but at least test your core features

  15. 1

    Hey Misir,

    When done right, testing produces a lot of value without much cost (i.e. your own time). What can be tough to answer for folks is usually not deciding whether or not to test (emphatically yes!) but, how much time should you invest in testing? My advice:

    • Investing the time to get some unit test coverage is a big win in my experience. Getting your code to be in state that's unit testable almost always leads to a better design, which in turns makes it easier to modify that area of the code in the future. There's a lot of strong opinions on this but my advice is: identify the areas of code that are (a) most critical to your business (b) would be best served with a well-designed foundation, and add unit tests there.

    • Manually test locally before every PR, smoke-test in prod after every deployment. We're a team of two and have gotten in the habit of listing the things we've manually tested in every PR. It only takes a few minutes but it forces you to not just focus on the features you're building but also what could break as a result of the change. We don't mention unit tests at all in this list btw - those are evident in the code so no need to summarize them in the PR description. Smoke-testing one or two things in prod (ideally something relevant to your change) just gives you piece of mind that something didn't majorly break.

    • End-to-end your critical flows. This is where my pitch comes in :) We built a tool that makes it easy to automate the end-to-end tests that you'd normally test manually. These are tests for things that absolutely can't break - things like add-to-cart/checkout on an e-comm store, or login and registration for any web app that requires a user login. Our product is called Reflect, it's a no-code testing tool, it's free to try and lets you get a few tests set up in minutes compared to hours or days for code-based tools.

    We're always happy to answer questions so feel free to reach out if you'd like to talk testing!

  16. 1

    Having a beta period of the release is a good idea to get feedback on a new app/release.

    Having unit tests can be handy as well they've saved me a couple of times.

  17. 1

    Find the critical parts of your app that would cause you and your customers much pain if they did not work correctly and start there. Write unit tests (for backend code) or UI tests (for front-end code) for these critical parts. You may feel you don't have time to do this now because other things are more important but I guarantee you, it will save you so much time in the long run.

    Also, use something Sentry.io to catch errors on the frontend that your users experience.

  18. 1

    Beside mentioned testing, creating beta groups etc.,
    Uou might also consider just outsourcing testing. You probably can find cheap service in Asia.

  19. 1

    As a solo developer, I can't afford to not write tests for my app. Write unit and UI tests. A lot of developers hate them, but they help you sleep at night.

    If you have beta user then choose distribution platform (firebase, testflight, play store) and give your application to your beta testers. Don't forget to implement a crash tracking tool like firebase or Instabug. Also, you can add some fancy shake for feedback libraries, or just communicate through emails with users.

    If you don't have beta users or any following skip straight to a soft release. Soft release separate topic. I strongly recommend doing a soft release.

    1. 1

      I strongly suggest you try to find some time to write some tests. The time invested now will save you so much time and pain later.

  20. 1

    We've also launched Pixelied a few days ago and have been posting about it in different communities and personal outreach for beta testing of our platform.

    The best way is to inform your users in advance and let them know the app is in beta mode. I personally think this is the best way to go where you fix and add new features after user feedback.

  21. 1

    Build up a number of scripts as you go that you can run on each build. For web apps Cyprus is decent way of doing integration tests. Get employees to hand-test for sanity. Accept limit to what you can/should test. Label release as beta. Cover-off business critical bugs in TOS. Release updates to subset of users first.

  22. 1

    Just ship it, users will test 😈

    1. 1

      🤣 that's what we did exactly on launch. but user reports usually not helpful. they just tell smth is not working. but I need to know how to reproduce to see what's the exact issue. Also lot's of them just left 1 ⭐ reviews on Google Play & AppStore.

  23. 1

    You can use something like TestFlight or Firebase Crashlytics and App Distribution to send out betas to a select list of people who you have recruited for feedback.

    Ask them to test out new features and set up some way of tracking bugs that they can use to post reports back to you (e.g. GitHub)

    You can also think about doing staged rollouts of features; put them behind a feature flag and then turn it on for percentages of users. That way you can minimise the number of people who might end up having a problem while you launch it (e.g. server suddenly spikes after feature launch? don't roll out to 100%)

  24. 1

    Yes, you should test your code. Users won't be too happy to use a buggy application. That being said, small bugs, as long as you address them quickly should be ok. I suggest you do a combination of following to start:

    1. Do code reviews.
    2. Ask second person who didn't write code to run some manual tests in your staging environment.
    3. If you can, have some "beta" testers from your users and ship the code to them first. Again, this shouldn't substitute your own testing.
    4. Write test plans if the quality is important.
    5. Unit-tests are nice, but they will help you from breaking things in future, not much with testing new features. Though, it does help developers to think more cases through.
    6. Generative unit-tests if your language/framework allows them.

    Hope this helps.