What automatic testing mechanisms do you have in place?
I just realized I had some major bug on my new project observely.net in the SignUp Form. In some browsers, the form could not be sent because of a CSS problem. So new users weren't able to sign up for the automatic competitor analysis service.
That's why I wondered what systems and mechanisms do you have in place to avoid such mistakes? Do you use automatic testing scripts? Unit Tests? Integration Tests? I'd love to know.
Also if you don't use automated testing, why?
Until my last business I was always reluctant to do any sort of testing. But after a few similar mistakes as you did, I am partially testing my application. I do some basic model/controller/routes tests to make sure the most important things work.
I then feature test (headless browser) the most critical tasks/paths. These have to absolutely be bug-free:
I also try to keep the tech for these tasks/path as simple as possible (ie. as little JS as possible). Based on my browser stats, I smoke test after each big deploy (deploys are only accepted if all tests pass) too. It's time-consuming, but loosing possible customers is worth more than that.
I still don't follow any “best-practices” for testing whatsoever and try to be as pragmatic about it as possible. I have only so much time.
If a bug arises, I'll get notified and I'll check in with the user. Based on the severance I'll pro-actively give a discount. This is still doable for me, since the business is in its early stages.
I have never bothered with unit tests after running into case after case where unit tests all passed, however the functional and integration tests failed. Made me realize that unit tests add little value yet have a high maintenance cost.
These days I focus only on functional and integration tests. Started using GhostIspector I couldn't be more happy. I used to code and maintain my own testing framework using CasperJS, but using a service like GhostInspector is saving the team tons of work.
Thats excactly what I thought! Becuase it was a CSS problem, Unit testing would have probably not even helped me.
Ghostinspector looks really interesting, thanks for the tip!
Phing/PHPUnit for unit tests and coverage reports. Run on CircleCI before deployment.
But... this is only for one particularly big project I have. I don't bother with automated testing for smaller projects. For that I just do manual tests. If we tested every small line of code you'd spend just as much time writing tests as you would coding the app itself :)
We are super early but we have model/view/controller specs (Rails stack using Rspec) that can account for Unit tests and feature tests. Thus, view specs would probably catch such an bug for us (and in fact we had such a case), but you can never be sure. I think there is a fine line between enough and too much tests that have a high maintenance cost. That said, I'd never pass on Unit tests. In my world, this is where testing begins.