The guilt of not testing everything

My CTO: “We’re really impressed with your testing solution, you’ve gone above and beyond!”

(Immediate feeling of guilt washes over me)

This happened a couple months ago after my CTO heard about a testing solution I put together for our web applications. My solution was to auto-generate the QA Selenium code instead of manually coding it by recording/configuring the tests. (Shameless plug: it’s available for free over at snaptest.io) . This tool has dramatically increased coverage - but why did I still feel guilty about accepting this compliment?

It’s because I unreasonably assume testing should be exhaustive.

Exhaustive testing: refers to running a test on every possible combination of actions/data.

The guilt comes when I say things like: “I’ve tested the new feature”, or “the tests are passing”. In the back of my mind, I know there are hundreds of thousands of combinations I didn’t test for.

We must give ourselves a break and remember that exhaustive testing is usually not possible. One must reconcile the cost/reward balance sheet of testing… Especially when you start thinking about negative tests.

Negative tests/Destructive tests/error path testing — refers to checking how gracefully a system handles errors or unexpected situations. This is when exhaustive testing gets exhausting… Don’t expect to get them all.

Here are my recommendations for handling negative test cases:

1. Pick off the easy negative tests like form validation.
2. Defer negative tests that will take a lot of work to create and maintain. Such as slow network connections on mobile applications. It may be more economical just to manually test those occasionally.
3. Spend time making negative tests on the most commonly experienced errors. Ask yourselves, which errors are users most likely to see? Cover those ones.
4. Don’t worry if you don’t cover every state of the app. Just remember, any test is helpful!