Fell in the trap of total coverage

We have a state-machine in the code at work. Nothing fancy as such – in graph theory terms: ~7 nodes, ~10 edges, 2 user roles – directional. State changes are always technical valid – an exception is thrown if it the state change is not valid. A clever choice of programming language allows us to express in in the code in a state transition diagram (yeah).  Not this one 🙂

But more like 10.000-ish total combinations, and the business legal ones a fraction of that. With automated tests we CAN implement total coverage, just by adding a code line to the state diagram. Cheaply – Neatly – yet a practically infinite number…

But we won’t 

We will let it depend on the context of the stories/features being developed, what business tests to include. We want business value over theoretic coverage.  We want the people in charge to take the decisions. But with that comes responsibility – to cheaply add any testcase that that’s just remotely interesting (in context). If that is all combinations, so be it. 🙂

[ U-test: The Complete Test Coverage Myth ]

What we do need to clearly communicate is what risks remain based on the (incomplete) test coverage we have achieved, so that business decisions can be made based on risk rather than on false assumptions that no risks remain due to “complete testing.”

[ Markus Gärtner – Code Coverage – useful or misuseful?]

In the past I have applied code coverage more often successful from inside the team as a measurement to measure pieces of code that we did not (yet) cover. Think about it as an information radiator. Code coverage makes then clear which pieces of your code are not covered, yet, and you can think through whether you need more tests. Code coverage then becomes a first-order measurement, which the team uses to bring forward their development efforts.

[ Anders Dinsen: An illustration of the resource vs coverage problem]

As a decision maker, I’d much rather have in depth knowledge about parts of the system, than to know very little about everyting. It will give me much better foundation for making good business decisions.

[ Anders Dinsen: Covering test coverage ]

Firstly, because the area outside the system is infinite and we can’t calculate the coverage of an infinite area. Secondly, because the checks don’t have an area – they are merely points – so any coverage calculation will beinfinitesimal.

See also: Eating wicked problems for breakfast, So Everything Must Be Tested?