We have a state-machine in the code at work. Nothing fancy as such – in graph theory terms: ~7 nodes, ~10 edges, 2 user roles – directional. State changes are always technical valid – an exception is thrown if it the state change is not valid. A clever choice of programming language allows us to express in in the code in a state transition diagram (yeah). Not this one 🙂
But more like 10.000-ish total combinations, and the business legal ones a fraction of that. With automated tests we CAN implement total coverage, just by adding a code line to the state diagram. Cheaply – Neatly – yet a practically infinite number…
But we won’t
We will let it depend on the context of the stories/features being developed, what business tests to include. We want business value over theoretic coverage. We want the people in charge to take the decisions. But with that comes responsibility – to cheaply add any testcase that that’s just remotely interesting (in context). If that is all combinations, so be it. 🙂
[ Michael Bolton: Handling an Overstructured Mission ]
Scripted test cases often reveal problems as those cases are developed. When those problems get fixed, the script loses much of its power. Thus variation on the script, rather than following the script rigourously, tends to reveal the actual problem. However, unless we’re clear that this is happening, managers will mistakenly give credit to the wrong thing—that is, the script—rather than to the mindset and the skill set of the tester.
[ U-test: The Complete Test Coverage Myth ]
What we do need to clearly communicate is what risks remain based on the (incomplete) test coverage we have achieved, so that business decisions can be made based on risk rather than on false assumptions that no risks remain due to “complete testing.”
In the past I have applied code coverage more often successful from inside the team as a measurement to measure pieces of code that we did not (yet) cover. Think about it as an information radiator. Code coverage makes then clear which pieces of your code are not covered, yet, and you can think through whether you need more tests. Code coverage then becomes a first-order measurement, which the team uses to bring forward their development efforts.
[ Anders Dinsen: An illustration of the resource vs coverage problem]
As a decision maker, I’d much rather have in depth knowledge about parts of the system, than to know very little about everyting. It will give me much better foundation for making good business decisions.
[ Anders Dinsen: Covering test coverage ]
Firstly, because the area outside the system is infinite and we can’t calculate the coverage of an infinite area. Secondly, because the checks don’t have an area – they are merely points – so any coverage calculation will beinfinitesimal.