We have a state-machine in the code at work. Nothing fancy as such – in graph theory terms: ~7 nodes, ~10 edges, 2 user roles – directional. State changes are always technical valid – an exception is thrown if it the state change is not valid. A clever choice of programming language allows us to express in in the code in a state transition diagram (yeah). Not this one 🙂
But more like 10.000-ish total combinations, and the business legal ones a fraction of that. With automated tests we CAN implement total coverage, just by adding a code line to the state diagram. Cheaply – Neatly – yet a practically infinite number…
But we won’t
We will let it depend on the context of the stories/features being developed, what business tests to include. We want business value over theoretic coverage. We want the people in charge to take the decisions. But with that comes responsibility – to cheaply add any testcase that that’s just remotely interesting (in context). If that is all combinations, so be it. 🙂
[ U-test: The Complete Test Coverage Myth ]
What we do need to clearly communicate is what risks remain based on the (incomplete) test coverage we have achieved, so that business decisions can be made based on risk rather than on false assumptions that no risks remain due to “complete testing.”
[ Markus Gärtner – Code Coverage – useful or misuseful?]
In the past I have applied code coverage more often successful from inside the team as a measurement to measure pieces of code that we did not (yet) cover. Think about it as an information radiator. Code coverage makes then clear which pieces of your code are not covered, yet, and you can think through whether you need more tests. Code coverage then becomes a first-order measurement, which the team uses to bring forward their development efforts.
[ Anders Dinsen: An illustration of the resource vs coverage problem]
As a decision maker, I’d much rather have in depth knowledge about parts of the system, than to know very little about everyting. It will give me much better foundation for making good business decisions.
[ Anders Dinsen: Covering test coverage ]
Firstly, because the area outside the system is infinite and we can’t calculate the coverage of an infinite area. Secondly, because the checks don’t have an area – they are merely points – so any coverage calculation will beinfinitesimal.
See also: Eating wicked problems for breakfast, So Everything Must Be Tested?
10 thoughts on “Fell in the trap of total coverage”
[…] Fell in the trap of total coverage […]
[…] it work – then ship it“. Enterprise settings and medical systems need a higher degree of conformity to contracts, standards, certifications and FDA approvals. Where does your context score on the […]
[…] goes for the testing efforts – agile scales to a certain point, and at that point the scrums, the state-models and so on are a part of the solution engine. It’s what’s tests something, but with […]
[…] management added metrics – like the number of testcases to pass pr. day (linear progress) or a real couch to have the project manager to sit on, until we found yet another […]
[…] the best CMMi or ISO29119 accredited company will stumble into new information, or test only limited subsets of total […]
[…] A project I was in regarding a website and API for trading property information had no testers, but had continuous build and deploy with even more user oriented test cases that I could ever grab. (see: Fell in the trap of total coverage) […]
[…] Read also: Many Bits under the Bridge, Less Software, more Testing, Test Criteria for Outsourced Software, The Expected, Fell in the trap of total coverage. […]
[…] For instance: In one project I did, we disabled the login entirely to make the CI/CD run feature testing. The plain login screen was temporary anyways, as the solution authentication would be based on certificates. We never spend much time on it, neither on total coverage. […]
[…] given as “Out of the Box”, we will probably not even be testing those explicitly. Our coverage criteria is not ALL OF […]
[…] Fell in the trap of total coverage https://jlottosen.wordpress.com/2012/11/05/fell-in-the-trap-of-total-coverage/ […]