Test ALL the things

TL;DR: We can add testing to all requirements and all business risks. Testing to document requirements and to debunk risks provides valuable information for the business. Let us not limit testing to things that can be coded. The intellectual activity of trial and learning is happening anyways, we might as well pitch in with ways to find important evidence for the decision makers.

Test all the requirements

Traditionally testing was all about testing the functional requirements that could be coded. Non-functional requirements was left for the specialists, or plainly disregarded. I know I have done my share of test planning, with a range of requirements left “N/A” with regards to testing. Especially performance scope, batch jobs, hardware specs, data base table expansions and virus scanning have been left out of my functional test plans…

When I look at a list of requirements now – I see that we can indeed test all the things, or we can at least work on how to document that the requirement is fulfilled. Some of the requirements are actually quite easy to document. If it’s on a screen somewhere, take a screen shot and attach it to a simple test case. Done deal really. Additionally with a testing mind-set I can think of ways to challenge the details. But do we really, really need to fill up a disk to establish if it’s exactly a 1 Gb allocation – probably not. Do we really really need to document all requirements – yes in some contracts/contexts it’s important for the customer to know that everything has indeed been established. Sometimes the customer doesn’t trust you otherwise, sometimes the tests are more about your ability to deliver and provide evidence that matters.

Test all the business risks

Look into the business case of your project and find the business risks. Sometimes they are explicitly stated and prioritized. A a recent Ministry of Testing Meetup we looked into a case for a large national health system. We looked at the tangible benefits, intangible benefits and on the scored business risks.  What worried the business and management most was budget, time and whether the new system would be used in a standardized way. There is an opportunity for testing here to help address, document and challenge the most important business  risks. Traditional testing would usually look at functional requirements that can be coded or configured, and miss totally to address what matters most to the business.

OK, how do we test the project costs? How about frequent checkpoints of expected spending – would that be similar to tracking test progress. Perhaps – let’s find out. Testing is all about asking questions for the stakeholders and solving the most important problems first. Can we help to analyse risks and setup mitigation activities – sure we can. We just have to step out of our traditional “software only bubble”.

MEME - Test ALL the things
Meme ALL the things








Read also: Many Bits under the Bridge, Less Software, more TestingTest Criteria for Outsourced SoftwareThe Expected, Fell in the trap of total coverage.

Links: “A Context-Driven Approach to Delivering Business Value”, Cynefin In Software TestingTesting during Application Transition Trials


Publications and Presentations

Presentations, Webinars & Podcasts

Articles for The Ministry of Testing, 2011+

  1. Robot Process Automation As A Power Tool For Testing [Ministry of Testing Dojo, Mar 2018]: While there are other power tools for web and API testing, the RPA tools are a class of their own, as RPA tools allow for codeless automation macros on the desktop. RPA tools can do some very handy things. They can be used for both test data and regression testing. In this article, we’ll walk through a real testing example and show how you can get started using RPA. [TOP 6 on the Ministry of testing 2018 article list]
  2. Testing is Shifting [Testing Planet 2017 by the Ministry of Testing, Mar 2017]: Change is the only constant, they say, but we still need to manage change – and cope with it. Coping not only means surviving mentally, but also adjusting to whatever happens and figuring out how to be productive and create value for our stakeholders when things change. [https://dojo.ministryoftesting.com/lessons/testing-is-shifting]
  3. About Closure [The Testing Planet by Ministry of Testing, Nov 2014] When I’m in a testing activity I want my test cases [Passed], my user stories [done] and my coffee [black].  Stuff may have a start point, some states in between and an end state. Let’s look at ways to represent states and articulate the meaning of states. [Reposted: Closing the Gaps]
  4. The Daily Defect Count and the Image of a Camel [The Testing Planet by The Ministry of Testing, April 2014] Count the defects daily – the ones that are part of the project work load. The number goes up and down during the cycle – why and what can you learn? [Reposted: A Track down History ]
  5. The Day Testing Died But Didn’t [The Testing Planet by Ministry of Testing, Jan 2014] To play according to textbooks is fine, up to a certain level. Perhaps up to master level, but not to grand masters. [Reposted: Chess and Testing ]
  6. One Test Case is All You Need [The Testing Planet by The Ministry of Testing, November 2013] If you can come up with just one business transaction – that crystallizes why the customer will be kicking and screaming to want to use your application, then you have a very good understanding of your customer and all you need is that one testcase. [On WebArchive.org]
  7. Recognize and Acknowledge Your Skills [The Testing Planet by the Ministry of Testing, June 2013] What you know and what you do is an important part of being you. Often it is required to rethink: What do I know? What are my skills? How strong are they? [https://jlottosen.wordpress.com/2013/06/18/tying-it-all-together/]
  8. The Build-A-Tester Workshop [The Testing Planet by The Ministry of Testing, June 2013] A small social game of Build-A-Tester can be used in a team to open the discussion, less formally than with Belbin and MBTI checklists. [on WebArchive.org]
  9. A Little Track History that goes a Long Way [The Testing Planet by Ministry of Testing, July 2011] The purpose of this tracking tool is to collect just enough data to answer the frequent question “Will we finish on time” [Reposted; A Track down History ]

Other publications

  • Could Modern Testing Work in The Enterprise [Guest blog for Panaya, May 2018] So far I have mostly thought that “Modern Testing” of the A/B testing podcast would never work in an enterprise context. But it seems some tools and existing approaches in the enterprises already fits well with the ideas of the concept. http://www.panaya.com/blog/testing/could-modern-testing-work-in-the-enterprise/
  • DevOps is cool, but get involved in OpsDev for Test Environment Management too! [Guest blog post for Plutora, Oct 2017] The hyped mnemonic “DevOps” is equally true the other way around: OpsDev http://www.plutora.com/blog/opsdev-test-environments-management
  • Testing during Transition: Test Criteria for Outsourced Software [Sticky Minds by TechWell, May 2017] In the world of IT outsourcing, it is not uncommon for a company to have its applications and infrastructure developed or maintained by others. How would you design acceptance criteria of a transition trial so that it is testable and clearly communicated? https://www.stickyminds.com/article/testing-during-transition-test-criteria-outsourced-software
  • Using Business Decisions to Drive Your Testing Coverage [Sticky Minds by TechWell, November 2014] In a business setting, software testers have a great challenge: to articulate how they support the business lines. One way to approach this is by addressing the business decisions—and there are plenty around. Use them to drive your testing activities and increase the business decisions being covered by testing. http://www.stickyminds.com/article/using-business-decisions-drive-your-testing-coverage 
  • The answer is: Why – because the answer depends on context.[The Testing Circus,vol.6 2.ed February 2015]: When asked about testing approaches, the options are so plentiful, that the reply is often “It depends” – and followed by a range of elaborations. But in our eager to reply, we forget to listen. http://www.testingcircus.com/february-2015/
  • The Testcases Template Trick – Getting One Testcase To Call Another [EuroStar TestHuddle, Nov 2014]: When doing test analysis I often find that we need to do test some customer feature over and over again for a range of combinations. I recently found myself able to redo a trick I learned a long time ago https://huddle.eurostarsoftwaretesting.com/the-testcases-template-trick/

Fell in the trap of total coverage

We have a state-machine in the code at work. Nothing fancy as such – in graph theory terms: ~7 nodes, ~10 edges, 2 user roles – directional. State changes are always technical valid – an exception is thrown if it the state change is not valid. A clever choice of programming language allows us to express in in the code in a state transition diagram (yeah).  Not this one 🙂

But more like 10.000-ish total combinations, and the business legal ones a fraction of that. With automated tests we CAN implement total coverage, just by adding a code line to the state diagram. Cheaply – Neatly – yet a practically infinite number…

But we won’t 

We will let it depend on the context of the stories/features being developed, what business tests to include. We want business value over theoretic coverage.  We want the people in charge to take the decisions. But with that comes responsibility – to cheaply add any testcase that that’s just remotely interesting (in context). If that is all combinations, so be it. 🙂

[ Michael Bolton: Handling an Overstructured Mission ]

Scripted test cases often reveal problems as those cases are developed. When those problems get fixed, the script loses much of its power. Thus variation on the script, rather than following the script rigourously, tends to reveal the actual problem. However, unless we’re clear that this is happening, managers will mistakenly give credit to the wrong thing—that is, the script—rather than to the mindset and the skill set of the tester.

[ U-test: The Complete Test Coverage Myth ]

What we do need to clearly communicate is what risks remain based on the (incomplete) test coverage we have achieved, so that business decisions can be made based on risk rather than on false assumptions that no risks remain due to “complete testing.”

Markus Gärtner – Code Coverage – useful or misuseful?]

In the past I have applied code coverage more often successful from inside the team as a measurement to measure pieces of code that we did not (yet) cover. Think about it as an information radiator. Code coverage makes then clear which pieces of your code are not covered, yet, and you can think through whether you need more tests. Code coverage then becomes a first-order measurement, which the team uses to bring forward their development efforts.

[ Anders Dinsen: An illustration of the resource vs coverage problem]

As a decision maker, I’d much rather have in depth knowledge about parts of the system, than to know very little about everyting. It will give me much better foundation for making good business decisions.

[ Anders Dinsen: Covering test coverage ]

Firstly, because the area outside the system is infinite and we can’t calculate the coverage of an infinite area. Secondly, because the checks don’t have an area – they are merely points – so any coverage calculation will beinfinitesimal.

See also: Eating wicked problems for breakfastSo Everything Must Be Tested? 

So Everything Must Be Tested?

Everything? Really? A 100% Coverage(1) of everything? As in E-v-e-r-y-t-h-i-n-g?  nah…

“Something must be tested” … oh wait (2) …

“Something – that matters to someone who matters – must be tested”.

So who matters – who decides what to test?

“Something – that matters to the stakeholders(3) of the project – must be tested”

Use both right and left brain parts (The Right Brain for the future) remember! 

“Something – that matters to the stakeholders of the project – must be explored and confirmed” (4,5)

So what is this something that matters?

“A solution that solves a problem for the stakeholders – must be explored and confirmed

Must? Always? MustShouldCanWould (MoSCoW) ? Everytime?  OK OK

“A solution that solves a problem for the stakeholders will be explored and confirmed within a given business context“. In short –The scope of testing is a business decision

Cartoon Tester: Coverage
Cartoon Tester: Coverage


1: http://blog.asym.dk/2011/03/29/covering-test-coverage/

2: “A bug is something that matters to someone who matters .. to me” (Michael Bolton, James Bach, Cem Kaner, Brett PettiChord et.al)

3: Stakeholders – in the broadest sense: developers, customers, users,…

4: Michael Bolton http://www.developsense.com/blog/2009/08/testing-vs-checking/

  • Checked as in confirming, using scripts and sequences
  • Tested as in learning and exploring, sapience

5; Rapid Software Testing http://www.satisfice.com/rst.pdf pages 59-67.