Wither the test manager?

[ Paul Gerrard | Will The Test Leaders Stand Up | The Testing Planet issue 10, Page 4, March 2013 , Paywall] paraphrased:

There are five broad choices for you to take if you are a test lead or test manager:

  1. Provide advice to the business leaders, as an independent agent cajole project leadership and review performance
  2. Take control of the knowledge required to define and build systems. Demand clarity and precision in requirements
  3. Help agile projects to recognise and react to risks, coach and mentor and manage testing
  4. Manage the information flow between the key groups and continuous integration system, control change and delivery
  5. Manage outsourced and offshore teams, manage relationships

It’s probably 6) All of the above


3D model for testing contexts

If software testing is solely about finding bugs, I would do nothing else to hack, break and attack the solution. In one context the testers job is to “positively destroy” the solution. Taking this to the extreme – this contradicts with the shared purpose to deliver value to some one“. If the context require that the tester is not allowed to report errors (#bad, but it happens) – that contexts is also not good taking to the extreme. If the acceptance criteria is that X severity bugs can be released, gaming the play will leads us not to look for them.  Where does your team score on the Bug hunters dimension

Bug hunters dimension
Mission: Break the solution
Mission: Confirm solution

In a Rapid Software Testing situation, testing is on the spot, with the given information at hand within a short time frame. I could imagine that a lot of mobile app development projects have very few formal/informal tests besides: “Does it work – then ship it“. Enterprise settings and medical systems need a higher degree of conformity to contracts, standards, certifications and FDA approvals. Where does your context score on the Exercise dimension:

Mission: Skipping Mission: Deep-dive
Mission: Skipping
Mission: Deep-dive

A lot of testing contexts have the job to verify the solution. Given so and so requirements, so and so predefined charters, testcases, scripts … verify that the thing works as predicted. To solve this problem brawn is usually enough, tasks are broken down and systematically applied (machine checking). On the other end of the scale is a more learning context, where sapient tests trigger exploration based on brains. Where does your context score on the Thinking dimension

Mission: Verify Mission: Learn
Mission: Prepared
Mission: Explored

When you start considering these dimensions it’s easier for you to get to the what, the where, the whom and how much. In some contexts the sliders are not easily moved. But even so consider to apply a bit of the dimensions in the various phases of your software testing – scoot over the application in “gold master testing” and snow plow testing. Verify and learn, test and check, break and confirm.

[ Rob Lambert | Thanks for the information, I’ll make up my own mind though ]

As professional skeptics we need to make up our own minds and come to our own conclusions. That should be done using any supporting material we can, but ultimately from our own information, decisions and activities

Please make your own model – what 3 dimensions characterizes the testing in your context? 

See also:

Without Timing – Quality, Schedule and Cost is nothing

A project usually has to be on target on Quality, or at least on agreed quality. 

A project usually has to be on Schedule, or at least on agreed schedule.

A project usually has to be on target regarding Costs, or at least agreed costs.

It is an ancient IT project manager mantra, to control these parameters using change requests and change management – typically due to events during the project. It takes to long, it’s too buggy, it costs more etc. etc. To change one parameter – you have to change one of the others. In example (and strictly in some context) – even adding more testcases will increase cost or influence the schedule (discuss).

But there are plenty of examples (of Government IT projects) that are on schedule, on cost, on quality – yet they fail the timing and after a number of years delivers a system on an obsolete platform [true story]. More recent examples – bank D launches their banking app first, get all the press – and a month later bank N comes around – on agreed schedule, agreed costs, agreed quality… and get very little of the “hype”.

When contemplating the business decisions – what seems right now right now might be wrong later – considering the Timing.


Acceptance criteria are more than what can be measuredAn Expected Gathering

I’ll fix this later 🙂

Black or white – it is the same box


  • Exploration, Finding, Testing, Setting up hypothesis, Manual, Bespoke, Check lists, What if
  • Steps, Confirmation, Checks, Proofs, Automated, Routine, Test cases, Does it matter

Does it indeed matter …

The “box” is a solution – if the box doesn’t fit the problem, it doesn’t work. (period) 

See also: Testing and checkingWhat if – and Does it MatterThe small changes in the big scripts Routinized and bespoke activities  left and right side of the brain

The problem with processes

I have designed corporate telco testing processes, I have taylored CMMI5 certified processes, I have studied at practiced context driven testing. Yet – the more I learn, the more it seems:

You can follow some of the processes all of the time,

and all of the processes some of the time

– but you cannot follow all the processes all of the time.

the processes are more what you’d call “guidelines” 

Testing is your sensory nerves

Remember how the brain reorganized it’s neurons when information stopped coming in? Just because the ‘wiring’ is fixed and the information flow is restarted doesn’t mean that the brain will start listening! Second, the brains’ map of sensory information coming from the hand has been distorted, and the brain will have to relearn the hand-map again.

It’s very much like a management board having to learn how to include information they haven’t had for a while, in their decision making.

Testing is your sensory nerves


The unknown unknown unexpressed expectations

  • There are things I know, I know
  • There are things I know, I don’t know
  • There are things I don’t know, I know
  • There are things I don’t know, I don’t know

I’m mostly worried about the last one.

  • Information I am assumed to have, but I don’t
  • Expectations to me, that are not expressed
  • Considerations that I aught to take, but I can’t see
  • Things I should remember, but I didn’t

and the consequence being  frustration that I am the one not knowing. #gofigure

See also Eating wicked problems for breakfastEven superheros need helpHow to spot defects Innovation is about the unknown – deal with it  All oracles are failable

Fell in the trap of total coverage

We have a state-machine in the code at work. Nothing fancy as such – in graph theory terms: ~7 nodes, ~10 edges, 2 user roles – directional. State changes are always technical valid – an exception is thrown if it the state change is not valid. A clever choice of programming language allows us to express in in the code in a state transition diagram (yeah).  Not this one 🙂

But more like 10.000-ish total combinations, and the business legal ones a fraction of that. With automated tests we CAN implement total coverage, just by adding a code line to the state diagram. Cheaply – Neatly – yet a practically infinite number…

But we won’t 

We will let it depend on the context of the stories/features being developed, what business tests to include. We want business value over theoretic coverage.  We want the people in charge to take the decisions. But with that comes responsibility – to cheaply add any testcase that that’s just remotely interesting (in context). If that is all combinations, so be it. 🙂

[ U-test: The Complete Test Coverage Myth ]

What we do need to clearly communicate is what risks remain based on the (incomplete) test coverage we have achieved, so that business decisions can be made based on risk rather than on false assumptions that no risks remain due to “complete testing.”

Markus Gärtner – Code Coverage – useful or misuseful?]

In the past I have applied code coverage more often successful from inside the team as a measurement to measure pieces of code that we did not (yet) cover. Think about it as an information radiator. Code coverage makes then clear which pieces of your code are not covered, yet, and you can think through whether you need more tests. Code coverage then becomes a first-order measurement, which the team uses to bring forward their development efforts.

[ Anders Dinsen: An illustration of the resource vs coverage problem]

As a decision maker, I’d much rather have in depth knowledge about parts of the system, than to know very little about everyting. It will give me much better foundation for making good business decisions.

[ Anders Dinsen: Covering test coverage ]

Firstly, because the area outside the system is infinite and we can’t calculate the coverage of an infinite area. Secondly, because the checks don’t have an area – they are merely points – so any coverage calculation will beinfinitesimal.

See also: Eating wicked problems for breakfastSo Everything Must Be Tested? 


Eating wicked problems for breakfast

Software testing is a wicked problem* – we eat the test environment problems and puzzling defects first thing in the morning. Like this one I’m Puzzled and Bugged: An odd mouse click problem on Win7 We try to understand what problem the sponsor really want is to provide information on.

There is no definitive formulation of a wicked problem Wicked Problems | Mary Poppendieck ]. Wikipedia have another, the key message is: Wicked problems cannot be solved in a traditional linear fashion, because the problem definition evolves as new possible solutions are considered and/or implemented.    

Testing at best have stopping heuristics, like running out of chocolate and other incentives. The results of testing is never passed/fail, it’s a finite sampling over an infinite space – complete coverage is a myth. And context is the most important factor in all testing activities. We learn as we test and explore the system.

So what better way to spend a morning than attending [ GOTO Aarhus 2012 | Problem-solving and Decision-making ]:

For those of us who struggle with complex problems for a living, unfortunately, don’t have time to keep up with the enormous amount of research in cognitive science that would help us be better thinkers. 

*(Originally coined by Michael Osborne, SQNZ April 2002). 

I’m an Official blogger for GOTO Aarhus 2012 – see also Testers are developers too – and the other way around