A Ratio between Tests

During one of my recent projects I was considering the ratio between the checks and the tests – that is the ratio between those tests, that are primarily simple binary confirmations, and those tests that are more tacit questions. This blog is about my considerations on the idea/experiment/model.

First I observed, that we have a range of different items in our requirements – some of them are [actual copy of the current specification]:

Binary Confirmations

  • It must be possible to add a customer ticket reference
  • It must be possible to copy the ticket number

Tacit Questions

  • You must be able to navigate displayed lists easily
  • It must be easy to compare work log and audit log

You could argue that they need refinement, more testability and less “easy“. But this is what we have to work with for now. Even if we had all the time in the world (we don’t) – we would not be able to write all of the requirements in a perfect form (if such a form exists).

As the system under test is a commercial standard system, some of the requirements are even given as “Out of the Box”, we will probably not even be testing those explicitly. Our coverage criteria is not ALL OF THEM.

Ordering the tests

It is a deliberate experiment from my side to divide the requirements (and hence the tests) into the piles of Closed and Open Questions. Perhaps there is even three piles – Rapid Software Testing has: human checking, machine checking and human/machine checking , Wardley has Pioneers, Settlers and Town Planners. Perhaps the Rule of Three applies here too.. perhaps it’s an continuum … let’s see.

Perhaps it’s a continuum

As part of the requirement workshops I will label the requirements and align with the stakeholders to get the expectations right – with the help of a few friends. This a context/project based “operationalization“.

I wrote about this ratio on my blog post around the Test Automation Pyramid, as I will use the labels to automate the confirmations (and only the confirmations). The assumption is, that there are significantly more of the binary requirements tested by machine checking – and more human tested tacit questions. If we can get all the tedious tasks automated – that is the really the end goal.

Automate all the things that should be automated

Alan Page

Every project/context will have it’s own ratio, depending on a range of factors. Saying there should always be more of one type than the other would not hold. As the above project is the configuration and implementation of a standard commercial business software package (like SAP, SalesForce etc), my expectation is that most of the requirements are binary. Also considering that this project is heavy on the right hand side of the Wardley Map scale of evolution.

It’s a Reduction in Context

I am well aware that the two/three piles are an approximation / reduction. Especially when looking at the “binary” requirements and “only” testing these by confirmation. They could as easily be explored to find further unknown unknowns. If we prioritize to do so – it all about our choice of risk.

It is also an limitation as “perfect testing” should consist of both testing and checking. I factor this into the test strategy, by insisting that all of the requirements are tested both explicitly and implicitly. First of all most of our binary requirements are on the configuration and customization of the out-of-the-box software solution. So when the subject matter experts are performing the testing of the business flows, they are also evaluating the configuration and customization. And I do want them to spot when something is odd

The binary configuration is ok, but human know-how tells us otherwise.

Ultimately I want to use the experts to do the thinking and the machines to do the both the confirmations and the tedious tasks.

The superpower that things will sort themselves out

Amongst my secret weapons are intuition, square lashings, preparing for the unexpected

… and that things will sort themselves out.

For instance:

  • I had planned to step into the parents group (aka PTA, aka forældrerådet) in one of my boys classes. But the day of the election meeting, I was pretty stressed and missed the meeting. Now a couple of months later, there’s a free spot, and I could step in and be very welcome.
  • At work I saw my boss had assigned me a new project for next month. I missed to talk to him about it, but when came around to it – the project allocation had been cancelled.

So recently I have come to value: letting things sort themselves out OVER looking into everything. THAT IS while there is value on preparing everything, I value the first opportunity more.  You might think of it of being sloppy, unprepared and not even tester like.. your loss… What is your secret weapon then?

dad blackbelt

The Expected

Many test processes and test tools mention that you have to establish the expected results to every test. It is at best risk ignorance not to take the notion of expected results with a kilo of salt #YMMV.

If you can establish the result of your solution in a deterministic, algorithmic and consistent way in a non-trivial problem, then you can solve the halting problem. But I doubt your requirements are trivial.. or even always right. Even the best business analyst or subject matter expert may be wrong. Your best oracle may fail too.  Or you may be focussing on only getting what you measure.

Nailed it
Nailed it

When working with validation in seemingly very controlled environments changes and rework happens a lot, as every new finding needs the document trail back to square one.. Stuff happens. Validation is not testing, but looking to show that the program did work as requested at some time. It is a race towards the lowest common denominator. IT suppliers can do better that just to look for “as designed” [1].

Still the Cynefin framework illustrates that there are repeatable and known projects, and in those contexts you should probably focus on looking to check as many as the simple binary questions in a tool supported way, and work on the open questions for your testing.

Speaking of open ends – every time I see an explicit expected result I tend to add the following disclaimer inspired by Michael Bolton (song to the tune of nothing else matters [2]):

And nothing odd happens … that I know of … on my machine, in this situation [3]

And odd is something that may harm my user, business or program result

Significantly…

But I’d rather skip this test step  and work on the description of the test and guidelines to mention:

And now to something completely different:

See also: The unknown unknown unexpressed expectationsEating wicked problems for breakfast

1: Anyone can beat us, unless we are the besttodays innovation becomes tomorrows commodity

2: https://www.youtube.com/watch?v=tAGnKpE4NCI 

3: I’ve heard that somewhere…

Can you see beyond the visible

As mentioned in “Diversity is important in testing” my view is that the best testers are those that know that testing can be done in many ways. “Testing practices appropriate to the first project will fail in the second. Practices appropriate to the second project would be criminally negligent in the first.” ( http://context-driven-testing.com/) There is no “one way to do testing” (despite what standards tells you). Granted there are – in context –  best practices.

The best testers are those that can see beyond the current project and company framework. Those that realize that there is a fundamental difference between life-science validation and modern enterprise IT projects – and for agile projects even more. If the company frameworks fails to keep current and allow clear tailoring, then “life finds a way“.

There will be contexts where UX is not very interesting, where there is no software as such, where they release directly to production (so what we have TitW). There will even be contexts where structured software testing has very little business value. As well as, there will be contexts, where it’s one-shot only and testing and dress-rehearsals are done, over and over again. (consider though that for space launches superstition and good-luck charms play a very large role).

But don’t confuse your one context and what you have seen in some domains to be directly applicable in others. See beyond the visible, extrapolate your testing knowledge and approaches for different contexts, and you are the better tester.

Iceberg Factory. Torsuut Tunoq Sound and Kulusuk Island. Southeastern Greenland. http://www.mikereyfman.com/

Asking Open Questions

It has always been a good interview technique to ask open questions. Then the person being interviewed have to elaborate and talk in full sentences. In contrast to closed questions, that replied to in binary [1]: yes, no, 42 – the red pill [2]. Until now I really didn’t understand how simple yet powerful this questioning technique is in testing. I might have done it all along, for some time :-).

The primary eye opener was the Copenhagen Context 2015 [4] workshop on Exploration Under Pressure by Jon Bach. One of the treats was that he showed us a list of things to find on the ebay.com website. Not specific items, but information about the items. Finding the most expensive item, and by that stumbling over a live production bug in the max value field. Finding the number of blue shoes available etc. What a fun “online scavenger hunt” – we could battle to find the oldest, longest and most odd details etc.

Later the same week eBay Classified hosted a local meetup of “QA Aarhus” with a live demo of how they do testing sessions of their app. They had to host the session twice,  due to popular demand, and what we got was an intro to a setting of exploration, thinking loud and doing pair testing. And I got to try my new-found quest to ask open questions. To search for things – but look out of the corner of the eye for oddities and what-ifs.

But how could I apply this technique in my current testing project of migrating an HR solution for a large IT outsourcing company. I did today. A staff member allocated to the project to test during UAT [3] specifically the processes they use in the old system and to act distribute this knowledge back to the team. For reasons the testing scope in this are had yet not been established, so she didn’t really know where to start – but I did… open questions 

  • What processes do you have?
  • What kind of events do you need to register on an employee
  • Tell me more about vacation calculation
  • Where, if any, are your current processes described (I’m fallible)
  • What has likely changed comparing the old and new solution

I asked her to go as deep until no new learning could be achieved, but not to detail it in scripts or discrete steps. Because from here we have test cases – test ideas – “a question that someone would like to ask (and presumably answer) about a program

Eureka!

 

[1]: Binary replies can be checked, open questions are testing. Testing is “Testing is the process of evaluating a product by learning about it through exploration and experimentation, which includes: questioning, study, modeling, observation and inference, output checking, etc.” http://www.satisfice.com/blog/archives/1509

[2]: I have seen how deep the rabbit hole goes…

[3] Let’s pretend there is such a thing as a “user acceptance test

[4]  Disclaimer: I was part of the program committee, and by chance most speakers hosts their own testing conferences. See more on http://copenhagencontext.com/blog/2015/01/meet-jesper-at-the-copenhagen-context-conference-venue/

QA Aarhus – Exploratory Testing How and When

QA Network Aarhus is a local non-affiliated network of testers (and good friends) in Aarhus. Where I had the great pleasure of talking about Exploratory Testing. This is the link collection, the slides are attached.

nnit

Continue reading