Closing the Gaps

[Previously on the Ministry of Testing, Nov 2014 – now only on the Web Archives]: About Closure

When I’m in a testing activity I want my test cases [Passed], my user stories [done] and my coffee [black].  Stuff may have a start point, some states in between and an end state. Lets look at ways to represent states and articulate the meaning of states.
One way to illustrate status about the product being tested is to model the activities we have with states. An agile user story may be [ready], [in progress] or [done]. A document may be [final], [approved] and a mind map may be iconified etc. States are so common that we sometimes forget the theory behind the model, and what benefits we have from the theories.

The representation of closure

For instance we can look to computer science graph theory[1]  to help us understand and control the states diagrams. It is the same graph theory that brings us state machines and state-transition diagrams, but that is another story[2] . In a graph theory state model we want one unique start state (Like [To do]), and one unique end state (like [passed]), everything in between is intermediate.

A (single) end state helps prevent the state machine from going on forever[3] , and us from going on forever too. [Deferred] and [Rejected] are temporary states to me. Setting [Rejected] back to “detected by” will aid that the tester reflects on the reasons. The reasons are then tested (in the brain). Sometimes it’s a “my bad” but quite often also the tester finds that issue is simply not “rejected” with more data and examples.

The understanding of closure

Similarly the agile “Definition of Done”[5]  and “Definition of Ready”[6]  helps the agile team phrase when the task is to change state, sometimes it’s explicit, sometimes it’s implied. The understanding of terms (the semantics) are usually more imperative than the syntax (the rules and representation). Sometimes it’s necessary to “connect the lines”.

There are a two related psychology theories on closure. One is the Gestalt law of closure[7]  – that is that we tend to self-organize items into an orderly structure. As the image above isn’t really about triangles – it’s about our human tendency to connect the dots. The other psychology part is the desire to close stuff to gain controllability[8]:

The need for closure is the motivation to find an answer to an ambiguous situation. This motivation is enhanced by the perceived benefits of obtaining closure, such as the increased ability to predict the world and a stronger basis for action.

Management and stakeholders often want a “firm and unambiguous” answer from our testing investigations. And this is often the business justification for setting states to our work products; that we have states to illustrate work progress in. Sometimes loose representations and a strong shared understanding goes well, sometimes a more strict representation and elaboration is required.

The syntax of states may be easily explained and codified (and checked), while the semantics and perception is less direct – and needs analysis (and testing). All work products (even for mind maps and test charters) have states and we must articulate both the syntax and the semantics to the team and stakeholders.

References

  1. Graph theory http://en.wikipedia.org/wiki/Graph_theory
  2. Fell in the trap of total coverage https://jlottosen.wordpress.com/2012/11/05/fell-in-the-trap-of-total-coverage/
  3. Finite state machines http://en.wikipedia.org/wiki/Finite-state_machine
  4. A Track down History
  5. definition of done https://www.scrum.org/Resources/Scrum-Glossary/Definition-of-Done
  6. definition of ready http://guide.agilealliance.org/guide/definition-of-ready.html
  7. Law of closure http://jeremybolton.com/2009/09/gestalt-design-principles-the-law-of-closure/
  8. Closure (Psychology) http://en.wikipedia.org/wiki/Closure_(psychology)

A Track down History

Currently I am doing a large V-model waterfall project with a three month testing period and 500+ requirements. To track the progress I want to dust off my old “test progress tracking” method that I matured and described in 2011 and 2014.

The approach was documented in two articles for “The Testing Planet” a paper magazine by the Software Testing Club. The “STC” later became the now world famous Ministry of Testing. Unfortunately the original articles are no longer available on the MoT site – they are on the WayBackMachine. So not to track down that path, here’s a recap of:

A Little Track History that goes a Long Way

For large (enterprise, waterfall) projects tracking test progress is important, as it helps us understand if we will finish a given test scope on time. Tracking many of projects have given me one key performance indicator: daily Percent Passed tests as compared to an s-curve. The data in the S-curve is based on the following data points, based on numerous projects:

Time progressExpected Passed Progress
10%0%
20%5%
30%10%
40%30%
50%45%
60%60%
70%75%
80%90%
90%95%
100%100%
Adding a 3rd order polynomial trend-curve gives the s-curve.

If the “Actual” curve is flat over more days or is below the blue trend line, then investigate what is blocking the test progress… defects perhaps:

The Defect Count and Camel Image

During a large project like this the active bug count goes up and down. No one can tell what we find tomorrow or how many we will find. In my experience tracking the daily count of active defects (i.e. not resolved) is key, and will oscillate like the humps on a camel:

Camel background is optional

If the curve doesn’t bend down after a few days there are bottlenecks in the timely resolution of defects found. When the count goes up – testing a new (buggy) area is usually happening. Over time the tops of the humps should be lower and lower and by the end of the project, steep down to 0.

And thus a little track history has come a long way.

The Testing Planet, July 2011

To Transform Testing

There is no doubt that our long lived testing narrative is under pressure. To continue to bring business value, both the IT field and testing is transforming to be about proactive business enabling.

The IT domain is currently buzzing with the word “IT transformation” – the idea that IT services should be more about “inspire, challenge and transform the digital businesses“. That it should be less about delivering IT products and artifacts and more about enabling digital business outcomes. Even for testing – it should be less about a product/service, and more about business necessity. As Anders writes:

Stop focusing on the things that bug people, and dare to do both IT and testing smarter and more business focused from the start. Build Quality in – smarter from start. That goes for IT services as a whole, and definitely also for the testing activities.

What can you do to transform your testing? I have three areas:

Discuss Business Strategy

Learn Wardley mapping – and use it like Chris McDermott to create context specific maturity models with Wardley Maps informed by Cynefin. Use the mapping to Broaden the scope of the system under test.

Align with the Business Strategy

Leading Quality [Cummings-John, Peer 2019] has a whole chapter on “Align your team to your company growth metric“. Consider if the company you work for is driven by Attention, Transaction or Productivity metrics, and arrange your test activities accordingly.

Dare to Deliver in New Ways

We are usually talk so much about optimizing the (IT and testing) delivery, that we forget other ways to be innovative and provide business enabling. One way could be to dare use new technology like RPA or a HoloLens to support tedious tasks in testing – to use an existing product to something new. Another approach to actually test “all the things” that matter or to apply testing to IT outside the realm of application delivery.

To Transform Testing I will discuss, align and dare so that test solutions can be proactive business enablers – (not only achieve shippable quality).

Mapping Mondays – Pioneers, Settlers, Town Planners

How Automation Affects the Business

As of writing I am managing the testing of a large enterprise IT program, where we are implementing a new commercial enterprise solution (COTS).

Over the last many months there have been requirement workshops upon requirement workshop to write down what the new system should be able to do for the various business units. We have had many representatives from the business units as part of the workshops and now have about 1000 specific business requirements that needs to be tested.

Some requirements are closed questions, others are more open-ended or similarly require some thinking. Currently the ratio is that 70% is done by test automation and 30% is for a few of the subject matter experts to test. Management was happy with this, as this made the project faster, the solution more robust and the project less reliant on taking the business people away from their “real work”.

So far so good

The other day I reached out by mail to more of the business people involved in the workshops to let them know that testing had started, and that they would be able to access the solution under test when it had been “hardened”. But so far, only a few “track leads” would be involved.

The feedback surprised me, as my message was both good and bad. Good in the sense that they would not be involved so much, but also bad that they would not be involved so much. One wrote back to me:

  • There is still a risk that the solution will not be as the workshops intended, as the requirements and solution might not capture precisely, what was agreed during the workshops
  • Having been part of the workshop, we are held responsible by our coworkers as to how well the new system supports the business
  • Why don’t the project want our involvement on this?

... but that was “just feelings”, he wrote in the end. And indeed it is – No matter how it looks at first, it’s always a people problem and even if we have a successful test automation effort – we can still fail to appreciate the experts knowledge and by that fail to solve the business problems.

More about “Leading when the experts test” at ConTest NYC 2019.

I was out hiking in April. But city management had locked the toilet up - out in the woods. As an END user my problem was then solved by doing it in the woods. And all fancy sheds where for naught.
I was out hiking in April. But city management had locked the toilet up – out in the woods. As an END user my problem was then solved by doing it in the woods. And all fancy sheds where for naught.

In Charge of Testing

As a Test Manager I oversee the testing in a project or program – I am usually the only testing specialist in the project, so, I need the right leadership skills and the right tools to succeed. I have to own the data about the testing and quality activities.

As the test manager I need to facilitate a quite a range of testing activities:

I need to balance that I need to know what’s going on (with regards to testing) but without micromanaging the people being involved in testing and quality activities. My role is to facilitate that testing things happen – like the project manager making project things happen. I cannot own the activities without owning the data about it. I need to cover the full spectrum of tests – from engineered (RDA and CI/CD) to people-based (scripts and exploration).

The most practical tool for a test manager with this scope is PractiTest, as there is more to testing than just the test cases [2]. The old term “ALM” [3] comes to mind – it is still relevant when I look for a full test management tool. I need to cover both the “inputs” to testing (requirements, tickets and user stories) and the “outputs” (bugs) in one location. I need the requirements and user stories in my tool, as I need to base my test analysis and planning on the delivery model (that may not always be agile). I need the bugs in the testing tool too, as bugs can happen in any work product of the project: documents, code base and even the tests. PractiTest acknowledge that there is more to IT projects than code.

I appreciate the key driver of PractiTest – that all activities happen in-flow. You don’t have to change window, stack pop-ups or go to another tool in order to run the tests or create bugs. Creating bugs happens in context of the test case and seamlessly moves all data about the run to the bug. Everything you need to do is context-based, and available to you on screen. And it has some cool features of read-only links to graphs for management reporting, and a smart built-in “rapid reporter” for exploratory testing notes.

It can be a challenge to switch to PractiTest if you are in a compliance setting, if you need on-Premise or if your team generally uses Azure DevOps (the tool formerly known as TFS). To get the full potential of Azure DevOps, though, you need the full Microsoft Test Pro licenses, so it’s not a free tool either – nor is DevOps intuitive for testing things doesn’t have the code available. As with Azure DevOps PractiTest is also SaaS only, with multiple data centers for regional data compliance. As there is always inertia towards a commodity it won’t be long before there is no good arguments to have test management tools on-Premise and for the tool vendors to provide the compliance certificates (ISO/SOC really should be sufficient, IMO).

Out of the box PractiTest supports the categories of testing above (engineered, scripted, exploratory) and has the necessary integrations too: Surefire for unit testing, Maven for CI/CD, Jira, ServiceNow or any other ITSM for requirement input. There is even a two way integration to Azue DevOps. As the web design is “responsive” it could probably run off a tablet. That would enable easier test documentation for field tests. It would be even better to have a small version of it on a phone and be able to use the camera for “screen shots”.

At work I am currently running a large project regarding customizing and implementing a standard commercial software system, PractiTest would fit right in, as we have the following test activities:

  • Unit test by the developers
  • Automation by test engineers
  • Exploratory test by Subject matter experts
  • Formal scripted testing with end users

And I need to own the data around all of this, if I want to in in charge of the testing (and not only the testers). We are very few software testing specialists on the project team, but as the manager of testing I need to cover many other people performing the testing. This transforms my role from test management to one about leadership, coaching, and facilitation of testing being performed by the SMEs – and anyone else really.

I will be talking about Leading When the Subject Matter Experts Test at ConTEST NYC 2019 until then read more about leadership:

  1. Anthropologists and similar humanities educations can be great BA’s
  2. looking at you Test Rail 😉
  3. ALM = Application Life Cycle, like Micro Focus Quality Center etc.

Disclaimer: This is an influencer review sponsored by PractiTest.

Visual Tests are Still Code

Among the currently shiny new test automation things are visual “script-less” test automation tools. But the visual test flows are still code – and thus require discipline to structure and maintain. Otherwise you are just adding yet another layer of spaghetti code.

Among the current shiny new test automation tools are visual “script-less” automation tools like LeapWork [9], Blue Prism [10] and UiPath [7]. These tools are a part of a new class of business process automation tools called “Robot Process Automation” (RPA) [4]. There are two sub types of – “RPA” which focuses on processing data and Robot Desktop Automation (RDA).

RDA is interesting in the context of test automation [9], as they can automate GUI interactions – also on top of enterprise package applications (SaaS, COTS, OOTB etc. [2]). The test automation challenge for most of these enterprise applications (SAP, MS Dynamics [6] etc.) is that they come with no access to the code-base, even if these are pure-play web based – the GUI is all there is.

All you can to these type of business solutions is usually to add customization and configurations by entering or editing data directly in through the GUI. Some of these systems allow configurations and customization in the form of config-file – they really should be under change control [3], as they are part of the pipeline. 

visual tests are code

part of the ship

part of the crew

Bootstrap Bill Turner

Using RDA tools for test automation [9] is a novel [1] uncharted approach [12]. The editing of the “tests”/flows is usually done in a stand-alone application studio (Graphical IDE) with interactions to the solution under test (across the GUI and over Citrix and RDP) and to any test management and issue tracking system.

Interestingly the other more “data processing” RPA tools like Automation Anywhere [5] uses a VB-Script like syntax. Writing and maintaining “scripts” like that is quite like the common approaches to GUI automation using frameworks like Siluki, tagUI, Applitools [11].

Applitools etc. are coding frameworks you can apply if you have the application code base or want to write test automation directly as code. There could be benefits in coding UI testing in all web-only projects directly using Selenium and Applitools. Most enterprise business solutions are often stand-alone applications, or their web code is horrible to hook into, as often the selectors seems randomly generated (been-there-done-that).

Hence the primary driver for RDA adoption in for test automation is to take the RDA & RPA [4] tools and apply their strengths in process automation of enterprise business solutions [2] to drive the test execution. And of a business flow could be “automating” activities during onboarding [7] or an SAP purchase order as below images:

Another key driver for adoption of RPA for test automation is their visual approach in presenting interactions/tests as flows. Some do it gracefully and user-friendly (LeapWork) – others have a more old-school workflow/swim lane approach (Blue Prism, UiPath). In both cases the visual flows illustrate an interaction across multiple GUI applications to perform business actions (yes, this still happens).

These drivers probably to make the barrier to entry seem more manageable. The visual ones very easily turn into visual spaghetti code if you don’t keep an eye on it and use sub flows, low coupling and high cohesion [13].  … as with any other non-trivial code (of a certain McCabe complexity [14]). One interesting way to go about a “coding” practice for visual test cases could be inspired by how BDD can be implemented in LeapWork [8] with annotation and self-referencing unit tests.

At the end of the day even a visual test automation project is a coding project, that should be part of the project code base like everything else [3]. And probably best maintained by software engineers within the project team (where possible) – unless you want a team of test engineers spending all day playing catch-up to maintain the automation code.

  1. Since 2017’ish.
  2. COTS/OOTB = Commercial of the shelf, out of the box
  3. https://twitter.com/mipsytipsy/status/1146968926493929472
  4. https://www.horsesforsources.com/2019_RTS_survey_070619
  5. https://www.linkedin.com/pulse/automation-anywhere-example-neil-kolban/
  6. https://www.leapwork.com/blog/automate-testing-microsoft-dynamics-365-crm
  7. https://www.uipath.com/blog/how-rpa-can-help-companies-rethink-hr-tasks
  8. https://www.capgemini.dk/bdd-in-leapwork/#tab5
  9. https://dojo.ministryoftesting.com/dojo/lessons/rpa-as-a-power-tool-for-testing  
  10. https://crunchytechbytz.wordpress.com/2018/03/13/automation-with-blue-prism/
  11. https://applitools.com/features
  12. https://jlottosen.wordpress.com/2019/04/20/broaden-the-scope-of-sut/
  13. https://medium.com/clarityhub/low-coupling-high-cohesion-3610e35ac4a6
  14. https://en.wikipedia.org/wiki/Cyclomatic_complexity

A Ratio between Tests

During one of my recent projects I was considering the ratio between the checks and the tests – that is the ratio between those tests, that are primarily simple binary confirmations, and those tests that are more tacit questions. This blog is about my considerations on the idea/experiment/model.

First I observed, that we have a range of different items in our requirements – some of them are [actual copy of the current specification]:

Binary Confirmations

  • It must be possible to add a customer ticket reference
  • It must be possible to copy the ticket number

Tacit Questions

  • You must be able to navigate displayed lists easily
  • It must be easy to compare work log and audit log

You could argue that they need refinement, more testability and less “easy“. But this is what we have to work with for now. Even if we had all the time in the world (we don’t) – we would not be able to write all of the requirements in a perfect form (if such a form exists).

As the system under test is a commercial standard system, some of the requirements are even given as “Out of the Box”, we will probably not even be testing those explicitly. Our coverage criteria is not ALL OF THEM.

Ordering the tests

It is a deliberate experiment from my side to divide the requirements (and hence the tests) into the piles of Closed and Open Questions. Perhaps there is even three piles – Rapid Software Testing has: human checking, machine checking and human/machine checking , Wardley has Pioneers, Settlers and Town Planners. Perhaps the Rule of Three applies here too.. perhaps it’s an continuum … let’s see.

Perhaps it’s a continuum

As part of the requirement workshops I will label the requirements and align with the stakeholders to get the expectations right – with the help of a few friends. This a context/project based “operationalization“.

I wrote about this ratio on my blog post around the Test Automation Pyramid, as I will use the labels to automate the confirmations (and only the confirmations). The assumption is, that there are significantly more of the binary requirements tested by machine checking – and more human tested tacit questions. If we can get all the tedious tasks automated – that is the really the end goal.

Automate all the things that should be automated

Alan Page

Every project/context will have it’s own ratio, depending on a range of factors. Saying there should always be more of one type than the other would not hold. As the above project is the configuration and implementation of a standard commercial business software package (like SAP, SalesForce etc), my expectation is that most of the requirements are binary. Also considering that this project is heavy on the right hand side of the Wardley Map scale of evolution.

It’s a Reduction in Context

I am well aware that the two/three piles are an approximation / reduction. Especially when looking at the “binary” requirements and “only” testing these by confirmation. They could as easily be explored to find further unknown unknowns. If we prioritize to do so – it all about our choice of risk.

It is also an limitation as “perfect testing” should consist of both testing and checking. I factor this into the test strategy, by insisting that all of the requirements are tested both explicitly and implicitly. First of all most of our binary requirements are on the configuration and customization of the out-of-the-box software solution. So when the subject matter experts are performing the testing of the business flows, they are also evaluating the configuration and customization. And I do want them to spot when something is odd

The binary configuration is ok, but human know-how tells us otherwise.

Ultimately I want to use the experts to do the thinking and the machines to do the both the confirmations and the tedious tasks.