Someone else will do it

The testing activity has been under change for long. And it’s clear that the testing activity has shifted. Even the test managers have to re-calibrate – as other roles will be doing the test management activity. Be prepared, as someone else will do your testing job. Work on building self-reliance in others and be prepared to hand-over what you can do.

There is more to testing than testing specialists punching test cases. The testing activity as such, has shifted (both left and right), and testing is being done by more roles than “testing people”. Depending on the context, the explicit testing activity is done by a mix of developers, testing specialists, end users and others.

I often find myself as the only testing person on the project. The testing activity is done by automation specialists and end users in one project, and by technical operations staff and end users in another. In these projects either the technology or the business knowledge is paramount, and not so much exploration, flaws and edge cases for specialized testers to explore.

me, 2020. YMMV

Similarly for the test managers – there’s a trend/shift, that sometimes the test management activity is shifting away from the test managers. Even to me – even if I’m sometimes more an a “project manager of the testing activity“, a “Test coach” or similar. The trend is already there – coined sometimes as “whole team approach to quality“. Yes, most of the test management activity can be done by scrum masters, Release Train Engineers and even project managers ….

Recently I was asked to assist a large transition project for a holding company with many brands. Each brand had their own applications and technology stack, but the holding company had decided to move the hosting. So the holding company’s Project Management Office (PMO) was put in charge of facilitating the brand’s testing activities – an activity they had never considered nor done before. My role would only be to provide guidance, not do the actual facilitation.

Which got me thinking….

And after some deep thinking. – I do have the privilege to be able to adapt. I don’t need to hoard knowledge or make power moves (anymore) or worry about health-coverage or any of the lower Maslow pyramid terms (anymore).

It’s very natural for me to hand over project approaches to my co-workers. I’m often on the “blue team” to outline the strategy, My best field of work is to bring clarity and consistency, not scalability or repeatability to the practise.

I naturally hand over learning anyways, so why not re-calibrate when the thing I do has reached a stage, where it’s repeatable. And then focus on building the skills in others, work myself out of the test management role as we know it.

And don’t worry that someone else will eventually do my (testing and test management) job. The first step is to acknowledge the trend/pattern, second to redefine and bring clarity! Let’s explore and see what we find!

Someone else will do the building, not Emmet. His task is repeatable.
Someone else will do the building, not Emmet. His task is repeatable.

A Better ROI for testing

In classic test techniques and test approaches the test activity is a scarce ressource. Due to time and money constraints a risk-based priority was always required to make ends meet. We now have the tools and approaches for a better Return of Investment on the testing activity, and it’s all about running more tests, more frequent and sooner.

You never have time to test everything. So in the context of classic test techniques and testing types (I’m looking at you, old fart) you had to prioritize and combine tests to make more with what you had.

  • MoScoW priorities” on test cases (Must-Should-Could-Would). Yet when management put a 20% max on must cases, the business just requested more tests until the level was achieved.
  • Pairwise combinatorics and equvivalence classes might reduce the number of tests, but always seemed to focus on low level input fields and formats, never about business outcomes.
  • Discussion on whether the test case was a regression test, an integration test or what not. Sometimes regression tests mattered more than new functionality. Sometimes SIT and UAT seems to be the very same thing, so why test it twice. What the business really wanted was not window dressing of words, but results no matter what the name.

Counting tests is like..

An analogy to testing everthing, is to count all possible decimal numbers. There is always one more, decimal position around the corner. For each number, I can select any amount of new numbers around it. As with tests, I can select any amount of new .. variations.. of the test (browser, time, user, preconditions…). It’s hard to count something that spills into one another, as two tests can cover much the same, but still be two different things in the end.

.. and the rocks overlap too.

The classic techniques above are filtering techniques to first reduce the infinite space of possible tests into something distinct (a “test case”) – where every test is seperated from one another (countable). A “rock” in Aarons analogy. Secondly to filter it into something finite. so that it can be completed and answer “when are we done testing“.

Filtering down from all possible numbers to a countable and finite set.

Old Cost of Software Testing

The above filtering is is under the pretext that every value/test counted has a high price. Similarly that every test has a high cost to prepare and run. Average cost to write a formal test case could easily be 3 hours, and 1 hour to run / perform for at tester – and the perhaps with a 50% rerun/defect rate. So with 100 test cases a simple cost model would be at least: 450 hours, or 3 hours pr. test including 50% rerun.

No wonder management want to reduce this, and race the cost to the lowest denominator. Also considering that this only covers – at best – all the tests once, and half the tests twice. Is that a sufficient safetynet to go live on?

A new view on ROI

Current tools and test approaches turns the approach around, and focusses on making testing faster, frequent and cheaper (pr. test). The practises are so prevalent and well-described, that it really is already should be considered general development best practise. (G x P). Consider:

Now every project will have it’s own ratio of automation, but for this simple model, let’s assume 75% can be automated/tools-supported to such a degree that running them is approximately costless. Ie. it runs as a part of the continous testing, a CI/CD pipeline with no hands or eyeballs on it.

Preparing tests still takes time, let’s assume the same 3 hours. So the 25 tests with no automation still needs 112,5 hours – but the automation, as running is zero only accounts for the 225 hours of preparation. Just this simple model of testing costs, reduces the cost for testing with 25% (from 450 to 337) – including reruning 50% of the tests once.

The modern approach is to make the tests plentifull and comoditice it, rather than make it scarce. (See also “Theory of Constraints” wrt bottlenecks). With the added benefits of CI/CD and whole team approach to quality – the research of Accelerate confirms the correlation to clear business benefits.

Since running the automated tests are cheap, we can run them “on demand”. Let’s run 25% daily – is this a regression test? Perhaps, it doesn’t really matter. Assuming we run 25% random tests a day for two weeks, aka 250 tests, we have increased the count of test runs, and the number of times each test has run. With this approach our test preparation effort of 225 hours above is now utilized for 250 runs… or under 1 hour/cost pr. run.

The whole idea is to (re)imagine the testing effort as fast and repeated sampling among all possible values, done multiple times. The more the tests are run the better – and the better ROI for testing. .. and if you dare an even better performance by the organization.

Fast, Repeated Sampling of numbers

Further reading for Contest NYC 2019

Materials used for the talk and workshop at Contest NYC 2019:

One page test plan

Wardley Maps

Research:

Test management / Test Coach

Subject Matter Experts

Practical tips:

The subject matter expert in LEGO knows the bigger pieces left goes into the model.
The subject matter expert in LEGO knows the bigger pieces left goes into the model.

Closing the Gaps

[Previously on the Ministry of Testing, Nov 2014 – now only on the Web Archives]: About Closure

When I’m in a testing activity I want my test cases [Passed], my user stories [done] and my coffee [black].  Stuff may have a start point, some states in between and an end state. Lets look at ways to represent states and articulate the meaning of states.
One way to illustrate status about the product being tested is to model the activities we have with states. An agile user story may be [ready], [in progress] or [done]. A document may be [final], [approved] and a mind map may be iconified etc. States are so common that we sometimes forget the theory behind the model, and what benefits we have from the theories.

The representation of closure

For instance we can look to computer science graph theory[1]  to help us understand and control the states diagrams. It is the same graph theory that brings us state machines and state-transition diagrams, but that is another story[2] . In a graph theory state model we want one unique start state (Like [To do]), and one unique end state (like [passed]), everything in between is intermediate.

A (single) end state helps prevent the state machine from going on forever[3] , and us from going on forever too. [Deferred] and [Rejected] are temporary states to me. Setting [Rejected] back to “detected by” will aid that the tester reflects on the reasons. The reasons are then tested (in the brain). Sometimes it’s a “my bad” but quite often also the tester finds that issue is simply not “rejected” with more data and examples.

The understanding of closure

Similarly the agile “Definition of Done”[5]  and “Definition of Ready”[6]  helps the agile team phrase when the task is to change state, sometimes it’s explicit, sometimes it’s implied. The understanding of terms (the semantics) are usually more imperative than the syntax (the rules and representation). Sometimes it’s necessary to “connect the lines”.

There are a two related psychology theories on closure. One is the Gestalt law of closure[7]  – that is that we tend to self-organize items into an orderly structure. As the image above isn’t really about triangles – it’s about our human tendency to connect the dots. The other psychology part is the desire to close stuff to gain controllability[8]:

The need for closure is the motivation to find an answer to an ambiguous situation. This motivation is enhanced by the perceived benefits of obtaining closure, such as the increased ability to predict the world and a stronger basis for action.

Management and stakeholders often want a “firm and unambiguous” answer from our testing investigations. And this is often the business justification for setting states to our work products; that we have states to illustrate work progress in. Sometimes loose representations and a strong shared understanding goes well, sometimes a more strict representation and elaboration is required.

The syntax of states may be easily explained and codified (and checked), while the semantics and perception is less direct – and needs analysis (and testing). All work products (even for mind maps and test charters) have states and we must articulate both the syntax and the semantics to the team and stakeholders.

References

  1. Graph theory http://en.wikipedia.org/wiki/Graph_theory
  2. Fell in the trap of total coverage https://jlottosen.wordpress.com/2012/11/05/fell-in-the-trap-of-total-coverage/
  3. Finite state machines http://en.wikipedia.org/wiki/Finite-state_machine
  4. A Track down History
  5. definition of done https://www.scrum.org/Resources/Scrum-Glossary/Definition-of-Done
  6. definition of ready http://guide.agilealliance.org/guide/definition-of-ready.html
  7. Law of closure http://jeremybolton.com/2009/09/gestalt-design-principles-the-law-of-closure/
  8. Closure (Psychology) http://en.wikipedia.org/wiki/Closure_(psychology)

To Transform Testing

There is no doubt that our long lived testing narrative is under pressure. To continue to bring business value, both the IT field and testing is transforming to be about proactive business enabling.

The IT domain is currently buzzing with the word “IT transformation” – the idea that IT services should be more about “inspire, challenge and transform the digital businesses“. That it should be less about delivering IT products and artifacts and more about enabling digital business outcomes. Even for testing – it should be less about a product/service, and more about business necessity. As Anders writes:

Stop focusing on the things that bug people, and dare to do both IT and testing smarter and more business focused from the start. Build Quality in – smarter from start. That goes for IT services as a whole, and definitely also for the testing activities.

What can you do to transform your testing? I have three areas:

Discuss Business Strategy

Learn Wardley mapping – and use it like Chris McDermott to create context specific maturity models with Wardley Maps informed by Cynefin. Use the mapping to Broaden the scope of the system under test.

Align with the Business Strategy

Leading Quality [Cummings-John, Peer 2019] has a whole chapter on “Align your team to your company growth metric“. Consider if the company you work for is driven by Attention, Transaction or Productivity metrics, and arrange your test activities accordingly.

Dare to Deliver in New Ways

We are usually talk so much about optimizing the (IT and testing) delivery, that we forget other ways to be innovative and provide business enabling. One way could be to dare use new technology like RPA or a HoloLens to support tedious tasks in testing – to use an existing product to something new. Another approach to actually test “all the things” that matter or to apply testing to IT outside the realm of application delivery.

To Transform Testing I will discuss, align and dare so that test solutions can be proactive business enablers – (not only achieve shippable quality).

Mapping Mondays – Pioneers, Settlers, Town Planners

How Automation Affects the Business

As of writing I am managing the testing of a large enterprise IT program, where we are implementing a new commercial enterprise solution (COTS).

Over the last many months there have been requirement workshops upon requirement workshop to write down what the new system should be able to do for the various business units. We have had many representatives from the business units as part of the workshops and now have about 1000 specific business requirements that needs to be tested.

Some requirements are closed questions, others are more open-ended or similarly require some thinking. Currently the ratio is that 70% is done by test automation and 30% is for a few of the subject matter experts to test. Management was happy with this, as this made the project faster, the solution more robust and the project less reliant on taking the business people away from their “real work”.

So far so good

The other day I reached out by mail to more of the business people involved in the workshops to let them know that testing had started, and that they would be able to access the solution under test when it had been “hardened”. But so far, only a few “track leads” would be involved.

The feedback surprised me, as my message was both good and bad. Good in the sense that they would not be involved so much, but also bad that they would not be involved so much. One wrote back to me:

  • There is still a risk that the solution will not be as the workshops intended, as the requirements and solution might not capture precisely, what was agreed during the workshops
  • Having been part of the workshop, we are held responsible by our coworkers as to how well the new system supports the business
  • Why don’t the project want our involvement on this?

... but that was “just feelings”, he wrote in the end. And indeed it is – No matter how it looks at first, it’s always a people problem and even if we have a successful test automation effort – we can still fail to appreciate the experts knowledge and by that fail to solve the business problems.

More about “Leading when the experts test” at ConTest NYC 2019.

I was out hiking in April. But city management had locked the toilet up - out in the woods. As an END user my problem was then solved by doing it in the woods. And all fancy sheds where for naught.
I was out hiking in April. But city management had locked the toilet up – out in the woods. As an END user my problem was then solved by doing it in the woods. And all fancy sheds where for naught.

In Charge of Testing

As a Test Manager I oversee the testing in a project or program – I am usually the only testing specialist in the project, so, I need the right leadership skills and the right tools to succeed. I have to own the data about the testing and quality activities.

As the test manager I need to facilitate a quite a range of testing activities:

I need to balance that I need to know what’s going on (with regards to testing) but without micromanaging the people being involved in testing and quality activities. My role is to facilitate that testing things happen – like the project manager making project things happen. I cannot own the activities without owning the data about it. I need to cover the full spectrum of tests – from engineered (RDA and CI/CD) to people-based (scripts and exploration).

The most practical tool for a test manager with this scope is PractiTest, as there is more to testing than just the test cases [2]. The old term “ALM” [3] comes to mind – it is still relevant when I look for a full test management tool. I need to cover both the “inputs” to testing (requirements, tickets and user stories) and the “outputs” (bugs) in one location. I need the requirements and user stories in my tool, as I need to base my test analysis and planning on the delivery model (that may not always be agile). I need the bugs in the testing tool too, as bugs can happen in any work product of the project: documents, code base and even the tests. PractiTest acknowledge that there is more to IT projects than code.

I appreciate the key driver of PractiTest – that all activities happen in-flow. You don’t have to change window, stack pop-ups or go to another tool in order to run the tests or create bugs. Creating bugs happens in context of the test case and seamlessly moves all data about the run to the bug. Everything you need to do is context-based, and available to you on screen. And it has some cool features of read-only links to graphs for management reporting, and a smart built-in “rapid reporter” for exploratory testing notes.

It can be a challenge to switch to PractiTest if you are in a compliance setting, if you need on-Premise or if your team generally uses Azure DevOps (the tool formerly known as TFS). To get the full potential of Azure DevOps, though, you need the full Microsoft Test Pro licenses, so it’s not a free tool either – nor is DevOps intuitive for testing things doesn’t have the code available. As with Azure DevOps PractiTest is also SaaS only, with multiple data centers for regional data compliance. As there is always inertia towards a commodity it won’t be long before there is no good arguments to have test management tools on-Premise and for the tool vendors to provide the compliance certificates (ISO/SOC really should be sufficient, IMO).

Out of the box PractiTest supports the categories of testing above (engineered, scripted, exploratory) and has the necessary integrations too: Surefire for unit testing, Maven for CI/CD, Jira, ServiceNow or any other ITSM for requirement input. There is even a two way integration to Azue DevOps. As the web design is “responsive” it could probably run off a tablet. That would enable easier test documentation for field tests. It would be even better to have a small version of it on a phone and be able to use the camera for “screen shots”.

At work I am currently running a large project regarding customizing and implementing a standard commercial software system, PractiTest would fit right in, as we have the following test activities:

  • Unit test by the developers
  • Automation by test engineers
  • Exploratory test by Subject matter experts
  • Formal scripted testing with end users

And I need to own the data around all of this, if I want to in in charge of the testing (and not only the testers). We are very few software testing specialists on the project team, but as the manager of testing I need to cover many other people performing the testing. This transforms my role from test management to one about leadership, coaching, and facilitation of testing being performed by the SMEs – and anyone else really.

I will be talking about Leading When the Subject Matter Experts Test at ConTEST NYC 2019 until then read more about leadership:

  1. Anthropologists and similar humanities educations can be great BA’s
  2. looking at you Test Rail 😉
  3. ALM = Application Life Cycle, like Micro Focus Quality Center etc.

Disclaimer: This is an influencer review sponsored by PractiTest.

Expectations around Testing

I usually mention that the work I do as a test manager is more around managing the testing activity, than managing testing specialists. “Managing the testing activity” to me is about:

  • Identifying the expectations are around the testing activities
  • Facilitating the performance/execution of the testing activities
  • Administration and documentation of the testing activity
  • Make the people doing the testing self-reliant

… in Context

The project context is the most important frame: it is all about the projects
story, risk profile, culture, traditions, deadline, budget etc. I am as Context-driven as contexts allow, in the classical “Seven Basic Principles of the Context-Driven School” sense*.

As I am motivated by finding solutions and making them work** my drive is more along the lines of “accelerate the achievement of shippable quality” [Modern Testing Mission] than “finding the problems that threaten the value of the product” [Rapid Software Testing].

Focussing on achievements over problems seems to work for me in the contexts I’m in, regarding enterprise transition, infrastructure projects and the implementation of commercial standard systems.

Setting a Frame for Expectations

Finding the “test solution” (or test strategy) that fits the project context is the key activity to me. The rest of it, is mostly about implementation – that too can be quite interesting. I like that too, but plan first!

First of all we have to realize that the testing activities we choose are limited and affected by or context (and biases). We can never test everything and think of everything to test. Based on the context restrictions (time, space, money, etc) the project gives me, I make a reduction of the testing theories and principles into a definition along the lines of:

In a specific context – testing will be a finite activity, to investigate if the shared interpretations of the requirements are implemented – at some time, for some configuration, evaluated by someone (that we trust), where nothing odd happens.

A reduction of the testing activity

Let me be the first to say: It’s not theoretical perfect! But it’s practical and based on context. The reduction gives me an achievable goal-oriented focus. It helps me to iron out what the relation is between the thing under test and someone whom it matters to.

Ironing out the Expectations

If there is an underlying risk that things will change a lot, then we can argue for test automation to multiply the configurations and the number of “runs” we can complete. Not all IT projects are around software development, so test automation practices and tooling might not be in place.

We can ask Open Questions to explore the boundaries of the shared understanding. We can discuss: how much total test coverage is needed here? We can challenge the requests for the kitchen sink – but also direct the testing to what matters. I have found that it is better to slowly impact the projects with questions from within, as discussed on the Guilty Tester Podcast, than break down traditions up front. We can look into “who” is doing the investigation and how much we trust them.

To make the agreements around the reduction of the testing activity explicit I establish a “Test Plan” document. I would often prefer to do without, and have a mutual team agreement – or even a mind map. But I know the enterprise contexts, too well to know, that shared expectations are best written down (even though it being an imperfective as well).

It’s all about the context and the expectations, really.

The expectations where that we could snorkel…

*: Even “CDT” is a context/model, and thus is flawed. One of the flaws of the model is that all test approaches are equally valid (as long as it adds value to someone who matters) and thus that no approach is never better than any other. Not even CDT..

**: See: Innovation in Testing, Less Software more Testing.

Broaden the scope of the SUT

When testers talk about SUT (System Under Test) there seem to be an implied context of it being software, developed, bespoke software to be specific. Let me broaden the notion of a SUT using Wardley Maps and with that illustrate how testing can add value across the board.

Bespoke software (aka Custom built) is where the solution (SUT) is built and maintained tailormade to a specific company by a specific team that answers to order giver. When you build and maintain an app or web site for a company and is embedded in the team delivering the code base – it’s usually in a bespoke context.

Experiment/emerging example: Built an internal web site to do some simple public service case management. Write it based on MicroSoft .Net and IIS technology. The solution is new and novel, so interaction with the user is important.

All the commotion on buiding MVP experiments and interaction with the Product Owner are usual symptoms of a genesis situation. As the processes mature and products emerge, the solution development becomes more an customization activity.

Customization example: Implement Dynamics 365, SiteCore, SalesForce etc – but taylor and code them to your specific purpose. I have worked in a project taking Dynamics 365O and creating custom forms to handle public sector health events.

The last class of software “development” is the pureplay configurations of standard solutions. This is the context of SaaS – pay the license and get started. Think SAP or Office Applications, anything that is so accepted that it’s almost free (OpenOffice) and kills the IT department.

Let me draw this on the axis of Wardley Maps Evolutions:

Similarly we can add the underlying infrastructure to the drawing. As solutions move to the cloud and infrastructure becomes code, the system under test could very well be code around infrastructure. Initially bespoke infrastructure experiments (in perl?), and as time moves – even infrastructure becomes a commodity in the form of Amazon S3.

So where is your SUT? – what is the path down the stack? Because there is a huge difference between testing custom code for cloud services, as compared to product customization on actual physical owned hardware.

Let’s think testing outside the bespoke areas on the map too. Some current examples I am working on are:

  • Infrastructure transition of 700 serves from being owned to being hosted
  • Application transition of 50+ applications from being owned to being outsourced
  • Transformation of a standard form management solution
  • Implementing a standard system for ITIL case management

These projects have no code, the SUT is either a server, an environment (collection of servers), a form, a process or something else. While we do know a lot about testing in bespoke software contexts, the practices for testing in transition and transformation are emerging practices! This gives us and extra layer. And this is where it gets interesting.

There are plenty of standard practices (SAFe, agile..) but the practices for testing in the context of transition is yet to materialize.

The same model can be applied to IT as a whole. IT support and end user computing (devices, desktop operation) are to the very far right as commodity services. While on the far left is the constant experimentation and tinkering (of AI, ML and RPA) to become actual products.

If we only see testing as part of building bespoke software we fail – we fail to see the horizontal and vertical contexts, where the testing disciplines can add similar value and impact.

Who is the tester?

In my current and primary projects the testing is not done by software testing professionals – and it’s probably for the better too! It is in contexts like these:

  1. A Microsoft Dynamics “D365O” implementation of health registration forms. Tested by public service clerks partly comparing to the previous solution, partly testing the new system platform.
  2. Moving 700+ servers running 50+ applications from one data center to another while keeping everything from mainframe to SaaS integrations live. Tested by the application staff that have maintained the system since for ever (10+ years).
  3. Implement at standard commercial off-the-shelf tool for 2000+ IT savvy users. To most users this tool is their primary work tracking system, so they get to test it too.

In contexts like these the act of testing done by subject matter experts of the field – infrastructure specialists, public service clerks, support staff, application developers and the like. These persons qualify as the “customer” in the Modern Testing Principle that “the customer is the only one capable to judge and evaluate the quality of our product“. They might have a testing /role/ during the project, but that is because of their high domain knowledge, but at the end of the project they continue with their “real business job” of using the system to produce stuff for the business.

It’s not their job to know ISTQB from “MT Principles” and “RST methodology“. That is up to me, as the manager of the testing. My role is more and more about the guidelines for the testing and the facilitation of the people doing the testing. My reach goes so far as to ask them to think about how the product fails and succeeds. But I cannot expect them to know checking from testing.

Long gone are the days of managing testers that put all their skill into the niches of the testing craft. There are less software testing professionals doing the testing in projects like the above. Part of it is, that the describing the whole system explicitly is simply to expensive in time and money. This makes the requirements inherently fuzzy and undefined. And part of it is that learning the skills simply takes to long. Some technical tests require skills of a certified VMware specialist, others having an eye for every unwritten tacit business rule.

Another angle is that the skills that the usual software testing specialist brings to the table is handled on a lower level. Testing is done by the organisation (like Microsoft) that builds the standard solutions and commercial off the self systems (SAP, D365O etc). Another is that the test techniques of the software testing field simply no longer applies. I mean how does boundary value analysis add value to enterprise data center transition executions, when the system under test it not even software?

The better tester is neither the software developer nor the software testing specialist. It’s the person who ponders:

  • How could this go wrong…
  • I wonder if…
  • For this to work, we need to do…

Come to think of it, everyone in the project does that! Some do it more explicitly, some do it more experimental. Everyone evaluates how their actions add value to the people that matter (at some time).