test_management

Where I Have Changed

jo2sn Softwaretesting , , , , , ,

The Ministry of Testing Bloggers Club suggested that I write a post based on “In testing, I have changed my mind about ________”. As this blog dates back to 2012 with consistent (220) articles about testing, and my career in the field dates back to 2002 – it seems a 20-year experience should give me a few things. Testing is still not dead – and it’s still about the context (lower-case context, not CDT).

It’s not about: Testers being the only ones doing Testing

yeah, not so much these days. Testing is an act that any role can do in context. It’s about the testing – not so much the testers. And I have realized that even classic test management tasks can be done by someone else. Testing is not owned by the testers – it might be stewarded/facilitated by us, but it’s performed by a team member (who could be a tester).

It’s not about: Perfect Requirements

After decades in IT, it’s clear that even requirements are never perfect. When we look closer we see the business requirements can vary from a profound idea to a rudimentary feature of the system under test. Even in regulated industries requirements can be both about a specific configuration in a SaaS system or a loose idea of a relevant dashboard. Sometimes a requirement can be by design of an underlying commodity product – there doesn’t need to be a test case for everything.

The more rigor you add to the requirements management – the more fragile it becomes. It’s key to understand the risks and bets of the person paying for your solution. – in that lies the true borders of the delivery. Much can go informally along if it aligns with stakeholder values.

It’s not about: Defects

Back in the day defects needed to be accounted for, tracked, and distributed. Besides testing documents – defects were the only tangible delivery of the testers. The defects needed to be raised and closed. I recently wrote a guideline that stated that only observations that couldn’t be fixed within a day should be raised to the project manager for shared handling. In that context fixing things is within the same team. If it’s for another team to fix, defects are simply something communicated between the teams (check team topologies for team interactions). Sure you can still find a blocker or a P1 – what matters is how fast you can fix things.

It’s not about: Month-Long Testing Phases

The more time there is from idea to implementation – the more the requirement risks not addressing an up-to-date business objective. Timing is key. Some tools provide epics and user stories – but the structure is often misused to be a simple work aggregation – and not goal aligned.

The counter-intuitive trick is not to add formality, and more time between releases – but less. Less time between feedback between idea and implementation, and less time between implementation and test. Less time between the various forms of feedback adds to better results faster.

It still happens, I’m sure, that a business needs a month-long testing phase before a release; having a range of business staff to participate in testing the latest release of the enterprise ERP or CRM. More often the testing phase is one sprint behind the development activity. I have pondered this a lot.

At best testing is an integrated activity in the team and in the sprint. But if testing is a more separate activity – it can be both agile and context-relevant. So I have changed my mind about this anti-pattern.

It’s ok for testing to be in the next sprint –

if that adds consistency and less stress to the team*

* Seperate boards needed. Your Mileage Will Vary

A More Advisory Role

jo2sn Enterprise IT, Softwaretesting , , , , , , , , , ,

Over the last year I have looking to work myself out of the test manager role and into a more advisory role. And by April 2021 I was given the formal title change from Senior Test Manager to Senior Advisory Consultant.

I have had the title “Test manager” probably since 2008, so it’s been a while. In the companies, where I have been employed, the Test manager title has never been with line management (hire/fire). Rather it has been similar to a project manager, with a focus on the testing deliveries of a project, release or program.

I will still be leading test activities, but my role for the future will be more about enabling someone else doing the testing or someone else having a testing problem to solve. There are plenty of test activities done by people in non-testing roles – it’s the activity that matters.

Continue reading →

Something About Leadership

jo2sn Autism, Enterprise IT, Softwaretesting , , , , , , , 
17yo: Dad, do you not know how old I am - and what I can do myself? 
Me: Oh, I know buddy. As you are learning new stuff, I am unlearning to help you

While this quote from my kitchen about a week ago, as all to do with the young man learning the ropes of life and me unlearning to always to help them and their 15yo sibling out – there is an key parallel to leadership and building self reliance in teams. My role these days are less about direction and (micro)managing a team of testers on a project, more about enabling others to succeed with their testing both in the delivery teams and in the board room.

Continue reading →

Visualize the Test Strategy

jo2sn Enterprise IT, Softwaretesting, Wardley's theories , , , , , , , ,

There are plenty of models/frameworks that list what you could have of testing activities – but there seems to be little assistance in WHAT to do in which situations. I have two visual ideas – let’s explore how to visualize the test strategy. And yes, there’s no size fits all – context matters.

Continue reading →

Someone else will do it

jo2sn LEGO, Softwaretesting , , , , , , , , ,

The testing activity has been under change for long. And it’s clear that the testing activity has shifted. Even the test managers have to re-calibrate – as other roles will be doing the test management activity. Be prepared, as someone else will do your testing job. Work on building self-reliance in others and be prepared to hand-over what you can do.

There is more to testing than testing specialists punching test cases. The testing activity as such, has shifted (both left and right), and testing is being done by more roles than “testing people”. Depending on the context, the explicit testing activity is done by a mix of developers, testing specialists, end users and others.

I often find myself as the only testing person on the project. The testing activity is done by automation specialists and end users in one project, and by technical operations staff and end users in another. In these projects either the technology or the business knowledge is paramount, and not so much exploration, flaws and edge cases for specialized testers to explore.

me, 2020. YMMV

Similarly for the test managers – there’s a trend/shift, that sometimes the test management activity is shifting away from the test managers. Even to me – even if I’m sometimes more an a “project manager of the testing activity“, a “Test coach” or similar. The trend is already there – coined sometimes as “whole team approach to quality“. Yes, most of the test management activity can be done by scrum masters, Release Train Engineers and even project managers ….

Recently I was asked to assist a large transition project for a holding company with many brands. Each brand had their own applications and technology stack, but the holding company had decided to move the hosting. So the holding company’s Project Management Office (PMO) was put in charge of facilitating the brand’s testing activities – an activity they had never considered nor done before. My role would only be to provide guidance, not do the actual facilitation.

Which got me thinking….

And after some deep thinking. – I do have the privilege to be able to adapt. I don’t need to hoard knowledge or make power moves (anymore) or worry about health-coverage or any of the lower Maslow pyramid terms (anymore).

It’s very natural for me to hand over project approaches to my co-workers. I’m often on the “blue team” to outline the strategy, My best field of work is to bring clarity and consistency, not scalability or repeatability to the practise.

I naturally hand over learning anyways, so why not re-calibrate when the thing I do has reached a stage, where it’s repeatable. And then focus on building the skills in others, work myself out of the test management role as we know it.

And don’t worry that someone else will eventually do my (testing and test management) job. The first step is to acknowledge the trend/pattern, second to redefine and bring clarity! Let’s explore and see what we find!

Someone else will do the building, not Emmet. His task is repeatable.
Someone else will do the building, not Emmet. Their task is repeatable.

A Better ROI for testing

jo2sn Softwaretesting , , ,

In classic test techniques and test approaches the test activity is a scarce ressource. Due to time and money constraints a risk-based priority was always required to make ends meet. We now have the tools and approaches for a better Return of Investment on the testing activity, and it’s all about running more tests, more frequent and sooner.

You never have time to test everything. So in the context of classic test techniques and testing types (I’m looking at you, old fart) you had to prioritize and combine tests to make more with what you had.

  • “MoScoW priorities” on test cases (Must-Should-Could-Would). Yet when management put a 20% max on must cases, the business just requested more tests until the level was achieved.
  • Pairwise combinatorics and equvivalence classes might reduce the number of tests, but always seemed to focus on low level input fields and formats, never about business outcomes.
  • Discussion on whether the test case was a regression test, an integration test or what not. Sometimes regression tests mattered more than new functionality. Sometimes SIT and UAT seems to be the very same thing, so why test it twice. What the business really wanted was not window dressing of words, but results no matter what the name.

Counting tests is like..

An analogy to testing everthing, is to count all possible decimal numbers. There is always one more, decimal position around the corner. For each number, I can select any amount of new numbers around it. As with tests, I can select any amount of new .. variations.. of the test (browser, time, user, preconditions…). It’s hard to count something that spills into one another, as two tests can cover much the same, but still be two different things in the end.

.. and the rocks overlap too.

The classic techniques above are filtering techniques to first reduce the infinite space of possible tests into something distinct (a “test case”) – where every test is seperated from one another (countable). A “rock” in Aarons analogy. Secondly to filter it into something finite. so that it can be completed and answer “when are we done testing“.

Filtering down from all possible numbers to a countable and finite set.

Old Cost of Software Testing

The above filtering is is under the pretext that every value/test counted has a high price. Similarly that every test has a high cost to prepare and run. Average cost to write a formal test case could easily be 3 hours, and 1 hour to run / perform for at tester – and the perhaps with a 50% rerun/defect rate. So with 100 test cases a simple cost model would be at least: 450 hours, or 3 hours pr. test including 50% rerun.

No wonder management want to reduce this, and race the cost to the lowest denominator. Also considering that this only covers – at best – all the tests once, and half the tests twice. Is that a sufficient safetynet to go live on?

A new view on ROI

Current tools and test approaches turns the approach around, and focusses on making testing faster, frequent and cheaper (pr. test). The practises are so prevalent and well-described, that it really is already should be considered general development best practise. (G x P). Consider:

Now every project will have it’s own ratio of automation, but for this simple model, let’s assume 75% can be automated/tools-supported to such a degree that running them is approximately costless. Ie. it runs as a part of the continous testing, a CI/CD pipeline with no hands or eyeballs on it.

Preparing tests still takes time, let’s assume the same 3 hours. So the 25 tests with no automation still needs 112,5 hours – but the automation, as running is zero only accounts for the 225 hours of preparation. Just this simple model of testing costs, reduces the cost for testing with 25% (from 450 to 337) – including reruning 50% of the tests once.

The modern approach is to make the tests plentifull and comoditice it, rather than make it scarce. (See also “Theory of Constraints” wrt bottlenecks). With the added benefits of CI/CD and whole team approach to quality – the research of Accelerate confirms the correlation to clear business benefits.

Since running the automated tests are cheap, we can run them “on demand”. Let’s run 25% daily – is this a regression test? Perhaps, it doesn’t really matter. Assuming we run 25% random tests a day for two weeks, aka 250 tests, we have increased the count of test runs, and the number of times each test has run. With this approach our test preparation effort of 225 hours above is now utilized for 250 runs… or under 1 hour/cost pr. run.

The whole idea is to (re)imagine the testing effort as fast and repeated sampling among all possible values, done multiple times. The more the tests are run the better – and the better ROI for testing. .. and if you dare an even better performance by the organization.

Fast, Repeated Sampling of numbers

Further reading for Contest NYC 2019

jo2sn Softwaretesting, Wardley's theories , , , , , , ,

Materials used for the talk and workshop at Contest NYC 2019:

One page test plan

Wardley Maps

Research:

Test management / Test Coach

Subject Matter Experts

Practical tips:

The subject matter expert in LEGO knows the bigger pieces left goes into the model.
The subject matter expert in LEGO knows the bigger pieces left goes into the model.

Closing the Gaps

jo2sn Softwaretesting , , , ,

[Previously on the Ministry of Testing, Nov 2014 – now only on the Web Archives]: About Closure

When I’m in a testing activity I want my test cases [Passed], my user stories [done] and my coffee [black].  Stuff may have a start point, some states in between and an end state. Lets look at ways to represent states and articulate the meaning of states.
One way to illustrate status about the product being tested is to model the activities we have with states. An agile user story may be [ready], [in progress] or [done]. A document may be [final], [approved] and a mind map may be iconified etc. States are so common that we sometimes forget the theory behind the model, and what benefits we have from the theories.

The representation of closure

For instance we can look to computer science graph theory[1]  to help us understand and control the states diagrams. It is the same graph theory that brings us state machines and state-transition diagrams, but that is another story[2] . In a graph theory state model we want one unique start state (Like [To do]), and one unique end state (like [passed]), everything in between is intermediate.

A (single) end state helps prevent the state machine from going on forever[3] , and us from going on forever too. [Deferred] and [Rejected] are temporary states to me. Setting [Rejected] back to “detected by” will aid that the tester reflects on the reasons. The reasons are then tested (in the brain). Sometimes it’s a “my bad” but quite often also the tester finds that issue is simply not “rejected” with more data and examples.

The understanding of closure

Similarly the agile “Definition of Done”[5]  and “Definition of Ready”[6]  helps the agile team phrase when the task is to change state, sometimes it’s explicit, sometimes it’s implied. The understanding of terms (the semantics) are usually more imperative than the syntax (the rules and representation). Sometimes it’s necessary to “connect the lines”.

There are a two related psychology theories on closure. One is the Gestalt law of closure[7]  – that is that we tend to self-organize items into an orderly structure. As the image above isn’t really about triangles – it’s about our human tendency to connect the dots. The other psychology part is the desire to close stuff to gain controllability[8]:

The need for closure is the motivation to find an answer to an ambiguous situation. This motivation is enhanced by the perceived benefits of obtaining closure, such as the increased ability to predict the world and a stronger basis for action.

Management and stakeholders often want a “firm and unambiguous” answer from our testing investigations. And this is often the business justification for setting states to our work products; that we have states to illustrate work progress in. Sometimes loose representations and a strong shared understanding goes well, sometimes a more strict representation and elaboration is required.

The syntax of states may be easily explained and codified (and checked), while the semantics and perception is less direct – and needs analysis (and testing). All work products (even for mind maps and test charters) have states and we must articulate both the syntax and the semantics to the team and stakeholders.

References

  1. Graph theory http://en.wikipedia.org/wiki/Graph_theory
  2. Fell in the trap of total coverage https://jlottosen.wordpress.com/2012/11/05/fell-in-the-trap-of-total-coverage/
  3. Finite state machines http://en.wikipedia.org/wiki/Finite-state_machine
  4. A Track down History
  5. definition of done https://www.scrum.org/Resources/Scrum-Glossary/Definition-of-Done
  6. definition of ready http://guide.agilealliance.org/guide/definition-of-ready.html
  7. Law of closure http://jeremybolton.com/2009/09/gestalt-design-principles-the-law-of-closure/
  8. Closure (Psychology) http://en.wikipedia.org/wiki/Closure_(psychology)

To Transform Testing

jo2sn Enterprise IT, Softwaretesting, Wardley's theories , , , , , , ,

There is no doubt that our long lived testing narrative is under pressure. To continue to bring business value, both the IT field and testing is transforming to be about proactive business enabling.

The IT domain is currently buzzing with the word “IT transformation” – the idea that IT services should be more about “inspire, challenge and transform the digital businesses“. That it should be less about delivering IT products and artifacts and more about enabling digital business outcomes. Even for testing – it should be less about a product/service, and more about business necessity.

Stop focusing on the things that bug people, and dare to do both IT and testing smarter and more business focused from the start. Build Quality in – smarter from start. That goes for IT services as a whole, and definitely also for the testing activities.

What can you do to transform your testing? I have three areas:

Discuss Business Strategy

Learn Wardley mapping – and use it like Chris McDermott to create context specific maturity models with Wardley Maps informed by Cynefin. Use the mapping to Broaden the scope of the system under test.

Align with the Business Strategy

Leading Quality [Cummings-John, Peer 2019] has a whole chapter on “Align your team to your company growth metric“. Consider if the company you work for is driven by Attention, Transaction or Productivity metrics, and arrange your test activities accordingly.

Dare to Deliver in New Ways

We are usually talk so much about optimizing the (IT and testing) delivery, that we forget other ways to be innovative and provide business enabling. One way could be to dare use new technology like RPA or a HoloLens to support tedious tasks in testing – to use an existing product to something new. Another approach to actually test “all the things” that matter or to apply testing to IT outside the realm of application delivery.

To Transform Testing I will discuss, align and dare so that test solutions can be proactive business enablers – (not only achieve shippable quality).

Mapping Mondays – Pioneers, Settlers, Town Planners

How Automation Affects the Business

jo2sn Enterprise IT, Softwaretesting , , , ,

As of writing I am managing the testing of a large enterprise IT program, where we are implementing a new commercial enterprise solution (COTS).

Over the last many months there have been requirement workshops upon requirement workshop to write down what the new system should be able to do for the various business units. We have had many representatives from the business units as part of the workshops and now have about 1000 specific business requirements that needs to be tested.

Some requirements are closed questions, others are more open-ended or similarly require some thinking. Currently the ratio is that 70% is done by test automation and 30% is for a few of the subject matter experts to test. Management was happy with this, as this made the project faster, the solution more robust and the project less reliant on taking the business people away from their “real work”.

So far so good

The other day I reached out by mail to more of the business people involved in the workshops to let them know that testing had started, and that they would be able to access the solution under test when it had been “hardened”. But so far, only a few “track leads” would be involved.

The feedback surprised me, as my message was both good and bad. Good in the sense that they would not be involved so much, but also bad that they would not be involved so much. One wrote back to me:

  • There is still a risk that the solution will not be as the workshops intended, as the requirements and solution might not capture precisely, what was agreed during the workshops
  • Having been part of the workshop, we are held responsible by our coworkers as to how well the new system supports the business
  • Why don’t the project want our involvement on this?

... but that was “just feelings”, they wrote in the end. And indeed it is – No matter how it looks at first, it’s always a people problem and even if we have a successful test automation effort – we can still fail to appreciate the experts knowledge and by that fail to solve the business problems.

More about “Leading when the experts test” at ConTest NYC 2019.

I was out hiking in April. But city management had locked the toilet up - out in the woods. As an END user my problem was then solved by doing it in the woods. And all fancy sheds where for naught.
I was out hiking in April. But city management had locked the toilet up – out in the woods. As an END user my problem was then solved by doing it in the woods. And all fancy sheds where for naught.