The key metric for any knowledge work – IT deliveries and testing in particular – is more than Mean Time to Repair (MTTR). While fixing fast matters – timing is everything. Timing in getting information to the people who needs it to make decisions. It’s no use if you can turn the ship around on a plate now, if you needed it yesterday. Key elements in calculating time to information is how far away the information is and how evolved the information is.
Continue readingcynefin
Focus on Goals over Risks
Looking into the discussion on what goes into a Test Plan and what goes into a Test Strategy – it’s my personal opinion that we can improve our business alignment. Risk-based testing and Product Risk Analysis have been around for long – but better models have emerged to address what will be more impactful.
Continue readingRelations – are about half of IT
You can’t have IT projects without relations. Relations matter more than it seem.
Continue readingTo Transform Testing
There is no doubt that our long lived testing narrative is under pressure. To continue to bring business value, both the IT field and testing is transforming to be about proactive business enabling.
The IT domain is currently buzzing with the word “IT transformation” – the idea that IT services should be more about “inspire, challenge and transform the digital businesses“. That it should be less about delivering IT products and artifacts and more about enabling digital business outcomes. Even for testing – it should be less about a product/service, and more about business necessity. As Anders writes:
Stop focusing on the things that bug people, and dare to do both IT and testing smarter and more business focused from the start. Build Quality in – smarter from start. That goes for IT services as a whole, and definitely also for the testing activities.
What can you do to transform your testing? I have three areas:
Discuss Business Strategy
Learn Wardley mapping – and use it like Chris McDermott to create context specific maturity models with Wardley Maps informed by Cynefin. Use the mapping to Broaden the scope of the system under test.
Align with the Business Strategy
Leading Quality [Cummings-John, Peer 2019] has a whole chapter on “Align your team to your company growth metric“. Consider if the company you work for is driven by Attention, Transaction or Productivity metrics, and arrange your test activities accordingly.
Dare to Deliver in New Ways
We are usually talk so much about optimizing the (IT and testing) delivery, that we forget other ways to be innovative and provide business enabling. One way could be to dare use new technology like RPA or a HoloLens to support tedious tasks in testing – to use an existing product to something new. Another approach to actually test “all the things” that matter or to apply testing to IT outside the realm of application delivery.
To Transform Testing I will discuss, align and dare so that test solutions can be proactive business enablers – (not only achieve shippable quality).
Test ALL the things
TL;DR: We can add testing to all requirements and all business risks. Testing to document requirements and to debunk risks provides valuable information for the business. Let us not limit testing to things that can be coded. The intellectual activity of trial and learning is happening anyways, we might as well pitch in with ways to find important evidence for the decision makers.
Test all the requirements
Traditionally testing was all about testing the functional requirements that could be coded. Non-functional requirements was left for the specialists, or plainly disregarded. I know I have done my share of test planning, with a range of requirements left “N/A” with regards to testing. Especially performance scope, batch jobs, hardware specs, data base table expansions and virus scanning have been left out of my functional test plans…
When I look at a list of requirements now – I see that we can indeed test all the things, or we can at least work on how to document that the requirement is fulfilled. Some of the requirements are actually quite easy to document. If it’s on a screen somewhere, take a screen shot and attach it to a simple test case. Done deal really. Additionally with a testing mind-set I can think of ways to challenge the details. But do we really, really need to fill up a disk to establish if it’s exactly a 1 Gb allocation – probably not. Do we really really need to document all requirements – yes in some contracts/contexts it’s important for the customer to know that everything has indeed been established. Sometimes the customer doesn’t trust you otherwise, sometimes the tests are more about your ability to deliver and provide evidence that matters.
Test all the business risks
Look into the business case of your project and find the business risks. Sometimes they are explicitly stated and prioritized. A a recent Ministry of Testing Meetup we looked into a case for a large national health system. We looked at the tangible benefits, intangible benefits and on the scored business risks. What worried the business and management most was budget, time and whether the new system would be used in a standardized way. There is an opportunity for testing here to help address, document and challenge the most important business risks. Traditional testing would usually look at functional requirements that can be coded or configured, and miss totally to address what matters most to the business.
OK, how do we test the project costs? How about frequent checkpoints of expected spending – would that be similar to tracking test progress. Perhaps – let’s find out. Testing is all about asking questions for the stakeholders and solving the most important problems first. Can we help to analyse risks and setup mitigation activities – sure we can. We just have to step out of our traditional “software only bubble”.
Read also: Many Bits under the Bridge, Less Software, more Testing, Test Criteria for Outsourced Software, The Expected, Fell in the trap of total coverage.
Links: “A Context-Driven Approach to Delivering Business Value”, Cynefin In Software Testing, Testing during Application Transition Trials
Less Software, more Testing
I rarely test software these days. I mostly lead testing of IT solutions.
Testing in the context of:
- Updating all corporate PC’s from windows 7/8 to Windows 10
- Consolidating network equipment from more devices to one box, on 80 global locations *
- Move 40 live business applications from one data center to another *
- Take over application maintenance for a specialized public organization
- Implement track and trace for pharma products from production to shops
- Migrate HR data for 2500 people from one system platform to another
Yes, it happens that I participate in a project that is about developing a new business application, but my activities are less about testing software and more about testing in IT solutions in general.
Mostly I manage test activities and describe testing in these contexts. My preferred way of working is in setting and implementing test strategies. I prefer complex and non-ordered projects (Complex and Chaotic – I’m looking at you), it fits well with my context-driven approach of finding the “test solution that fits the context”.
Testing is in it self a solution, that must solve a business problem. Great testing is all about providing information to the stakeholders. I don’t care especially if this is done by someone TESTING or a TESTER. It is my responsibility to setup the testing activities (information gathering) that supports the team, faces the business & technology and challenges the product “sufficiently“.
Sometimes “sufficiently” is merely confirming and going through the motions of explicit requirement coverage. This is a special challenge to me, as I know of many effective and Rapid approaches, that could add valuable information. When I face this challenge, I try to look at the full picture of the project, and what the business want’s to achieve.
The business of the business is business. What matters is not software or projects, but the solutions to the challenges the business have. And the context of testing is similarly so much more than the software.
*: As mentioned in “How to test in IT Operations” at Nordic Testing Days 2016.
The Expected
Many test processes and test tools mention that you have to establish the expected results to every test. It is at best risk ignorance not to take the notion of expected results with a kilo of salt #YMMV.
If you can establish the result of your solution in a deterministic, algorithmic and consistent way in a non-trivial problem, then you can solve the halting problem. But I doubt your requirements are trivial.. or even always right. Even the best business analyst or subject matter expert may be wrong. Your best oracle may fail too. Or you may be focussing on only getting what you measure.
When working with validation in seemingly very controlled environments changes and rework happens a lot, as every new finding needs the document trail back to square one.. Stuff happens. Validation is not testing, but looking to show that the program did work as requested at some time. It is a race towards the lowest common denominator. IT suppliers can do better that just to look for “as designed” [1].
Still the Cynefin framework illustrates that there are repeatable and known projects, and in those contexts you should probably focus on looking to check as many as the simple binary questions in a tool supported way, and work on the open questions for your testing.
Speaking of open ends – every time I see an explicit expected result I tend to add the following disclaimer (song to the tune of nothing else matters [2]):
And nothing odd happens … that I know of … on my machine, in this situation [3]
And odd is something that may harm my user, business or program result
Significantly…
But I’d rather skip this test step and work on the description of the test and guidelines to mention:
- Think! Use your head – use what you know.
- You are the expert.
- If it smells fishy, there is something fishy going on
- Wonder, Wander, Galumph if you may
- Analyse, consider,.. if you get an idea, try it!
- Learn, Explore, and by all means test
See also: The unknown unknown unexpressed expectations, Eating wicked problems for breakfast
1: Anyone can beat us, unless we are the best, todays innovation becomes tomorrows commodity
2: https://www.youtube.com/watch?v=tAGnKpE4NCI
3: I’ve heard that somewhere…
Complexity is a Matter of Perspective 