Using Wardley Maps to visualize test strategies and align the test strategies with business goals.
-> Available on LeanPub: https://leanpub.com/goatsbook
business
Old Test Types Need Not Apply
I primarily work in situations that are less about application delivery and more about moving the whole system stacks, implementing a standard system, or similarly changing the organizational IT landscape. Some would list these projects as “staff projects” if that helps you. I often find the terms Regressiontest, SIT, and UAT to be misleading and not helpful to what kind of test is needed for my examples.
Example 1: We are rebuilding all the environments in the delivery stack. All the Dev-, Test-, Integration-, Preprod-, and Prod- environments, and their underlying databases, brokers, and websites. Every time we are constructing, an environment we will be testing the setup from a baseline version of the existing running systems. A baseline that we know is functional already. The classic software development testing types don’t really help us in this situation, as neither regression test, UAT, or SIT conveys the things we want to confirm and the learnings we want to explore.
Example 2: We are setting up a new expense platform for employee reimbursements to go live with a new branding of the company. It’s a SaaS system and we load it every month with data about the organization. So while it’s needed for various purposes – the risk is low and the mean time to repair is similarly low. The testing we will do will be a limited confirmation of an initial data load. Snow-plow style – not a full system-integration test and similar user-acceptance test. After all, this is a SaaS – not a custom solution. It’s OK to shift right.
SIT and UAT has become generic term that it has lost the strength to convey the needed quality narrative. If you do CI/CD (which you should) for your application development that such be sufficient. If you figure out, you need to do a “connection alive” test for your third-party integrations that you move from one environment stack to another that should be accepted with the acknowledgment that you actually considered the challenges ahead.
It’s all about the risks and the mitigations – less about testing everything to the dot. One tool to read the landscape is to communicate curiously about what the stakeholders value more – and value less. And on the other hand, consider the nature of the solution being proposed.
Example 3: Setting up a cloud-based Azure Active Directory – the solution comes with a given security level out of the box (OOTB). As with other OOTB and Software-as-a-service solutions you have little impact on the security features of the solution, besides some simple configurations. While you might think that all security requirements would require 100% acceptance testing coverage, what you want to accept is that they might be provided “by design” – or by a solution decision made long ago.
I would prefer that we can call things what they are and not blindly apply old testing types
Communicate Curiosity
A “testing mindset” of inquiry applies when you are a principal tester or even a senior advisor in testing. Asking open questions and approaching a challenge with exploration proves better than command and control. People are people – be mindful of where they are.
Currently, I’m involved in an extensive change program for a company. It’s not your usual agile feature factory but more about the technology stack needed to run a 3000 people company (payment, finances, etc). While testing professionals could be onboarded – their learning curve would not scale – not even being equipped with all the heuristics and testing vocabulary in the world. As it is a one-off, automation wouldn’t scale either.
Fortunately, most applications have a system owner (or product owner, or manager in charge) and usually a team around maintaining the application. To some degree, we could frame these teams as stream-aligned teams. The experts in testing the applications are the superuser – hence they do the testing.
Trust must be extended before it is given
Rachel Happe – @mastodon.social/@rhappe
One such team is the XYZ team anchored by a VP-level manager. A colleague and I had been trying to communicate with them to align on the testing needed but had met reluctance. They would not have time to help us. We discussed the matter and realized that we needed to approach them differently. Since the change program was such a fundamental shift for the team, they had already considered the implications. They were already testing and thinking about the impact. We just had to trust them – and build on what they already had instead of imposing specific ways of working.
Without requiring them to earn it first, tell everyone who works for you that you #trust them implicitly. From that point forward, just ask them to never give you reason to withhold it. (In my experience, most won’t).
Mark C. Crowley
Communicating curiosity can be a little thing like “Remind me, how does XYZ work” or simply “sitting on your hands” during meetings that you would previously fill with questions and requests. The tables of the great program test lead are turning, it’s about being an enabling function for others to succeed. Similar to the freedom you should give your growing up kids – it’s about leadership.
Factor in the Ripple Effects
TL;DR: Investing in basic tooling and automation improves your team besides expected metrics.
I work mostly with the implementation of enterprise SaaS systems these days. Large global companies are consolidating custom-built applications and on-premise applications with web-based standard solutions in the cloud aiming for “one standardized source of information to enable digital transformation”.
Yet the testing tooling hasn’t caught up. One company with €5000 million in sales is still using Word documents for test cases and “party like it’s 1999“. They are reluctantly considering tooling to support more agile ways of working. The whole “automate the knowns-knowns” is still pending an evaluation of return on investment (ROI) into technology from 2015. As of writing, Anno Domini 2022.
Assumptions
- Writing test cases in documents takes about as long as writing automation
- Maintaining automation is a more explicit task, humans can more easily apply a bit of fuzziness
- When automation is in place, the execution requires limited efforts to run
- The alternative to automated test execution is hours of people following and filling out the documents
With the investment in the tool, there’s a break-even around XX hours of document-based testing a month. That is if we plan for more than XX hours of document-based testing a month, the investment pays off. Your Mileage May Vary
But there’s more to it
First of all, when automated test execution is at limited costs to run and it can run independently at night, you will get the same effects as Continuous Integration and nightly builds have had in software development: you tend to run them more and more often.
This enables faster feedback both with regards to confirming new features and sums up to more effective regression testing. I have seen this happen in both custom application development and configuration of web-based standard solutions. In one project where I added automation, we have run nearly 8000 automated runs in a year (and 200 SME-based). We actually run the tests more often, and we cover the important things every day – and everything often enough. We do in fact get more testing, and broader coverage than any document-supported testing could ever scale to do.
Believe the experts
While there is some vendor basis in the following two webinars, the story is the same: Test automation can accelerate IT deliveries:
- We Must Address Testing in Low-Code Development
- Best Practices: Measuring the Business Value of Test Automation
Alternatively, look into the research from Accelerate – and the DevOps handbooks. The ripple effects of automated test execution are plenty and go beyond the math of the testing effort. One thing to keep in mind is that test automation itself is not enough. At first, you need transformational leadership.
Tester Aided in Two-Digit Million Dollar Deal
Yes indeed. It has happened for me in the last couple of months. While my role is not tester anymore (but advisor in testing) – it just wouldn’t make the headline as click-baity. Sorry for that, though it does help to prove the point that testing specialists can be a part of bids and tender teams. A testing mindset is needed even before there is an “system development life cycle”.
The Dealing
In the context of bids and tenders the testing activities are mostly about technical writing around how the testing will happen when the dealing’s done. It’s not so much about finding issues – but more about a coherent analytic viewpoint. The customer of the deal often set up “requirements” that the supplier must answer and is scored against:
- Elaborate on a test strategy
- Elaborate on the suggested test process
- Describe relevant testing documents – don’t overdo them!
- Describe testing types and environments in use
- Describe test tools and approach to automation
If you don’t reply to all “requirements” you get a sub-par score, so being able to find information in the organization is key. The contractor uploads the final documents to the customer and the content is evaluated. The evaluation is usually a balanced scoring between the individual reply documents and the price point. Often price wins, even if the scoring of the (testing) documents where at 100% score.
More and more often I see outsourcing contracts that requests 10-15 test phases. It looks like someone has simply thrown the book at it, and not considered if it is an infrastructure project, a software development project or COTS implementation or – what on earth, they actually want to learn from the testing.
So how do you go about to be in on deals like this? – business context and a seat at the table seems key.
Business Context
First of all you need to be part of a company that cater to this size of deals. The deals I have been involved recently have mostly been about national IT solutions for public and semi-public organisations. The national government rarely have their own IT divisions but hire outsourcing companies to develop new solutions, maintain existing solutions along with hosting and cloud journeys. The more you add into the deal the larger the sums rack up. And similar if it’s a eight year contract for full IT operations, devices/laptops, support and application management services – the deal sum easily ends around the ballpark amount in the headline.
It’s probably different where you live and where you work. You might work on a consumer facing app that is paid pr subscription or perhaps in a team that develop a specific business-to-business product. And that’s cool – context matters. But I know even product houses have to go out and close deals with their business customers ever so often.
A Seat at the Table
Being part of bid and tender teams can be a key role for individual contributors in the staff levels. Staff levels are the senior and principal testing roles – that do not have management responsibilities. The term comes from the book “Staff Engineer: Leadership beyond the management track by Will Larson” and well as from “The Staff Engineer’s Path by Tanya Reilly“. The former book has some excellent chapters on getting a seat at the table and staying relevant there.
While having the role does not guarantee you a seat in the bid teams, neither is a staff role a prerequisite to be in. What matters most is probably management’s willingness to step out and let the experts in on the details. This could also be work that was done by heads of testing and managers of testing. Though as manager you should really focus on servant leadership and let your testing pro’s aid in closing the deal.
#270 – But what if we can’t release often?
With digital solutions there is a ongoing urge to release often. A quest for feature toggles and continuous deliveries of new features and fixes. Automation of tedious tasks do help to drive consistent deliveries and aids in driving high-performing teams. There is good research to support that.
Recently I have worked on a solution, where the system had only to work for a month each year – and be closed down the other months. Some solutions I have similarly looked at, have years between being active. We can’t wait to ship features in the next release – the system has to be at 100% features that specific month.
Examples of business situations and domains:
- Performances, like Eurovision
- Sport events, like the Olympics
- Elections
- Black friday, Christmas shopping?
How do you test when you can only perform the act once – and if it fails it will have serious consequences? You practise and rehearse until it becomes safely repeatable. You have stage-moms and support teams to train with you.
Let’s look at the ultimate example: rock-climbing with no rope. How do you test for climbing up Yosemite’s El Capitan 3000 feet / 900 meters?
The accomplishment is more preparation than performance. Honnold climbed El Capitan roughly 50 times in the decade before his free soloing of the rock formation on July 3, 2017. While he is famous for the ridiculously fast 3-hour, 56-minute ascent, 99% of Honnold’s time on the wall was spent roped up, practicing the route. Knowing where and how to move was the culmination of hundreds of hours on that granite in advance.
The Seven Lessons From ‘Free Solo’ On Working Without A Rope
Besides the scaling of the IT infrastructure for peak load – the test strategy has to consider the fact that the event itself will be a one-off, where there show must go on – and there’s no fallback, only fall forward.
There’s a huge difference between continuously delivering web features every day in a business to consumer setting, as compared to one-off projects of migration legacy platforms. This is why my approach to creating situational aware test plans starts with looking at delivery speed:
| Rare | Regular | Often | Pervasive |
| One-off | Quarterly | Weekly | So often you dont notice |
In one of the projects I looked had we had extensive user rehearsals and dress rehearsals. Well, they are called something more IT fancy. But at the end of the day it was about training to make the performance muscle knowledge in the people performing the event. Much like the training Honnold did.
Lastly, my experience is that you get more organizational traction by aligning with goals rather than risks and issues. It’s a behavioral trick simply to talk about the thing you want to give attention. And at the end of the day the CEO wants to talk about goals rather than risks. She rather wants a successful performance with a few flaws that an delay due to bruised hands (and egos) during waxing on and waxing off.
#265 – Using MTTR to Understand When to Test
It interests me deeply to explore why testing is happening. Often it’s because some decision-maker or framework dictates – “This is the Way“. And off we go on the quest to slay the dragon – or move items from point A to point B. Without much thinking about how the side quests help to move the main risks of the story.
The main risks are usually around something irreplaceable – and hence we test and try our best to shield it. But not all risks are equally dangerous. In IT we can build implicit testing into repeatable deliveries and reduce the time to fix things. The faster things are fixed, the better is time to information for the business needs.
#263: There is a Model for your Trouble
Often directors, managers and other decision makers talks about an advanced challenge they have: What can we automate, who should automate and what tools to pick. There more and more I listen, the more I hear – they have not applied any models of the problem at hand. And there usually is a model of the problem space already. Any old model is preferable over no model at all. But it can be hard to see in the middle of chaos.
Continue readingFocus on Goals over Risks
Looking into the discussion on what goes into a Test Plan and what goes into a Test Strategy – it’s my personal opinion that we can improve our business alignment. Risk-based testing and Product Risk Analysis have been around for long – but better models have emerged to address what will be more impactful.
Continue readingStrategy is about Making a Map
Strategy is not where you are heading, but how you’re getting somewhere in the long run. That goes for all strategies, and even for test strategies. Though for test strategies we often get caught up in mechanics of selector strategies, testing types and techniques that we lose track of the higher purpose: Moving the business towards a vision.
Continue reading
Complexity is a Matter of Perspective 