Using Wardley Maps to visualize test strategies and align the test strategies with business goals.
-> Available on LeanPub: https://leanpub.com/goatsbook
Using Wardley Maps to visualize test strategies and align the test strategies with business goals.
-> Available on LeanPub: https://leanpub.com/goatsbook
For reasons of regulations and data restrictions, we were recently doing a cloud project for a large customer of ours. Cloud resources had to be spun up, granted access to, utilized, and taken down (CRUD-like). But we had to do it without granting any of the customer’s employers access to the cloud controls. They couldn’t even have a cloud admin. So for various organizational reasons, we designed the solution below.
The user would order cloud activities in a “shop” and the requests sent first to the customer ticketing system, then to our ticketing system. And finally to the cloud pipelines and scripts. And there my friend is where the testing started.
The first script was for initializing the cloud services: Ressource Name. First of I found out the (cloud) developers had already tested the happy paths. I loaded Bugmagnet and gave some funny names, as the requirements for that field hadn’t really been that specific.
I quickly learned that the cloud provider was already sanitizing the input, so no Little Bobby Tables were allowed. It made the customer think, though, and she added a rule for the shop with regard to the Ressource names. This was coded as a regular expression (Start with a capital letter, then some letters, perhaps some numbers) – not that we or the cloud cared for that strict naming scheme, it was more of a business usage rule. Was that a bug? No mostly something that testing finds – not a bug.
The next request could assign privileges to the cloud resources. Based on the resource name and Access Level, the cloud resources would be enabled. Among the access level options was “admin” rights. Which was directly opposite to the whole purpose of the project, and the customer risked breaching the regulatory inertia/controls.
And that was really the only formal bug we logged. And I don’t go home – I quickly reached out to the team, had the Accces level parameter removed and all was good. The intention was all along that only one contributor access level should be granted. That level was then hardcoded in the code repository.
And then we tested further around creating things already created, revoking resources that were not there, and race conditions for revoking the services. The only thing that really gave us a challenge was when a request failed (say you request something already there). Having the two ticking systems pick up the response and send the error message back to the users was more trick than expected as they were more REQUEST focussed – but we managed. Without creating a bug the team worked together and resolved the issues.
As always this is a story retold from actual experience, there are details I can’t share and elements I preferred to have done differently – but due to the organizational culture had its own way.
If something as seemingly simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.
Once upon a time, there was a professional chocolate shop. It was an actual chocolate shop, full of products for many needs, and a staffed counter, where you could go and order. One fine day a customer arrived and asked if the shop could create a chocolate car. The shop chocolatier said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car”. “Ah, very nice chocolate car. But please could the wheels turn.”The shop chocolatier a little annoyingly said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn”. “Ah, very nice chocolate car. But please could the doors open.” The shop chocolatier annoyingly said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open”. “Ah, very nice chocolate car. But please could it have a sunroof.” The shop chocolatier a little more annoyingly said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open, see the sunroof”. Just as the shop chocolatier was going to get even more annoyed, the customer said: “thank you, no more updates”. Baffled the shop owner forgot all about steering, ABS, and passenger seats, and asked: “How would you like it packaged”. And the customer said “ah! No need, I’ll eat it right away” And so he did. The End.
Despite all the product demos and product development, the customer consumed the product in a heart beat – or probably two.
The origin of this story is a two person sketch from my local youth work[link in Danish] in the tradition of Abbott and Costello’s “Who is on first base” – so it’s probably from the 1960’es. The sketch is usually acted with elaborate gesturing building up to the absurd punchline on the final day.
I was reminded about it recently – in a project where external reviewers kept coming back for more. More requests for bells and whistles that where not originally stated. While we did share the test approach initially, the feedback from the customer is coming towards the end of the project delivery.
All the extra effort seems absurd in contrast to communicating goals and collaborating up front. While each party might be trustworthy, it only takes one of them to extend trust and improve the relationship significantly. Extending trust would obviously defuse the absurdity of the exchange, and there would be no sketch. And we need the sketch to provide the backdrop to the morale:
If something as simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.
There are many ways for continuous customer feedback. In coding deliverables, you can demonstrate how the solution works so far and monitor the usage to get actual data on the end-user experience. It’s proven over and over again that faster feedback loops are essential to optimal deliverables – and to improve business performance. Go read Accelerate.. again! Here are some of my tricks for the non-coding deliverables as a staff-level testing consultant.
One task I have is to help others manage the testing activities. It’s usually people fresh out of Uni – as long as there is curiosity the rest can be learned. The way I usually do it is to have an intro session, break down the first few steps and then have recurring meetings to align and dig even further into the tasks. My outcome as a leader is to make them self-reliable in doing the tasks. The usual struggle is for me to sit on my hands and to pause more.
Simon Koerner suggests these five ways to lead when not in charge:
I’m currently leading a testing activity to deliver some IT services to company A. There’s a huge contract that stipulates all the mechanics of delivering these services. And for each testing activity, there is a good old-fashioned test plan document that needs to be delivered and approved by company A. (sigh)
The old-fashioned way would be for me to prepare the document in detail and forward it formally. If you have to write too much down and debate the documents over and over – it might be an organizational maturity issue, but it’s most likely a people problem and a trust issue.
Recently I have started stating that the test plan will be presented continuously. At my weekly meetings with the Customer A test responsible persons I present the current draft of the document. This way they see it being developed step-by-step and we can align – and the approval will be a formality in the end. I prefer this way of working – but have also failed where the customer expected detailed hand-overs and I preferred to work entrepreneurially to set things in motion.
When working with large IT services contracts it’s unfortunately customary that there are requirements for the content and headlines of test plans and other test documents. These content requirements are often straight out of old templates listing testing deliverables, testing types, and physical conditions for the testing activities. No test documents will be approved if there are sections missing, as the list items are contractual minimum requirements. No one wants to be non-compliant with a contract based on a missing headline in a document somewhere.
The contract authors and their lawyers would probably argue that it provides business value – at least in the sense of comparing competing companies submitting for the contract. That best-practice templates are market-leading. Yet, the case is over and over again that best practices will only give you standard solutions that are not tailored to you.
In the year 2022, it’s so awkward that you have to address physical rooms and equipment as a required element. Sure if it’s special in this case, by all means. But usually, it probably isn’t that important. Preferably we should make situational aware test plans – but if your content isn’t as innovative I recently wrote a reply in a big contract tender for customer B reply that solved this in a good way.
For the contract, customer B requested a maximum of 4 pages – where we had to elaborate on what we would advise the customer and our experience in doing so. See we don’t need 100 pages to convey a quality narrative. And based on our reply they could still evaluate and score us on how well we understood their situation.
I primarily work in situations that are less about application delivery and more about moving the whole system stacks, implementing a standard system, or similarly changing the organizational IT landscape. Some would list these projects as “staff projects” if that helps you. I often find the terms Regressiontest, SIT, and UAT to be misleading and not helpful to what kind of test is needed for my examples.
Example 1: We are rebuilding all the environments in the delivery stack. All the Dev-, Test-, Integration-, Preprod-, and Prod- environments, and their underlying databases, brokers, and websites. Every time we are constructing, an environment we will be testing the setup from a baseline version of the existing running systems. A baseline that we know is functional already. The classic software development testing types don’t really help us in this situation, as neither regression test, UAT, or SIT conveys the things we want to confirm and the learnings we want to explore.
Example 2: We are setting up a new expense platform for employee reimbursements to go live with a new branding of the company. It’s a SaaS system and we load it every month with data about the organization. So while it’s needed for various purposes – the risk is low and the mean time to repair is similarly low. The testing we will do will be a limited confirmation of an initial data load. Snow-plow style – not a full system-integration test and similar user-acceptance test. After all, this is a SaaS – not a custom solution. It’s OK to shift right.
SIT and UAT has become generic term that it has lost the strength to convey the needed quality narrative. If you do CI/CD (which you should) for your application development that such be sufficient. If you figure out, you need to do a “connection alive” test for your third-party integrations that you move from one environment stack to another that should be accepted with the acknowledgment that you actually considered the challenges ahead.
It’s all about the risks and the mitigations – less about testing everything to the dot. One tool to read the landscape is to communicate curiously about what the stakeholders value more – and value less. And on the other hand, consider the nature of the solution being proposed.
Example 3: Setting up a cloud-based Azure Active Directory – the solution comes with a given security level out of the box (OOTB). As with other OOTB and Software-as-a-service solutions you have little impact on the security features of the solution, besides some simple configurations. While you might think that all security requirements would require 100% acceptance testing coverage, what you want to accept is that they might be provided “by design” – or by a solution decision made long ago.
I would prefer that we can call things what they are and not blindly apply old testing types
A “testing mindset” of inquiry applies when you are a principal tester or even a senior advisor in testing. Asking open questions and approaching a challenge with exploration proves better than command and control. People are people – be mindful of where they are.
Currently, I’m involved in an extensive change program for a company. It’s not your usual agile feature factory but more about the technology stack needed to run a 3000 people company (payment, finances, etc). While testing professionals could be onboarded – their learning curve would not scale – not even being equipped with all the heuristics and testing vocabulary in the world. As it is a one-off, automation wouldn’t scale either.
Fortunately, most applications have a system owner (or product owner, or manager in charge) and usually a team around maintaining the application. To some degree, we could frame these teams as stream-aligned teams. The experts in testing the applications are the superuser – hence they do the testing.
Trust must be extended before it is givenRachel Happe – @mastodon.social/@rhappe
One such team is the XYZ team anchored by a VP-level manager. A colleague and I had been trying to communicate with them to align on the testing needed but had met reluctance. They would not have time to help us. We discussed the matter and realized that we needed to approach them differently. Since the change program was such a fundamental shift for the team, they had already considered the implications. They were already testing and thinking about the impact. We just had to trust them – and build on what they already had instead of imposing specific ways of working.
Without requiring them to earn it first, tell everyone who works for you that you #trust them implicitly. From that point forward, just ask them to never give you reason to withhold it. (In my experience, most won’t).Mark C. Crowley
Communicating curiosity can be a little thing like “Remind me, how does XYZ work” or simply “sitting on your hands” during meetings that you would previously fill with questions and requests. The tables of the great program test lead are turning, it’s about being an enabling function for others to succeed. Similar to the freedom you should give your growing up kids – it’s about leadership.
… one piece of the time. Really. Don’t worry about it – it’s similar to eating an elephant.
While Discord and Slack sites can be places to go for communities with the faill of Twitter – they are closed systems. You need an invite and communication closed within the instance. This is good for closed discussions you want to gatekeep. Mastodon seems to be the place for serendipity and cross-collaboration across topics and communities. It’s not new, the framework started in 2016 gaining 1 million users in 2017. There are a number of great and extensive guides – one of them is fedi.tips. Another great guide is Mastodon Lessons Learned by Lisi Hocke (@firstname.lastname@example.org).
Hi I’m Jesper🧷🌻🐐📕🗺🇩🇰 (@Jo2sn@mastodon.social)
For Mastodon, you create accounts on a server (or instance) in the Mastodon-based network. Through a shared protocol the people on the servers can communicate with posts, threads, likes, and resharing as known from other platforms. Mastodon is really the software framework (on GitHub) and the network called “Fediverse” – but people mostly call it “on Mastodon”.
Someone wrote that picking a Mastodon Server feels like selecting a character in a new video game. You don’t really know what you’re looking into and how you will play the game. Luckily you are not fixed to one account on the Mastodon network. You can have multiple and easily redirect from the old to the newer. I have tried jumping profiles myself, as the first one I made was on a server with too few general topics. One of the elements of Mastodon is that each server has a timeline – besides the ability to communicate across.
Some servers are topic-based on Open-source software, others on art and LGBT, you can pick a specific server and expect the local content mostly on that topic. Or you can pick a general-purpose server – if one topic doesn’t define you. There are some tools, like debirdify that can help you in the process – but not yet a 1-click tool to move over.
Two ways to select a server:
People can and will move to many places – there is yet to be a primary test community instance.
As there is no algorithm to drive engagement, so use hashtags and reshare content from others. Likes are only posted to the author – so reshare/retoot great posts. Similarly to following people you can follow hashtags like #MinistryOfTesting and #TwitterTestingPeople. The culture of Mastodon is to be inclusive. Use Content-Warnings and hide images as a common rule – and ALT-text on images as a general rule.
Mastodon has free apps (Android, iOS) for casual use and some pay-to-use apps. Currently, it seems you can’t follow hashtags in the iOS app, so I tend to use the browser more. There is a range of choices – see what bests fit you. You can get a long way with just the primary site.
Servers are run by small teams of admins, so in peak periods there might be suboptimal performance. Most servers seem to be funded by patreons or similar fundraising.
In theory, an instance could close down and take everything away. If you have content that is more permanent and you want to own, put it somewhere you control. The same challenge is true for ad-based social media (Facebook) and subscription-based platforms like Medium.
If you feel like organizing the content coming your way, enabling the advanced web GUI setting will give you columns like “TweetDeck” – but the setting is global for your login. Sometimes I just want a quick read over the timelines, other times I want to monitor specific topics. To monitor topics I have set up columns in Sengi https://sengi.nicolas-constant.com/. It’s a per-browser setting, so it’s more of a topic radar that can be different across my devices. Personal lists are a feature too, but it seems based on people not so much on topics.
Do notice that direct messages aren’t private messages. All the people you tag can read the messages. So don’t tag the people you want to rant about.
The visibility of a toot/post follows the flowchart below.
As mentioned I have written a book. Looking through my notes I have around 30 versions, one for each significant session of working on it. I have been asked to share tips on writing a book, so here you go, Simon. The key lessons are:
One of my book’s themes was moving something from a hunch to a hard truth. And the same really applies here. When I finished my master’s degree while working (back in 2002), I had two slots a week at a “study office”. From that, I learned that not all study days are equally productive “on paper” – but that’s ok. Each session had its purpose.
Similarly for my book project. I set a weekly evening booking in my Calendar – family chores were arranged around it. After dinner, I work start working on the book and work for 2-2½ hours. I used my personal computer in my work-from-home setup.
There is so much great content out there already, my focus was on writing about my experience and my vision of better test strategies. But to do that I needed to stand on the shoulders of others to set the scene and describe the techniques I applied.
While writing I did want to bring in loads of existing content to elaborate and provide a foundation for my thinking. While editing I removed most of it, partly because I didn’t want to sound like a high school book report (thank you for that one Tristan). What I did leave in were quotes, recommendations, and listings of the work of others. The book is full of footnotes directly on the page (as compared to end notes) to highlight everyone in context.
Initially, I outlined the chapters and subchapters and it was important to get the right “flow” and storyline into the content. My base model was inspired by “Situation, Complication, Question, Answer” from the guide “How to present to executives” (StaffEng Book) and similar lessons on taking the first steps.
One thing I have worked a lot on is the flow of the text: Sections would be 4-6 lines long, with empty lines in between. I also worked to reduce “dangling lines”, so no two lines would linger into the next page. No pages should be text only, so quotes and illustrations are important for readability – as well as making the book content visible. Lastly, I worked a lot on having one section end with words that tap into the next section.
The below recommendation is cool, I could have done that too.
Often I started my “book evening” by reading the book from the start all over again. As I’m not a native English writer, I installed Grammarly and paid for the premium version for a period to rewrite phrases that were in a passive form. Fun Fact: the opposite of passive voice, is not an aggressive voice but an active voice.
The tool you use to write your book isn’t that important. I used Google Docs with embedded Google Drawings, others might prefer Word or other editing platforms. The biggest challenge in publishing was getting the embedded pictures in a resolution that could be printed by the print shop. The book exists in a printed format – but it’s primarily an electronic book.
Self-publishing on LeanPub is intuitive and easy, it also has tools for incremental versions, previews, and updates. Let people sign up for it in advance with price suggestions. That allows you to set a price on the book based on audience-based quotes.
They also have a great system for coupons. I have coupons for those that helped in preparing the book, those I have referenced, and those that participated in the virtual launch. Let me know with the phrase “LeanPubbing” if you want a coupon too.
In my book “Goal-Aligned Test Strategies,” I draw extensively on the work of others. This is a reference post to further credit my reviewers and to list the 112 references in the book.
Special thank you to Lonnie for my writing evenings and to Simon Wardley for finding a path.
Recently I had the chance to apply my own templates to myself and my active project – as I had to mentor a new test manager. I was challenged in explaining how I read the upcoming IT environment project. After looking into resources for new test leads, I realized I could take my own medicine.
A year ago, I created a new test plan format – the Situational Aware Test Plan. While mind-maps and one-page test plan canvases exist, I wanted to elaborate using the evolution principles from Wardley mapping and stop writing test plan documents.
The table structure is there to provide guard rails for the elaboration. I will use the Darlings, Pets, Cattle, and GUID -mnemonic as headlines. Our strategic decisions emerge as we use the worksheet based on the current situation and state. The strategies will be the decisions to push a field in the grid to another state.
Delivery and Situation
|New project||Fixed date|
|Existing delivery speed||Scheduled|
|Test Environments, internal||Repeatable|
|Test environments with integrations||Crafted||Some existing know-how|
|Environment Infrastructure||Hosted data center practices|
|Test data||Known but cumbersome|
While this project introduces new test environments, there is an existing environment with a quarterly delivery pace. This is a classic example of the core chronic conflict of pursuing both: responding to the rapidly changing competitive landscape and providing stable, reliable, and secure services (DevOps handbook introduction xxv) as elaborated on Align your Test Strategy to your Business Strategy.
The test team allocated beside me and the new test lead is a new junior and senior tester. We are in the same team, and most are even in the same office. So collaboration will be collaborative and pervasive, with a focus on helping the new people grow.
The test team
|Test team collaboration||Growing||Pervasive|
|Domain know-how||Getting there|
Test tools and approach
|Test activity||Explore integrations||Confirm internal requirements|
|Test cases||Existing can be updated.|
|Test case repro||Create new repository|
As mentioned in the blog post about visualization, we can now use the map to discuss why we need CT and ET for the project. Based on the project’s layout, I would advise having an expert exploration of the integrations and more standard scripts for the known construction of the internal environments.