On the Shoulders of 112 Giants

In my book “Goal-Aligned Test Strategies,” I draw extensively on the work of others. This is a reference post to further credit my reviewers and to list the 112 references in the book.

Special thank you to Lonnie for my writing evenings and to Simon Wardley for finding a path.

Reviewers and Feedback

Referenced Quotes

“Making work visible is one of the most fundamental things we can do to improve our work because the human brain is designed to find meaningful patterns and structures in what is perceived through vision. Thus, it makes sense that we have difficulty managing our work when we can’t see it.”

Dominica DeGrandis, Making Work Visible: Exposing Time Theft to Optimize Work & Flow, 2017

Today, we are offered a bewildering variety of tools and concepts to aid in analyzing and constructing strategies. Each of these tools envisions the challenge slightly differently. For some, it recognizes advantages; for others, it understands industry structure. For some, it identifies important trends; for others, it erects barriers to imitation. Yet, there is a more fundamental challenge common to all contexts. That is the challenge of working around one’s own cognitive limitations and biases—one’s myopia. Our myopia is the obstacle common to all strategic situations.

Good Strategy/Bad Strategy: The difference and why it matters” by Richard Rumelt, 2017

“Take your strategy. Apply it to 3 random companies. If it sounds like it would be something they would want (“more money,” “more customers,” etc.), then it’s not a strategy,”

Adrian Howard, 2022

“Instead of well-designed strategies plotted out in sealed boardrooms, executives needed strategies dynamic enough to receive market feedback and flexible enough to evolve.”

How to Make Big Old Companies Act Fast, Pia Lauritzen 2021

“Anticipate your boss’ needs before they do. Help them with prioritization as much as possible. In the same way, they shield you from upper management, shield them from details that are not important – a boss that wants insights into every detail needs micromanagement coaching, and you can help too.”

Tristan Lombard, 2022

The way we have always thought about strategy is very centralized and top-down. That’s controlling and directive – and leads to very detailed strategic plans. We need to move towards something lighter driven by a shared purpose.

Rachel Happe, Control Is For Amateurs: Empower The Community To Change The Culture In Your Organization, 2018

“When you learn Wardley Mapping, you may build better situational awareness than your boss has and a better one than anyone in your time. Unfortunately, you are doomed unless you know how to sell your strategy to other people and whether you can sell it in the first place. Nobody will act on your thinking.”

Krzysztof ‘Chris’ Daniel: You need to sell your strategy! 2022

Referenced Links

Referenced Books

Blog posts

#270 – But what if we can’t release often?

With digital solutions there is a ongoing urge to release often. A quest for feature toggles and continuous deliveries of new features and fixes. Automation of tedious tasks do help to drive consistent deliveries and aids in driving high-performing teams. There is good research to support that.

Recently I have worked on a solution, where the system had only to work for a month each year – and be closed down the other months. Some solutions I have similarly looked at, have years between being active. We can’t wait to ship features in the next release – the system has to be at 100% features that specific month.

Examples of business situations and domains:

  • Performances, like Eurovision
  • Sport events, like the Olympics
  • Elections
  • Black friday, Christmas shopping?

How do you test when you can only perform the act once – and if it fails it will have serious consequences? You practise and rehearse until it becomes safely repeatable. You have stage-moms and support teams to train with you.

Let’s look at the ultimate example: rock-climbing with no rope. How do you test for climbing up Yosemite’s El Capitan 3000 feet / 900 meters?

Photo by Tobias Aeppli on Pexels.com

The accomplishment is more preparation than performance. Honnold climbed El Capitan roughly 50 times in the decade before his free soloing of the rock formation on July 3, 2017. While he is famous for the ridiculously fast 3-hour, 56-minute ascent, 99% of Honnold’s time on the wall was spent roped up, practicing the route. Knowing where and how to move was the culmination of hundreds of hours on that granite in advance.

The Seven Lessons From ‘Free Solo’ On Working Without A Rope

Besides the scaling of the IT infrastructure for peak load – the test strategy has to consider the fact that the event itself will be a one-off, where there show must go on – and there’s no fallback, only fall forward.

There’s a huge difference between continuously delivering web features every day in a business to consumer setting, as compared to one-off projects of migration legacy platforms. This is why my approach to creating situational aware test plans starts with looking at delivery speed:

RareRegularOftenPervasive
One-offQuarterlyWeeklySo often you dont notice
A scale of delivery speed

In one of the projects I looked had we had extensive user rehearsals and dress rehearsals. Well, they are called something more IT fancy. But at the end of the day it was about training to make the performance muscle knowledge in the people performing the event. Much like the training Honnold did.

Lastly, my experience is that you get more organizational traction by aligning with goals rather than risks and issues. It’s a behavioral trick simply to talk about the thing you want to give attention. And at the end of the day the CEO wants to talk about goals rather than risks. She rather wants a successful performance with a few flaws that an delay due to bruised hands (and egos) during waxing on and waxing off.

Strategy is about Making a Map

Strategy is not where you are heading, but how you’re getting somewhere in the long run. That goes for all strategies, and even for test strategies. Though for test strategies we often get caught up in mechanics of selector strategies, testing types and techniques that we lose track of the higher purpose: Moving the business towards a vision.

Continue reading

3 Visual Models on Quality

Besides the visualization of the delivery pipeline it can be a good idea to visualize your quality model. There’s more to it than V-model, Agile/Scrum and other old models. The following models are at least as established and practice based as any old book

Continue reading

Visualize the Test Strategy

There are plenty of models/frameworks that list what you could have of testing activities – but there seems to be little assistance in WHAT to do in which situations. I have two visual ideas – let’s explore how to visualize the test strategy. And yes, there’s no size fits all – context matters.

Continue reading

To Transform Testing

There is no doubt that our long lived testing narrative is under pressure. To continue to bring business value, both the IT field and testing is transforming to be about proactive business enabling.

The IT domain is currently buzzing with the word “IT transformation” – the idea that IT services should be more about “inspire, challenge and transform the digital businesses“. That it should be less about delivering IT products and artifacts and more about enabling digital business outcomes. Even for testing – it should be less about a product/service, and more about business necessity. As Anders writes:

Stop focusing on the things that bug people, and dare to do both IT and testing smarter and more business focused from the start. Build Quality in – smarter from start. That goes for IT services as a whole, and definitely also for the testing activities.

What can you do to transform your testing? I have three areas:

Discuss Business Strategy

Learn Wardley mapping – and use it like Chris McDermott to create context specific maturity models with Wardley Maps informed by Cynefin. Use the mapping to Broaden the scope of the system under test.

Align with the Business Strategy

Leading Quality [Cummings-John, Peer 2019] has a whole chapter on “Align your team to your company growth metric“. Consider if the company you work for is driven by Attention, Transaction or Productivity metrics, and arrange your test activities accordingly.

Dare to Deliver in New Ways

We are usually talk so much about optimizing the (IT and testing) delivery, that we forget other ways to be innovative and provide business enabling. One way could be to dare use new technology like RPA or a HoloLens to support tedious tasks in testing – to use an existing product to something new. Another approach to actually test “all the things” that matter or to apply testing to IT outside the realm of application delivery.

To Transform Testing I will discuss, align and dare so that test solutions can be proactive business enablers – (not only achieve shippable quality).

Mapping Mondays – Pioneers, Settlers, Town Planners

A Ratio between Tests

During one of my recent projects I was considering the ratio between the checks and the tests – that is the ratio between those tests, that are primarily simple binary confirmations, and those tests that are more tacit questions. This blog is about my considerations on the idea/experiment/model.

First I observed, that we have a range of different items in our requirements – some of them are [actual copy of the current specification]:

Binary Confirmations

  • It must be possible to add a customer ticket reference
  • It must be possible to copy the ticket number

Tacit Questions

  • You must be able to navigate displayed lists easily
  • It must be easy to compare work log and audit log

You could argue that they need refinement, more testability and less “easy“. But this is what we have to work with for now. Even if we had all the time in the world (we don’t) – we would not be able to write all of the requirements in a perfect form (if such a form exists).

As the system under test is a commercial standard system, some of the requirements are even given as “Out of the Box”, we will probably not even be testing those explicitly. Our coverage criteria is not ALL OF THEM.

Ordering the tests

It is a deliberate experiment from my side to divide the requirements (and hence the tests) into the piles of Closed and Open Questions. Perhaps there is even three piles – Rapid Software Testing has: human checking, machine checking and human/machine checking , Wardley has Pioneers, Settlers and Town Planners. Perhaps the Rule of Three applies here too.. perhaps it’s an continuum … let’s see.

Perhaps it’s a continuum

As part of the requirement workshops I will label the requirements and align with the stakeholders to get the expectations right – with the help of a few friends. This a context/project based “operationalization“.

I wrote about this ratio on my blog post around the Test Automation Pyramid, as I will use the labels to automate the confirmations (and only the confirmations). The assumption is, that there are significantly more of the binary requirements tested by machine checking – and more human tested tacit questions. If we can get all the tedious tasks automated – that is the really the end goal.

Automate all the things that should be automated

Alan Page

Every project/context will have it’s own ratio, depending on a range of factors. Saying there should always be more of one type than the other would not hold. As the above project is the configuration and implementation of a standard commercial business software package (like SAP, SalesForce etc), my expectation is that most of the requirements are binary. Also considering that this project is heavy on the right hand side of the Wardley Map scale of evolution.

It’s a Reduction in Context

I am well aware that the two/three piles are an approximation / reduction. Especially when looking at the “binary” requirements and “only” testing these by confirmation. They could as easily be explored to find further unknown unknowns. If we prioritize to do so – it all about our choice of risk.

It is also an limitation as “perfect testing” should consist of both testing and checking. I factor this into the test strategy, by insisting that all of the requirements are tested both explicitly and implicitly. First of all most of our binary requirements are on the configuration and customization of the out-of-the-box software solution. So when the subject matter experts are performing the testing of the business flows, they are also evaluating the configuration and customization. And I do want them to spot when something is odd

The binary configuration is ok, but human know-how tells us otherwise.

Ultimately I want to use the experts to do the thinking and the machines to do the both the confirmations and the tedious tasks.

Broaden the scope of the SUT

When testers talk about SUT (System Under Test) there seem to be an implied context of it being software, developed, bespoke software to be specific. Let me broaden the notion of a SUT using Wardley Maps and with that illustrate how testing can add value across the board.

Bespoke software (aka Custom built) is where the solution (SUT) is built and maintained tailormade to a specific company by a specific team that answers to order giver. When you build and maintain an app or web site for a company and is embedded in the team delivering the code base – it’s usually in a bespoke context.

Experiment/emerging example: Built an internal web site to do some simple public service case management. Write it based on MicroSoft .Net and IIS technology. The solution is new and novel, so interaction with the user is important.

All the commotion on buiding MVP experiments and interaction with the Product Owner are usual symptoms of a genesis situation. As the processes mature and products emerge, the solution development becomes more an customization activity.

Customization example: Implement Dynamics 365, SiteCore, SalesForce etc – but taylor and code them to your specific purpose. I have worked in a project taking Dynamics 365O and creating custom forms to handle public sector health events.

The last class of software “development” is the pureplay configurations of standard solutions. This is the context of SaaS – pay the license and get started. Think SAP or Office Applications, anything that is so accepted that it’s almost free (OpenOffice) and kills the IT department.

Let me draw this on the axis of Wardley Maps Evolutions:

Similarly we can add the underlying infrastructure to the drawing. As solutions move to the cloud and infrastructure becomes code, the system under test could very well be code around infrastructure. Initially bespoke infrastructure experiments (in perl?), and as time moves – even infrastructure becomes a commodity in the form of Amazon S3.

So where is your SUT? – what is the path down the stack? Because there is a huge difference between testing custom code for cloud services, as compared to product customization on actual physical owned hardware.

Let’s think testing outside the bespoke areas on the map too. Some current examples I am working on are:

  • Infrastructure transition of 700 serves from being owned to being hosted
  • Application transition of 50+ applications from being owned to being outsourced
  • Transformation of a standard form management solution
  • Implementing a standard system for ITIL case management

These projects have no code, the SUT is either a server, an environment (collection of servers), a form, a process or something else. While we do know a lot about testing in bespoke software contexts, the practices for testing in transition and transformation are emerging practices! This gives us and extra layer. And this is where it gets interesting.

There are plenty of standard practices (SAFe, agile..) but the practices for testing in the context of transition is yet to materialize.

The same model can be applied to IT as a whole. IT support and end user computing (devices, desktop operation) are to the very far right as commodity services. While on the far left is the constant experimentation and tinkering (of AI, ML and RPA) to become actual products.

If we only see testing as part of building bespoke software we fail – we fail to see the horizontal and vertical contexts, where the testing disciplines can add similar value and impact.

When testing web features doesn’t matter

(aka not every testing problem can be solved by a webdriver)

Web features doesn’t matter as much in the contexts I usually work in. While some may be delivered over the web, the focus for testing is on the whole system’s fit for the business. Adding automation in testing to the mix gives additional challenges as there is no source code in the solution to interact with, and we have to find other solutions to solve the tedious tasks in testing with.

One area where this is the case, is when implementing standard commercial software packages (COTS) for the enterprise or public sector. Solutions like SAP for retail CRM and ERP, Microsoft Dynamics 365 for Finance and Operations, EPIC for hospitals, Service Now for IT Service management etc. These are standard solutions that can be configured and customized, but the general source code is not available.

Thus the “test automation pyramid” falls short to help us automate things as only the GUI is available for interacting with the solution. Test engineers might want to setup CI/CD but the success of that depends on the system architecture and the provided as-is methods of releasing updates. Some of the solutions above are provided as SaaS but quite often they still run “on premise” and the business still wants releases tested before launching things on a corporate scale.

Screenshot of an RPA test on SAP

Another example is the many bespoke software solutions that are still in operation. My local electronics store has two interfaces for the sales persons: web to look things up (specs and availability) and a mainframe system to set up the actual purchase (Point of sales, POC). Many public organisations and enterprises are only now transitioning from the desktop applications of the 1990’es to more up to date solutions. Unfortunately systems that are 10+ years old have very little of live and relevant specifications and neither CI/CD suites.

While some COTS and POC solutions are being delivered over the web, testing web particularities the very least of our focus areas. The web particularities seems to matter more if the solutions are business-to-consumer but not so much when it’s business-to-employer or business-to-business.

In a business-to-employer and business-to-business context, usually only one browser is in scope. And that’s it. There is little interest for HTTP status codes, broken links, browser compatibility or login forms.

The primary testing challenges of these projects cannot be solved by Selenium, Cypress or neat tweaks in the latest JavaScript library.

Focusing only on testing the web in contexts like these we fail in

  • covering the whole system landscape across applications of different technologies
  • addressing the real questions of the business subject matter experts and the business

It’s in this context that RPA probably has some benefits in providing automation of tedious testing tasks to the tester with a business background. That is, they are business people first – and then they do the testing that matters to the context.

Don’t request the kitchen sink

More and more often I see outsourcing contracts that requests 10-15 test phases. It looks like someone has simply thrown the book at it, and not considered if it is an infrastructure project, a software development project or COTS implementation or – what on earth, they actually want to learn from the testing.

These are the phases of a recent project:

  • Environment Acceptance testing
  • Hardware and integration testing
  • Component testing
  • Component- integration testing
  • Installation test
  • System testing
  • Functional testing
  • Regression testing
  • Security testing
  • Performance testing
  • Operational acceptance testing
  • Service Level testing

It’s a challenge in the vendor reply. The vendor will want to reply to all test phases in order to be compliant with the tender, and not loose points. There is no room for elaboration or discussion if you want in on the bid process.

Quite often the requester comes back and say “we didn’t want all those weird testing things, we just want something that works for us”. But when the contract is signed and the work set in motion the project team have challenge to make the testing practical within the framework of the contract. This goes from both sides. Many good hours can be wasted with unwinding cumbersome contractual terms.

What I usually do in such a situation is to bundle the contract’s testing scope into fewer activities, and setup a mapping so that everything is covered. That is if the client allows me to make the activities practical and context-driven. If not – my hands are tied, and we deliver according to spec – even if the chapters of the test plans are set in stone.

Let’s work towards better deals for testing activities. If you are looking to prepare a BID include a test manager – and have a discussion of the value-add and learning of testing up front. There is no one book of how to do testing. Instead spend the time and money figuring out your context. Figure out what phases are on the client side, and what is on the vendor side. Have a test management consultant on retainer for before and after the bid process. Do something to discuss your test strategy and put the guidelines in the contracts, so that the vendors can propose a solution.

Don’t request everything and the kitchen sink too

Everything and the kitchen sink
Everything and the kitchen too