What About Expected Results?

TL;DR: We should know better than to require expected results in test case steps.

Welcome to 2022 – where old wisdom surfaces yet again. Recently I considered expected results in test cases. It seems the topic is still relevant – I wrote about it in “The Expected” back in 2015.

As always we have to backup a bit and consider: What purpose does the detailed steps have to our stakeholders. How visible is the value provided? We have to start from their viewpoint – not build a chain of reasoning up from our own wishful thinking.

Currently I’m working on a project for payments of benefits. We have daily CI/CD and loads of classic manual testing. The key message from the stakeholders is that they expect things to be tested and evaluated. They generally trust us and collaborate with us on the work, so there is not so much discussion on the details as in other projects. Yet the discussion of expected results did come up in our internal dialog: What is the reasoning behind expected results on the test case steps – besides coming from an old book.

Testing is a performance

One of the usual arguments is that it enables hard-over from person to another. While handover can be relevant to consider if the team members changes often. It seems to me just to create redundant information based on requirements and other oracles. I mean where does it it end, why should we add every project information into the test cases and into the test plan. Stop it already.

Let the test cases be evaluated by the person performing the test – to see the whole picture. And not get biased by expected results. I want the persons performing the tests to use their brain and based on the oracles available evaluate the system under test. Also if everything can be planned – why would there be a need for retakes of movie scenes? It turns out even simple instructions are not possible:

Exact Instructions Challenge – THIS is why my kids hate me. | Josh Darnit

Let people do what they are best at

Let’s have testing being a balanced approach, where people evaluate and scripts confirm.

When the confirmation can be determined by an algorithm or by assertions towards an oracle we can code a test for it – and test it continuously. Adding as much automation as (practically) possible, aids the delivery in multiple ways. It aids to build regression testing and help us test that things not only worked once somewhere, but that it works again and again.

On the other hand when there’s no algorithm or explicit oracle in the form of specifications, acceptance criteria or similar authority we can only do a subjective evaluation: at that point in time – given the information available. do not neglect the wisdom of subject matter experts testing.

And in the later situation expected results hinder the exploration of whether the system under test fulfills not on the explicit requirements but also the implicit. We really should know better.

Photo by Karolina Grabowska on Pexels.com

#270 – But what if we can’t release often?

With digital solutions there is a ongoing urge to release often. A quest for feature toggles and continuous deliveries of new features and fixes. Automation of tedious tasks do help to drive consistent deliveries and aids in driving high-performing teams. There is good research to support that.

Recently I have worked on a solution, where the system had only to work for a month each year – and be closed down the other months. Some solutions I have similarly looked at, have years between being active. We can’t wait to ship features in the next release – the system has to be at 100% features that specific month.

Examples of business situations and domains:

  • Performances, like Eurovision
  • Sport events, like the Olympics
  • Elections
  • Black friday, Christmas shopping?

How do you test when you can only perform the act once – and if it fails it will have serious consequences? You practise and rehearse until it becomes safely repeatable. You have stage-moms and support teams to train with you.

Let’s look at the ultimate example: rock-climbing with no rope. How do you test for climbing up Yosemite’s El Capitan 3000 feet / 900 meters?

Photo by Tobias Aeppli on Pexels.com

The accomplishment is more preparation than performance. Honnold climbed El Capitan roughly 50 times in the decade before his free soloing of the rock formation on July 3, 2017. While he is famous for the ridiculously fast 3-hour, 56-minute ascent, 99% of Honnold’s time on the wall was spent roped up, practicing the route. Knowing where and how to move was the culmination of hundreds of hours on that granite in advance.

The Seven Lessons From ‘Free Solo’ On Working Without A Rope

Besides the scaling of the IT infrastructure for peak load – the test strategy has to consider the fact that the event itself will be a one-off, where there show must go on – and there’s no fallback, only fall forward.

There’s a huge difference between continuously delivering web features every day in a business to consumer setting, as compared to one-off projects of migration legacy platforms. This is why my approach to creating situational aware test plans starts with looking at delivery speed:

RareRegularOftenPervasive
One-offQuarterlyWeeklySo often you dont notice
A scale of delivery speed

In one of the projects I looked had we had extensive user rehearsals and dress rehearsals. Well, they are called something more IT fancy. But at the end of the day it was about training to make the performance muscle knowledge in the people performing the event. Much like the training Honnold did.

Lastly, my experience is that you get more organizational traction by aligning with goals rather than risks and issues. It’s a behavioral trick simply to talk about the thing you want to give attention. And at the end of the day the CEO wants to talk about goals rather than risks. She rather wants a successful performance with a few flaws that an delay due to bruised hands (and egos) during waxing on and waxing off.

#269 – We Have Outsmarted Ourselves Again

Links from my talk at “map-camp-use-case-edition” Nov 2021:

Blogposts:

Photo: Kimiya Oveisi on Unsplash, https://unsplash.com/photos/rzsBKBb96HA

#268 – Who Brings in New Knowledge?

Well, if you are reading this – there’s a good chance it’s you. Especially if you read this with the intention of sharing this with your team. I hope you do, obviously 😊. But perhaps it’s unclear whose responsibility it is, to bring new knowledge to the team. Is it always the team manager job – or is it a dedicated person that by role, or by habit, that bring in new knowledge?

Photo by William Fortunato on Pexels.com
Continue reading

#267 – Reminder – it’s about Story Shaping

In agile delivery teams, it’s recommended that stories and features are discussed before being developed. A checklist called “Definition of Done” can be used as a reminder to check that the delivery team has everything needed to prepare the delivery of the feature. A key ingredient is a shared understanding of how to confirm the delivery – for the whole team.

Story shaping requires at least three people to get together. They are not amigos nor amigas. They are often three, sometimes two people, and often more. They represent the different viewpoints needed to deliver: builders mindset, testing mindset, security mindset, operations mindset, business mindset, etc.

No one person can hold all these viewpoints – at least a builders mindset and a requester/business mindset is needed.

If you skip doing story shaping your acceptance criteria and testing activities will be misaligned. And you will either have overdone the effort needed or underestimated the challenges of the story.

Thank you to all who have replied to the tweet below.

One customer, three people to do the story shaping

#266 – Retraining Yourself for the Future

The local university college has a big sign outside saying: “Retrain Yourself for the Future“. The thing is – what is offered is at best years behind the current practice in the industry. If you certify towards a body of knowledge, you forget about the learning journey and only on the outcome. Innovation yet happens in many ways – and industry practitioners are only one form to train for.

Continue reading

#265 – Using MTTR to Understand When to Test

It interests me deeply to explore why testing is happening. Often it’s because some decision-maker or framework dictates – “This is the Way“. And off we go on the quest to slay the dragon – or move items from point A to point B. Without much thinking about how the side quests help to move the main risks of the story.

The main risks are usually around something irreplaceable – and hence we test and try our best to shield it. But not all risks are equally dangerous. In IT we can build implicit testing into repeatable deliveries and reduce the time to fix things. The faster things are fixed, the better is time to information for the business needs.

Grogu agrees
Continue reading

#264 – Create Situational Aware Test Plans

From the endless discussions on the proper content and contexts of a test plan, it’s apparently still needed – but what goes in it? Let’s create situational aware test plans inspired by Wardley Mapping.

ISTQB template-based test plan documents are in my personal opinion no longer industry best practice. First of all it’s bloatware. While they intend to be a springboard into considering what is relevant we have ended up with 8 page templates – where every single of the 20 topics are required information. While it looks dazzling – it’s like frosting puffed with empty calories.

What most people delivering effectively software are using is 1) modern test automation and 2) modern test case management tools to lead and manage the test activities. And there is clear research on what 24 factors correlates to high-performing teams. It seems to me the templates have been frozen stale since 2012 – and are hindering us more than helping.

Continue reading

#263: There is a Model for your Trouble

Often directors, managers and other decision makers talks about an advanced challenge they have: What can we automate, who should automate and what tools to pick. There more and more I listen, the more I hear – they have not applied any models of the problem at hand. And there usually is a model of the problem space already. Any old model is preferable over no model at all. But it can be hard to see in the middle of chaos.

Continue reading

Hunches and Hard Truths

Recently I was in a network call on the use of automation and machine learning in detection of skin issues (EDB 5.0 in Danish only). Similarly I was reading about automation in the legal space. Both these stories align with the struggles we see in the discussions around how much we can automate. We can model it on this simple continuum between hunches and hard truths:

Continue reading