What About Expected Results?

TL;DR: We should know better than to require expected results in test case steps.

Welcome to 2022 – where old wisdom surfaces yet again. Recently I considered expected results in test cases. It seems the topic is still relevant – I wrote about it in “The Expected” back in 2015.

As always we have to backup a bit and consider: What purpose does the detailed steps have to our stakeholders. How visible is the value provided? We have to start from their viewpoint – not build a chain of reasoning up from our own wishful thinking.

Currently I’m working on a project for payments of benefits. We have daily CI/CD and loads of classic manual testing. The key message from the stakeholders is that they expect things to be tested and evaluated. They generally trust us and collaborate with us on the work, so there is not so much discussion on the details as in other projects. Yet the discussion of expected results did come up in our internal dialog: What is the reasoning behind expected results on the test case steps – besides coming from an old book.

Testing is a performance

One of the usual arguments is that it enables hard-over from person to another. While handover can be relevant to consider if the team members changes often. It seems to me just to create redundant information based on requirements and other oracles. I mean where does it it end, why should we add every project information into the test cases and into the test plan. Stop it already.

Let the test cases be evaluated by the person performing the test – to see the whole picture. And not get biased by expected results. I want the persons performing the tests to use their brain and based on the oracles available evaluate the system under test. Also if everything can be planned – why would there be a need for retakes of movie scenes? It turns out even simple instructions are not possible:

Exact Instructions Challenge – THIS is why my kids hate me. | Josh Darnit

Let people do what they are best at

Let’s have testing being a balanced approach, where people evaluate and scripts confirm.

When the confirmation can be determined by an algorithm or by assertions towards an oracle we can code a test for it – and test it continuously. Adding as much automation as (practically) possible, aids the delivery in multiple ways. It aids to build regression testing and help us test that things not only worked once somewhere, but that it works again and again.

On the other hand when there’s no algorithm or explicit oracle in the form of specifications, acceptance criteria or similar authority we can only do a subjective evaluation: at that point in time – given the information available. do not neglect the wisdom of subject matter experts testing.

And in the later situation expected results hinder the exploration of whether the system under test fulfills not on the explicit requirements but also the implicit. We really should know better.

Photo by Karolina Grabowska on Pexels.com