What About Expected Results?

TL;DR: We should know better than to require expected results in test case steps.

Welcome to 2022 – where old wisdom surfaces yet again. Recently I considered expected results in test cases. It seems the topic is still relevant – I wrote about it in “The Expected” back in 2015.

As always we have to backup a bit and consider: What purpose does the detailed steps have to our stakeholders. How visible is the value provided? We have to start from their viewpoint – not build a chain of reasoning up from our own wishful thinking.

Currently I’m working on a project for payments of benefits. We have daily CI/CD and loads of classic manual testing. The key message from the stakeholders is that they expect things to be tested and evaluated. They generally trust us and collaborate with us on the work, so there is not so much discussion on the details as in other projects. Yet the discussion of expected results did come up in our internal dialog: What is the reasoning behind expected results on the test case steps – besides coming from an old book.

Testing is a performance

One of the usual arguments is that it enables hard-over from person to another. While handover can be relevant to consider if the team members changes often. It seems to me just to create redundant information based on requirements and other oracles. I mean where does it it end, why should we add every project information into the test cases and into the test plan. Stop it already.

Let the test cases be evaluated by the person performing the test – to see the whole picture. And not get biased by expected results. I want the persons performing the tests to use their brain and based on the oracles available evaluate the system under test. Also if everything can be planned – why would there be a need for retakes of movie scenes? It turns out even simple instructions are not possible:

Exact Instructions Challenge – THIS is why my kids hate me. | Josh Darnit

Let people do what they are best at

Let’s have testing being a balanced approach, where people evaluate and scripts confirm.

When the confirmation can be determined by an algorithm or by assertions towards an oracle we can code a test for it – and test it continuously. Adding as much automation as (practically) possible, aids the delivery in multiple ways. It aids to build regression testing and help us test that things not only worked once somewhere, but that it works again and again.

On the other hand when there’s no algorithm or explicit oracle in the form of specifications, acceptance criteria or similar authority we can only do a subjective evaluation: at that point in time – given the information available. do not neglect the wisdom of subject matter experts testing.

And in the later situation expected results hinder the exploration of whether the system under test fulfills not on the explicit requirements but also the implicit. We really should know better.

Photo by Karolina Grabowska on Pexels.com

It all starts with an odd piece

One of my coworkers had gotten himself a LEGO 10242 MINI Cooper, and by the help of the other consultants it had been build (to spec?). We look over the remaining pieces and discuss how come. All the 1×1 plates are quite expected, there are extras of these because the weights aren’t that precise and the pieces are cheap. Also customers easily loose them, so it’s cheaper to send some extra than to handle through customer service. On BrickLink inventory there is even a fan made list of the usual extra items….

But an extra black 2×4 plate – naa. That’s odd.. And surely it missed on the bottom of the car. I had prior knowledge .. but have not built this exact set.

Now I have another hunch that the two gray 1×3 tiles and 1×1 dark green bricks in to the rear are missing somewhere. A good thing those consultants have a test department, one could say…  Still the pieces seem not to be 1-CRITICALLY missing, so the model is DONE and accepted. So even if the LEGO tester gets to ask “what if” – we have to remember to hear the answer to “does it matter” – even if it is our favorite hobbyhorse

2015-12-18 13.49.28