What About Expected Results?

TL;DR: We should know better than to require expected results in test case steps.

Welcome to 2022 – where old wisdom surfaces yet again. Recently I considered expected results in test cases. It seems the topic is still relevant – I wrote about it in “The Expected” back in 2015.

As always we have to backup a bit and consider: What purpose does the detailed steps have to our stakeholders. How visible is the value provided? We have to start from their viewpoint – not build a chain of reasoning up from our own wishful thinking.

Currently I’m working on a project for payments of benefits. We have daily CI/CD and loads of classic manual testing. The key message from the stakeholders is that they expect things to be tested and evaluated. They generally trust us and collaborate with us on the work, so there is not so much discussion on the details as in other projects. Yet the discussion of expected results did come up in our internal dialog: What is the reasoning behind expected results on the test case steps – besides coming from an old book.

Testing is a performance

One of the usual arguments is that it enables hard-over from person to another. While handover can be relevant to consider if the team members changes often. It seems to me just to create redundant information based on requirements and other oracles. I mean where does it it end, why should we add every project information into the test cases and into the test plan. Stop it already.

Let the test cases be evaluated by the person performing the test – to see the whole picture. And not get biased by expected results. I want the persons performing the tests to use their brain and based on the oracles available evaluate the system under test. Also if everything can be planned – why would there be a need for retakes of movie scenes? It turns out even simple instructions are not possible:

Exact Instructions Challenge – THIS is why my kids hate me. | Josh Darnit

Let people do what they are best at

Let’s have testing being a balanced approach, where people evaluate and scripts confirm.

When the confirmation can be determined by an algorithm or by assertions towards an oracle we can code a test for it – and test it continuously. Adding as much automation as (practically) possible, aids the delivery in multiple ways. It aids to build regression testing and help us test that things not only worked once somewhere, but that it works again and again.

On the other hand when there’s no algorithm or explicit oracle in the form of specifications, acceptance criteria or similar authority we can only do a subjective evaluation: at that point in time – given the information available. do not neglect the wisdom of subject matter experts testing.

And in the later situation expected results hinder the exploration of whether the system under test fulfills not on the explicit requirements but also the implicit. We really should know better.

Photo by Karolina Grabowska on Pexels.com

Explore, Code and Business

There are plenty of contexts where testing happens that doesn’t formally involve testers, or formally involve a testing phase. Still contexts like those have testing as an implicit activity.

We are neglecting the business know-how when all we talk about wrt. smarter testing is either the practices of exploration or of coding. Let’s bring in smarter testing for the business and coding sides too.

Visualize the Test Strategy

There are plenty of models/frameworks that list what you could have of testing activities – but there seems to be little assistance in WHAT to do in which situations. I have two visual ideas – let’s explore how to visualize the test strategy. And yes, there’s no size fits all – context matters.

In Charge of Testing

As a Test Manager I oversee the testing in a project or program – I am usually the only testing specialist in the project, so, I need the right leadership skills and the right tools to succeed. I have to own the data about the testing and quality activities.

As the test manager I need to facilitate a quite a range of testing activities:

• Developer’s unit test
• Continuous integration tests
• Robot Desktop test automation
• Infrastructure specialists testing
• Any professional testing specialists?
• Business Analysts [1] or similar within the project
• End users or end user representatives

I need to balance that I need to know what’s going on (with regards to testing) but without micromanaging the people being involved in testing and quality activities. My role is to facilitate that testing things happen – like the project manager making project things happen. I cannot own the activities without owning the data about it. I need to cover the full spectrum of tests – from engineered (RDA and CI/CD) to people-based (scripts and exploration).

The most practical tool for a test manager with this scope is PractiTest, as there is more to testing than just the test cases [2]. The old term “ALM” [3] comes to mind – it is still relevant when I look for a full test management tool. I need to cover both the “inputs” to testing (requirements, tickets and user stories) and the “outputs” (bugs) in one location. I need the requirements and user stories in my tool, as I need to base my test analysis and planning on the delivery model (that may not always be agile). I need the bugs in the testing tool too, as bugs can happen in any work product of the project: documents, code base and even the tests. PractiTest acknowledge that there is more to IT projects than code.

I appreciate the key driver of PractiTest – that all activities happen in-flow. You don’t have to change window, stack pop-ups or go to another tool in order to run the tests or create bugs. Creating bugs happens in context of the test case and seamlessly moves all data about the run to the bug. Everything you need to do is context-based, and available to you on screen. And it has some cool features of read-only links to graphs for management reporting, and a smart built-in “rapid reporter” for exploratory testing notes.

It can be a challenge to switch to PractiTest if you are in a compliance setting, if you need on-Premise or if your team generally uses Azure DevOps (the tool formerly known as TFS). To get the full potential of Azure DevOps, though, you need the full Microsoft Test Pro licenses, so it’s not a free tool either – nor is DevOps intuitive for testing things doesn’t have the code available. As with Azure DevOps PractiTest is also SaaS only, with multiple data centers for regional data compliance. As there is always inertia towards a commodity it won’t be long before there is no good arguments to have test management tools on-Premise and for the tool vendors to provide the compliance certificates (ISO/SOC really should be sufficient, IMO).

Out of the box PractiTest supports the categories of testing above (engineered, scripted, exploratory) and has the necessary integrations too: Surefire for unit testing, Maven for CI/CD, Jira, ServiceNow or any other ITSM for requirement input. There is even a two way integration to Azue DevOps. As the web design is “responsive” it could probably run off a tablet. That would enable easier test documentation for field tests. It would be even better to have a small version of it on a phone and be able to use the camera for “screen shots”.

At work I am currently running a large project regarding customizing and implementing a standard commercial software system, PractiTest would fit right in, as we have the following test activities:

• Unit test by the developers
• Automation by test engineers
• Exploratory test by Subject matter experts
• Formal scripted testing with end users

And I need to own the data around all of this, if I want to in in charge of the testing (and not only the testers). We are very few software testing specialists on the project team, but as the manager of testing I need to cover many other people performing the testing. This transforms my role from test management to one about leadership, coaching, and facilitation of testing being performed by the SMEs – and anyone else really.

I will be talking about Leading When the Subject Matter Experts Test at ConTEST NYC 2019 until then read more about leadership:

• You don’t have to be a boss to be a leader
• Expectations around Testing
• Who is the tester?

1. Anthropologists and similar humanities educations can be great BA’s
2. looking at you Test Rail 😉
3. ALM = Application Life Cycle, like Micro Focus Quality Center etc.

Disclaimer: This is an influencer review sponsored by PractiTest.

The Expected

Many test processes and test tools mention that you have to establish the expected results to every test. It is at best risk ignorance not to take the notion of expected results with a kilo of salt #YMMV.

If you can establish the result of your solution in a deterministic, algorithmic and consistent way in a non-trivial problem, then you can solve the halting problem. But I doubt your requirements are trivial.. or even always right. Even the best business analyst or subject matter expert may be wrong. Your best oracle may fail too.  Or you may be focussing on only getting what you measure.

Nailed it

When working with validation in seemingly very controlled environments changes and rework happens a lot, as every new finding needs the document trail back to square one.. Stuff happens. Validation is not testing, but looking to show that the program did work as requested at some time. It is a race towards the lowest common denominator. IT suppliers can do better that just to look for “as designed” [1].

Still the Cynefin framework illustrates that there are repeatable and known projects, and in those contexts you should probably focus on looking to check as many as the simple binary questions in a tool supported way, and work on the open questions for your testing.

Speaking of open ends – every time I see an explicit expected result I tend to add the following disclaimer  (song to the tune of nothing else matters [2]):

And nothing odd happens … that I know of … on my machine, in this situation [3]

And odd is something that may harm my user, business or program result

Significantly…

But I’d rather skip this test step  and work on the description of the test and guidelines to mention:

• Think! Use your head – use what you know.
• You are the expert.
• If it smells fishy, there is something fishy going on
• Wonder, Wander, Galumph if you may
• Analyse, consider,.. if you get an idea, try it!
• Learn, Explore, and by all means test

See also: The unknown unknown unexpressed expectations, Eating wicked problems for breakfast

1: Anyone can beat us, unless we are the best, todays innovation becomes tomorrows commodity

2: https://www.youtube.com/watch?v=tAGnKpE4NCI 

3: I’ve heard that somewhere…

Test Like Sherlock

Sometimes testing is like being Sherlock Holmes – You find your clues hidden in plain sight: Where the users scratch their nails; how the application user interface is cobbled together; odd patterns in the error logs….

But seldom without experimenting, seldom without pushing the subject under test or consulting the weather report, time tables – and getting out in the rain, doing some footwork.

They always seems to know better, always asking questions. They are so passionate about the problem solving skills that their standard by default seems arrogant [1]    (but that is usually not on purpose).

This is very clear in the recent BBC TV Series “Sherlock” – that illustrates and mentions their Asperger clearly. Almost on par with the The Curious Incident of the Dog in the Night-Time. Still when they are out solving mysteries they are a hero – if there ever was one. [2]

—

1: If your standard is to never be called arrogant, you’ve probably walked away from your calling. http://sethgodin.typepad.com/seths_blog/2014/12/in-search-of-arrogance.html

2: Don’t make people into heroes, John. Heroes don’t exist, and if they did, I wouldn’t be one of them. http://en.wikiquote.org/wiki/Sherlock_(TV_series)

Related: The yardstick of mythical normality, People are people – despite their labels,

QA Aarhus – Exploratory Testing How and When

QA Network Aarhus is a local non-affiliated network of testers (and good friends) in Aarhus. Where I had the great pleasure of talking about Exploratory Testing. This is the link collection, the slides are attached.

When good enough is the perfect fit

Similar to scope creep we may also experience “test creep“. Test Creep is when the tester adds more tests than what is in scope. Just as well as more business functionality is added during scope creep, more testing is added in test creep. Both aren’t necessarily bad, but in time-boxed or similar (budgeted) constraint projects creeping isn’t necessarily value adding. This is probably easier to understand in an agile project focusing on minimal viable product , but may happen in other contexts too.

It is test creep, when the tester feels an obligation to run an extra drill down into browser and OS configurations, when scope is less broad. It is creeping the scope of testing, if the testers feels a “/need/ to write testcases for this first” when exploratory sessions fits the mission. Consider test creep like gold plating, in that way that it tries to refine and perfect the product – when good enough is the perfect fit.

Test creep can happen intentionally, happen by management or by product owner request. It may happen unintentionally, and usually it is with the best intentions – as more testing always is better testing – right? (But it Depends) Sometimes yes, we as testers are to blame that we add more scenarios, rigor and details, because a testing mindset drives us to investigate the product.

In discussing this with Mohinder and Darren, we found that – it’s not only a matter of removing wait time for testers. This may add more time, to test but the scope creep in testing may happen none the less. A Lean mindset with focus on what adds value to the business and a discussion on the minimum viable testing will assist the project in avoiding test creep.

 

Mindmaps for 400

Finally non-profit self-organizing software testing is happening in Denmark. On may 21 2014 we actually had two events:

• Copenhagen Software Testing Meetup at Podio
• QA Network Aarhus at eBay Aarhus

At the first I was glad to share my experiences using mind maps in software testing, note taking and information structuring. (Get the PDF Xmind mind map here: https://jlottosen.files.wordpress.com/2014/05/mindmaps-400.pdf)

• Mind maps for 100: Read all about mind maps here:  http://www.ministryoftesting.com/resources/mindmaps/
• Mind maps for 200: Examples of mind map guidance:
  • http://lisacrispin.com/2011/02/28/using-mind-maps-for-test-planning/
  • http://visible-quality.blogspot.de/2014/05/test-documentation-and-mindmaps.html?
  • http://testconsultant.blogspot.co.uk/2010/04/ewt12-mind-mapping-and-testing.html
• Mind maps for 300:  Real life examples from my collection
• Mind maps for 400: 
  • Mind maps helps you explore your ideas
  • Mind maps helps you structure ideas
  • Mind maps help you visualize
  • Mind maps helps you see areas of depth

You stop going deeper down the tree, when there is nothing more knowledge to gain, just like good (exploratory) testing.

—

Cultural context of the “for 100″ comes from the Jeopardy TV Quiz, where the questions come in 4 levels: 100, 200,300, 400 for the  increasingly harder questions. The prize is similarly $100 for level 100 etc.  

FDA, Exploration and time to information

A key driver in implementing enterprise knowledge management is to reduce time to information (77% are seeing faster access to knowledge). But that goes for LinkedIn and Twitter too. Using twitter professionally helps you meet the famous people and help you see the communication layers at conferences. Case story: Today I was reading about test processes in a regulated environment, and got curious towards exploratory testing in that context. So I reached out to the #twitterbrain and asked the giants, whose shoulders I am standing on*:

• Griffin Jones ‏@Griff0Jones, help clients struggling with regulatory compliance and context-driven software testing problems.
  • CAST 2011: Cast 2011 What Do Auditors Expect From Testers
  • What is good evidence – Let’s Test 2013
  • WREST – Workshop on REgulated Software Testing
• Johan Åtting ‏@JohanAtting Chief Quality Officer -atSectra’s Medical Operation
  • turned the testing from a traditional scripted approach into a context driven approach and introduced exploratory testing.
  • ensuring that the company are regulatory compliant with e.g. FDA, MDD, CMDR, ISO13485, ISO14971 etc.
• James Christie ‏@james_christie
  •  interested in testing’s relationship with audit and governance.
  • dedicated to the audit, control, and security of information systems.
• Claire Moss ‏@aclairefication (my favorite retweeter) and many others retweeted

Within 2 hours I had both relevant references, a debate on the pitfalls and base for further details. Follow the tread of this tweet: https://twitter.com/jlottosen/status/411473074312052736 

@Griff0Jones you got links to FDA approved context & exploratory testing (besides #CAST 2011)?

— Jesper (@jlottosen) December 13, 2013

*: really, not to brag – I have met both Griffin, Johan and James, and they know me too 🙂