The Expected

3 Comments

Many test processes and test tools mention that you have to establish the expected results to every test. It is at best risk ignorance not to take the notion of expected results with a kilo of salt #YMMV.

If you can establish the result of your solution in a deterministic, algorithmic and consistent way in a non-trivial problem, then you can solve the halting problem. But I doubt your requirements are trivial.. or even always right. Even the best business analyst or subject matter expert may be wrong. Your best oracle may fail too.  Or you may be focussing on only getting what you measure.

When working with validation in seemingly very controlled environments changes and rework happens a lot, as every new finding needs the document trail back to square one.. Stuff happens. Validation is not testing, but looking to show that the program did work as requested at some time. It is a race towards the lowest common denominator. IT suppliers can do better that just to look for “as designed” [1].

Still the Cynefin framework illustrates that there are repeatable and known projects, and in those contexts you should probably focus on looking to check as many as the simple binary questions in a tool supported way, and work on the open questions for your testing.

Speaking of open ends – every time I see an explicit expected result I tend to add the following disclaimer inspired by Michael Bolton (song to the tune of nothing else matters [2]):

And nothing odd happens … that I know of … on my machine, in this situation [3]

And odd is something that may harm my user, business or program result

Significantly…

But I’d rather skip this test step  and work on the description of the test and guidelines to mention:

And now to something completely different:

See also: The unknown unknown unexpressed expectationsEating wicked problems for breakfast

1: Anyone can beat us, unless we are the besttodays innovation becomes tomorrows commodity

2: https://www.youtube.com/watch?v=tAGnKpE4NCI 

3: I’ve heard that somewhere…

Test Like Sherlock

3 Comments

Sometimes testing is like being Sherlock Holmes – You find your clues hidden in plain sight: Where the users scratch their nails; how the application user interface is cobbled together; odd patterns in the error logs….

But seldom without experimenting, seldom without pushing the subject under test or consulting the weather report, time tables – and getting out in the rain, doing some footwork.

He always seems to know better, always asking questions. He is so passionate about his problem solving skills that his standard by default seems arrogant [1]    (but that is usually not on purpose).

This is very clear in the recent BBC TV Series “Sherlock” – that illustrates and mentions his Asperger clearly. Almost on par with the The Curious Incident of the Dog in the Night-Time. Still when he is out solving mysteries he is a hero – if there ever was one. [2]

1: If your standard is to never be called arrogant, you’ve probably walked away from your calling. http://sethgodin.typepad.com/seths_blog/2014/12/in-search-of-arrogance.html

2: Don’t make people into heroes, John. Heroes don’t exist, and if they did, I wouldn’t be one of them. http://en.wikiquote.org/wiki/Sherlock_(TV_series)

deerstalker

Related: The yardstick of mythical normalityPeople are people – despite their labels,

QA Aarhus – Exploratory Testing How and When

1 Comment

QA Network Aarhus is a local non-affiliated network of testers (and good friends) in Aarhus. Where I had the great pleasure of talking about Exploratory Testing. This is the link collection, the slides are attached.

nnit

More

When good enough is the perfect fit

2 Comments

Similar to scope creep we may also experience “test creep“. Test Creep is when the tester adds more tests than what is in scope. Just as well as more business functionality is added during scope creep, more testing is added in test creep. Both aren’t necessarily bad, but in time-boxed or similar (budgeted) constraint projects creeping isn’t necessarily value adding. This is probably easier to understand in an agile project focusing on minimal viable product , but may happen in other contexts too.

It is test creep, when the tester feels an obligation to run an extra drill down into browser and OS configurations, when scope is less broad. It is creeping the scope of testing, if the testers feels a “/need/ to write testcases for this first” when exploratory sessions fits the mission. Consider test creep like gold plating, in that way that it tries to refine and perfect the product – when good enough is the perfect fit.

Test creep can happen intentionally, happen by management or by product owner request. It may happen unintentionally, and usually it is with the best intentions – as more testing always is better testing – right? (But it Depends) Sometimes yes, we as testers are to blame that we add more scenarios, rigor and details, because a testing mindset drives us to investigate the product.

In discussing this with Mohinder and Darren, we found that – it’s not only a matter of removing wait time for testers. This may add more time, to test but the scope creep in testing may happen none the less. A Lean mindset with focus on what adds value to the business and a discussion on the minimum viable testing will assist the project in avoiding test creep.

 

minecraft_creeper_wallpaper_by_lynchmob10-1440x900

Mindmaps for 400

1 Comment

Finally non-profit self-organizing software testing is happening in Denmark. On may 21 2014 we actually had two events:

At the first I was glad to share my experiences using mind maps in software testing, note taking and information structuring. (Get the PDF Xmind mind map here: https://jlottosen.files.wordpress.com/2014/05/mindmaps-400.pdf)

You stop going deeper down the tree, when there is nothing more knowledge to gain, just like good (exploratory) testing.

Cultural context of the “for 100″ comes from the Jeopardy TV Quiz, where the questions come in 4 levels: 100, 200,300, 400 for the  increasingly harder questions. The prize is similarly $100 for level 100 etc.  

FDA, Exploration and time to information

1 Comment

A key driver in implementing enterprise knowledge management is to reduce time to information (77% are seeing faster access to knowledge). But that goes for LinkedIn and Twitter too. Using twitter professionally helps you meet the famous people and help you see the communication layers at conferences. Case story: Today I was reading about test processes in a regulated environment, and got curious towards exploratory testing in that context. So I reached out to the #twitterbrain and asked the giants, whose shoulders I am standing on*:

  • Griffin Jones ‏@Griff0Jones, help clients struggling with regulatory compliance and context-driven software testing problems.
    • CAST 2011: Cast 2011 What Do Auditors Expect From Testers
    • What is good evidence – Let’s Test 2013
    • WREST – Workshop on REgulated Software Testing
  • Johan Åtting ‏@JohanAtting Chief Quality Officer -atSectra’s Medical Operation
    • turned the testing from a traditional scripted approach into a context driven approach and introduced exploratory testing.
    • ensuring that the company are regulatory compliant with e.g. FDA, MDD, CMDR, ISO13485, ISO14971 etc.
  • James Christie ‏@james_christie
    •  interested in testing’s relationship with audit and governance.
    • dedicated to the audit, control, and security of information systems.
  • Michael Bolton ‏@michaelbolton Program Chair of EuroStar 2013 and key guru pointed me to an article by James Marcus Bach ‏@jamesmarcusbach on the topic.
  • Keith Klain ‏@KeithKlain Head of Barclays Test Service retweeted on the fallacy on the evidence of scripted testing.
  • Claire Moss ‏@aclairefication (my favorite retweeter) and many others retweeted

Within 2 hours I had both relevant references, a debate on the pitfalls and base for further details. Follow the tread of this tweet: https://twitter.com/jlottosen/status/411473074312052736 

*: really, not to brag – I have met both Griffin, Johan and James, and they know me too 🙂

Uncovering better ways

4 Comments

I am uncovering better ways of developing solutions – by doing it and helping others do it. Through this work I have come to value:

  • Apply the costs to add business value – over cutting costs
  • Being flexible and open  – over adding predictability 
  • Providing information for decisions – over ensuring the reliability

More

Older Entries