softwaretestingclub

A Track down History

jo2sn Softwaretesting , , , , , , , ,

Currently I am doing a large V-model waterfall project with a three month testing period and 500+ requirements. To track the progress I want to dust off my old “test progress tracking” method that I matured and described in 2011 and 2014.

The approach was documented in two articles for “The Testing Planet” a paper magazine by the Software Testing Club. The “STC” later became the now world famous Ministry of Testing. Unfortunately the original articles are no longer available on the MoT site – they are on the WayBackMachine. So not to track down that path, here’s a recap of:

A Little Track History that goes a Long Way

For large (enterprise, waterfall) projects tracking test progress is important, as it helps us understand if we will finish a given test scope on time. Tracking many of projects have given me one key performance indicator: daily Percent Passed tests as compared to an s-curve. The data in the S-curve is based on the following data points, based on numerous projects:

Time progressExpected Passed Progress
10%0%
20%5%
30%10%
40%30%
50%45%
60%60%
70%75%
80%90%
90%95%
100%100%
Adding a 3rd order polynomial trend-curve gives the s-curve.

If the “Actual” curve is flat over more days or is below the blue trend line, then investigate what is blocking the test progress… defects perhaps:

The Defect Count and Camel Image

During a large project like this the active bug count goes up and down. No one can tell what we find tomorrow or how many we will find. In my experience tracking the daily count of active defects (i.e. not resolved) is key, and will oscillate like the humps on a camel:

Camel background is optional

If the curve doesn’t bend down after a few days there are bottlenecks in the timely resolution of defects found. When the count goes up – testing a new (buggy) area is usually happening. Over time the tops of the humps should be lower and lower and by the end of the project, steep down to 0.

And thus a little track history has come a long way.

The Testing Planet, July 2011

Test ALL the things

jo2sn Enterprise IT, Family and fun, Softwaretesting , , , , , , , , , ,

TL;DR: We can add testing to all requirements and all business risks. Testing to document requirements and to debunk risks provides valuable information for the business. Let us not limit testing to things that can be coded. The intellectual activity of trial and learning is happening anyways, we might as well pitch in with ways to find important evidence for the decision makers.

Test all the requirements

Traditionally testing was all about testing the functional requirements that could be coded. Non-functional requirements was left for the specialists, or plainly disregarded. I know I have done my share of test planning, with a range of requirements left “N/A” with regards to testing. Especially performance scope, batch jobs, hardware specs, data base table expansions and virus scanning have been left out of my functional test plans…

When I look at a list of requirements now – I see that we can indeed test all the things, or we can at least work on how to document that the requirement is fulfilled. Some of the requirements are actually quite easy to document. If it’s on a screen somewhere, take a screen shot and attach it to a simple test case. Done deal really. Additionally with a testing mind-set I can think of ways to challenge the details. But do we really, really need to fill up a disk to establish if it’s exactly a 1 Gb allocation – probably not. Do we really really need to document all requirements – yes in some contracts/contexts it’s important for the customer to know that everything has indeed been established. Sometimes the customer doesn’t trust you otherwise, sometimes the tests are more about your ability to deliver and provide evidence that matters.

Test all the business risks

Look into the business case of your project and find the business risks. Sometimes they are explicitly stated and prioritized. A a recent Ministry of Testing Meetup we looked into a case for a large national health system. We looked at the tangible benefits, intangible benefits and on the scored business risks.  What worried the business and management most was budget, time and whether the new system would be used in a standardized way. There is an opportunity for testing here to help address, document and challenge the most important business  risks. Traditional testing would usually look at functional requirements that can be coded or configured, and miss totally to address what matters most to the business.

OK, how do we test the project costs? How about frequent checkpoints of expected spending – would that be similar to tracking test progress. Perhaps – let’s find out. Testing is all about asking questions for the stakeholders and solving the most important problems first. Can we help to analyse risks and setup mitigation activities – sure we can. We just have to step out of our traditional “software only bubble”.

MEME - Test ALL the things
Meme ALL the things

 

 

 

 

 

 

 

Read also: Many Bits under the Bridge, Less Software, more Testing, Test Criteria for Outsourced Software, The Expected, Fell in the trap of total coverage.

Links: “A Context-Driven Approach to Delivering Business Value”, Cynefin In Software Testing, Testing during Application Transition Trials

 

Testing across the IT landscape

jo2sn Enterprise IT, Softwaretesting , , ,

Good testing is much more than confirmation of explicit requirements. It’s figuring out the implicit requirements, what blocks the business and what drives the business. Businesses are not driven by SDLC’s but by decisions and strategies. IT road maps are just a representation of the business strategy, an engine to build business solutions on. The is much more to the business than the software solutions and related risk mitigation.

Very often the biggest business risks are outside the project scope. When we look at it this way we see that testers and testing activities has an opportunity outside the classic project life cycle. Testing is about experimenting with a IT solution, to evaluate if it fits the business requests. IT solutions that supports the business exists in many forms, I am certain that explicit testing (*) can add value in other parts of the IT landscape.

Here is a model of an enterprise IT landscape consisting of business ideas, solution development, operations and end user devices + support. Solution delivery is boring in the sense of well-known software creation and maintenance. What if the item under test and the requirements are around network, servers, end-user devices and IT support tickets. I am certain that it’s valuable to test business cases before the project is even formally assembled.

 

*: implicit testing in the form of critical evaluation always happen. .. similarly does checking.

Co-creating smarter testers

jo2sn Softwaretesting , , , , , , ,

“Co-creating smarter testers” is the byline of the Ministry of Testing, a small company with a great impact that I have been following and supporting for 7 years* now . I have attended TestBash’es, webinars, challenges, discussions and memes. And now for the first time in Denmark – Anders Dinsen and I are bringing the world known Meetups to Copenhagen (Aarhus 2017 you’re next).

Ministry of Testing – Copenhagen

Copenhagen, DK
224 Members

The Ministry of Testing exists to advance the software testing industry in a fun, safe, professional and forward thinking way.Our meetups exist as a way to bring people toget…

Check out this Meetup Group →

The topics so far are:

At the first meetup we split into three groups, discussed risks and how to TEST THEM RISKS. Dearest to me was the discussion of stakeholders and new places to test. Great to see that even with very little information, we can still do a rapid testing based on business objectives. There is so much more to testing these days.

600_457785753

1: since a EuroStar 2010 t-shirt competition 🙂

Publications and Presentations

jo2sn Softwaretesting , , , , , ,

Presentations, Webinars & Podcasts

Other publications

  • 21st Century Skills For Testers [Collection by Kramer & Ayadi, jan 2021]: apply the 21st-century skills: critical thinking, communication,  collaboration, and creativity.
  • Test Automation for Software-as-a-Service Solutions [Synapse QA SuperReads, Jan 2021]: New tools are breaking ground in this space and have some clear benefits, yet also have to apply some good-old automation practices. https://synapse-qa.com/2021/01/13/automation-for-saas/
  • Could Modern Testing Work in The Enterprise [Guest blog for Panaya, May 2018] So far I have mostly thought that “Modern Testing” of the A/B testing podcast would never work in an enterprise context. But it seems some tools and existing approaches in the enterprises already fits well with the ideas of the concept. http://www.panaya.com/blog/testing/could-modern-testing-work-in-the-enterprise/
  • DevOps is cool, but get involved in OpsDev for Test Environment Management too! [Guest blog post for Plutora, Oct 2017] The hyped mnemonic “DevOps” is equally true the other way around: OpsDev http://www.plutora.com/blog/opsdev-test-environments-management
  • Testing during Transition: Test Criteria for Outsourced Software [Sticky Minds by TechWell, May 2017] In the world of IT outsourcing, it is not uncommon for a company to have its applications and infrastructure developed or maintained by others. How would you design acceptance criteria of a transition trial so that it is testable and clearly communicated? https://www.stickyminds.com/article/testing-during-transition-test-criteria-outsourced-software
  • Using Business Decisions to Drive Your Testing Coverage [Sticky Minds by TechWell, November 2014] In a business setting, software testers have a great challenge: to articulate how they support the business lines. One way to approach this is by addressing the business decisions—and there are plenty around. Use them to drive your testing activities and increase the business decisions being covered by testing. http://www.stickyminds.com/article/using-business-decisions-drive-your-testing-coverage 
  • The answer is: Why – because the answer depends on context.[The Testing Circus,vol.6 2.ed February 2015]: When asked about testing approaches, the options are so plentiful, that the reply is often “It depends” – and followed by a range of elaborations. But in our eager to reply, we forget to listen. http://www.testingcircus.com/february-2015/
  • The Testcases Template Trick – Getting One Testcase To Call Another [EuroStar TestHuddle, Nov 2014]: When doing test analysis I often find that we need to do test some customer feature over and over again for a range of combinations. I recently found myself able to redo a trick I learned a long time ago https://huddle.eurostarsoftwaretesting.com/the-testcases-template-trick/

Articles for The Ministry of Testing, 2011+

  1. Robot Process Automation As A Power Tool For Testing [Ministry of Testing Dojo, Mar 2018]: While there are other power tools for web and API testing, the RPA tools are a class of their own, as RPA tools allow for codeless automation macros on the desktop. RPA tools can do some very handy things. They can be used for both test data and regression testing. In this article, we’ll walk through a real testing example and show how you can get started using RPA. [TOP 6 on the Ministry of testing 2018 article list]
  2. Testing is Shifting [Testing Planet 2017 by the Ministry of Testing, Mar 2017]: Change is the only constant, they say, but we still need to manage change – and cope with it. Coping not only means surviving mentally, but also adjusting to whatever happens and figuring out how to be productive and create value for our stakeholders when things change. [https://dojo.ministryoftesting.com/lessons/testing-is-shifting]
  3. About Closure [The Testing Planet by Ministry of Testing, Nov 2014] When I’m in a testing activity I want my test cases [Passed], my user stories [done] and my coffee [black].  Stuff may have a start point, some states in between and an end state. Let’s look at ways to represent states and articulate the meaning of states. [Reposted: Closing the Gaps]
  4. The Daily Defect Count and the Image of a Camel [The Testing Planet by The Ministry of Testing, April 2014] Count the defects daily – the ones that are part of the project work load. The number goes up and down during the cycle – why and what can you learn? [Reposted: A Track down History ]
  5. The Day Testing Died But Didn’t [The Testing Planet by Ministry of Testing, Jan 2014] To play according to textbooks is fine, up to a certain level. Perhaps up to master level, but not to grand masters. [Reposted: Chess and Testing ]
  6. One Test Case is All You Need [The Testing Planet by The Ministry of Testing, November 2013] If you can come up with just one business transaction – that crystallizes why the customer will be kicking and screaming to want to use your application, then you have a very good understanding of your customer and all you need is that one testcase. [Reposted: One Test Case is All You Need. Original on WebArchive.org]
  7. Recognize and Acknowledge Your Skills [The Testing Planet by the Ministry of Testing, June 2013] What you know and what you do is an important part of being you. Often it is required to rethink: What do I know? What are my skills? How strong are they? [Reposted: https://jlottosen.wordpress.com/2013/06/18/tying-it-all-together/]
  8. The Build-A-Tester Workshop [The Testing Planet by The Ministry of Testing, June 2013] A small social game of Build-A-Tester can be used in a team to open the discussion, less formally than with Belbin and MBTI checklists. [on WebArchive.org]
  9. A Little Track History that goes a Long Way [The Testing Planet by Ministry of Testing, July 2011] The purpose of this tracking tool is to collect just enough data to answer the frequent question “Will we finish on time” [Reposted; A Track down History ]

QA Aarhus – Exploratory Testing How and When

jo2sn Enterprise IT, Softwaretesting , , , , , , , , , ,

QA Network Aarhus is a local non-affiliated network of testers (and good friends) in Aarhus. Where I had the great pleasure of talking about Exploratory Testing. This is the link collection, the slides are attached.

nnit

Continue reading →

Testing for coffee, testing for ǝnןɐʌ

jo2sn Softwaretesting , , , , ,

We focus on both testing and the office coffee a lot and may be looking at its real value too little. If testing does not provide direct business value – as the office coffee does – how can we elaborate and talk about it as a value to the business?

Time: October 1st 2013  Cafe Stiften Banegårdspladsen 11, 8000 Aarhus, Denmark

If you attend #GOTO aarhus 2013 this will be a good time to continue discussing “ǝnןɐʌ: Why we have it backwards”http://gotocon.com/aarhus-2013/presentation/ǝnןɐʌ:%20Why%20we%20have%20it%20backwards

The first 25 participants gets

meetup

All oracles are failable

jo2sn Enterprise IT, Softwaretesting , , , ,

[Software Testing Club Blog | Oct 6 2011 | myself ]

All oracles are fail-able to a certain level of confidence.

Recently I had the opportunity to participate in the acclaimed dice game. I also had the chance to be game master on a variation of the dice game session for a small test team. Reflecting on the experiences I had two considerations: (Some spoilers apply)

When are you confident enough?

The dice game is played by a loop of theories/ideas and tries/tests on the idea. The goal is to produce a theory/algorithm that can successfully predict the number that the game master presents. How many tries/tests/checks would give you confidence in the theory you have in mind? Options:

  • When you successfully predict one throw. Ie you say 7, game master say 7. Do you yell “LOOSERS, Sea ya”?
  • When you have 7 successfully predicted in a row? (why 7)
  • All 7776 combinations of 5 pieces of 6-sided one-colored standard dices?
  • Do you try every throw just once, or more times?
  • Would you know if every trial number divisible by 100, the game master would say “pi” (think leap years)

The dice game seems simple, but the problem domain of even the dice game is infinite. Or at least practically infinite (7776 is practically infinite in the dice game IMHO). The number of tests doesn’t matter, but the character of the test, relative to your mental models, matters a great deal. My purpose is not to find a fixed number of tries, but to make you consider the underlying assumption on confidence levels. That is you have a confidence in your model until it fails. You are confident to the level of x successful predictions, where the x+1 prediction fails. All you know at that time is that your theory is “incomplete” (not wrong, not right) – and this calls for more learning and more ideas…

All oracles are failable

The oracle in software projects is the ressource of answers – the documents, the mindmap, the subject matter expert. In the dice game the game master is the oracle. We are humans hence failable. The physical oracles (docs, …) even more. This made me ponder:

  • Would you approach the dice game differently actively knowing that the dice game master is failable?
  • If the game master (aka oracle) made a deliberate error ever once in a while – would you know?
  • If there is a bug (non deliberate) in the game masters algorithm would you know?
  • How would you test for the oracle making mistakes?
  • Do you test the dice game different if it was a human or machine oracle
    • First think about the dice game being computer based
    • Secondly what if it was a human behind a computer based interface
    • Consider the implications of the Turing Test
  • Oh, did you forget that I could make mistakes? – was that a rule, or an assumption?

The key framing of the dice game is usually a lesson in learning, in theory setting and trials of that theory – still under the underlying assumption that the game master can deal with any test that is offered. What would happen if the game master was blindfolded? What would the case if the algorithm was more complex – less humanly processable in a short time. There will always be a level of capability of the oracle, and it will fail – eventually.

http://www.thinkgeek.com/geektoys/plush/7ccc/?srp=13
20 Sided Fuzzy Dice Danglers

Testing a new version

jo2sn Enterprise IT, Softwaretesting , , , ,

[The Software Testing Club blog | November 29, 2011 | Jesper Lindholt Ottosen ]

Enterprise applications often come in packages and are purchased as (Commercial Off the Shelf – COTS). Every now and then a new version of SharePoint,SAP, Jive, OeBS, Microsoft Windows…. is made available and the business and product owner decides to implement the upgrade.

Usually the setting is that there is a “Factory Acceptance Test” by the vendor of the COTS package and a “Site Acceptance Test” by the implementing IT service organization. Here are some ideas that come to my mind, the couple of times I have had look into a testing strategy for a Enterprise COTS upgrade project. It’s not a best practice – at best it’s a heuristic:-)

 

Regression testing first – it might be considered to examine that quality didn’t get worse. Select some of the key existing features that is most important to the product owner, and examine them. Also involving the super users or application advocates in an exploratory testing activity will provide benefits for both the testers, the super users and the other project participants.

Interfaces – in an enterprise environment there is always interfaces to legacy systems and new “bud shots” to the IT tree. SOA services makes it even more important to look for known and unknown interfaces to the application. Similarly context specific customization’s (additions and removals) and applied “production hot fixes” having been applied or constructed based on v2.5. Analyzing the intermediate versions (above example 2.3, 2.4 and finally 2.5) including known fixes and known new features, can be another approach to identify the required levels of testing. Discuss with the product manager and business representative – the key is to find a level of test that they are OK with.