In which our hero must revive old wisdom to battle empires and where the band of front runners will be supported by fuzzy friends to bring down legacy structures.
Continue readinghistory
A Track down History
Currently I am doing a large V-model waterfall project with a three month testing period and 500+ requirements. To track the progress I want to dust off my old “test progress tracking” method that I matured and described in 2011 and 2014.
The approach was documented in two articles for “The Testing Planet” a paper magazine by the Software Testing Club. The “STC” later became the now world famous Ministry of Testing. Unfortunately the original articles are no longer available on the MoT site – they are on the WayBackMachine. So not to track down that path, here’s a recap of:
- A Little Track History that goes a Long Way [The Testing Planet by Ministry of Testing, July 2011] The purpose of this tracking tool is to collect just enough data to answer the frequent question “Will we finish on time” [https://web.archive.org/web/20160410200818/https://www.ministryoftesting.com/2011/07/a-little-track-history-that-goes-a-long-way/]
- The Daily Defect Count and the Image of a Camel [The Testing Planet by The Ministry of Testing, April 2014] Count the defects daily – the ones that are part of the project work load. The number goes up and down during the cycle – why and what can you learn? [https://web.archive.org/web/20140718052546/https://www.ministryoftesting.com/2014/04/daily-defect-count-image-camel/]
A Little Track History that goes a Long Way
For large (enterprise, waterfall) projects tracking test progress is important, as it helps us understand if we will finish a given test scope on time. Tracking many of projects have given me one key performance indicator: daily Percent Passed tests as compared to an s-curve. The data in the S-curve is based on the following data points, based on numerous projects:
| Time progress | Expected Passed Progress |
| 10% | 0% |
| 20% | 5% |
| 30% | 10% |
| 40% | 30% |
| 50% | 45% |
| 60% | 60% |
| 70% | 75% |
| 80% | 90% |
| 90% | 95% |
| 100% | 100% |
If the “Actual” curve is flat over more days or is below the blue trend line, then investigate what is blocking the test progress… defects perhaps:
The Defect Count and Camel Image
During a large project like this the active bug count goes up and down. No one can tell what we find tomorrow or how many we will find. In my experience tracking the daily count of active defects (i.e. not resolved) is key, and will oscillate like the humps on a camel:
If the curve doesn’t bend down after a few days there are bottlenecks in the timely resolution of defects found. When the count goes up – testing a new (buggy) area is usually happening. Over time the tops of the humps should be lower and lower and by the end of the project, steep down to 0.
And thus a little track history has come a long way.
Without Timing – Quality, Schedule and Cost is nothing
A project usually has to be on target on Quality, or at least on agreed quality.
A project usually has to be on Schedule, or at least on agreed schedule.
A project usually has to be on target regarding Costs, or at least agreed costs.
It is an ancient IT project manager mantra, to control these parameters using change requests and change management – typically due to events during the project. It takes to long, it’s too buggy, it costs more etc. etc. To change one parameter – you have to change one of the others. In example (and strictly in some context) – even adding more testcases will increase cost or influence the schedule (discuss).
But there are plenty of examples (of Government IT projects) that are on schedule, on cost, on quality – yet they fail the timing and after a number of years delivers a system on an obsolete platform [true story]. More recent examples – bank D launches their banking app first, get all the press – and a month later bank N comes around – on agreed schedule, agreed costs, agreed quality… and get very little of the “hype”.
When contemplating the business decisions – what seems right now right now might be wrong later – considering the Timing.
Acceptance criteria are more than what can be measured, An Expected Gathering
…
Softwaretesting is only dead, if it stands still
[ Software Testing Club Blog | January 23, 2012 ]
Here are some quotes from actual test mission statements from the last 10 years. Test always is changing. Come to think of it, it’s only dead if it stands still.
2003
We discover and measure the quality of the system:
– to test everything is utopia,
– we should test something, something new, something old (and something blue?)
2006
The primary objective for testing is to reveal the quality of the delivery. The revealed quality is documented in test reports, that serves as suggestions for decisions on approval or rejection of the delivery. Key principles: early involvement, structured testing, based on V-model, documented, COTS tools (TestDirector)
2008
Testing is the structured activity of identifying the quality and improving the quality of the an IT-supported product or project
– Identification of quality = Preparing and executing a test activity
– Improving the quality = Evaluation, fixing and retesting of raised incidentTest provides information about the quality of IT-supported product or project, with that information:
– Quality of products can be improved
– Project decisions can be based on facts
– Processes can be investigated for improvement
2011
Testing has the mission of providing information to the project about the quality of the business features of the solution.
Testing is about learning about the quality solution, not about confirming, verifying or assuring.
Test is about adjusting, not about following a strict script and knowing everything up front.
Originally at my SoftwareTestingClub blog – more from my STC Blog here: All oracles are failable, Testing decommissioning of ICBMs and software and Testing a new version.
Complexity is a Matter of Perspective 