During one of my recent projects I was considering the ratio between the checks and the tests – that is the ratio between those tests, that are primarily simple binary confirmations, and those tests that are more tacit questions. This blog is about my considerations on the idea/experiment/model.
First I observed, that we have a range of different items in our requirements – some of them are [actual copy of the current specification]:
It must be possible to add a customer ticket reference
It must be possible to copy the ticket number
You must be able to navigate displayed lists easily
It must be easy to compare work log and audit log
You could argue that they need refinement, more testability and less “easy“. But this is what we have to work with for now. Even if we had all the time in the world (we don’t) – we would not be able to write all of the requirements in a perfect form (if such a form exists).
I wrote about this ratio on my blog post around the Test Automation Pyramid, as I will use the labels to automate the confirmations (and only the confirmations). The assumption is, that there are significantly more of the binary requirements tested by machine checking – and more human tested tacit questions. If we can get all the tedious tasks automated – that is the really the end goal.
Automate all the things that should be automated
Every project/context will have it’s own ratio, depending on a range of factors. Saying there should always be more of one type than the other would not hold. As the above project is the configuration and implementation of a standard commercial business software package (like SAP, SalesForce etc), my expectation is that most of the requirements are binary. Also considering that this project is heavy on the right hand side of the Wardley Map scale of evolution.
It’s a Reduction in Context
I am well aware that the two/three piles are an approximation / reduction. Especially when looking at the “binary” requirements and “only” testing these by confirmation. They could as easily be explored to find further unknown unknowns. If we prioritize to do so – it all about our choice of risk.
It is also an limitation as “perfect testing” should consist of both testing and checking. I factor this into the test strategy, by insisting that all of the requirements are tested both explicitly and implicitly. First of all most of our binary requirements are on the configuration and customization of the out-of-the-box software solution. So when the subject matter experts are performing the testing of the business flows, they are also evaluating the configuration and customization. And I do want them to spot when something is odd
Ultimately I want to use the experts to do the thinking and the machines to do the both the confirmations and the tedious tasks.
As I am motivated by finding solutions and making them work** my drive is more along the lines of “accelerate the achievement of shippable quality” [Modern Testing Mission] than “finding the problems that threaten the value of the product” [Rapid Software Testing].
Focussing on achievements over problems seems to work for me in the contexts I’m in, regarding enterprise transition, infrastructure projects and the implementation of commercial standard systems.
Setting a Frame for Expectations
Finding the “test solution” (or test strategy) that fits the project context is the key activity to me. The rest of it, is mostly about implementation – that too can be quite interesting. I like that too, but plan first!
First of all we have to realize that the testing activities we choose are limited and affected by or context (and biases). We can never test everything and think of everything to test. Based on the context restrictions (time, space, money, etc) the project gives me, I make a reduction of the testing theories and principles into a definition along the lines of:
In a specific context – testing will be a finite activity, to investigate if the shared interpretations of the requirements are implemented – at some time, for some configuration, evaluated by someone (that we trust), where nothing odd happens.
A reduction of the testing activity
Let me be the first to say: It’s not theoretical perfect! But it’s practical and based on context. The reduction gives me an achievable goal-oriented focus. It helps me to iron out what the relation is between the thing under test and someone whom it matters to.
Ironing out the Expectations
If there is an underlying risk that things will change a lot, then we can argue for test automation to multiply the configurations and the number of “runs” we can complete. Not all IT projects are around software development, so test automation practices and tooling might not be in place.
To make the agreements around the reduction of the testing activity explicit I establish a “Test Plan” document. I would often prefer to do without, and have a mutual team agreement – or even a mind map. But I know the enterprise contexts, too well to know, that shared expectations are best written down (even though it being an imperfective as well).
It’s all about the context and the expectations, really.
Looking at all the podcast, conference and community chatter you could get the impression that everyone else’s projects always follows the latest trends and hottest principles. That everything is perfect, and everything is a success – and that all new ideas are working right off the bat.
It almost sounds as if testing has to be in a specific way, and that what you experience is wrong or less worthy. That there are no failures, no scrum-fall projects or old legacy systems. It sounds like everything runs smoothly on an up-to-date CI/CD K8 technology stack with all the bells and whistles.
Don’t worry, listen to the “Guilty tester podcast”
Every single project/company/context is both it’s own mess and it’s own best. There is a huge difference between all the worlds companies and all the countries traditions around IT. Sure, it may happen – let me tell you the world of IT projects is a weird place, and that’s OK. So take all the stories of successes with the added American car commercial catch phrase goes “your mileage may vary” [YMMV] or as we tend to say coming from a Context Driven point of view “It depends“.
It depends on your context if modern testing is a thing for you, it might work in an enterprise setting of commercial standard systems. Using Robot Desktop automation in testing might work better in that setting, but then again it will probably not be a good fit in your average software development project. In a context of developing software business-to-consumer the web features is more importation than in an enterprise setting. And round and round the practices and approaches goes and goes…
If you look a little beyond the borders of your own project you will see similarities to others doing the same, but also the diversity in approaches, successes and failures. You are not doing all of it wrong.
(aka not every testing problem can be solved by a webdriver)
Web features doesn’t matter as much in the contexts I usually work in. While some may be delivered over the web, the focus for testing is on the whole system’s fit for the business. Adding automation in testing to the mix gives additional challenges as there is no source code in the solution to interact with, and we have to find other solutions to solve the tedious tasks in testing with.
One area where this is the case, is when implementing standard commercial software packages (COTS) for the enterprise or public sector. Solutions like SAP for retail CRM and ERP, Microsoft Dynamics 365 for Finance and Operations, EPIC for hospitals, Service Now for IT Service management etc. These are standard solutions that can be configured and customized, but the general source code is not available.
Thus the “test automation pyramid” falls short to help us automate things as only the GUI is available for interacting with the solution. Test engineers might want to setup CI/CD but the success of that depends on the system architecture and the provided as-is methods of releasing updates. Some of the solutions above are provided as SaaS but quite often they still run “on premise” and the business still wants releases tested before launching things on a corporate scale.
Another example is the many bespoke software solutions that are still in operation. My local electronics store has two interfaces for the sales persons: web to look things up (specs and availability) and a mainframe system to set up the actual purchase (Point of sales, POC). Many public organisations and enterprises are only now transitioning from the desktop applications of the 1990’es to more up to date solutions. Unfortunately systems that are 10+ years old have very little of live and relevant specifications and neither CI/CD suites.
While some COTS and POC solutions are being delivered over the web, testing web particularities the very least of our focus areas. The web particularities seems to matter more if the solutions are business-to-consumer but not so much when it’s business-to-employer or business-to-business.
In a business-to-employer and business-to-business context, usually only one browser is in scope. And that’s it. There is little interest for HTTP status codes, broken links, browser compatibility or login forms.
Focusing only on testing the web in contexts like these we fail in
covering the whole system landscape across applications of different technologies
It’s in this context that RPA probably has some benefits in providing automation of tedious testing tasks to the tester with a business background. That is, they are business people first – and then they do the testing that matters to the context.
Do you ever feel guilty for not meeting the standards set by others in the Software Testing community? You’re in the right place then.
In this episode I talk to Dave (@theguiltytester). We discuss traditions, open questions and how to work within contracts which are specifically requesting traditional test practices based on large numbers of test cases.
So far I have mostly thought that “Modern Testing” of the A/B testing podcast would never work in an enterprise context. But it seems some tools and existing approaches in the enterprises already fits well with the ideas of the concept.
The enterprise is all the privately owned companies that usually manufacture (non-IT) things – for either the consumer or other businesses. The enterprise sell and produce tangible products like windmills, power tools, dairy consumer products etc. The interest with regards to IT for the enterprise is that it just works, and supports business processes around order setup, order tracking and invoicing – and many other moving parts.
While I have heard of some organisations that have successfully implemented some agile and SaFe methods (in banking), the enterprise have a hard time to change mode of operations, as it usually comes down to actual production of things, logistics and hierarchies of command-and-control … and culture, most of all culture.
Some enterprises change towards being learning organisations, but still treat their IT in general as low-value and an annoying cost. It seems the IT departments and IT contractors have a challenge in talking about what they do to achieve the right quality for the businesses….
Que: The Modern testing mission on “Accelerate the achievement of Shippable Quality”. While MT is mostly a concept around transition of testing activities, it seems the concept applies to IT delivery teams in general. MT has 7 principles and some of these are:
5. We believe that the customer is the only one capable to judge and evaluate the quality of our product
Most enterprise projects I know off around implementing SAP, MS dynamics 365, EPIC hospital solutions etc, have a large reliance on end-user testing and UAT. Often there is no professional testers involved, as the best tester is the business experts themselves. Interestingly the principle #5 fit’s well with existing practices from the UAT space.
Another interesting MT principle is #6 around data analysis of actual customer usage. This requires some totally different tooling for the tester, than previously generally available (…besides shifting-right perhaps…).
6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
Yet recently I was investigating Panaya Autonomous Testing for SAP. One thing I realized is that what the tool collects real user usage of SAP and then provides the ability to balance the testing activities based on that analysis. It is interesting to see a commercial test management product for the enterprises following new trends like the “modern tester”.
While it’s interesting how some of the concepts of modern testing are reflected in testing in the enterprise – and vise versa – the challenge remains for both the tools and concepts to be applied and accepted in more organisations.
It might not fit everywhere, but it might be a good fit in more places than you think it would.
TL;DR: We can add testing to all requirements and all business risks. Testing to document requirements and to debunk risks provides valuable information for the business. Let us not limit testing to things that can be coded. The intellectual activity of trial and learning is happening anyways, we might as well pitch in with ways to find important evidence for the decision makers.
Test all the requirements
Traditionally testing was all about testing the functional requirements that could be coded. Non-functional requirements was left for the specialists, or plainly disregarded. I know I have done my share of test planning, with a range of requirements left “N/A” with regards to testing. Especially performance scope, batch jobs, hardware specs, data base table expansions and virus scanning have been left out of my functional test plans…
When I look at a list of requirements now – I see that we can indeed test all the things, or we can at least work on how to document that the requirement is fulfilled. Some of the requirements are actually quite easy to document. If it’s on a screen somewhere, take a screen shot and attach it to a simple test case. Done deal really. Additionally with a testing mind-set I can think of ways to challenge the details. But do we really, really need to fill up a disk to establish if it’s exactly a 1 Gb allocation – probably not. Do we really really need to document all requirements – yes in some contracts/contexts it’s important for the customer to know that everything has indeed been established. Sometimes the customer doesn’t trust you otherwise, sometimes the tests are more about your ability to deliver and provide evidence that matters.
Test all the business risks
Look into the business case of your project and find the business risks. Sometimes they are explicitly stated and prioritized. A a recent Ministry of Testing Meetup we looked into a case for a large national health system. We looked at the tangible benefits, intangible benefits and on the scored business risks. What worried the business and management most was budget, time and whether the new system would be used in a standardized way. There is an opportunity for testing here to help address, document and challenge the most important business risks. Traditional testing would usually look at functional requirements that can be coded or configured, and miss totally to address what matters most to the business.
OK, how do we test the project costs? How about frequent checkpoints of expected spending – would that be similar to tracking test progress. Perhaps – let’s find out. Testing is all about asking questions for the stakeholders and solving the most important problems first. Can we help to analyse risks and setup mitigation activities – sure we can. We just have to step out of our traditional “software only bubble”.
To me Shift-Left is still an active trend and change how to do testing. This goes along with Shift-Right,Shift-Coach and Shift-Deliver discussed separately. I discussed these trend labels at Nordic Testing Days 2016 during the talk “How to Test in IT operations“.
Microsoft have similar “Software Design Engineer in Test” as discussed by Alan Page in “The SDET Pendulum” and in the e-book “A-word“
A project I was regarding pharmaceutical Track and Trace, had no testers. I didn’t even test but did compliance documentation of test activities. The developers tested. First via peer review, then via peer execution of story tests and then validation activities. No testers, just the same team – for various reasons.
A project I was in regarding a website and API for trading property information had no testers, but had continuous build and deploy with even more user oriented test cases that I could ever grab. (see: Fell in the trap of total coverage)
The general approach to Shift-Left is that “checking” moves earlier in the cycle in form of automation. More BDD, more TDD, more automated tests, continuous builds, frequent feedback and green bars. More based on “Test automation pyramid” (blog discussion, whiteboard testing video). Discussing the pyramid model reveals that testing and checking goes together in the lower levels too. I’m certain that (exploratory) testing happens among technicians and service-level developers; – usually not explicitly, but still.
To have “no QA” is not easy. Not easy on the testers because they need to shift and become more SET/SDET-like or shift something else (Shift-Right and Shift-Coach and Shift-Deliver). Neither is it easy on the team, as the team has to own the quality activities – as discussed in “So we’re going “No QA’s”. How do we get the devs to do enough testing?”
Testers and test managers cannot complain, when testing and checking is performed in new ways. When tool-supported testing take over the boring less-complex checks, we can either own these checks or move to facilitate that these checks are in place. Similarly when the (exploratory) brain-based testing of the complex and unknown is being handed over to some other person. Come to think of it I always prefer testing done by subject matter experts in the project, be it users, clients, testers or other specialists.
We need to shift to adapt to new contexts and new ways of aiding in delivering working solutions to our clients.
I rarely test software these days. I mostly lead testing of IT solutions.
Testing in the context of:
Updating all corporate PC’s from windows 7/8 to Windows 10
Consolidating network equipment from more devices to one box, on 80 global locations *
Move 40 live business applications from one data center to another *
Take over application maintenance for a specialized public organization
Implement track and trace for pharma products from production to shops
Migrate HR data for 2500 people from one system platform to another
Yes, it happens that I participate in a project that is about developing a new business application, but my activities are less about testing software and more about testing in IT solutions in general.
Mostly I manage test activities and describe testing in these contexts. My preferred way of working is in setting and implementing test strategies. I prefer complex and non-ordered projects (Complex and Chaotic – I’m looking at you), it fits well with my context-driven approach of finding the “test solution that fits the context”.
Testing is in it self a solution, that must solve a business problem. Great testing is all about providing information to the stakeholders. I don’t care especially if this is done by someone TESTING or a TESTER. It is my responsibility to setup the testing activities (information gathering) that supports the team, faces the business & technology and challenges the product “sufficiently“.
Sometimes “sufficiently” is merely confirming and going through the motions of explicit requirement coverage. This is a special challenge to me, as I know of many effective and Rapid approaches, that could add valuable information. When I face this challenge, I try to look at the full picture of the project, and what the business want’s to achieve.
The business of the business is business. What matters is not software or projects, but the solutions to the challenges the business have. And the context of testing is similarly so much more than the software.