A Ratio between Tests

During one of my recent projects I was considering the ratio between the checks and the tests – that is the ratio between those tests, that are primarily simple binary confirmations, and those tests that are more tacit questions. This blog is about my considerations on the idea/experiment/model.

First I observed, that we have a range of different items in our requirements – some of them are [actual copy of the current specification]:

Binary Confirmations

  • It must be possible to add a customer ticket reference
  • It must be possible to copy the ticket number

Tacit Questions

  • You must be able to navigate displayed lists easily
  • It must be easy to compare work log and audit log

You could argue that they need refinement, more testability and less “easy“. But this is what we have to work with for now. Even if we had all the time in the world (we don’t) – we would not be able to write all of the requirements in a perfect form (if such a form exists).

As the system under test is a commercial standard system, some of the requirements are even given as “Out of the Box”, we will probably not even be testing those explicitly. Our coverage criteria is not ALL OF THEM.

Ordering the tests

It is a deliberate experiment from me side to divide the requirements (and hence the tests) into the piles of Closed and Open Questions. Perhaps there is even three piles – Rapid Software Testing has: human checking, machine checking and human/machine checking , Wardley has Pioneers, Settlers and Town Planners. Perhaps the Rule of Three applies here too.. perhaps it’s an continuum … let’s see.

Perhaps it’s a continuum

As part of the requirement workshops I will label the requirements and align with the stakeholders to get the expectations right – with the help of a few friends. This a context/project based “operationalization“.

I wrote about this ratio on my blog post around the Test Automation Pyramid, as I will use the labels to automate the confirmations (and only the confirmations). The assumption is, that there are significantly more of the binary requirements tested by machine checking – and more human tested tacit questions. If we can get all the tedious tasks automated – that is the really the end goal.

Automate all the things that should be automated

Alan Page

Every project/context will have it’s own ratio, depending on a range of factors. Saying there should always be more of one type than the other would not hold. As the above project is the configuration and implementation of a standard commercial business software package (like SAP, SalesForce etc), my expectation is that most of the requirements are binary. Also considering that this project is heavy on the right hand side of the Wardley Map scale of evolution.

It’s a Reduction in Context

I am well aware that the two/three piles are an approximation / reduction. Especially when looking at the “binary” requirements and “only” testing these by confirmation. They could as easily be explored to find further unknown unknowns. If we prioritize to do so – it all about our choice of risk.

It is also an limitation as “perfect testing” should consist of both testing and checking. I factor this into the test strategy, by insisting that all of the requirements are tested both explicitly and implicitly. First of all most of our binary requirements are on the configuration and customization of the out-of-the-box software solution. So when the subject matter experts are performing the testing of the business flows, they are also evaluating the configuration and customization. And I do want them to spot when something is odd

The binary configuration is ok, but human know-how tells us otherwise.

Ultimately I want to use the experts to do the thinking and the machines to do the both the confirmations and the tedious tasks.

Advertisements

Career paths for testing specialists

I believe it is possible to have challenging opportunities and career advancement within the broad field of testing – for all kinds of people and backgrounds.

I’m probably both spoiled and privileged, so see down below for context of the following model. It is a model for career paths that is in active use as of writing. Some might consider it old fashioned or limited, but I do hope that you can learn a bit from it with regards to defining career paths for testing specialists.

Let’s look at the following titles:

  • Tester,
    • prepares test cases in a test case management tool
    • performs the testing activity
  • Technical Test Analyst,
    • prepares and initiate engineered tests
  • Test Manager,
    • prepares test plans, test strategies
    • lead the testing activity

You might have other titles at your place – the point is to identify the titles and not take the work areas too literally. On smaller delivery teams the testing specialist is both the analyst, test manager and the one performing the tests. On larger projects there may be more testing professionals with more defined roles/titles. On other projects the test managers job is more around leading SMEs in testing (and less the testers).

Notice that the test manager manages the testing, she is not a people/line manager of the testing specialists. All the testing professionals may have the same manager or be distributed into the delivery teams. That usually depends on if the company’s focus is on consultancy/projects or on products/deliveries.

There are strong trends that much test engineering is a developers discipline and that management of testing is more about coaching. Still some development organisations strive with having exploratory testers on all teams, where they dig into the specific domain and application. But the field is not all dead yet, and many organisations will have the above titles represented for years.

Based on the titles above you can identify the work usually being done, but not the skill level or span of control. This is where we usually add (promotion) levels like:

  • Junior
  • Advanced/Associate
  • Senior
  • Principal

Do use the promotions that your company uses for developers (and similar titles) and other roles! If you have ninja developers as a formal promotion level over lead developers, the by all means add that to your testing titles as well. Do insist and argue! If you fail, move away and let that company deal with not wanting to improve their people. (having the option to turn away can be a privilege too).

The levels of formal training could follow the levels of promotion. ISTQB training is one approach that is similarly scaled. That can be helpful if your organisation has a quest for certificates (for some business reason). Certificates are though just a race to the bottom.

The advancement from one level to the next could be on the basis of independence of the person. A junior level is an entry level and will usually require that the person tries it out and have a skills mentor. Advance and associate levels apply the know-how consistently on one project/delivery team.

The higher up the level, the more teams the person can apply the knowledge to at the same time (span of control, see also the law of raspberry jam). Alternatively it could also be to be able to generalize practices learned in one project and apply it to a project/delivery team that works in a different way. Senior and principal roles is more into the strategic work or work as a adviser. They could be the advisers on bids & tenders for new projects or be more of an test architect working with implementing principles for the testing activities.

Context: I work in the Danish IT outsourcing sector in a global IT company of 3000+ people. The software testing team working across projects is 30+ people (globally). The title “promotions” are used consistently in the company for various job titles: Developers, project managers etc. I have applied a similar model for 20+ testing people at other outsourcing companies and the job titles are consistent with similar software development companies in Denmark.

Denmark is a country where trust in others is very high, where there is a high wealth equality level (Gini) and where women have equal privileges to men. In the company 30% is female, with many female “middle managers” and also high in the technical hierarchy.

When testing web features doesn’t matter

(aka not every testing problem can be solved by a webdriver)

Web features doesn’t matter as much in the contexts I usually work in. While some may be delivered over the web, the focus for testing is on the whole system’s fit for the business. Adding automation in testing to the mix gives additional challenges as there is no source code in the solution to interact with, and we have to find other solutions to solve the tedious tasks in testing with.

One area where this is the case, is when implementing standard commercial software packages (COTS) for the enterprise or public sector. Solutions like SAP for retail CRM and ERP, Microsoft Dynamics 365 for Finance and Operations, EPIC for hospitals, Service Now for IT Service management etc. These are standard solutions that can be configured and customized, but the general source code is not available.

Thus the “test automation pyramid” falls short to help us automate things as only the GUI is available for interacting with the solution. Test engineers might want to setup CI/CD but the success of that depends on the system architecture and the provided as-is methods of releasing updates. Some of the solutions above are provided as SaaS but quite often they still run “on premise” and the business still wants releases tested before launching things on a corporate scale.

Screenshot of an RPA test on SAP

Another example is the many bespoke software solutions that are still in operation. My local electronics store has two interfaces for the sales persons: web to look things up (specs and availability) and a mainframe system to set up the actual purchase (Point of sales, POC). Many public organisations and enterprises are only now transitioning from the desktop applications of the 1990’es to more up to date solutions. Unfortunately systems that are 10+ years old have very little of live and relevant specifications and neither CI/CD suites.

While some COTS and POC solutions are being delivered over the web, testing web particularities the very least of our focus areas. The web particularities seems to matter more if the solutions are business-to-consumer but not so much when it’s business-to-employer or business-to-business.

In a business-to-employer and business-to-business context, usually only one browser is in scope. And that’s it. There is little interest for HTTP status codes, broken links, browser compatibility or login forms.

The primary testing challenges of these projects cannot be solved by Selenium, Cypress or neat tweaks in the latest JavaScript library.

Focusing only on testing the web in contexts like these we fail in

  • covering the whole system landscape across applications of different technologies
  • addressing the real questions of the business subject matter experts and the business

It’s in this context that RPA probably has some benefits in providing automation of tedious testing tasks to the tester with a business background. That is, they are business people first – and then they do the testing that matters to the context.

Who is the tester?

In my current and primary projects the testing is not done by software testing professionals – and it’s probably for the better too! It is in contexts like these:

  1. A Microsoft Dynamics “D365O” implementation of health registration forms. Tested by public service clerks partly comparing to the previous solution, partly testing the new system platform.
  2. Moving 700+ servers running 50+ applications from one data center to another while keeping everything from mainframe to SaaS integrations live. Tested by the application staff that have maintained the system since for ever (10+ years).
  3. Implement at standard commercial off-the-shelf tool for 2000+ IT savvy users. To most users this tool is their primary work tracking system, so they get to test it too.

In contexts like these the act of testing done by subject matter experts of the field – infrastructure specialists, public service clerks, support staff, application developers and the like. These persons qualify as the “customer” in the Modern Testing Principle that “the customer is the only one capable to judge and evaluate the quality of our product“. They might have a testing /role/ during the project, but that is because of their high domain knowledge, but at the end of the project they continue with their “real business job” of using the system to produce stuff for the business.

It’s not their job to know ISTQB from “MT Principles” and “RST methodology“. That is up to me, as the manager of the testing. My role is more and more about the guidelines for the testing and the facilitation of the people doing the testing. My reach goes so far as to ask them to think about how the product fails and succeeds. But I cannot expect them to know checking from testing.

Long gone are the days of managing testers that put all their skill into the niches of the testing craft. There are less software testing professionals doing the testing in projects like the above. Part of it is, that the describing the whole system explicitly is simply to expensive in time and money. This makes the requirements inherently fuzzy and undefined. And part of it is that learning the skills simply takes to long. Some technical tests require skills of a certified VMware specialist, others having an eye for every unwritten tacit business rule.

Another angle is that the skills that the usual software testing specialist brings to the table is handled on a lower level. Testing is done by the organisation (like Microsoft) that builds the standard solutions and commercial off the self systems (SAP, D365O etc). Another is that the test techniques of the software testing field simply no longer applies. I mean how does boundary value analysis add value to enterprise data center transition executions, when the system under test it not even software?

The better tester is neither the software developer nor the software testing specialist. It’s the person who ponders:

  • How could this go wrong…
  • I wonder if…
  • For this to work, we need to do…

Come to think of it, everyone in the project does that! Some do it more explicitly, some do it more experimental. Everyone evaluates how their actions add value to the people that matter (at some time).

Rant on Login Screen examples

If you are demonstrating testing technologies or testing examples around RPA, ML, Selenium and so on – Please: DO NOT USE A LOGIN FORM!

The test scenarios I usually deal with are not this… mundane. While a few testers probably still have to build login forms from scratch, a login feature is a commodity by now. Use OAuth for public facing sites and Active Directory Federation internally in the organisation. Really – there’s no need to reinvent the wheel. To the end user and even the Product Owner logging in is just a stepping stone. 

I just want to log in, and then I’m done for the day

Said no user ever*

Showing that you can train an AI network or other framework to login might solve a tedius testing task, but is usually not the thing I’m after. When a user is logged in, they are there to solve something, to process something, to do something – to engage with something. And this is where the best tests are heading too – this is the tests that adds value to the business and tells something about the product. 

For instance: In one project I did, we disabled the login entirely to make the CI/CD run feature testing. The plain login screen was temporary anyways, as the solution authentication would be based on certificates. We never spend much time on it, neither on total coverage.

Less combinations – More Real Life Scenarios

So what can you use as an example instead of login boxes and combinatoric bar stories? How about anonymizing the latest test you had to do on your latest live-action testing project? This will tell me about your challenges, your business domain and when the last time was – among other things.

Let me start, as of writing the latest test case I touched (same day as writing this) was for a new public data registration project. The tester and end user “subject matter expert” was testing the data registration form from both a GUI and web-services perspective. Export the entered data form as XML and import it again via web-services to see the consistency.

Could that be tool supported – maybe. The knowledge about the system is not very explicit. It’s a bit complicated, actually. Could it be trained by an ML/AI system – I doubt it. There is no global training set for this class of systems – we have the old version but are adding new things. 

If you are (a tool vendor) demonstrating something – do try to understand the context and problems your customers are trying to solve first. Ask them what their latest tests were and where the challenges are. If it’s a login box you’re good to go, but I doubt it. 

*: Unless you are somehow measured on “logging in”. For instance to claim unemployment benefit, login to job site daily…Snap! OTOH shallow measure.

http://rebrick.lego.com

With A Little Help From New Friends

Do you ever feel guilty for not meeting the standards set by others in the Software Testing community? You’re in the right place then.

In this episode I talk to Dave (@theguiltytester). We discuss traditions, open questions and how to work within contracts which are specifically requesting traditional test practices based on large numbers of test cases. 

Listen and read all about it here: http://theguiltytester.libsyn.com/the-guilty-tester-episode-4-jesper-ottosen-with-a-little-help-from-new-friends 

Some of the blog posts mentioned are: 

New friends - the subject matter experts of all trades
New friends – the subject matter experts of all trades

You don’t have to be a boss to be a leader

It’s really that simple, yet awesomely profound. And a typical Gerald Weinberg quote, like my other favorites of his points:

  • No matter how it looks at first, it’s always a people problem (The second law of consulting)
  • You’ll never accomplish anything if you care who gets the credit
  • Things are the way they are because they got that way
  • Quality is value to some person

Regarding the last quote; which was later extended with “who matters, at some time” by Bach, Bolton. Once I had an argument about how to deliver quality. The other side held towards IEEE definition of delivering the expected. But even when he did, he failed to see that the unmeasured and irrational parts affected the value to the customer. I agree completely with The Cowboy Tester that knowing works of Weinberg is a measure of seriousness.

Weinberg worked not only with testing, but among other things also consulting and organisational change management. I did not know that when reading “Making Sense of Change Management” (Cameron & Green 2012). I literally jumped up and started laughing while reading the quite serious elaborations to the Satir Change model – the authors found that Quality Software Management: Anticipating Change (1997) is a “masterly book on change, but with a title that might not appeal to everyone“. It might not appeal to change scholars, but definitely appealed indirectly to a lot of people in testing.

Recently (August 2018) Jerry died aged 84. Not a boss – yet a leader.