In Charge of Testing

As a Test Manager I oversee the testing in a project or program – I am usually the only testing specialist in the project, so, I need the right leadership skills and the right tools to succeed. I have to own the data about the testing and quality activities.

As the test manager I need to facilitate a quite a range of testing activities:

I need to balance that I need to know what’s going on (with regards to testing) but without micromanaging the people being involved in testing and quality activities. My role is to facilitate that testing things happen – like the project manager making project things happen. I cannot own the activities without owning the data about it. I need to cover the full spectrum of tests – from engineered (RDA and CI/CD) to people-based (scripts and exploration).

The most practical tool for a test manager with this scope is PractiTest, as there is more to testing than just the test cases [2]. The old term “ALM” [3] comes to mind – it is still relevant when I look for a full test management tool. I need to cover both the “inputs” to testing (requirements, tickets and user stories) and the “outputs” (bugs) in one location. I need the requirements and user stories in my tool, as I need to base my test analysis and planning on the delivery model (that may not always be agile). I need the bugs in the testing tool too, as bugs can happen in any work product of the project: documents, code base and even the tests. PractiTest acknowledge that there is more to IT projects than code.

I appreciate the key driver of PractiTest – that all activities happen in-flow. You don’t have to change window, stack pop-ups or go to another tool in order to run the tests or create bugs. Creating bugs happens in context of the test case and seamlessly moves all data about the run to the bug. Everything you need to do is context-based, and available to you on screen. And it has some cool features of read-only links to graphs for management reporting, and a smart built-in “rapid reporter” for exploratory testing notes.

It can be a challenge to switch to PractiTest if you are in a compliance setting, if you need on-Premise or if your team generally uses Azure DevOps (the tool formerly known as TFS). To get the full potential of Azure DevOps, though, you need the full Microsoft Test Pro licenses, so it’s not a free tool either – nor is DevOps intuitive for testing things doesn’t have the code available. As with Azure DevOps PractiTest is also SaaS only, with multiple data centers for regional data compliance. As there is always inertia towards a commodity it won’t be long before there is no good arguments to have test management tools on-Premise and for the tool vendors to provide the compliance certificates (ISO/SOC really should be sufficient, IMO).

Out of the box PractiTest supports the categories of testing above (engineered, scripted, exploratory) and has the necessary integrations too: Surefire for unit testing, Maven for CI/CD, Jira, ServiceNow or any other ITSM for requirement input. There is even a two way integration to Azue DevOps. As the web design is “responsive” it could probably run off a tablet. That would enable easier test documentation for field tests. It would be even better to have a small version of it on a phone and be able to use the camera for “screen shots”.

At work I am currently running a large project regarding customizing and implementing a standard commercial software system, PractiTest would fit right in, as we have the following test activities:

  • Unit test by the developers
  • Automation by test engineers
  • Exploratory test by Subject matter experts
  • Formal scripted testing with end users

And I need to own the data around all of this, if I want to in in charge of the testing (and not only the testers). We are very few software testing specialists on the project team, but as the manager of testing I need to cover many other people performing the testing. This transforms my role from test management to one about leadership, coaching, and facilitation of testing being performed by the SMEs – and anyone else really.

I will be talking about Leading When the Subject Matter Experts Test at ConTEST NYC 2019 until then read more about leadership:

  1. Anthropologists and similar humanities educations can be great BA’s
  2. looking at you Test Rail 😉
  3. ALM = Application Life Cycle, like Micro Focus Quality Center etc.

Disclaimer: This is an influencer review sponsored by PractiTest.


Visual Tests are Still Code

Among the currently shiny new test automation things are visual “script-less” test automation tools. But the visual test flows are still code – and thus require discipline to structure and maintain. Otherwise you are just adding yet another layer of spaghetti code.

Among the current shiny new test automation tools are visual “script-less” automation tools like LeapWork [9], Blue Prism [10] and UiPath [7]. These tools are a part of a new class of business process automation tools called “Robot Process Automation” (RPA) [4]. There are two sub types of – “RPA” which focuses on processing data and Robot Desktop Automation (RDA).

RDA is interesting in the context of test automation [9], as they can automate GUI interactions – also on top of enterprise package applications (SaaS, COTS, OOTB etc. [2]). The test automation challenge for most of these enterprise applications (SAP, MS Dynamics [6] etc.) is that they come with no access to the code-base, even if these are pure-play web based – the GUI is all there is.

All you can to these type of business solutions is usually to add customization and configurations by entering or editing data directly in through the GUI. Some of these systems allow configurations and customization in the form of config-file – they really should be under change control [3], as they are part of the pipeline. 

visual tests are code

part of the ship

part of the crew

Bootstrap Bill Turner

Using RDA tools for test automation [9] is a novel [1] uncharted approach [12]. The editing of the “tests”/flows is usually done in a stand-alone application studio (Graphical IDE) with interactions to the solution under test (across the GUI and over Citrix and RDP) and to any test management and issue tracking system.

Interestingly the other more “data processing” RPA tools like Automation Anywhere [5] uses a VB-Script like syntax. Writing and maintaining “scripts” like that is quite like the common approaches to GUI automation using frameworks like Siluki, tagUI, Applitools [11].

Applitools etc. are coding frameworks you can apply if you have the application code base or want to write test automation directly as code. There could be benefits in coding UI testing in all web-only projects directly using Selenium and Applitools. Most enterprise business solutions are often stand-alone applications, or their web code is horrible to hook into, as often the selectors seems randomly generated (been-there-done-that).

Hence the primary driver for RDA adoption in for test automation is to take the RDA & RPA [4] tools and apply their strengths in process automation of enterprise business solutions [2] to drive the test execution. And of a business flow could be “automating” activities during onboarding [7] or an SAP purchase order as below images:

Another key driver for adoption of RPA for test automation is their visual approach in presenting interactions/tests as flows. Some do it gracefully and user-friendly (LeapWork) – others have a more old-school workflow/swim lane approach (Blue Prism, UiPath). In both cases the visual flows illustrate an interaction across multiple GUI applications to perform business actions (yes, this still happens).

These drivers probably to make the barrier to entry seem more manageable. The visual ones very easily turn into visual spaghetti code if you don’t keep an eye on it and use sub flows, low coupling and high cohesion [13].  … as with any other non-trivial code (of a certain McCabe complexity [14]). One interesting way to go about a “coding” practice for visual test cases could be inspired by how BDD can be implemented in LeapWork [8] with annotation and self-referencing unit tests.

At the end of the day even a visual test automation project is a coding project, that should be part of the project code base like everything else [3]. And probably best maintained by software engineers within the project team (where possible) – unless you want a team of test engineers spending all day playing catch-up to maintain the automation code.

  1. Since 2017’ish.
  2. COTS/OOTB = Commercial of the shelf, out of the box

A Ratio between Tests

During one of my recent projects I was considering the ratio between the checks and the tests – that is the ratio between those tests, that are primarily simple binary confirmations, and those tests that are more tacit questions. This blog is about my considerations on the idea/experiment/model.

First I observed, that we have a range of different items in our requirements – some of them are [actual copy of the current specification]:

Binary Confirmations

  • It must be possible to add a customer ticket reference
  • It must be possible to copy the ticket number

Tacit Questions

  • You must be able to navigate displayed lists easily
  • It must be easy to compare work log and audit log

You could argue that they need refinement, more testability and less “easy“. But this is what we have to work with for now. Even if we had all the time in the world (we don’t) – we would not be able to write all of the requirements in a perfect form (if such a form exists).

As the system under test is a commercial standard system, some of the requirements are even given as “Out of the Box”, we will probably not even be testing those explicitly. Our coverage criteria is not ALL OF THEM.

Ordering the tests

It is a deliberate experiment from my side to divide the requirements (and hence the tests) into the piles of Closed and Open Questions. Perhaps there is even three piles – Rapid Software Testing has: human checking, machine checking and human/machine checking , Wardley has Pioneers, Settlers and Town Planners. Perhaps the Rule of Three applies here too.. perhaps it’s an continuum … let’s see.

Perhaps it’s a continuum

As part of the requirement workshops I will label the requirements and align with the stakeholders to get the expectations right – with the help of a few friends. This a context/project based “operationalization“.

I wrote about this ratio on my blog post around the Test Automation Pyramid, as I will use the labels to automate the confirmations (and only the confirmations). The assumption is, that there are significantly more of the binary requirements tested by machine checking – and more human tested tacit questions. If we can get all the tedious tasks automated – that is the really the end goal.

Automate all the things that should be automated

Alan Page

Every project/context will have it’s own ratio, depending on a range of factors. Saying there should always be more of one type than the other would not hold. As the above project is the configuration and implementation of a standard commercial business software package (like SAP, SalesForce etc), my expectation is that most of the requirements are binary. Also considering that this project is heavy on the right hand side of the Wardley Map scale of evolution.

It’s a Reduction in Context

I am well aware that the two/three piles are an approximation / reduction. Especially when looking at the “binary” requirements and “only” testing these by confirmation. They could as easily be explored to find further unknown unknowns. If we prioritize to do so – it all about our choice of risk.

It is also an limitation as “perfect testing” should consist of both testing and checking. I factor this into the test strategy, by insisting that all of the requirements are tested both explicitly and implicitly. First of all most of our binary requirements are on the configuration and customization of the out-of-the-box software solution. So when the subject matter experts are performing the testing of the business flows, they are also evaluating the configuration and customization. And I do want them to spot when something is odd

The binary configuration is ok, but human know-how tells us otherwise.

Ultimately I want to use the experts to do the thinking and the machines to do the both the confirmations and the tedious tasks.

Career paths for testing specialists

I believe it is possible to have challenging opportunities and career advancement within the broad field of testing – for all kinds of people and backgrounds.

I’m probably both spoiled and privileged, so see down below for context of the following model. It is a model for career paths that is in active use as of writing. Some might consider it old fashioned or limited, but I do hope that you can learn a bit from it with regards to defining career paths for testing specialists.

Let’s look at the following titles:

  • Tester,
    • prepares test cases in a test case management tool
    • performs the testing activity
  • Technical Test Analyst,
    • prepares and initiate engineered tests
  • Test Manager,
    • prepares test plans, test strategies
    • lead the testing activity

You might have other titles at your place – the point is to identify the titles and not take the work areas too literally. On smaller delivery teams the testing specialist is both the analyst, test manager and the one performing the tests. On larger projects there may be more testing professionals with more defined roles/titles. On other projects the test managers job is more around leading SMEs in testing (and less the testers).

Notice that the test manager manages the testing, she is not a people/line manager of the testing specialists. All the testing professionals may have the same manager or be distributed into the delivery teams. That usually depends on if the company’s focus is on consultancy/projects or on products/deliveries.

There are strong trends that much test engineering is a developers discipline and that management of testing is more about coaching. Still some development organisations strive with having exploratory testers on all teams, where they dig into the specific domain and application. But the field is not all dead yet, and many organisations will have the above titles represented for years.

Based on the titles above you can identify the work usually being done, but not the skill level or span of control. This is where we usually add (promotion) levels like:

  • Junior
  • Advanced/Associate
  • Senior
  • Principal

Do use the promotions that your company uses for developers (and similar titles) and other roles! If you have ninja developers as a formal promotion level over lead developers, the by all means add that to your testing titles as well. Do insist and argue! If you fail, move away and let that company deal with not wanting to improve their people. (having the option to turn away can be a privilege too).

The levels of formal training could follow the levels of promotion. ISTQB training is one approach that is similarly scaled. That can be helpful if your organisation has a quest for certificates (for some business reason). Certificates are though just a race to the bottom.

The advancement from one level to the next could be on the basis of independence of the person. A junior level is an entry level and will usually require that the person tries it out and have a skills mentor. Advance and associate levels apply the know-how consistently on one project/delivery team.

The higher up the level, the more teams the person can apply the knowledge to at the same time (span of control, see also the law of raspberry jam). Alternatively it could also be to be able to generalize practices learned in one project and apply it to a project/delivery team that works in a different way. Senior and principal roles is more into the strategic work or work as a adviser. They could be the advisers on bids & tenders for new projects or be more of an test architect working with implementing principles for the testing activities.

Context: I work in the Danish IT outsourcing sector in a global IT company of 3000+ people. The software testing team working across projects is 30+ people (globally). The title “promotions” are used consistently in the company for various job titles: Developers, project managers etc. I have applied a similar model for 20+ testing people at other outsourcing companies and the job titles are consistent with similar software development companies in Denmark.

Denmark is a country where trust in others is very high, where there is a high wealth equality level (Gini) and where women have equal privileges to men. In the company 30% is female, with many female “middle managers” and also high in the technical hierarchy.

When testing web features doesn’t matter

(aka not every testing problem can be solved by a webdriver)

Web features doesn’t matter as much in the contexts I usually work in. While some may be delivered over the web, the focus for testing is on the whole system’s fit for the business. Adding automation in testing to the mix gives additional challenges as there is no source code in the solution to interact with, and we have to find other solutions to solve the tedious tasks in testing with.

One area where this is the case, is when implementing standard commercial software packages (COTS) for the enterprise or public sector. Solutions like SAP for retail CRM and ERP, Microsoft Dynamics 365 for Finance and Operations, EPIC for hospitals, Service Now for IT Service management etc. These are standard solutions that can be configured and customized, but the general source code is not available.

Thus the “test automation pyramid” falls short to help us automate things as only the GUI is available for interacting with the solution. Test engineers might want to setup CI/CD but the success of that depends on the system architecture and the provided as-is methods of releasing updates. Some of the solutions above are provided as SaaS but quite often they still run “on premise” and the business still wants releases tested before launching things on a corporate scale.

Screenshot of an RPA test on SAP

Another example is the many bespoke software solutions that are still in operation. My local electronics store has two interfaces for the sales persons: web to look things up (specs and availability) and a mainframe system to set up the actual purchase (Point of sales, POC). Many public organisations and enterprises are only now transitioning from the desktop applications of the 1990’es to more up to date solutions. Unfortunately systems that are 10+ years old have very little of live and relevant specifications and neither CI/CD suites.

While some COTS and POC solutions are being delivered over the web, testing web particularities the very least of our focus areas. The web particularities seems to matter more if the solutions are business-to-consumer but not so much when it’s business-to-employer or business-to-business.

In a business-to-employer and business-to-business context, usually only one browser is in scope. And that’s it. There is little interest for HTTP status codes, broken links, browser compatibility or login forms.

The primary testing challenges of these projects cannot be solved by Selenium, Cypress or neat tweaks in the latest JavaScript library.

Focusing only on testing the web in contexts like these we fail in

  • covering the whole system landscape across applications of different technologies
  • addressing the real questions of the business subject matter experts and the business

It’s in this context that RPA probably has some benefits in providing automation of tedious testing tasks to the tester with a business background. That is, they are business people first – and then they do the testing that matters to the context.

Who is the tester?

In my current and primary projects the testing is not done by software testing professionals – and it’s probably for the better too! It is in contexts like these:

  1. A Microsoft Dynamics “D365O” implementation of health registration forms. Tested by public service clerks partly comparing to the previous solution, partly testing the new system platform.
  2. Moving 700+ servers running 50+ applications from one data center to another while keeping everything from mainframe to SaaS integrations live. Tested by the application staff that have maintained the system since for ever (10+ years).
  3. Implement at standard commercial off-the-shelf tool for 2000+ IT savvy users. To most users this tool is their primary work tracking system, so they get to test it too.

In contexts like these the act of testing done by subject matter experts of the field – infrastructure specialists, public service clerks, support staff, application developers and the like. These persons qualify as the “customer” in the Modern Testing Principle that “the customer is the only one capable to judge and evaluate the quality of our product“. They might have a testing /role/ during the project, but that is because of their high domain knowledge, but at the end of the project they continue with their “real business job” of using the system to produce stuff for the business.

It’s not their job to know ISTQB from “MT Principles” and “RST methodology“. That is up to me, as the manager of the testing. My role is more and more about the guidelines for the testing and the facilitation of the people doing the testing. My reach goes so far as to ask them to think about how the product fails and succeeds. But I cannot expect them to know checking from testing.

Long gone are the days of managing testers that put all their skill into the niches of the testing craft. There are less software testing professionals doing the testing in projects like the above. Part of it is, that the describing the whole system explicitly is simply to expensive in time and money. This makes the requirements inherently fuzzy and undefined. And part of it is that learning the skills simply takes to long. Some technical tests require skills of a certified VMware specialist, others having an eye for every unwritten tacit business rule.

Another angle is that the skills that the usual software testing specialist brings to the table is handled on a lower level. Testing is done by the organisation (like Microsoft) that builds the standard solutions and commercial off the self systems (SAP, D365O etc). Another is that the test techniques of the software testing field simply no longer applies. I mean how does boundary value analysis add value to enterprise data center transition executions, when the system under test it not even software?

The better tester is neither the software developer nor the software testing specialist. It’s the person who ponders:

  • How could this go wrong…
  • I wonder if…
  • For this to work, we need to do…

Come to think of it, everyone in the project does that! Some do it more explicitly, some do it more experimental. Everyone evaluates how their actions add value to the people that matter (at some time).

Rant on Login Screen examples

If you are demonstrating testing technologies or testing examples around RPA, ML, Selenium and so on – Please: DO NOT USE A LOGIN FORM!

The test scenarios I usually deal with are not this… mundane. While a few testers probably still have to build login forms from scratch, a login feature is a commodity by now. Use OAuth for public facing sites and Active Directory Federation internally in the organisation. Really – there’s no need to reinvent the wheel. To the end user and even the Product Owner logging in is just a stepping stone. 

I just want to log in, and then I’m done for the day

Said no user ever*

Showing that you can train an AI network or other framework to login might solve a tedius testing task, but is usually not the thing I’m after. When a user is logged in, they are there to solve something, to process something, to do something – to engage with something. And this is where the best tests are heading too – this is the tests that adds value to the business and tells something about the product. 

For instance: In one project I did, we disabled the login entirely to make the CI/CD run feature testing. The plain login screen was temporary anyways, as the solution authentication would be based on certificates. We never spend much time on it, neither on total coverage.

Less combinations – More Real Life Scenarios

So what can you use as an example instead of login boxes and combinatoric bar stories? How about anonymizing the latest test you had to do on your latest live-action testing project? This will tell me about your challenges, your business domain and when the last time was – among other things.

Let me start, as of writing the latest test case I touched (same day as writing this) was for a new public data registration project. The tester and end user “subject matter expert” was testing the data registration form from both a GUI and web-services perspective. Export the entered data form as XML and import it again via web-services to see the consistency.

Could that be tool supported – maybe. The knowledge about the system is not very explicit. It’s a bit complicated, actually. Could it be trained by an ML/AI system – I doubt it. There is no global training set for this class of systems – we have the old version but are adding new things. 

If you are (a tool vendor) demonstrating something – do try to understand the context and problems your customers are trying to solve first. Ask them what their latest tests were and where the challenges are. If it’s a login box you’re good to go, but I doubt it. 

*: Unless you are somehow measured on “logging in”. For instance to claim unemployment benefit, login to job site daily…Snap! OTOH shallow measure.