Assumptions of the Test Pyramid

This may be a heresy to some… While the Test Automation Pyramid as a model may be right in many contexts, – but the model will be similarly wrong in other test automation contexts.

First let’s look at one of the assumptions of the Test Automation Pyramid:

Martin Fowler, 2012

Fowlers assumption (2012) is that UI automation is slow and expensive. Similarly Cohen (2009) writes that testing in the UI is “brittle, expensive, and time consuming“. Recently (2019) there have been developed at least two types of tools that break those assumptions – and make it relatively faster and cheaper to have automated GUI tests than before.

Example 1: Tools like Applitools Eyes let’s you do prepare test automation code that compare images of the UI. Angie Jones has an excellent code example of how to compare PDF files.

Example 2: Robot Desktop Automation tools gives the possibility of automating and autotomize end user business processes. These kind of tools can be used to write, maintain and schedule end user activities.

I have performed an analysis that shows that using RDA for test automation has similar costs and speed as with using Selenium for test automation … but then not all projects are web projects.

Still, the underlying assumption of both Applitools,the pyramid above and even Bach’s earth model is that the system under test consists of accessible code on the service and unit layers.

UI testing may be all there is!

In the context of Software-as-a-Service, standard commercially packaged applications and solutions – the business still want to test the system they are starting to use, but they have no access to the code. While they must reasonably expect the vendor to have tested the solution, the business implementing the IT package would want to test it in their setting using their own people.

As testing professionals we can help the business both not to request the kitchen sink, while also test all the things (that matter). As with all other testing – even the dreaded UAT – some of it is simple repeatable tasks (checks) while others are more subtle experiments (tests).

Perhaps we can estimate a ratio between the checks and the tests? Perhaps that ratio has more checks..? That would depend on what the business would like to know (what is their perception of quality) and how well the domain is codified (Genesis / Commodity).

There is a discussion and collection of alternative pyramids on “The Club”.


Advertisements