Automation of business tasks is no longer for the software developers only. Similarly test automation is no longer for testers and test engineers only. Both these trends help to create smarter testing performed by non-testers.
Continue readingenterprise
Episode V: Return of the ESN
In which our hero must revive old wisdom to battle empires and where the band of front runners will be supported by fuzzy friends to bring down legacy structures.
Continue readingWhen to use RPA in testing
This post is my current take on using Robot Process Automation (RPA) tools for automation in testing. RPA tools comes in different shapes, some are better at some things than others. Similarly it has to do with the tool stack of the system under test.
First some definitions/terms.
RPA/RDA definiton: I use the original Horses for Sources definitions: “increase productivity, support human operators in completing tasks and activities” (RDA) and “increase process efficiency, reduce manual labor by automating transaction intensive activities” (RPA). In more practical terms RDA is across the desktop while RPA is more about background processing – a poor man integration between systems.
The class of RPA tools: To me there are at least 4 groups of tools that all claim the RPA label: Test automation (LeapWork, Tricentis), Business Process Optimization (UiPath, BluePrism, Kofax kapow/RPA, Automation Anywhere…), Web only (Testim.io, Cypress.io) and Coding frameworks (Robot Framework, RoboCloud, AppliTools).
System under Test: There is more to a system under test than a web page with backends or an app for a smart phone. Explore the notion of System under Test. You might not have access to the source code – as the test pyramid assumes.
The enterprise challenge: Large businesses and organizations unfortunately struggle, as their IT stack is much more that web only. They want the benefits of continuous testing to their whole technology stack. Existing automation best practices doesn’t seem to address testing on top of IT systems that are older than SAFe, agile and Test-Driven Development.
Automation in Testing: See the AiT definition and namespace: “using automation to support their testing outside of automated checks“. Use tools and automation to handle all the tedious tasks of your active testing activity. Use automated checks to cover the binary requirement confirmations.
When to use RPA tools
Putting RPA tools in play in test automation do enable digital transformation and acceleration of the test activities. There are many parameters to evaluate when to use RPA tools in a testing context:
Hence, the sweet spot for using RPA tools is a as an execution muscle for mainframe solutions, commercial standard applications and legacy systems with inactive or unavailable codebases. The test management system is still key in providing an overview over all testing activities across CI/CD pipelines, RPA and tests based on human evaluation.
If your system under test is web only, you can follow the modern testing principles and build in Observability in (https://charity.wtf/) and a lot of the things in the code. Plenty of best practices around ci/cd for web systems. Obviously it depends on how well the knowledge about the system is codified – but you can work on that within your org/team too. It’s more tricky of the source code of your web SUT is not available to you and render new locators every time you deploy or refresh. … for that consider to move up the stack and use Cypress or Testim.
Remember – there are no silver bullets.
Go read Accelerate!
ACCELERATE – The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations
by NICOLE FORSGREN, JEZ HUMBLE, GENE KIM
The authors have “multiple examples of applying these practices within mainframe environments, traditional packaged software application delivery teams, and product teams“. It’s not just for business-to-consumer web-based organizations.
The book is a tour the force into software delivery performance – the research and statistics shows a clear correlation from DevOps and Lean principles to high achieving organisations. Every arrow on the below model is backed with research. Read the arrow clearly as “drives”, “improves” and “lead to”. E.g. Continuous Delivery leads to Less Burn out.
A last thing to highlight: High performing organisations have lower manual work percentages in areas like: configuration management, testing, deployments and in the (ITIL) change approval process.
So – if you want to increase the boxes on the right, go do the stuff on the left.
Read the book and act on it.
Visual Tests are Still Code
Among the currently shiny new test automation things are visual “script-less” test automation tools. But the visual test flows are still code – and thus require discipline to structure and maintain. Otherwise you are just adding yet another layer of spaghetti code.
Among the current shiny new test automation tools are visual “script-less” automation tools like LeapWork [9], Blue Prism [10] and UiPath [7]. These tools are a part of a new class of business process automation tools called “Robot Process Automation” (RPA) [4]. There are two sub types of – “RPA” which focuses on processing data and Robot Desktop Automation (RDA).
RDA is interesting in the context of test automation [9], as they can automate GUI interactions – also on top of enterprise package applications (SaaS, COTS, OOTB etc. [2]). The test automation challenge for most of these enterprise applications (SAP, MS Dynamics [6] etc.) is that they come with no access to the code-base, even if these are pure-play web based – the GUI is all there is.
All you can to these type of business solutions is usually to add customization and configurations by entering or editing data directly in through the GUI. Some of these systems allow configurations and customization in the form of config-file – they really should be under change control [3], as they are part of the pipeline.
visual tests are code
part of the ship
part of the crew
Bootstrap Bill Turner
Using RDA tools for test automation [9] is a novel [1] uncharted approach [12]. The editing of the “tests”/flows is usually done in a stand-alone application studio (Graphical IDE) with interactions to the solution under test (across the GUI and over Citrix and RDP) and to any test management and issue tracking system.
Interestingly the other more “data processing” RPA tools like Automation Anywhere [5] uses a VB-Script like syntax. Writing and maintaining “scripts” like that is quite like the common approaches to GUI automation using frameworks like Siluki, tagUI, Applitools [11].
Applitools etc. are coding frameworks you can apply if you have the application code base or want to write test automation directly as code. There could be benefits in coding UI testing in all web-only projects directly using Selenium and Applitools. Most enterprise business solutions are often stand-alone applications, or their web code is horrible to hook into, as often the selectors seems randomly generated (been-there-done-that).
Hence the primary driver for RDA adoption in for test automation is to take the RDA & RPA [4] tools and apply their strengths in process automation of enterprise business solutions [2] to drive the test execution. And of a business flow could be “automating” activities during onboarding [7] or an SAP purchase order as below images:
Another key driver for adoption of RPA for test automation is their visual approach in presenting interactions/tests as flows. Some do it gracefully and user-friendly (LeapWork) – others have a more old-school workflow/swim lane approach (Blue Prism, UiPath). In both cases the visual flows illustrate an interaction across multiple GUI applications to perform business actions (yes, this still happens).
These drivers probably to make the barrier to entry seem more manageable. The visual ones very easily turn into visual spaghetti code if you don’t keep an eye on it and use sub flows, low coupling and high cohesion [13]. … as with any other non-trivial code (of a certain McCabe complexity [14]). One interesting way to go about a “coding” practice for visual test cases could be inspired by how BDD can be implemented in LeapWork [8] with annotation and self-referencing unit tests.
At the end of the day even a visual test automation project is a coding project, that should be part of the project code base like everything else [3]. And probably best maintained by software engineers within the project team (where possible) – unless you want a team of test engineers spending all day playing catch-up to maintain the automation code.
- Since 2017’ish.
- COTS/OOTB = Commercial of the shelf, out of the box
- https://twitter.com/mipsytipsy/status/1146968926493929472
- https://www.horsesforsources.com/2019_RTS_survey_070619
- https://www.linkedin.com/pulse/automation-anywhere-example-neil-kolban/
- https://www.leapwork.com/blog/automate-testing-microsoft-dynamics-365-crm
- https://www.uipath.com/blog/how-rpa-can-help-companies-rethink-hr-tasks
- https://www.capgemini.dk/bdd-in-leapwork/#tab5
- https://dojo.ministryoftesting.com/dojo/lessons/rpa-as-a-power-tool-for-testing
- https://crunchytechbytz.wordpress.com/2018/03/13/automation-with-blue-prism/
- https://applitools.com/features
- https://jlottosen.wordpress.com/2019/04/20/broaden-the-scope-of-sut/
- https://medium.com/clarityhub/low-coupling-high-cohesion-3610e35ac4a6
- https://en.wikipedia.org/wiki/Cyclomatic_complexity
Innovation in Testing
Let’s look at testing and test management as something you can build expertise in, thus it can be placed in various places on a wardley map. Similarly innovation activities in the field of testing can be modelled by “Pioneers, Settlers, Town Planners” [also originally swardley, article by Itamar Goldminz].
The model has three types of talent: those that experiment, those that build products and those that optimize the products/commodity. Shortly put:
Each group innovates, but there is also an built-in drive from experiment to product, to optimal commodity and back again as components to experiment on. As stated in the original article (from 2015) all three kinds are brilliant people. We can relate the model both to what value the customer looks for and what kind of activity the organization strives for. We can apply it for the broader testing field as not all testing is pure play experiments and not all testing is a commodity.
Examples of Pioneer experiments could be all the fuzz around RPA, AI and ML.. and square lashings on the System Under Test – on the technology side. On the practise side, it could be emerging practices of how to test in the space of infrastructure or IT service transition. It’s the “Pippi Longstocking” of – “I have never done that before, so I probably can“.
The settler activities are all about taking the emerging practises mature them and make them repeatable. Shortcutting the time to learn something or repeat some novel practise in a new setting. Some examples could be: A Practical Guide to Testing in DevOps, the shifts of testing (at their time of writing) as ways to codify emerging practices.
Example: In 2018 I did management of testing of a large enterprise IT transition of 700 servers, running 100+ applications – it was a novel first time, so we put together some testing practices that seemed to work (for that context). In 2019 I’m doing a similar transition of similar size, where we try to repeat the practices and approaches.
The brilliant quest of the settlers is to take ideas and built innovative and established solutions for the broader audience. Most settlers are probably framework (and content) creators .. not framework maintainers.
As soon as a practise has been established it’s up to the Town Planners to maintain and optimize the practices. To me, examples in this space includes:
- Using Selenium to test web applications with
- Using BDD/Gherkins for collaboration
- Using agile practices and embed testing in the agile teams
- … following the ISTQB cook book
You mind find it harsh that I group all of those practices together. To me, they are so established by now that they can be purchased. It’s a commodity market and it’s frowned upon if you don’t use it. But still – innovation happens and town planners do a brilliant job. It’s about faster, better, smarter – and especially about building more effective teams.
Also the Town Planners build the components that the Pioneers stand upon for their next novel idea. One example could be that to test web applications with code-less test an RDA tool utilizes the Selenium framework.
Could Modern Testing work in the enterprise
So far I have mostly thought that “Modern Testing” of the A/B testing podcast would never work in an enterprise context. But it seems some tools and existing approaches in the enterprises already fits well with the ideas of the concept.
The enterprise is all the privately owned companies that usually manufacture (non-IT) things – for either the consumer or other businesses. The enterprise sell and produce tangible products like windmills, power tools, dairy consumer products etc. The interest with regards to IT for the enterprise is that it just works, and supports business processes around order setup, order tracking and invoicing – and many other moving parts.
While I have heard of some organisations that have successfully implemented some agile and SaFe methods (in banking), the enterprise have a hard time to change mode of operations, as it usually comes down to actual production of things, logistics and hierarchies of command-and-control … and culture, most of all culture.
#LLKD18 via @HelenLisowski
Some enterprises change towards being learning organisations, but still treat their IT in general as low-value and an annoying cost. It seems the IT departments and IT contractors have a challenge in talking about what they do to achieve the right quality for the businesses….
Que: The Modern testing mission on “Accelerate the achievement of Shippable Quality”. While MT is mostly a concept around transition of testing activities, it seems the concept applies to IT delivery teams in general. MT has 7 principles and some of these are:
5. We believe that the customer is the only one capable to judge and evaluate the quality of our product
Most enterprise projects I know off around implementing SAP, MS dynamics 365, EPIC hospital solutions etc, have a large reliance on end-user testing and UAT. Often there is no professional testers involved, as the best tester is the business experts themselves. Interestingly the principle #5 fit’s well with existing practices from the UAT space.
Another interesting MT principle is #6 around data analysis of actual customer usage. This requires some totally different tooling for the tester, than previously generally available (…besides shifting-right perhaps…).
6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.
Yet recently I was investigating Panaya Autonomous Testing for SAP. One thing I realized is that what the tool collects real user usage of SAP and then provides the ability to balance the testing activities based on that analysis. It is interesting to see a commercial test management product for the enterprises following new trends like the “modern tester”.
While it’s interesting how some of the concepts of modern testing are reflected in testing in the enterprise – and vise versa – the challenge remains for both the tools and concepts to be applied and accepted in more organisations.
It might not fit everywhere, but it might be a good fit in more places than you think it would.
The domain expert is the tester
Sometimes the best tester is the domain expert, the person that knows all the in’s and out’s and corners of the system. I have worked with testers that have had hands-on on a system since the late 1970’es, but I also know testers of mobile app’s that marvel in being the subject matter expert of the domain. Sometimes the professional tester doubles as agile Product Owner(1) too given their vast knowledge. The tester becomes the the SME …
The subject matter expert, though, is usually a business analyst, or perhaps a User Experience expert. Those persons might have a better stand to be testing the system, than testers with no prior knowledge. Often the SME is the best tester available. I see this happening in a shift-left setting – but also in settings with a heavy user and business involvement. Like SAP releases to enterprise systems – where the business users and SAP architects still spend a month off their “actual” work (user acceptance) testing corporate configurations and customization.
The UAT is not dead, but the classic role of the tester testing on behalf of the business is declining. The business would rather test their own, with in-house subject matter experts. The field is active, as there is tool support for this activity. Panaya(2) is a tool that specializes in managing the UAT of a corporate system like SAP, and one of the key elements is that test cases can be broken up in steps and handed over between persons. Not even classic HP ALM’s handle hand-over between testers well. While ALM’s support that the tester does the testing, Panaya supports that tests are distributed across many people. People that have other (“real business”) tasks during the work day.
Testing can also be pushed even further out to the users with crowd-based testing, beta releases etc. In both crowd-based and UAT-based testing, the role of the pro’ tester is missing but the testing is still happening. IT’s being done by the most skilled – most valuable for the task.
So what can we as testers do when our tasks are gone – skill up, go with the change and become the expert – or move out to other skills: Coaches, delivery leads etc.
- If they double as scrum master, she’s probably more a Test Coach
- this post is not sponsored. I’m just making observations – not recommendations.
Testing a new version
[The Software Testing Club blog | November 29, 2011 | Jesper Lindholt Ottosen ]
Enterprise applications often come in packages and are purchased as (Commercial Off the Shelf – COTS). Every now and then a new version of SharePoint,SAP, Jive, OeBS, Microsoft Windows…. is made available and the business and product owner decides to implement the upgrade.
Usually the setting is that there is a “Factory Acceptance Test” by the vendor of the COTS package and a “Site Acceptance Test” by the implementing IT service organization. Here are some ideas that come to my mind, the couple of times I have had look into a testing strategy for a Enterprise COTS upgrade project. It’s not a best practice – at best it’s a heuristic:-)
Regression testing first – it might be considered to examine that quality didn’t get worse. Select some of the key existing features that is most important to the product owner, and examine them. Also involving the super users or application advocates in an exploratory testing activity will provide benefits for both the testers, the super users and the other project participants.
Interfaces – in an enterprise environment there is always interfaces to legacy systems and new “bud shots” to the IT tree. SOA services makes it even more important to look for known and unknown interfaces to the application. Similarly context specific customization’s (additions and removals) and applied “production hot fixes” having been applied or constructed based on v2.5. Analyzing the intermediate versions (above example 2.3, 2.4 and finally 2.5) including known fixes and known new features, can be another approach to identify the required levels of testing. Discuss with the product manager and business representative – the key is to find a level of test that they are OK with.
Complexity is a Matter of Perspective 