Your Mileage Will Vary

Looking at all the podcast, conference and community chatter you could get the impression that everyone else’s projects always follows the latest trends and hottest principles. That everything is perfect, and everything is a success – and that all new ideas are working right off the bat.

First Try… ish

It almost sounds as if testing has to be in a specific way, and that what you experience is wrong or less worthy. That there are no failures, no scrum-fall projects or old legacy systems. It sounds like everything runs smoothly on an up-to-date CI/CD K8 technology stack with all the bells and whistles.

Hmm.. no.

Don’t worry, listen to the “Guilty tester podcast”

Every single project/company/context is both it’s own mess and it’s own best. There is a huge difference between all the worlds companies and all the countries traditions around IT. Sure, it may happen – let me tell you the world of IT projects is a weird place, and that’s OK. So take all the stories of successes with the added American car commercial catch phrase goes “your mileage may vary” [YMMV] or as we tend to say coming from a Context Driven point of view “It depends“.

It depends on your context if modern testing is a thing for you, it might work in an enterprise setting of commercial standard systems. Using Robot Desktop automation in testing might work better in that setting, but then again it will probably not be a good fit in your average software development project. In a context of developing software business-to-consumer the web features is more importation than in an enterprise setting. And round and round the practices and approaches goes and goes…

If you look a little beyond the borders of your own project you will see similarities to others doing the same, but also the diversity in approaches, successes and failures. You are not doing all of it wrong.

You are not doing it wrong

– you’r mileage is just different.

Advertisements

When testing web features doesn’t matter

(aka not every testing problem can be solved by a webdriver)

Web features doesn’t matter as much in the contexts I usually work in. While some may be delivered over the web, the focus for testing is on the whole system’s fit for the business. Adding automation in testing to the mix gives additional challenges as there is no source code in the solution to interact with, and we have to find other solutions to solve the tedious tasks in testing with.

One area where this is the case, is when implementing standard commercial software packages (COTS) for the enterprise or public sector. Solutions like SAP for retail CRM and ERP, Microsoft Dynamics 365 for Finance and Operations, EPIC for hospitals, Service Now for IT Service management etc. These are standard solutions that can be configured and customized, but the general source code is not available.

Thus the “test automation pyramid” falls short to help us automate things as only the GUI is available for interacting with the solution. Test engineers might want to setup CI/CD but the success of that depends on the system architecture and the provided as-is methods of releasing updates. Some of the solutions above are provided as SaaS but quite often they still run “on premise” and the business still wants releases tested before launching things on a corporate scale.

Screenshot of an RPA test on SAP

Another example is the many bespoke software solutions that are still in operation. My local electronics store has two interfaces for the sales persons: web to look things up (specs and availability) and a mainframe system to set up the actual purchase (Point of sales, POC). Many public organisations and enterprises are only now transitioning from the desktop applications of the 1990’es to more up to date solutions. Unfortunately systems that are 10+ years old have very little of live and relevant specifications and neither CI/CD suites.

While some COTS and POC solutions are being delivered over the web, testing web particularities the very least of our focus areas. The web particularities seems to matter more if the solutions are business-to-consumer but not so much when it’s business-to-employer or business-to-business.

In a business-to-employer and business-to-business context, usually only one browser is in scope. And that’s it. There is little interest for HTTP status codes, broken links, browser compatibility or login forms.

The primary testing challenges of these projects cannot be solved by Selenium, Cypress or neat tweaks in the latest JavaScript library.

Focusing only on testing the web in contexts like these we fail in

  • covering the whole system landscape across applications of different technologies
  • addressing the real questions of the business subject matter experts and the business

It’s in this context that RPA probably has some benefits in providing automation of tedious testing tasks to the tester with a business background. That is, they are business people first – and then they do the testing that matters to the context.

The Guilty Tester Episode 7 – – With A Little Help From New Friends

Do you ever feel guilty for not meeting the standards set by others in the Software Testing community? You’re in the right place then.

In this episode I talk to Dave (@theguiltytester). We discuss traditions, open questions and how to work within contracts which are specifically requesting traditional test practices based on large numbers of test cases. 

Listen and read all about it here: http://theguiltytester.libsyn.com/the-guilty-tester-episode-4-jesper-ottosen-with-a-little-help-from-new-friends 

Some of the blog posts mentioned are: 

New friends - the subject matter experts of all trades
New friends – the subject matter experts of all trades

Could Modern Testing work in the enterprise

So far I have mostly thought that “Modern Testing” of the A/B testing podcast would never work in an enterprise context. But it seems some tools and existing approaches in the enterprises already fits well with the ideas of the concept. 

The enterprise is all the privately owned companies that usually manufacture (non-IT) things –  for either the consumer or other businesses. The enterprise sell and produce tangible products like windmills, power tools, dairy consumer products etc. The interest with regards to IT for the enterprise is that it just works, and supports business processes around order setup, order tracking and invoicing – and many other moving parts.

While I have heard of some organisations that have successfully implemented some agile and SaFe methods (in banking), the enterprise have a hard time to change mode of operations, as it usually comes down to actual production of things, logistics and hierarchies of command-and-control … and culture, most of all culture.

 

@  via @HelenLisowski

 

Some enterprises change towards being learning organisations, but still treat their IT in general as low-value and an annoying cost. It seems the IT departments and IT contractors have a challenge in talking about what they do to achieve the right quality for the businesses….

 

 

Que: The Modern testing mission on “Accelerate the achievement of Shippable Quality”. While MT is mostly a concept around transition of testing activities, it seems the concept applies to IT delivery teams in general. MT has 7 principles and some of these are:

5. We believe that the customer is the only one capable to judge and evaluate the quality of our product

Most enterprise projects I know off around implementing SAP, MS dynamics 365, EPIC hospital solutions etc, have a large reliance on end-user testing and UAT. Often there is no professional testers involved, as the best tester is the business experts themselves. Interestingly the principle #5 fit’s well with existing practices from the UAT space.

Another interesting MT principle is #6 around data analysis of actual customer usage. This requires some totally different tooling for the tester, than previously generally available (…besides shifting-right perhaps…).

6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.

Yet recently I was investigating Panaya Autonomous Testing for SAP. One thing I realized is that what the tool collects real user usage of SAP and then provides the ability to balance the testing activities based on that analysis. It is interesting to see a commercial test management product for the enterprises following new trends like the “modern tester”.

While it’s interesting how some of the concepts of modern testing are reflected in testing in the enterprise – and vise versa –  the challenge remains for both the tools and concepts to be applied and accepted in more organisations. 

It might not fit everywhere, but it might be a good fit in more places than you think it would. 

 

Test ALL the things

TL;DR: We can add testing to all requirements and all business risks. Testing to document requirements and to debunk risks provides valuable information for the business. Let us not limit testing to things that can be coded. The intellectual activity of trial and learning is happening anyways, we might as well pitch in with ways to find important evidence for the decision makers.

Test all the requirements

Traditionally testing was all about testing the functional requirements that could be coded. Non-functional requirements was left for the specialists, or plainly disregarded. I know I have done my share of test planning, with a range of requirements left “N/A” with regards to testing. Especially performance scope, batch jobs, hardware specs, data base table expansions and virus scanning have been left out of my functional test plans…

When I look at a list of requirements now – I see that we can indeed test all the things, or we can at least work on how to document that the requirement is fulfilled. Some of the requirements are actually quite easy to document. If it’s on a screen somewhere, take a screen shot and attach it to a simple test case. Done deal really. Additionally with a testing mind-set I can think of ways to challenge the details. But do we really, really need to fill up a disk to establish if it’s exactly a 1 Gb allocation – probably not. Do we really really need to document all requirements – yes in some contracts/contexts it’s important for the customer to know that everything has indeed been established. Sometimes the customer doesn’t trust you otherwise, sometimes the tests are more about your ability to deliver and provide evidence that matters.

Test all the business risks

Look into the business case of your project and find the business risks. Sometimes they are explicitly stated and prioritized. A a recent Ministry of Testing Meetup we looked into a case for a large national health system. We looked at the tangible benefits, intangible benefits and on the scored business risks.  What worried the business and management most was budget, time and whether the new system would be used in a standardized way. There is an opportunity for testing here to help address, document and challenge the most important business  risks. Traditional testing would usually look at functional requirements that can be coded or configured, and miss totally to address what matters most to the business.

OK, how do we test the project costs? How about frequent checkpoints of expected spending – would that be similar to tracking test progress. Perhaps – let’s find out. Testing is all about asking questions for the stakeholders and solving the most important problems first. Can we help to analyse risks and setup mitigation activities – sure we can. We just have to step out of our traditional “software only bubble”.

MEME - Test ALL the things
Meme ALL the things

 

 

 

 

 

 

 

Read also: Many Bits under the Bridge, Less Software, more TestingTest Criteria for Outsourced SoftwareThe Expected, Fell in the trap of total coverage.

Links: “A Context-Driven Approach to Delivering Business Value”, Cynefin In Software TestingTesting during Application Transition Trials

 

Trending: Shift-Left

TL;DR: Shift-Left is about testing early and automated. Shift technical with this trend or facilitate that testing happens.

Shift-Left is the label we apply when testing moves closer to development and integrated into the development activities. The concept is many IT years old, and there are already some excellent articles out there: What the Shift Left in Testing Means (Smart Bear, no date), “Shift left” has become “drop right” (Test Plant 2014), Shift Left QA. How to do it. Why it matters (Work Soft, 2015).

To me Shift-Left is still an active trend and change how to do testing. This goes along with Shift-Right, Shift-Coach and Shift-Deliver discussed separately. I discussed these trend labels at Nordic Testing Days 2016 during the talk “How to Test in IT operations“.

Here are some contexts where Shift-Left happens:

  • Google have “Software Engineer in Test” as job title according to the book “How we Test Software at Google“.
  • Microsoft have similar “Software Design Engineer in Test” as discussed by Alan Page in “The SDET Pendulum” and in the e-book “A-word
  • A project I was regarding pharmaceutical  Track and Trace, had no testers. I didn’t even test but did compliance documentation of test activities. The developers tested. First via peer review, then via peer execution of story tests and then validation activities. No testers, just the same team – for various reasons.
  • A project I was in regarding a website and API for trading property information had no testers, but had continuous build and deploy with even more user oriented test cases that I could ever grab. (see: Fell in the trap of total coverage)

The general approach to Shift-Left is that “checking” moves earlier in the cycle in form of automation. More BDD, more TDD, more automated tests, continuous builds, frequent feedback and green bars. More based on “Test automation pyramid” (blog discussion, whiteboard testing video). Discussing the pyramid model reveals that testing and checking goes together in the lower levels too. I’m certain that (exploratory) testing happens among technicians and service-level developers; – usually not explicitly, but still.

To have “no QA” is not easy. Not easy on the testers because they need to shift and become more SET/SDET-like or shift something else (Shift-Right and Shift-Coach and Shift-Deliver). Neither is it easy on the team, as the team has to own the quality activities – as discussed in “So we’re going “No QA’s”. How do we get the devs to do enough testing?

Testers and test managers cannot complain, when testing and checking is performed in new ways. When tool-supported testing take over the boring less-complex checks, we can either own these checks or  move to facilitate that these checks are in place. Similarly when the (exploratory) brain-based testing of the complex and unknown is being handed over to some other person. Come to think of it I always prefer testing done by subject matter experts in the project, be it users, clients, testers or other specialists.

We need to shift to adapt to new contexts and new ways of aiding in delivering working solutions to our clients.

jollyrum

Less Software, more Testing

I rarely test software these days. I mostly lead testing of IT solutions.

Testing in the context of:

  • Updating all corporate PC’s from windows 7/8 to Windows 10
  • Consolidating network equipment from more devices to one box, on 80 global locations *
  • Move 40 live business applications from one data center to another *
  • Take over application maintenance for a specialized public organization
  • Implement track and trace for pharma products from production to shops
  • Migrate HR data for 2500 people from one system platform to another

Yes, it happens that I participate in a project that is about developing a new business application, but my activities are less about testing software and more about testing in IT solutions in general.

Mostly I manage test activities and describe testing in these contexts. My preferred way of working is in setting and implementing test strategies. I prefer complex and non-ordered projects (Complex and Chaotic – I’m looking at you), it fits well with my context-driven approach of finding the “test solution that fits the context”.

Testing is in it self a solution, that must solve a business problem. Great testing is all about providing information to the stakeholders. I don’t care especially if this is done by someone TESTING or a TESTER. It is my responsibility to setup the testing activities (information gathering) that supports the team, faces the business & technology and challenges the product “sufficiently“.

Sometimes “sufficiently” is merely confirming and going through the motions of explicit requirement coverage. This is a special challenge to me, as I know of many effective and Rapid approaches, that could add valuable information. When I face this challenge, I try to look at the full picture of the project, and what the business want’s to achieve.

The business of the business is business. What matters is not software or projects, but the solutions to the challenges the business have. And the context of testing is similarly so much more than the software.

*: As mentioned in “How to test in IT Operations” at Nordic Testing Days 2016.

13346977_10153714694356849_242947497430826688_n