Could Modern Testing work in the enterprise

So far I have mostly thought that “Modern Testing” of the A/B testing podcast would never work in an enterprise context. But it seems some tools and existing approaches in the enterprises already fits well with the ideas of the concept. 

The enterprise is all the privately owned companies that usually manufacture (non-IT) things –  for either the consumer or other businesses. The enterprise sell and produce tangible products like windmills, power tools, dairy consumer products etc. The interest with regards to IT for the enterprise is that it just works, and supports business processes around order setup, order tracking and invoicing – and many other moving parts.

While I have heard of some organisations that have successfully implemented some agile and SaFe methods (in banking), the enterprise have a hard time to change mode of operations, as it usually comes down to actual production of things, logistics and hierarchies of command-and-control … and culture, most of all culture.

 

@  via @HelenLisowski

 

Some enterprises change towards being learning organisations, but still treat their IT in general as low-value and an annoying cost. It seems the IT departments and IT contractors have a challenge in talking about what they do to achieve the right quality for the businesses….

 

 

Que: The Modern testing mission on “Accelerate the achievement of Shippable Quality”. While MT is mostly a concept around transition of testing activities, it seems the concept applies to IT delivery teams in general. MT has 7 principles and some of these are:

5. We believe that the customer is the only one capable to judge and evaluate the quality of our product

Most enterprise projects I know off around implementing SAP, MS dynamics 365, EPIC hospital solutions etc, have a large reliance on end-user testing and UAT. Often there is no professional testers involved, as the best tester is the business experts themselves. Interestingly the principle #5 fit’s well with existing practices from the UAT space.

Another interesting MT principle is #6 around data analysis of actual customer usage. This requires some totally different tooling for the tester, than previously generally available (…besides shifting-right perhaps…).

6. We use data extensively to deeply understand customer usage and then close the gaps between product hypotheses and business impact.

Yet recently I was investigating Panaya Autonomous Testing for SAP. One thing I realized is that what the tool collects real user usage of SAP and then provides the ability to balance the testing activities based on that analysis. It is interesting to see a commercial test management product for the enterprises following new trends like the “modern tester”.

While it’s interesting how some of the concepts of modern testing are reflected in testing in the enterprise – and vise versa –  the challenge remains for both the tools and concepts to be applied and accepted in more organisations. 

It might not fit everywhere, but it might be a good fit in more places than you think it would. 

 

Advertisements

Test ALL the things

TL;DR: We can add testing to all requirements and all business risks. Testing to document requirements and to debunk risks provides valuable information for the business. Let us not limit testing to things that can be coded. The intellectual activity of trial and learning is happening anyways, we might as well pitch in with ways to find important evidence for the decision makers.

Test all the requirements

Traditionally testing was all about testing the functional requirements that could be coded. Non-functional requirements was left for the specialists, or plainly disregarded. I know I have done my share of test planning, with a range of requirements left “N/A” with regards to testing. Especially performance scope, batch jobs, hardware specs, data base table expansions and virus scanning have been left out of my functional test plans…

When I look at a list of requirements now – I see that we can indeed test all the things, or we can at least work on how to document that the requirement is fulfilled. Some of the requirements are actually quite easy to document. If it’s on a screen somewhere, take a screen shot and attach it to a simple test case. Done deal really. Additionally with a testing mind-set I can think of ways to challenge the details. But do we really, really need to fill up a disk to establish if it’s exactly a 1 Gb allocation – probably not. Do we really really need to document all requirements – yes in some contracts/contexts it’s important for the customer to know that everything has indeed been established. Sometimes the customer doesn’t trust you otherwise, sometimes the tests are more about your ability to deliver and provide evidence that matters.

Test all the business risks

Look into the business case of your project and find the business risks. Sometimes they are explicitly stated and prioritized. A a recent Ministry of Testing Meetup we looked into a case for a large national health system. We looked at the tangible benefits, intangible benefits and on the scored business risks.  What worried the business and management most was budget, time and whether the new system would be used in a standardized way. There is an opportunity for testing here to help address, document and challenge the most important business  risks. Traditional testing would usually look at functional requirements that can be coded or configured, and miss totally to address what matters most to the business.

OK, how do we test the project costs? How about frequent checkpoints of expected spending – would that be similar to tracking test progress. Perhaps – let’s find out. Testing is all about asking questions for the stakeholders and solving the most important problems first. Can we help to analyse risks and setup mitigation activities – sure we can. We just have to step out of our traditional “software only bubble”.

MEME - Test ALL the things
Meme ALL the things

 

 

 

 

 

 

 

Read also: Many Bits under the Bridge, Less Software, more TestingTest Criteria for Outsourced SoftwareThe Expected, Fell in the trap of total coverage.

Links: “A Context-Driven Approach to Delivering Business Value”, Cynefin In Software TestingTesting during Application Transition Trials

 

Trending: Shift-Left

TL;DR: Shift-Left is about testing early and automated. Shift technical with this trend or facilitate that testing happens.

Shift-Left is the label we apply when testing moves closer to development and integrated into the development activities. The concept is many IT years old, and there are already some excellent articles out there: What the Shift Left in Testing Means (Smart Bear, no date), “Shift left” has become “drop right” (Test Plant 2014), Shift Left QA. How to do it. Why it matters (Work Soft, 2015).

To me Shift-Left is still an active trend and change how to do testing. This goes along with Shift-Right, Shift-Coach and Shift-Deliver discussed separately. I discussed these trend labels at Nordic Testing Days 2016 during the talk “How to Test in IT operations“.

Here are some contexts where Shift-Left happens:

  • Google have “Software Engineer in Test” as job title according to the book “How we Test Software at Google“.
  • Microsoft have similar “Software Design Engineer in Test” as discussed by Alan Page in “The SDET Pendulum” and in the e-book “A-word
  • A project I was regarding pharmaceutical  Track and Trace, had no testers. I didn’t even test but did compliance documentation of test activities. The developers tested. First via peer review, then via peer execution of story tests and then validation activities. No testers, just the same team – for various reasons.
  • A project I was in regarding a website and API for trading property information had no testers, but had continuous build and deploy with even more user oriented test cases that I could ever grab. (see: Fell in the trap of total coverage)

The general approach to Shift-Left is that “checking” moves earlier in the cycle in form of automation. More BDD, more TDD, more automated tests, continuous builds, frequent feedback and green bars. More based on “Test automation pyramid” (blog discussion, whiteboard testing video). Discussing the pyramid model reveals that testing and checking goes together in the lower levels too. I’m certain that (exploratory) testing happens among technicians and service-level developers; – usually not explicitly, but still.

To have “no QA” is not easy. Not easy on the testers because they need to shift and become more SET/SDET-like or shift something else (Shift-Right and Shift-Coach and Shift-Deliver). Neither is it easy on the team, as the team has to own the quality activities – as discussed in “So we’re going “No QA’s”. How do we get the devs to do enough testing?

Testers and test managers cannot complain, when testing and checking is performed in new ways. When tool-supported testing take over the boring less-complex checks, we can either own these checks or  move to facilitate that these checks are in place. Similarly when the (exploratory) brain-based testing of the complex and unknown is being handed over to some other person. Come to think of it I always prefer testing done by subject matter experts in the project, be it users, clients, testers or other specialists.

We need to shift to adapt to new contexts and new ways of aiding in delivering working solutions to our clients.

jollyrum

Less Software, more Testing

I rarely test software these days. I mostly lead testing of IT solutions.

Testing in the context of:

  • Updating all corporate PC’s from windows 7/8 to Windows 10
  • Consolidating network equipment from more devices to one box, on 80 global locations *
  • Move 40 live business applications from one data center to another *
  • Take over application maintenance for a specialized public organization
  • Implement track and trace for pharma products from production to shops
  • Migrate HR data for 2500 people from one system platform to another

Yes, it happens that I participate in a project that is about developing a new business application, but my activities are less about testing software and more about testing in IT solutions in general.

Mostly I manage test activities and describe testing in these contexts. My preferred way of working is in setting and implementing test strategies. I prefer complex and non-ordered projects (Complex and Chaotic – I’m looking at you), it fits well with my context-driven approach of finding the “test solution that fits the context”.

Testing is in it self a solution, that must solve a business problem. Great testing is all about providing information to the stakeholders. I don’t care especially if this is done by someone TESTING or a TESTER. It is my responsibility to setup the testing activities (information gathering) that supports the team, faces the business & technology and challenges the product “sufficiently“.

Sometimes “sufficiently” is merely confirming and going through the motions of explicit requirement coverage. This is a special challenge to me, as I know of many effective and Rapid approaches, that could add valuable information. When I face this challenge, I try to look at the full picture of the project, and what the business want’s to achieve.

The business of the business is business. What matters is not software or projects, but the solutions to the challenges the business have. And the context of testing is similarly so much more than the software.

*: As mentioned in “How to test in IT Operations” at Nordic Testing Days 2016.

13346977_10153714694356849_242947497430826688_n

Chicken for Christmas – Tradition is a choice

TL;DR: Testing is always something that we choose to do – and how we test is similarly a matter of choice. As is Christmas traditions. .. it’s just man-made rules, we can choose to change them.

So I was discussing what we should have for Christmas with the stakeholders. One of them wants the traditional rice pudding, the other wants – untraditionally: chicken. And you know what – that’s ok. Traditions are guidelines, not rules – we can make new traditions, just by choosing to.

Testing in production used to be a great no-no. I’m still feeling odd when we do it, but I have come to see it as another tool of the trade. It has a name now “testing in the wild” TTitW as has been presented at EuroStar 2015. Also this is how Netflix have been testing for years, from GitHub, as it is open source too (!).

You might argue that changing testing (in the wild) is not allowed. I will challenge that assumption – being allowed to do something is a choice too. You choose to follow the the process frameworks, requirements, rules… and you can choose not to. The tradition of manual predefined testcases are so four years ago!

Sometimes it’s just a matter of saying up-front, that you are tailoring the process. So choose an approach that actually gives meaning and value to the stakeholders and context. Deconstruct the traditions and commercial bodies of knowledge and make some new!

http://www.eurobricks.com/forum/index.php?showtopic=64241
http://www.eurobricks.com/forum/index.php?showtopic=64241

The Expected

Many test processes and test tools mention that you have to establish the expected results to every test. It is at best risk ignorance not to take the notion of expected results with a kilo of salt #YMMV.

If you can establish the result of your solution in a deterministic, algorithmic and consistent way in a non-trivial problem, then you can solve the halting problem. But I doubt your requirements are trivial.. or even always right. Even the best business analyst or subject matter expert may be wrong. Your best oracle may fail too.  Or you may be focussing on only getting what you measure.

Nailed it
Nailed it

When working with validation in seemingly very controlled environments changes and rework happens a lot, as every new finding needs the document trail back to square one.. Stuff happens. Validation is not testing, but looking to show that the program did work as requested at some time. It is a race towards the lowest common denominator. IT suppliers can do better that just to look for “as designed” [1].

Still the Cynefin framework illustrates that there are repeatable and known projects, and in those contexts you should probably focus on looking to check as many as the simple binary questions in a tool supported way, and work on the open questions for your testing.

Speaking of open ends – every time I see an explicit expected result I tend to add the following disclaimer inspired by Michael Bolton (song to the tune of nothing else matters [2]):

And nothing odd happens … that I know of … on my machine, in this situation [3]

And odd is something that may harm my user, business or program result

Significantly…

But I’d rather skip this test step  and work on the description of the test and guidelines to mention:

And now to something completely different:

See also: The unknown unknown unexpressed expectationsEating wicked problems for breakfast

1: Anyone can beat us, unless we are the besttodays innovation becomes tomorrows commodity

2: https://www.youtube.com/watch?v=tAGnKpE4NCI 

3: I’ve heard that somewhere…

Read for your kids – special interest edition

If you are a parent to (early) school children you should know that it is important to read  to your kids. Reading the words out trains vocabulary, recognition, imagination, wondering etc etc. So I read subtitles from movies… because

The boys currently have Star Wars as their special interest [1], and wanted to see the “people” movies. The have played the scenes via the LEGO Video Games (GC) and have a range of the LEGO sets – so they had the basic plot already. Feature movies like Star Wars are usually subtitled in Denmark – while animation movies are dubbed [2]. So in order both to keep up with “PG” [3] and helping them read the titles – I get to watch the movies and read the subtitles…

Poor daddy, it’s almost as hard as when he has to finish the ice cream they can’t 😉

In the last months the (soon to be) 9yo have cracked the reading code and have gone from LIX11 books to the shorter subtitles. The 11yo have rest covered, but some of the longer texts are tricky (I’m looking at you – opening Scroll).

2015-04-04 16.51.08

I tried reading Harry Potter (in Danish) but even if the story was very elaborate and detailed it didn’t catch their interest. Neither did classics from when I was a kid (Sorry Bjarne Reuter), so I had to rethink the acceptance criteria for “read for your kids“.

See these two boys are not as easily motivated – it has to tie into something they can see a direct interest in. Their autism makes them very picky on the choice of subject. What I try is to meet them where they are, expand their competencies and give them a lot of positive feedback until they master it on their own.

Links: The yardstick of mythical normalityAcceptance is more than what can be measured

  1. special interest, as in overly dedicated into the topic and cannot talk about anything else.
  2. The Danish “dubbers” are usually world class, luckily.
  3. Episode 3 is still to come, though.