A Chocolate Car Calamity

If something as seemingly simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.

Once upon a time, there was a professional chocolate shop. It was an actual chocolate shop, full of products for many needs, and a staffed counter, where you could go and order. One fine day a customer arrived and asked if the shop could create a chocolate car. The shop chocolatier said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car”. “Ah, very nice chocolate car. But please could the wheels turn.”The shop chocolatier a little annoyingly said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn”. “Ah, very nice chocolate car. But please could the doors open.” The shop chocolatier annoyingly said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open”. “Ah, very nice chocolate car. But please could it have a sunroof.” The shop chocolatier a little more annoyingly said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open, see the sunroof”. Just as the shop chocolatier was going to get even more annoyed, the customer said: “thank you, no more updates”. Baffled the shop owner forgot all about steering, ABS, and passenger seats, and asked: “How would you like it packaged”. And the customer said “ah! No need, I’ll eat it right away” And so he did. The End.

Despite all the product demos and product development, the customer consumed the product in a heart beat – or probably two.

The Morale, as There is Always a Morale

The origin of this story is a two person sketch from my local youth work[link in Danish] in the tradition of Abbott and Costello’s “Who is on first base” – so it’s probably from the 1960’es. The sketch is usually acted with elaborate gesturing building up to the absurd punchline on the final day.

I was reminded about it recently – in a project where external reviewers kept coming back for more. More requests for bells and whistles that where not originally stated. While we did share the test approach initially, the feedback from the customer is coming towards the end of the project delivery.

All the extra effort seems absurd in contrast to communicating goals and collaborating up front. While each party might be trustworthy, it only takes one of them to extend trust and improve the relationship significantly. Extending trust would obviously defuse the absurdity of the exchange, and there would be no sketch. And we need the sketch to provide the backdrop to the morale:

• As a customer state your end goal clearly, don’t be a push-over
• As a development team, stop and ask. If it’s outside your usual range it’s ok to say no.

If something as simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.

Making chocolate dough by Jakub Kapusnak is licensed under CC-CC0 1.0

Old Test Types Need Not Apply

I primarily work in situations that are less about application delivery and more about moving the whole system stacks, implementing a standard system, or similarly changing the organizational IT landscape. Some would list these projects as “staff projects” if that helps you. I often find the terms Regressiontest, SIT, and UAT to be misleading and not helpful to what kind of test is needed for my examples.

Everything and the kitchen sink

Example 1: We are rebuilding all the environments in the delivery stack. All the Dev-, Test-, Integration-, Preprod-, and Prod- environments, and their underlying databases, brokers, and websites. Every time we are constructing, an environment we will be testing the setup from a baseline version of the existing running systems. A baseline that we know is functional already. The classic software development testing types don’t really help us in this situation, as neither regression test, UAT, or SIT conveys the things we want to confirm and the learnings we want to explore.

Example 2: We are setting up a new expense platform for employee reimbursements to go live with a new branding of the company. It’s a SaaS system and we load it every month with data about the organization. So while it’s needed for various purposes – the risk is low and the mean time to repair is similarly low. The testing we will do will be a limited confirmation of an initial data load. Snow-plow style – not a full system-integration test and similar user-acceptance test. After all, this is a SaaS – not a custom solution. It’s OK to shift right.

SIT and UAT has become generic term that it has lost the strength to convey the needed quality narrative. If you do CI/CD (which you should) for your application development that such be sufficient. If you figure out, you need to do a “connection alive” test for your third-party integrations that you move from one environment stack to another that should be accepted with the acknowledgment that you actually considered the challenges ahead.

It’s all about the risks and the mitigations – less about testing everything to the dot. One tool to read the landscape is to communicate curiously about what the stakeholders value more – and value less. And on the other hand, consider the nature of the solution being proposed.

Example 3: Setting up a cloud-based Azure Active Directory – the solution comes with a given security level out of the box (OOTB). As with other OOTB and Software-as-a-service solutions you have little impact on the security features of the solution, besides some simple configurations. While you might think that all security requirements would require 100% acceptance testing coverage, what you want to accept is that they might be provided “by design” – or by a solution decision made long ago.

I would prefer that we can call things what they are and not blindly apply old testing types

#270 – But what if we can’t release often?

With digital solutions there is a ongoing urge to release often. A quest for feature toggles and continuous deliveries of new features and fixes. Automation of tedious tasks do help to drive consistent deliveries and aids in driving high-performing teams. There is good research to support that.

Recently I have worked on a solution, where the system had only to work for a month each year – and be closed down the other months. Some solutions I have similarly looked at, have years between being active. We can’t wait to ship features in the next release – the system has to be at 100% features that specific month.

Examples of business situations and domains:

• Performances, like Eurovision
• Sport events, like the Olympics
• Elections
• Black friday, Christmas shopping?

How do you test when you can only perform the act once – and if it fails it will have serious consequences? You practise and rehearse until it becomes safely repeatable. You have stage-moms and support teams to train with you.

Let’s look at the ultimate example: rock-climbing with no rope. How do you test for climbing up Yosemite’s El Capitan 3000 feet / 900 meters?

Photo by Tobias Aeppli on Pexels.com

The accomplishment is more preparation than performance. Honnold climbed El Capitan roughly 50 times in the decade before his free soloing of the rock formation on July 3, 2017. While he is famous for the ridiculously fast 3-hour, 56-minute ascent, 99% of Honnold’s time on the wall was spent roped up, practicing the route. Knowing where and how to move was the culmination of hundreds of hours on that granite in advance.

The Seven Lessons From ‘Free Solo’ On Working Without A Rope

Besides the scaling of the IT infrastructure for peak load – the test strategy has to consider the fact that the event itself will be a one-off, where there show must go on – and there’s no fallback, only fall forward.

There’s a huge difference between continuously delivering web features every day in a business to consumer setting, as compared to one-off projects of migration legacy platforms. This is why my approach to creating situational aware test plans starts with looking at delivery speed:

Rare	Regular	Often	Pervasive
One-off	Quarterly	Weekly	So often you dont notice

A scale of delivery speed

In one of the projects I looked had we had extensive user rehearsals and dress rehearsals. Well, they are called something more IT fancy. But at the end of the day it was about training to make the performance muscle knowledge in the people performing the event. Much like the training Honnold did.

Lastly, my experience is that you get more organizational traction by aligning with goals rather than risks and issues. It’s a behavioral trick simply to talk about the thing you want to give attention. And at the end of the day the CEO wants to talk about goals rather than risks. She rather wants a successful performance with a few flaws that an delay due to bruised hands (and egos) during waxing on and waxing off.

#265 – Using MTTR to Understand When to Test

It interests me deeply to explore why testing is happening. Often it’s because some decision-maker or framework dictates – “This is the Way“. And off we go on the quest to slay the dragon – or move items from point A to point B. Without much thinking about how the side quests help to move the main risks of the story.

The main risks are usually around something irreplaceable – and hence we test and try our best to shield it. But not all risks are equally dangerous. In IT we can build implicit testing into repeatable deliveries and reduce the time to fix things. The faster things are fixed, the better is time to information for the business needs.

Grogu agrees

Calculating Time To Information

The key metric for any knowledge work – IT deliveries and testing in particular – is more than Mean Time to Repair (MTTR). While fixing fast matters – timing is everything. Timing in getting information to the people who needs it to make decisions. It’s no use if you can turn the ship around on a plate now, if you needed it yesterday. Key elements in calculating time to information is how far away the information is and how evolved the information is.

Focus on Goals over Risks

Looking into the discussion on what goes into a Test Plan and what goes into a Test Strategy – it’s my personal opinion that we can improve our business alignment. Risk-based testing and Product Risk Analysis have been around for long – but better models have emerged to address what will be more impactful.

Strategy is about Making a Map

Strategy is not where you are heading, but how you’re getting somewhere in the long run. That goes for all strategies, and even for test strategies. Though for test strategies we often get caught up in mechanics of selector strategies, testing types and techniques that we lose track of the higher purpose: Moving the business towards a vision.

The Testing, not the Testers

Occasionally I see posts and discussions, where testers are indignated that this and this company has no testers. How could they! Or similarly when a product is released publicly with significant issues: See – it’s because they have no testers! Or that the testers are not taken seriously. OMG!

Relations – are about half of IT

You can’t have IT projects without relations. Relations matter more than it seem.

One Test Case is All You Need

[Originally on Ministry of Testing, November 20, 2013: One Test Case is All You Need. Reposted from the web archives]