Taking My Own Medicine

Recently I had the chance to apply my own templates to myself and my active project – as I had to mentor a new test manager. I was challenged in explaining how I read the upcoming IT environment project. After looking into resources for new test leads, I realized I could take my own medicine.

Photo by PhotoMIX Company on Pexels.com
Photo by PhotoMIX Company on Pexels.com

A year ago, I created a new test plan format – the Situational Aware Test Plan. While mind-maps and one-page test plan canvases exist, I wanted to elaborate using the evolution principles from Wardley mapping and stop writing test plan documents.

The table structure is there to provide guard rails for the elaboration. I will use the Darlings, Pets, Cattle, and GUID -mnemonic as headlines. Our strategic decisions emerge as we use the worksheet based on the current situation and state. The strategies will be the decisions to push a field in the grid to another state. 

Delivery and Situation

DarlingsPetsCattleGUID’s
New projectFixed date
Existing delivery speedScheduled
Quarterly
Test Environments, internalRepeatable
Test environments with integrationsCraftedSome existing know-how
Environment InfrastructureHosted data center practices
Test dataKnown but cumbersome

While this project introduces new test environments, there is an existing environment with a quarterly delivery pace. This is a classic example of the core chronic conflict of pursuing both: responding to the rapidly changing competitive landscape and providing stable, reliable, and secure services (DevOps handbook introduction xxv) as elaborated on Align your Test Strategy to your Business Strategy.

The test team allocated beside me and the new test lead is a new junior and senior tester. We are in the same team, and most are even in the same office. So collaboration will be collaborative and pervasive, with a focus on helping the new people grow.

The test team

DarlingsPetsCattleGUID’s
Test team collaborationGrowingPervasive
Test leadGrowing
MentoringEnabling
Domain know-howGetting there

Test tools and approach

DarlingsPetsCattleGUID’s
Test activityExplore integrationsConfirm internal requirements
Test casesExisting can be updated.
Test case reproCreate new repository

As mentioned in the blog post about visualization, we can now use the map to discuss why we need CT and ET for the project. Based on the project’s layout, I would advise having an expert exploration of the integrations and more standard scripts for the known construction of the internal environments.

Where I Have Changed

The Ministry of Testing Bloggers Club suggested that I write a post based on “In testing, I have changed my mind about ________”. As this blog dates back to 2012 with consistent (220) articles about testing, and my career in the field dates back to 2002 – it seems a 20-year experience should give me a few things. Testing is still not dead – and it’s still about the context (lower-case context, not CDT).

It’s not about: Testers being the only ones doing Testing

yeah, not so much these days. Testing is an act that any role can do in context. It’s about the testing – not so much the testers. And I have realized that even classic test management tasks can be done by someone else. Testing is not owned by the testers – it might be stewarded/facilitated by us, but it’s performed by a team member (who could be a tester).

It’s not about: Perfect Requirements

After decades in IT, it’s clear that even requirements are never perfect. When we look closer we see the business requirements can vary from a profound idea to a rudimentary feature of the system under test. Even in regulated industries requirements can be both about a specific configuration in a SaaS system or a loose idea of a relevant dashboard. Sometimes a requirement can be by design of an underlying commodity product – there doesn’t need to be a test case for everything.

The more rigor you add to the requirements management – the more fragile it becomes. It’s key to understand the risks and bets of the person paying for your solution. – in that lies the true borders of the delivery. Much can go informally along if it aligns with stakeholder values.

It’s not about: Defects

Back in the day defects needed to be accounted for, tracked, and distributed. Besides testing documents – defects were the only tangible delivery of the testers. The defects needed to be raised and closed. I recently wrote a guideline that stated that only observations that couldn’t be fixed within a day should be raised to the project manager for shared handling. In that context fixing things is within the same team. If it’s for another team to fix, defects are simply something communicated between the teams (check team topologies for team interactions). Sure you can still find a blocker or a P1 – what matters is how fast you can fix things.

It’s not about: Month-Long Testing Phases

The more time there is from idea to implementation – the more the requirement risks not addressing an up-to-date business objective. Timing is key. Some tools provide epics and user stories – but the structure is often misused to be a simple work aggregation – and not goal aligned.

The counter-intuitive trick is not to add formality, and more time between releases – but less. Less time between feedback between idea and implementation, and less time between implementation and test. Less time between the various forms of feedback adds to better results faster.

It still happens, I’m sure, that a business needs a month-long testing phase before a release; having a range of business staff to participate in testing the latest release of the enterprise ERP or CRM. More often the testing phase is one sprint behind the development activity. I have pondered this a lot.

At best testing is an integrated activity in the team and in the sprint. But if testing is a more separate activity – it can be both agile and context-relevant. So I have changed my mind about this anti-pattern.

It’s ok for testing to be in the next sprint –

if that adds consistency and less stress to the team*

* Seperate boards needed. Your Mileage Will Vary

Why Do We Fall, Master Bruce?

… So that we can learn to pick ourselves up, Alfred! I was recently reminded of this quote from watching the “Batman Begins” movie with my 16yo. I really needed that reminder. Then I read the two blog posts by Beren on “Those who Failed” and “Versus the Endboss“. Let’s put it out there that we fall – and fail to remember that we fall.

I had been part of a large project – but had read the culture all wrong and we had failed hard. For a number of reasons and maybe mostly for systemic reasons. The team expected one mindset and one way of tooling – we provided another one. Even with all my best intentions and know-how of change management, this crashed. As Hannes elegantly put it, we had cycled too far ahead of the team:

  • The team expected minutes of meetings and agendas, we worked for making things visible and shared
  • The team expected testing to be checking the requirements, we worked for testing to support critical decision making
  • The team worked political with back channels, we worked open and power-lifting
  • The team participants had agendas that didn’t align with the project’s purpose
  • The team expected all things equally important, we worked by priority and deadline
  • The team expected detailed test cases to be approved at all steps, we provided intentions and purpose
  • The team expected detailed handovers, we worked entrepreneurially to set things in motion
  • The team expected an error-free lead, we worked knowing I wouldn’t remember everything

At one point I was arguing that the team needed to read Hofstede’s cultural dimensions theory – to read up on the different cultures we would be interacting with (our customers). In retrospect, we should have used it on ourselves first of all.

Hofstede’s cultural dimensions theory

It’s a little more detailed than Westrum – and even Westrum might have helped. That is if we had been able to articulate the conflict well in advance. Perhaps a senior hire should have spotted the signals beforehand. As an outsider, I relied on people telling me things. I couldn’t hear or see the back-channel communications. This is a struggle for many staff people when switching roles:

Initially, no one from the operations organization and latest implementation opted for the leading the activity. As we had no playbook or project plan (only the produced artifacts) – I made a scrum-board-inspired work tracking system. Perhaps I should have used a Wardley map first of all as recommended by John Cutler in “TBM 18/52: We Need Someone Who Has Done “It” Before

What is Wardley Mapping doing for us here? It is letting us explore a more nuanced view of the problem space. Instead of treating things as one problem, we break the problem apart into a bunch of capabilities. When we do this exercise we typically find:

Not everything is an existing playbook. Not everything is a new playbook.

To solve new problems, we need a foundation of stable playbooks. For example, to solve that crazy new problem, the team might need a foundation of trustworthy data.

Yes, you can break things apart to see them better. But you’re also dealing with the whole thing.

But then again the team would probably have stalled over the very concept of a strategy map. People are weird. No matter how it looks at first, it’s always a people problem. And even if you do try to take the first steps – your steps could be in the wrong direction. Even Master Bruce will fall in that situation.

Factor in the Ripple Effects

TL;DR: Investing in basic tooling and automation improves your team besides expected metrics.

I work mostly with the implementation of enterprise SaaS systems these days. Large global companies are consolidating custom-built applications and on-premise applications with web-based standard solutions in the cloud aiming for “one standardized source of information to enable digital transformation”.

Yet the testing tooling hasn’t caught up. One company with €5000 million in sales is still using Word documents for test cases and “party like it’s 1999“. They are reluctantly considering tooling to support more agile ways of working. The whole “automate the knowns-knowns” is still pending an evaluation of return on investment (ROI) into technology from 2015. As of writing, Anno Domini 2022.

Assumptions

  • Writing test cases in documents takes about as long as writing automation
  • Maintaining automation is a more explicit task, humans can more easily apply a bit of fuzziness
  • When automation is in place, the execution requires limited efforts to run
  • The alternative to automated test execution is hours of people following and filling out the documents

With the investment in the tool, there’s a break-even around XX hours of document-based testing a month. That is if we plan for more than XX hours of document-based testing a month, the investment pays off. Your Mileage May Vary

But there’s more to it

First of all, when automated test execution is at limited costs to run and it can run independently at night, you will get the same effects as Continuous Integration and nightly builds have had in software development: you tend to run them more and more often.

This enables faster feedback both with regards to confirming new features and sums up to more effective regression testing. I have seen this happen in both custom application development and configuration of web-based standard solutions. In one project where I added automation, we have run nearly 8000 automated runs in a year (and 200 SME-based). We actually run the tests more often, and we cover the important things every day – and everything often enough. We do in fact get more testing, and broader coverage than any document-supported testing could ever scale to do.

Believe the experts

While there is some vendor basis in the following two webinars, the story is the same: Test automation can accelerate IT deliveries:

Alternatively, look into the research from Accelerate – and the DevOps handbooks. The ripple effects of automated test execution are plenty and go beyond the math of the testing effort. One thing to keep in mind is that test automation itself is not enough. At first, you need transformational leadership.

Tester Aided in Two-Digit Million Dollar Deal

Yes indeed. It has happened for me in the last couple of months. While my role is not tester anymore (but advisor in testing) – it just wouldn’t make the headline as click-baity. Sorry for that, though it does help to prove the point that testing specialists can be a part of bids and tender teams. A testing mindset is needed even before there is an “system development life cycle”.

The Dealing

In the context of bids and tenders the testing activities are mostly about technical writing around how the testing will happen when the dealing’s done. It’s not so much about finding issues – but more about a coherent analytic viewpoint. The customer of the deal often set up “requirements” that the supplier must answer and is scored against:

  • Elaborate on a test strategy
  • Elaborate on the suggested test process
  • Describe relevant testing documents – don’t overdo them!
  • Describe testing types and environments in use
  • Describe test tools and approach to automation

If you don’t reply to all “requirements” you get a sub-par score, so being able to find information in the organization is key. The contractor uploads the final documents to the customer and the content is evaluated. The evaluation is usually a balanced scoring between the individual reply documents and the price point. Often price wins, even if the scoring of the (testing) documents where at 100% score.

More and more often I see outsourcing contracts that requests 10-15 test phases. It looks like someone has simply thrown the book at it, and not considered if it is an infrastructure project, a software development project or COTS implementation or – what on earth, they actually want to learn from the testing.

So how do you go about to be in on deals like this? – business context and a seat at the table seems key.

Business Context

First of all you need to be part of a company that cater to this size of deals. The deals I have been involved recently have mostly been about national IT solutions for public and semi-public organisations. The national government rarely have their own IT divisions but hire outsourcing companies to develop new solutions, maintain existing solutions along with hosting and cloud journeys. The more you add into the deal the larger the sums rack up. And similar if it’s a eight year contract for full IT operations, devices/laptops, support and application management services – the deal sum easily ends around the ballpark amount in the headline.

It’s probably different where you live and where you work. You might work on a consumer facing app that is paid pr subscription or perhaps in a team that develop a specific business-to-business product. And that’s cool – context matters. But I know even product houses have to go out and close deals with their business customers ever so often.

A Seat at the Table

Being part of bid and tender teams can be a key role for individual contributors in the staff levels. Staff levels are the senior and principal testing roles – that do not have management responsibilities. The term comes from the book “Staff Engineer: Leadership beyond the management track by Will Larson” and well as from “The Staff Engineer’s Path by Tanya Reilly“. The former book has some excellent chapters on getting a seat at the table and staying relevant there.

While having the role does not guarantee you a seat in the bid teams, neither is a staff role a prerequisite to be in. What matters most is probably management’s willingness to step out and let the experts in on the details. This could also be work that was done by heads of testing and managers of testing. Though as manager you should really focus on servant leadership and let your testing pro’s aid in closing the deal.

Photo by Pixabay on Pexels.com

What About Expected Results?

TL;DR: We should know better than to require expected results in test case steps.

Welcome to 2022 – where old wisdom surfaces yet again. Recently I considered expected results in test cases. It seems the topic is still relevant – I wrote about it in “The Expected” back in 2015.

As always we have to backup a bit and consider: What purpose does the detailed steps have to our stakeholders. How visible is the value provided? We have to start from their viewpoint – not build a chain of reasoning up from our own wishful thinking.

Currently I’m working on a project for payments of benefits. We have daily CI/CD and loads of classic manual testing. The key message from the stakeholders is that they expect things to be tested and evaluated. They generally trust us and collaborate with us on the work, so there is not so much discussion on the details as in other projects. Yet the discussion of expected results did come up in our internal dialog: What is the reasoning behind expected results on the test case steps – besides coming from an old book.

Testing is a performance

One of the usual arguments is that it enables hard-over from person to another. While handover can be relevant to consider if the team members changes often. It seems to me just to create redundant information based on requirements and other oracles. I mean where does it it end, why should we add every project information into the test cases and into the test plan. Stop it already.

Let the test cases be evaluated by the person performing the test – to see the whole picture. And not get biased by expected results. I want the persons performing the tests to use their brain and based on the oracles available evaluate the system under test. Also if everything can be planned – why would there be a need for retakes of movie scenes? It turns out even simple instructions are not possible:

Exact Instructions Challenge – THIS is why my kids hate me. | Josh Darnit

Let people do what they are best at

Let’s have testing being a balanced approach, where people evaluate and scripts confirm.

When the confirmation can be determined by an algorithm or by assertions towards an oracle we can code a test for it – and test it continuously. Adding as much automation as (practically) possible, aids the delivery in multiple ways. It aids to build regression testing and help us test that things not only worked once somewhere, but that it works again and again.

On the other hand when there’s no algorithm or explicit oracle in the form of specifications, acceptance criteria or similar authority we can only do a subjective evaluation: at that point in time – given the information available. do not neglect the wisdom of subject matter experts testing.

And in the later situation expected results hinder the exploration of whether the system under test fulfills not on the explicit requirements but also the implicit. We really should know better.

Photo by Karolina Grabowska on Pexels.com

#270 – But what if we can’t release often?

With digital solutions there is a ongoing urge to release often. A quest for feature toggles and continuous deliveries of new features and fixes. Automation of tedious tasks do help to drive consistent deliveries and aids in driving high-performing teams. There is good research to support that.

Recently I have worked on a solution, where the system had only to work for a month each year – and be closed down the other months. Some solutions I have similarly looked at, have years between being active. We can’t wait to ship features in the next release – the system has to be at 100% features that specific month.

Examples of business situations and domains:

  • Performances, like Eurovision
  • Sport events, like the Olympics
  • Elections
  • Black friday, Christmas shopping?

How do you test when you can only perform the act once – and if it fails it will have serious consequences? You practise and rehearse until it becomes safely repeatable. You have stage-moms and support teams to train with you.

Let’s look at the ultimate example: rock-climbing with no rope. How do you test for climbing up Yosemite’s El Capitan 3000 feet / 900 meters?

Photo by Tobias Aeppli on Pexels.com

The accomplishment is more preparation than performance. Honnold climbed El Capitan roughly 50 times in the decade before his free soloing of the rock formation on July 3, 2017. While he is famous for the ridiculously fast 3-hour, 56-minute ascent, 99% of Honnold’s time on the wall was spent roped up, practicing the route. Knowing where and how to move was the culmination of hundreds of hours on that granite in advance.

The Seven Lessons From ‘Free Solo’ On Working Without A Rope

Besides the scaling of the IT infrastructure for peak load – the test strategy has to consider the fact that the event itself will be a one-off, where there show must go on – and there’s no fallback, only fall forward.

There’s a huge difference between continuously delivering web features every day in a business to consumer setting, as compared to one-off projects of migration legacy platforms. This is why my approach to creating situational aware test plans starts with looking at delivery speed:

RareRegularOftenPervasive
One-offQuarterlyWeeklySo often you dont notice
A scale of delivery speed

In one of the projects I looked had we had extensive user rehearsals and dress rehearsals. Well, they are called something more IT fancy. But at the end of the day it was about training to make the performance muscle knowledge in the people performing the event. Much like the training Honnold did.

Lastly, my experience is that you get more organizational traction by aligning with goals rather than risks and issues. It’s a behavioral trick simply to talk about the thing you want to give attention. And at the end of the day the CEO wants to talk about goals rather than risks. She rather wants a successful performance with a few flaws that an delay due to bruised hands (and egos) during waxing on and waxing off.

#268 – Who Brings in New Knowledge?

Well, if you are reading this – there’s a good chance it’s you. Especially if you read this with the intention of sharing this with your team. I hope you do, obviously 😊. But perhaps it’s unclear whose responsibility it is, to bring new knowledge to the team. Is it always the team manager job – or is it a dedicated person that by role, or by habit, that bring in new knowledge?

Photo by William Fortunato on Pexels.com
Continue reading

#267 – Reminder – it’s about Story Shaping

In agile delivery teams, it’s recommended that stories and features are discussed before being developed. A checklist called “Definition of Done” can be used as a reminder to check that the delivery team has everything needed to prepare the delivery of the feature. A key ingredient is a shared understanding of how to confirm the delivery – for the whole team.

Story shaping requires at least three people to get together. They are not amigos nor amigas. They are often three, sometimes two people, and often more. They represent the different viewpoints needed to deliver: builders mindset, testing mindset, security mindset, operations mindset, business mindset, etc.

No one person can hold all these viewpoints – at least a builders mindset and a requester/business mindset is needed.

If you skip doing story shaping your acceptance criteria and testing activities will be misaligned. And you will either have overdone the effort needed or underestimated the challenges of the story.

Thank you to all who have replied to the tweet below.

One customer, three people to do the story shaping

#264 – Create Situational Aware Test Plans

From the endless discussions on the proper content and contexts of a test plan, it’s apparently still needed – but what goes in it? Let’s create situational aware test plans inspired by Wardley Mapping.

ISTQB template-based test plan documents are in my personal opinion no longer industry best practice. First of all it’s bloatware. While they intend to be a springboard into considering what is relevant we have ended up with 8 page templates – where every single of the 20 topics are required information. While it looks dazzling – it’s like frosting puffed with empty calories.

What most people delivering effectively software are using is 1) modern test automation and 2) modern test case management tools to lead and manage the test activities. And there is clear research on what 24 factors correlates to high-performing teams. It seems to me the templates have been frozen stale since 2012 – and are hindering us more than helping.

Continue reading