Softwaretesting

Why Do We Fall, Master Bruce?

jo2sn Enterprise IT, Softwaretesting, Wardley's theories , , , , ,

… So that we can learn to pick ourselves up, Alfred! I was recently reminded of this quote from watching the “Batman Begins” movie with my 16yo. I really needed that reminder. Then I read the two blog posts by Beren on “Those who Failed” and “Versus the Endboss“. Let’s put it out there that we fall – and fail to remember that we fall.

I had been part of a large project – but had read the culture all wrong and we had failed hard. For a number of reasons and maybe mostly for systemic reasons. The team expected one mindset and one way of tooling – we provided another one. Even with all my best intentions and know-how of change management, this crashed. As Hannes elegantly put it, we had cycled too far ahead of the team:

  • The team expected minutes of meetings and agendas, we worked for making things visible and shared
  • The team expected testing to be checking the requirements, we worked for testing to support critical decision making
  • The team worked political with back channels, we worked open and power-lifting
  • The team participants had agendas that didn’t align with the project’s purpose
  • The team expected all things equally important, we worked by priority and deadline
  • The team expected detailed test cases to be approved at all steps, we provided intentions and purpose
  • The team expected detailed handovers, we worked entrepreneurially to set things in motion
  • The team expected an error-free lead, we worked knowing I wouldn’t remember everything

At one point I was arguing that the team needed to read Hofstede’s cultural dimensions theory – to read up on the different cultures we would be interacting with (our customers). In retrospect, we should have used it on ourselves first of all.

Hofstede’s cultural dimensions theory

It’s a little more detailed than Westrum – and even Westrum might have helped. That is if we had been able to articulate the conflict well in advance. Perhaps a senior hire should have spotted the signals beforehand. As an outsider, I relied on people telling me things. I couldn’t hear or see the back-channel communications. This is a struggle for many staff people when switching roles:

Initially, no one from the operations organization and latest implementation opted for the leading the activity. As we had no playbook or project plan (only the produced artifacts) – I made a scrum-board-inspired work tracking system. Perhaps I should have used a Wardley map first of all as recommended by John Cutler in “TBM 18/52: We Need Someone Who Has Done “It” Before“

What is Wardley Mapping doing for us here? It is letting us explore a more nuanced view of the problem space. Instead of treating things as one problem, we break the problem apart into a bunch of capabilities. When we do this exercise we typically find:

Not everything is an existing playbook. Not everything is a new playbook.

To solve new problems, we need a foundation of stable playbooks. For example, to solve that crazy new problem, the team might need a foundation of trustworthy data.

Yes, you can break things apart to see them better. But you’re also dealing with the whole thing.

But then again the team would probably have stalled over the very concept of a strategy map. People are weird. No matter how it looks at first, it’s always a people problem. And even if you do try to take the first steps – your steps could be in the wrong direction. Even Master Bruce will fall in that situation.

Low-code – the Bigger Picture

jo2sn Enterprise IT, Softwaretesting , , , ,

Low-code test automation is part of a bigger trend in IT. Forresters and Gartner call it the “Citizen Developer” – the general idea that many business activities can be achieved by business users and citizens directly without the need for big IT projects… initially.

We Have Outsmarted Ourselves Again

For the last 10 years we have mockingly called Robot Process Automation (RPA) a “poor man’s integration”, in the sense that instead of building sic “real” integration, we build an RPA robot to handle the interface. But it’s equally low-code when your Apple Shortcuts trigger application events or you use Airtables or SmartSheets instead of MS Office tools.

In the mocking from IT teams, we do tend to forget that low-code tools are a short-term efficient and user-friendly way for organizations without a big IT budget to solve some common problems. That it can very well make business sense to RPA data between systems until the last silo has been cemented over.

There is a clear trend that the business units of large enterprises are getting more tech-savvy and can do more IT things on their own: order a new OS, configure a new form, populate tables, and configure collaborative work products. Previously these actions would have mockingly been called “shadow IT” when outside the realm of the IT units. Now it’s more out in the open – and where the IT spend money is.

Low code is, when you squint at it, all about the visualization and abstraction of something that previously took coding in IDEs, tinkering in Excel sheets, and similarly skilled IT labor to configure. It’s really nothing new in the history of IT. For large global enterprise companies, it has always been about consolidating business IT systems and redefining new coherent ways of working.

Replace the existing system suite of 10+ tools with one Software-as-a-Service Solution to create and maintain product information, so that it can be kept in one place and inspire additional digital transformation.

The strategic objective for a large global company in 2022

The current journey for these global enterprises is to move the IT savvy-ness into the business units and make the business units more autonomous in their IT spending. There’s no need to hire an external outsourcing company to maintain the IT operations when most can be done by a few internal staff inside the cloud dashboards or similar admin modules of Salesforce and ServiceNow.

Give it some time, though.

While low-code and RPA can in some cases be effectively coded by business experts – they will soon need some good old computer science techniques to maintain the RPA and low-code shoe strings. At the end of the day, visual code is still code. And low-code test automation is just part of a bigger picture.

The Mechanics of Modern Meetings

jo2sn Enterprise IT, Softwaretesting , , , ,

In these days of virtual meetings, the very structures of formal meetings are under change. It’s definitely forged by extensive work-from-home and working with people not in the same locations. It challenges the people that are used to having everything in documents and actions/assignments tracked as part of a “Minutes of Meetings” document. They seem to mistake the absence of document artifacts with no structure. But if you look closer you will see that even a circus is a choreographed act.

The Agenda is always the Current State of Affairs

A key observation from the agile and collaborative way of working is the principle of making work visible. Put tasks and assignments on a shared board for the team. The tool is not so important, as long as it reasonably supports the kanban/scrum-board mechanics. You can use Trello, Podio, Miro, Azure DevOps, or Jira – whatever is available to you in your organization.

Among the benefits of a shared digital board is that it additionally supports the team with the ability to work on items asynchronously, independent of timezones, working hours, and locations. The state of affairs is whatever state the board depicts – so make sure it’s always as truthful as it can be. It takes practice for the team members to learn to update the board outside of the meeting. But this small step is really key in making the meetings more effective and reducing the time to information.

The status board challenges the fact that an agenda can be locked prior to a meeting. All items are moving pieces – so the agenda can only be “look at the board“. If someone is working on something – put it on the board. This also helps if a team member runs off to join a circus – or is temporarily away from this very circus.

Boards help to streamline getting things done. Items might not be perfect – but the focus is on getting them done. “Stop starting – Start stopping” is a recurring mantra. Secondly using a board and agile backlogs and work limits help to prioritize the work according to the team’s availability and speed of delivery. Bottlenecks and overloaded staff can be more easily identified.

Recurring touchpoints, though, are still needed for the team, but the latest status of the work items is no longer at the end of a Minutes of Meetings document.

Recorded Minutes of Meetings

Originally, the MoM (Minutes of Meetings) documents hold the decision items and action items after every meeting. As discussed a shared task board can replace much of the MoM. Is Alice joining Bob on a task? Did Charles agree to deliver X by Friday? All of those actions can be activities on the task board, as long as it’s added during the meeting. A meeting notetaker could do this during the meeting on the board, and not focus on writing down every minute. Adding ideas to the board’s “to-do” column is also a powerful way to remember things for the future.

A strong trend I see in the use of virtual meeting platforms is a default recording of most meetings. You have to get used to it, also privacy vise. Be careful in political organizations – the spoken word is now recorded. Among the benefits of recorded meetings is that everyone can rewind into the meetings and that previous meeting content is available for new team members. This goes especially well for content that is more “show and tell” than status calls.

My preferred leadership style is to set direction, provide what I have of relevant information, and follow up indirectly via the board. I don’t need to meet for a status message that can be read from the board. But I will use the information on the board to reflect on where we are and where we’re supposed to be heading.

Reframe meetings as Collaborative Conversations

When I set up a meeting in someone’s calendar, it’s not always with the intention to have the formal mechanics of a Capital-M Meeting. The scheduling in the calendar is a way to respect people’s time and to make sure key participants can be available at the same time. It’s out of the same respect for people’s variety of availability that meetings need to be effective.

I rarely invoke the formalities of a Meeting. When we (small-m) meet it’s to collaborate and interact and discover serendipity. Sometimes it seems that the name “meeting” is taken literally as a formal structure, while to me it’s more like a placeholder for collaborative conversations.

It may look like a circus – but that is on purpose. There is a choreography behind it all.

The Circus by Alex Herreru00edas is licensed under CC-BY-NC-ND 4.0

Factor in the Ripple Effects

jo2sn Enterprise IT, Softwaretesting , , , ,

TL;DR: Investing in basic tooling and automation improves your team besides expected metrics.

I work mostly with the implementation of enterprise SaaS systems these days. Large global companies are consolidating custom-built applications and on-premise applications with web-based standard solutions in the cloud aiming for “one standardized source of information to enable digital transformation”.

Yet the testing tooling hasn’t caught up. One company with €5000 million in sales is still using Word documents for test cases and “party like it’s 1999“. They are reluctantly considering tooling to support more agile ways of working. The whole “automate the knowns-knowns” is still pending an evaluation of return on investment (ROI) into technology from 2015. As of writing, Anno Domini 2022.

Assumptions

  • Writing test cases in documents takes about as long as writing automation
  • Maintaining automation is a more explicit task, humans can more easily apply a bit of fuzziness
  • When automation is in place, the execution requires limited efforts to run
  • The alternative to automated test execution is hours of people following and filling out the documents

With the investment in the tool, there’s a break-even around XX hours of document-based testing a month. That is if we plan for more than XX hours of document-based testing a month, the investment pays off. Your Mileage May Vary

But there’s more to it

First of all, when automated test execution is at limited costs to run and it can run independently at night, you will get the same effects as Continuous Integration and nightly builds have had in software development: you tend to run them more and more often.

This enables faster feedback both with regards to confirming new features and sums up to more effective regression testing. I have seen this happen in both custom application development and configuration of web-based standard solutions. In one project where I added automation, we have run nearly 8000 automated runs in a year (and 200 SME-based). We actually run the tests more often, and we cover the important things every day – and everything often enough. We do in fact get more testing, and broader coverage than any document-supported testing could ever scale to do.

Believe the experts

While there is some vendor basis in the following two webinars, the story is the same: Test automation can accelerate IT deliveries:

Alternatively, look into the research from Accelerate – and the DevOps handbooks. The ripple effects of automated test execution are plenty and go beyond the math of the testing effort. One thing to keep in mind is that test automation itself is not enough. At first, you need transformational leadership.

Tester Aided in Two-Digit Million Dollar Deal

jo2sn Enterprise IT, Softwaretesting , , , ,

Yes indeed. It has happened for me in the last couple of months. While my role is not tester anymore (but advisor in testing) – it just wouldn’t make the headline as click-baity. Sorry for that, though it does help to prove the point that testing specialists can be a part of bids and tender teams. A testing mindset is needed even before there is an “system development life cycle”.

The Dealing

In the context of bids and tenders the testing activities are mostly about technical writing around how the testing will happen when the dealing’s done. It’s not so much about finding issues – but more about a coherent analytic viewpoint. The customer of the deal often set up “requirements” that the supplier must answer and is scored against:

  • Elaborate on a test strategy
  • Elaborate on the suggested test process
  • Describe relevant testing documents – don’t overdo them!
  • Describe testing types and environments in use
  • Describe test tools and approach to automation

If you don’t reply to all “requirements” you get a sub-par score, so being able to find information in the organization is key. The contractor uploads the final documents to the customer and the content is evaluated. The evaluation is usually a balanced scoring between the individual reply documents and the price point. Often price wins, even if the scoring of the (testing) documents where at 100% score.

More and more often I see outsourcing contracts that requests 10-15 test phases. It looks like someone has simply thrown the book at it, and not considered if it is an infrastructure project, a software development project or COTS implementation or – what on earth, they actually want to learn from the testing.

So how do you go about to be in on deals like this? – business context and a seat at the table seems key.

Business Context

First of all you need to be part of a company that cater to this size of deals. The deals I have been involved recently have mostly been about national IT solutions for public and semi-public organisations. The national government rarely have their own IT divisions but hire outsourcing companies to develop new solutions, maintain existing solutions along with hosting and cloud journeys. The more you add into the deal the larger the sums rack up. And similar if it’s a eight year contract for full IT operations, devices/laptops, support and application management services – the deal sum easily ends around the ballpark amount in the headline.

It’s probably different where you live and where you work. You might work on a consumer facing app that is paid pr subscription or perhaps in a team that develop a specific business-to-business product. And that’s cool – context matters. But I know even product houses have to go out and close deals with their business customers ever so often.

A Seat at the Table

Being part of bid and tender teams can be a key role for individual contributors in the staff levels. Staff levels are the senior and principal testing roles – that do not have management responsibilities. The term comes from the book “Staff Engineer: Leadership beyond the management track by Will Larson” and well as from “The Staff Engineer’s Path by Tanya Reilly“. The former book has some excellent chapters on getting a seat at the table and staying relevant there.

While having the role does not guarantee you a seat in the bid teams, neither is a staff role a prerequisite to be in. What matters most is probably management’s willingness to step out and let the experts in on the details. This could also be work that was done by heads of testing and managers of testing. Though as manager you should really focus on servant leadership and let your testing pro’s aid in closing the deal.

Photo by Pixabay on Pexels.com

What About Expected Results?

jo2sn Family and fun, Softwaretesting , , , , , ,

TL;DR: We should know better than to require expected results in test case steps.

Welcome to 2022 – where old wisdom surfaces yet again. Recently I considered expected results in test cases. It seems the topic is still relevant – I wrote about it in “The Expected” back in 2015.

As always we have to backup a bit and consider: What purpose does the detailed steps have to our stakeholders. How visible is the value provided? We have to start from their viewpoint – not build a chain of reasoning up from our own wishful thinking.

Currently I’m working on a project for payments of benefits. We have daily CI/CD and loads of classic manual testing. The key message from the stakeholders is that they expect things to be tested and evaluated. They generally trust us and collaborate with us on the work, so there is not so much discussion on the details as in other projects. Yet the discussion of expected results did come up in our internal dialog: What is the reasoning behind expected results on the test case steps – besides coming from an old book.

Testing is a performance

One of the usual arguments is that it enables hard-over from person to another. While handover can be relevant to consider if the team members changes often. It seems to me just to create redundant information based on requirements and other oracles. I mean where does it it end, why should we add every project information into the test cases and into the test plan. Stop it already.

Let the test cases be evaluated by the person performing the test – to see the whole picture. And not get biased by expected results. I want the persons performing the tests to use their brain and based on the oracles available evaluate the system under test. Also if everything can be planned – why would there be a need for retakes of movie scenes? It turns out even simple instructions are not possible:

Exact Instructions Challenge – THIS is why my kids hate me. | Josh Darnit

Let people do what they are best at

Let’s have testing being a balanced approach, where people evaluate and scripts confirm.

When the confirmation can be determined by an algorithm or by assertions towards an oracle we can code a test for it – and test it continuously. Adding as much automation as (practically) possible, aids the delivery in multiple ways. It aids to build regression testing and help us test that things not only worked once somewhere, but that it works again and again.

On the other hand when there’s no algorithm or explicit oracle in the form of specifications, acceptance criteria or similar authority we can only do a subjective evaluation: at that point in time – given the information available. do not neglect the wisdom of subject matter experts testing.

And in the later situation expected results hinder the exploration of whether the system under test fulfills not on the explicit requirements but also the implicit. We really should know better.

Photo by Karolina Grabowska on Pexels.com

#270 – But what if we can’t release often?

jo2sn Softwaretesting , , , ,

With digital solutions there is a ongoing urge to release often. A quest for feature toggles and continuous deliveries of new features and fixes. Automation of tedious tasks do help to drive consistent deliveries and aids in driving high-performing teams. There is good research to support that.

Recently I have worked on a solution, where the system had only to work for a month each year – and be closed down the other months. Some solutions I have similarly looked at, have years between being active. We can’t wait to ship features in the next release – the system has to be at 100% features that specific month.

Examples of business situations and domains:

  • Performances, like Eurovision
  • Sport events, like the Olympics
  • Elections
  • Black friday, Christmas shopping?

How do you test when you can only perform the act once – and if it fails it will have serious consequences? You practise and rehearse until it becomes safely repeatable. You have stage-moms and support teams to train with you.

Let’s look at the ultimate example: rock-climbing with no rope. How do you test for climbing up Yosemite’s El Capitan 3000 feet / 900 meters?

Photo by Tobias Aeppli on Pexels.com

The accomplishment is more preparation than performance. Honnold climbed El Capitan roughly 50 times in the decade before his free soloing of the rock formation on July 3, 2017. While he is famous for the ridiculously fast 3-hour, 56-minute ascent, 99% of Honnold’s time on the wall was spent roped up, practicing the route. Knowing where and how to move was the culmination of hundreds of hours on that granite in advance.

The Seven Lessons From ‘Free Solo’ On Working Without A Rope

Besides the scaling of the IT infrastructure for peak load – the test strategy has to consider the fact that the event itself will be a one-off, where there show must go on – and there’s no fallback, only fall forward.

There’s a huge difference between continuously delivering web features every day in a business to consumer setting, as compared to one-off projects of migration legacy platforms. This is why my approach to creating situational aware test plans starts with looking at delivery speed:

RareRegularOftenPervasive
One-offQuarterlyWeeklySo often you dont notice
A scale of delivery speed

In one of the projects I looked had we had extensive user rehearsals and dress rehearsals. Well, they are called something more IT fancy. But at the end of the day it was about training to make the performance muscle knowledge in the people performing the event. Much like the training Honnold did.

Lastly, my experience is that you get more organizational traction by aligning with goals rather than risks and issues. It’s a behavioral trick simply to talk about the thing you want to give attention. And at the end of the day the CEO wants to talk about goals rather than risks. She rather wants a successful performance with a few flaws that an delay due to bruised hands (and egos) during waxing on and waxing off.

#269 – We Have Outsmarted Ourselves Again

jo2sn Enterprise IT, Softwaretesting, Wardley's theories , , , ,

Links from my talk at “map-camp-use-case-edition” Nov 2021:

Blogposts:

Photo: Kimiya Oveisi on Unsplash, https://unsplash.com/photos/rzsBKBb96HA

#268 – Who Brings in New Knowledge?

jo2sn Enterprise IT, Family and fun, Softwaretesting , , , , , , ,

Well, if you are reading this – there’s a good chance it’s you. Especially if you read this with the intention of sharing this with your team. I hope you do, obviously 😊. But perhaps it’s unclear whose responsibility it is, to bring new knowledge to the team. Is it always the team manager job – or is it a dedicated person that by role, or by habit, that bring in new knowledge?

Photo by William Fortunato on Pexels.com
Continue reading →

#267 – Reminder – it’s about Story Shaping

jo2sn LEGO, Softwaretesting , , ,

In agile delivery teams, it’s recommended that stories and features are discussed before being developed. A checklist called “Definition of Ready” can be used as a reminder to check that the delivery team has everything needed to prepare the delivery of the feature. A key ingredient is a shared understanding of how to confirm the delivery – for the whole team.

Story shaping requires at least three people to get together. They are not amigos nor amigas. They are often three, sometimes two people, and often more. They represent the different viewpoints needed to deliver: builders mindset, testing mindset, security mindset, operations mindset, business mindset, etc.

No one person can hold all these viewpoints – at least a builders mindset and a requester/business mindset is needed.

If you skip doing story shaping your acceptance criteria and testing activities will be misaligned. And you will either have overdone the effort needed or underestimated the challenges of the story.

One customer, three people to do the story shaping