Lessons Learned in Cloud Testing

For reasons of regulations and data restrictions, we were recently doing a cloud project for a large customer of ours. Cloud resources had to be spun up, granted access to, utilized, and taken down (CRUD-like). But we had to do it without granting any of the customer’s employers access to the cloud controls. They couldn’t even have a cloud admin. So for various organizational reasons, we designed the solution below.

The user would order cloud activities in a “shop” and the requests sent first to the customer ticketing system, then to our ticketing system. And finally to the cloud pipelines and scripts. And there my friend is where the testing started.

Testing Input Sanity

The first script was for initializing the cloud services: Ressource Name. First of I found out the (cloud) developers had already tested the happy paths. I loaded Bugmagnet and gave some funny names, as the requirements for that field hadn’t really been that specific.

https://bugmagnet.org/

I quickly learned that the cloud provider was already sanitizing the input, so no Little Bobby Tables were allowed. It made the customer think, though, and she added a rule for the shop with regard to the Ressource names. This was coded as a regular expression (Start with a capital letter, then some letters, perhaps some numbers) – not that we or the cloud cared for that strict naming scheme, it was more of a business usage rule. Was that a bug? No mostly something that testing finds – not a bug.

I Found A Bug- Can I Go Home Now?

The next request could assign privileges to the cloud resources. Based on the resource name and Access Level, the cloud resources would be enabled. Among the access level options was “admin” rights. Which was directly opposite to the whole purpose of the project, and the customer risked breaching the regulatory inertia/controls.

And that was really the only formal bug we logged. And I don’t go home – I quickly reached out to the team, had the Accces level parameter removed and all was good. The intention was all along that only one contributor access level should be granted. That level was then hardcoded in the code repository.

And then we tested further around creating things already created, revoking resources that were not there, and race conditions for revoking the services. The only thing that really gave us a challenge was when a request failed (say you request something already there). Having the two ticking systems pick up the response and send the error message back to the users was more trick than expected as they were more REQUEST focussed – but we managed. Without creating a bug the team worked together and resolved the issues.

Key Learnings

As always this is a story retold from actual experience, there are details I can’t share and elements I preferred to have done differently – but due to the organizational culture had its own way.

• A mindset toward input sanity and access control is key to supporting business goals
• Bugs, issues, and observations will be found they are learnings for the team, not problems
• Hold end-2-end rehearsals of the solution and play out the scenarios
• Don’t restrict the testing activities to software development, look at the whole solution

A Chocolate Car Calamity

If something as seemingly simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.

Once upon a time, there was a professional chocolate shop. It was an actual chocolate shop, full of products for many needs, and a staffed counter, where you could go and order. One fine day a customer arrived and asked if the shop could create a chocolate car. The shop chocolatier said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car”. “Ah, very nice chocolate car. But please could the wheels turn.”The shop chocolatier a little annoyingly said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn”. “Ah, very nice chocolate car. But please could the doors open.” The shop chocolatier annoyingly said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open”. “Ah, very nice chocolate car. But please could it have a sunroof.” The shop chocolatier a little more annoyingly said: “Come by tomorrow and I will have it for you”.

The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open, see the sunroof”. Just as the shop chocolatier was going to get even more annoyed, the customer said: “thank you, no more updates”. Baffled the shop owner forgot all about steering, ABS, and passenger seats, and asked: “How would you like it packaged”. And the customer said “ah! No need, I’ll eat it right away” And so he did. The End.

Despite all the product demos and product development, the customer consumed the product in a heart beat – or probably two.

The Morale, as There is Always a Morale

The origin of this story is a two person sketch from my local youth work[link in Danish] in the tradition of Abbott and Costello’s “Who is on first base” – so it’s probably from the 1960’es. The sketch is usually acted with elaborate gesturing building up to the absurd punchline on the final day.

I was reminded about it recently – in a project where external reviewers kept coming back for more. More requests for bells and whistles that where not originally stated. While we did share the test approach initially, the feedback from the customer is coming towards the end of the project delivery.

All the extra effort seems absurd in contrast to communicating goals and collaborating up front. While each party might be trustworthy, it only takes one of them to extend trust and improve the relationship significantly. Extending trust would obviously defuse the absurdity of the exchange, and there would be no sketch. And we need the sketch to provide the backdrop to the morale:

• As a customer state your end goal clearly, don’t be a push-over
• As a development team, stop and ask. If it’s outside your usual range it’s ok to say no.

If something as simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.

Making chocolate dough by Jakub Kapusnak is licensed under CC-CC0 1.0

How Do you Eat a Mastodon?

… one piece of the time. Really. Don’t worry about it – it’s similar to eating an elephant.

While Discord and Slack sites can be places to go for communities with the faill of Twitter – they are closed systems. You need an invite and communication closed within the instance. This is good for closed discussions you want to gatekeep. Mastodon seems to be the place for serendipity and cross-collaboration across topics and communities. It’s not new, the framework started in 2016 gaining 1 million users in 2017. There are a number of great and extensive guides – one of them is fedi.tips. Another great guide is Mastodon Lessons Learned by Lisi Hocke (@lisihocke@mastodon.social).

Hi I’m Jesper🧷🌻🐐📕🗺🇩🇰 (@Jo2sn@mastodon.social)

Signup and Servers

For Mastodon, you create accounts on a server (or instance) in the Mastodon-based network. Through a shared protocol the people on the servers can communicate with posts, threads, likes, and resharing as known from other platforms. Mastodon is really the software framework (on GitHub) and the network called “Fediverse” – but people mostly call it “on Mastodon”.

Someone wrote that picking a Mastodon Server feels like selecting a character in a new video game. You don’t really know what you’re looking into and how you will play the game. Luckily you are not fixed to one account on the Mastodon network. You can have multiple and easily redirect from the old to the newer. I have tried jumping profiles myself, as the first one I made was on a server with too few general topics. One of the elements of Mastodon is that each server has a timeline – besides the ability to communicate across.  

Some servers are topic-based on Open-source software, others on art and LGBT, you can pick a specific server and expect the local content mostly on that topic. Or you can pick a general-purpose server – if one topic doesn’t define you. There are some tools, like debirdify that can help you in the process – but not yet a 1-click tool to move over. 

Two ways to select a server:

• Browse the directory at https://joinmastodon.org/servers 
• Use https://pruvisto.org/debirdify/ to see where your Twitter people have moved

People can and will move to many places – there is yet to be a primary test community instance. 

Profile and Settings

• Your username is text-based, but you can go crazy with emojis in the profile name field
• Create your profile with a picture, banner, and profile text
• Profile meta-data and hashtags will help people find you and recognize you
• Add 2-factor authentication and save the password and keys
• If English is not your native language, there are other languages available (German is rather prominent). Selecting a language seems to give you more content in that language.
• Enabling the “slow mode” setting is recommended. It forces you manually to reload the feeds and not have the dreaded firehose. 
• Make an #introduction post and write your favorite hashtags, that will help people find you.
• Write your Mastodon username @user@server.name in your Twitter bio so that others can follow you

General Use

As there is no algorithm to drive engagement, so use hashtags and reshare content from others. Likes are only posted to the author – so reshare/retoot great posts. Similarly to following people you can follow hashtags like #MinistryOfTesting and #TwitterTestingPeople. The culture of Mastodon is to be inclusive. Use Content-Warnings and hide images as a common rule – and ALT-text on images as a general rule. 

Mastodon has free apps (Android, iOS) for casual use and some pay-to-use apps. Currently, it seems you can’t follow hashtags in the iOS app, so I tend to use the browser more. There is a range of choices – see what bests fit you. You can get a long way with just the primary site.

Servers are run by small teams of admins, so in peak periods there might be suboptimal performance. Most servers seem to be funded by patreons or similar fundraising. 

In theory, an instance could close down and take everything away. If you have content that is more permanent and you want to own, put it somewhere you control. The same challenge is true for ad-based social media (Facebook) and subscription-based platforms like Medium. 

If you feel like organizing the content coming your way, enabling the advanced web GUI setting will give you columns like “TweetDeck” – but the setting is global for your login. Sometimes I just want a quick read over the timelines, other times I want to monitor specific topics. To monitor topics I have set up columns in Sengi https://sengi.nicolas-constant.com/. It’s a per-browser setting, so it’s more of a topic radar that can be different across my devices. Personal lists are a feature too, but it seems based on people not so much on topics. 

Do notice that direct messages aren’t private messages. All the people you tag can read the messages. So don’t tag the people you want to rant about. 

The visibility of a toot/post follows the flowchart below. 

Via @geekyMary@Mastodon.social: https://mastodon.social/@Geekymary/109290888512800541

Taking My Own Medicine

Recently I had the chance to apply my own templates to myself and my active project – as I had to mentor a new test manager. I was challenged in explaining how I read the upcoming IT environment project. After looking into resources for new test leads, I realized I could take my own medicine.

Photo by PhotoMIX Company on Pexels.com

A year ago, I created a new test plan format – the Situational Aware Test Plan. While mind-maps and one-page test plan canvases exist, I wanted to elaborate using the evolution principles from Wardley mapping and stop writing test plan documents.

The table structure is there to provide guard rails for the elaboration. I will use the Darlings, Pets, Cattle, and GUID -mnemonic as headlines. Our strategic decisions emerge as we use the worksheet based on the current situation and state. The strategies will be the decisions to push a field in the grid to another state. 

Delivery and Situation

	Darlings	Pets	Cattle	GUID’s
New project	Fixed date
Existing delivery speed		Scheduled Quarterly
Test Environments, internal			Repeatable
Test environments with integrations	Crafted	Some existing know-how
Environment Infrastructure			Hosted data center practices
Test data		Known but cumbersome

While this project introduces new test environments, there is an existing environment with a quarterly delivery pace. This is a classic example of the core chronic conflict of pursuing both: responding to the rapidly changing competitive landscape and providing stable, reliable, and secure services (DevOps handbook introduction xxv) as elaborated on Align your Test Strategy to your Business Strategy.

The test team allocated beside me and the new test lead is a new junior and senior tester. We are in the same team, and most are even in the same office. So collaboration will be collaborative and pervasive, with a focus on helping the new people grow.

The test team

	Darlings	Pets	Cattle	GUID’s
Test team collaboration			Growing	Pervasive
Test lead			Growing
Mentoring		Enabling
Domain know-how		Getting there

Test tools and approach

	Darlings	Pets	Cattle	GUID’s
Test activity	Explore integrations		Confirm internal requirements
Test cases			Existing can be updated.
Test case repro			Create new repository

As mentioned in the blog post about visualization, we can now use the map to discuss why we need CT and ET for the project. Based on the project’s layout, I would advise having an expert exploration of the integrations and more standard scripts for the known construction of the internal environments.

Why Do We Fall, Master Bruce?

… So that we can learn to pick ourselves up, Alfred! I was recently reminded of this quote from watching the “Batman Begins” movie with my 16yo. I really needed that reminder. Then I read the two blog posts by Beren on “Those who Failed” and “Versus the Endboss“. Let’s put it out there that we fall – and fail to remember that we fall.

I had been part of a large project – but had read the culture all wrong and we had failed hard. For a number of reasons and maybe mostly for systemic reasons. The team expected one mindset and one way of tooling – we provided another one. Even with all my best intentions and know-how of change management, this crashed. As Hannes elegantly put it, we had cycled too far ahead of the team:

Something I learned trying to be a change agent: it is kind of like team cycling. If you want to help your organization forward, going too far ahead of the group will only wear you out. Riding in front of the group absorbing some of the air resistance is much more effective.

— Hannes Lindblom (@HannesLindblom) April 21, 2022

• The team expected minutes of meetings and agendas, we worked for making things visible and shared
• The team expected testing to be checking the requirements, we worked for testing to support critical decision making
• The team worked political with back channels, we worked open and power-lifting
• The team participants had agendas that didn’t align with the project’s purpose
• The team expected all things equally important, we worked by priority and deadline
• The team expected detailed test cases to be approved at all steps, we provided intentions and purpose
• The team expected detailed handovers, we worked entrepreneurially to set things in motion
• The team expected an error-free lead, we worked knowing I wouldn’t remember everything

At one point I was arguing that the team needed to read Hofstede’s cultural dimensions theory – to read up on the different cultures we would be interacting with (our customers). In retrospect, we should have used it on ourselves first of all.

Hofstede’s cultural dimensions theory

It’s a little more detailed than Westrum – and even Westrum might have helped. That is if we had been able to articulate the conflict well in advance. Perhaps a senior hire should have spotted the signals beforehand. As an outsider, I relied on people telling me things. I couldn’t hear or see the back-channel communications. This is a struggle for many staff people when switching roles:

People underestimate how much of their success is simply that they’ve been working at the same company for a long time and know the people & culture.

A recipe for success in switching companies as a senior hire is entering with a beginner’s mindset. When you don’t it’s painful. https://t.co/RJceWwKmm5

— Dare Obasanjo (@Carnage4Life) April 23, 2022

Blogged: Why Success is Often Elusive at the Highest Echelonshttps://t.co/6JnAs5BEXy

TLDR:

– innovation is hard without the right support structure and context
– execution is hard without being well-versed in the culture
– culture often has a steep learning curve https://t.co/AR2LtUpk27

— Cindy Sridharan (@copyconstruct) April 25, 2022

Initially, no one from the operations organization and latest implementation opted for the leading the activity. As we had no playbook or project plan (only the produced artifacts) – I made a scrum-board-inspired work tracking system. Perhaps I should have used a Wardley map first of all as recommended by John Cutler in “TBM 18/52: We Need Someone Who Has Done “It” Before“

What is Wardley Mapping doing for us here? It is letting us explore a more nuanced view of the problem space. Instead of treating things as one problem, we break the problem apart into a bunch of capabilities. When we do this exercise we typically find:

Not everything is an existing playbook. Not everything is a new playbook.

To solve new problems, we need a foundation of stable playbooks. For example, to solve that crazy new problem, the team might need a foundation of trustworthy data.

Yes, you can break things apart to see them better. But you’re also dealing with the whole thing.

But then again the team would probably have stalled over the very concept of a strategy map. People are weird. No matter how it looks at first, it’s always a people problem. And even if you do try to take the first steps – your steps could be in the wrong direction. Even Master Bruce will fall in that situation.

The Mechanics of Modern Meetings

In these days of virtual meetings, the very structures of formal meetings are under change. It’s definitely forged by extensive work-from-home and working with people not in the same locations. It challenges the people that are used to having everything in documents and actions/assignments tracked as part of a “Minutes of Meetings” document. They seem to mistake the absence of document artifacts with no structure. But if you look closer you will see that even a circus is a choreographed act.

The Agenda is always the Current State of Affairs

A key observation from the agile and collaborative way of working is the principle of making work visible. Put tasks and assignments on a shared board for the team. The tool is not so important, as long as it reasonably supports the kanban/scrum-board mechanics. You can use Trello, Podio, Miro, Azure DevOps, or Jira – whatever is available to you in your organization.

Among the benefits of a shared digital board is that it additionally supports the team with the ability to work on items asynchronously, independent of timezones, working hours, and locations. The state of affairs is whatever state the board depicts – so make sure it’s always as truthful as it can be. It takes practice for the team members to learn to update the board outside of the meeting. But this small step is really key in making the meetings more effective and reducing the time to information.

The status board challenges the fact that an agenda can be locked prior to a meeting. All items are moving pieces – so the agenda can only be “look at the board“. If someone is working on something – put it on the board. This also helps if a team member runs off to join a circus – or is temporarily away from this very circus.

Boards help to streamline getting things done. Items might not be perfect – but the focus is on getting them done. “Stop starting – Start stopping” is a recurring mantra. Secondly using a board and agile backlogs and work limits help to prioritize the work according to the team’s availability and speed of delivery. Bottlenecks and overloaded staff can be more easily identified.

Recurring touchpoints, though, are still needed for the team, but the latest status of the work items is no longer at the end of a Minutes of Meetings document.

Recorded Minutes of Meetings

Originally, the MoM (Minutes of Meetings) documents hold the decision items and action items after every meeting. As discussed a shared task board can replace much of the MoM. Is Alice joining Bob on a task? Did Charles agree to deliver X by Friday? All of those actions can be activities on the task board, as long as it’s added during the meeting. A meeting notetaker could do this during the meeting on the board, and not focus on writing down every minute. Adding ideas to the board’s “to-do” column is also a powerful way to remember things for the future.

A strong trend I see in the use of virtual meeting platforms is a default recording of most meetings. You have to get used to it, also privacy vise. Be careful in political organizations – the spoken word is now recorded. Among the benefits of recorded meetings is that everyone can rewind into the meetings and that previous meeting content is available for new team members. This goes especially well for content that is more “show and tell” than status calls.

My preferred leadership style is to set direction, provide what I have of relevant information, and follow up indirectly via the board. I don’t need to meet for a status message that can be read from the board. But I will use the information on the board to reflect on where we are and where we’re supposed to be heading.

Reframe meetings as Collaborative Conversations

When I set up a meeting in someone’s calendar, it’s not always with the intention to have the formal mechanics of a Capital-M Meeting. The scheduling in the calendar is a way to respect people’s time and to make sure key participants can be available at the same time. It’s out of the same respect for people’s variety of availability that meetings need to be effective.

I rarely invoke the formalities of a Meeting. When we (small-m) meet it’s to collaborate and interact and discover serendipity. Sometimes it seems that the name “meeting” is taken literally as a formal structure, while to me it’s more like a placeholder for collaborative conversations.

It may look like a circus – but that is on purpose. There is a choreography behind it all.

The Circus by Alex Herreru00edas is licensed under CC-BY-NC-ND 4.0

#267 – Reminder – it’s about Story Shaping

In agile delivery teams, it’s recommended that stories and features are discussed before being developed. A checklist called “Definition of Ready” can be used as a reminder to check that the delivery team has everything needed to prepare the delivery of the feature. A key ingredient is a shared understanding of how to confirm the delivery – for the whole team.

Story shaping requires at least three people to get together. They are not amigos nor amigas. They are often three, sometimes two people, and often more. They represent the different viewpoints needed to deliver: builders mindset, testing mindset, security mindset, operations mindset, business mindset, etc.

No one person can hold all these viewpoints – at least a builders mindset and a requester/business mindset is needed.

If you skip doing story shaping your acceptance criteria and testing activities will be misaligned. And you will either have overdone the effort needed or underestimated the challenges of the story.

One customer, three people to do the story shaping

Stop Writing Overdone Test Plans

While I have previously talked about writing down expectations and alignments – I would much prefer a more lean and up-to-date approach to test plan documents. Looking at what we know now, an separate test is more of a sign of missing trust between parties than a collaborative value add for the business needs.

You Cannot Be Everywhere

A key lesson with the whole situation is, that there is no Nirvana, no steady state nor closure to everything. The only constant is that life is ever changing. Coming up around the corner is a new change – some can be planned for and others not so much. Imagining that things can be different is the first step to dealing with change (Thank you, Virginia Satir).

There will always be more things needing attention. Sometimes it’s based on the fear of missing out (FOMO) and the urge to be involved in everything. You will very easily find yourself stretched too thin – and needing a way to scale the effort, both professionally and personally. Clocking more hours will only work in the short run. Remember, it’s ok not to scale – sometimes it’s actually the best solution.

Darlings, Pets, Cattle and GUID’s

Kill your darlings and treat your tests more like cattle than pets, are among some of the heuristics currently around for managing your environments and automation test suites. These heuristics tells me that the environments and automation are in a state of product or even commodity, while previously the tests and environments where like darlings and pets – named and nurtured.