Continuous Customer Feedback

There are many ways for continuous customer feedback. In coding deliverables, you can demonstrate how the solution works so far and monitor the usage to get actual data on the end-user experience. It’s proven over and over again that faster feedback loops are essential to optimal deliverables – and to improve business performance. Go read Accelerate.. again! Here are some of my tricks for the non-coding deliverables as a staff-level testing consultant.

Provide Continuous Training

One task I have is to help others manage the testing activities. It’s usually people fresh out of Uni – as long as there is curiosity the rest can be learned. The way I usually do it is to have an intro session, break down the first few steps and then have recurring meetings to align and dig even further into the tasks. My outcome as a leader is to make them self-reliable in doing the tasks. The usual struggle is for me to sit on my hands and to pause more.

Simon Koerner suggests these five ways to lead when not in charge:

  • Be proactive
  • Motivate those around you
  • Look beyond your job role
  • Recognize others
  • Share your knowledge

Continuous Test Plan Alignment

I’m currently leading a testing activity to deliver some IT services to company A. There’s a huge contract that stipulates all the mechanics of delivering these services. And for each testing activity, there is a good old-fashioned test plan document that needs to be delivered and approved by company A. (sigh)

The old-fashioned way would be for me to prepare the document in detail and forward it formally. If you have to write too much down and debate the documents over and over – it might be an organizational maturity issue, but it’s most likely a people problem and a trust issue.

Recently I have started stating that the test plan will be presented continuously. At my weekly meetings with the Customer A test responsible persons I present the current draft of the document. This way they see it being developed step-by-step and we can align – and the approval will be a formality in the end. I prefer this way of working – but have also failed where the customer expected detailed hand-overs and I preferred to work entrepreneurially to set things in motion.

Limiting Contracts Replies to a Few Pages

When working with large IT services contracts it’s unfortunately customary that there are requirements for the content and headlines of test plans and other test documents. These content requirements are often straight out of old templates listing testing deliverables, testing types, and physical conditions for the testing activities. No test documents will be approved if there are sections missing, as the list items are contractual minimum requirements. No one wants to be non-compliant with a contract based on a missing headline in a document somewhere.

The contract authors and their lawyers would probably argue that it provides business value – at least in the sense of comparing competing companies submitting for the contract. That best-practice templates are market-leading. Yet, the case is over and over again that best practices will only give you standard solutions that are not tailored to you.

In the year 2022, it’s so awkward that you have to address physical rooms and equipment as a required element. Sure if it’s special in this case, by all means. But usually, it probably isn’t that important. Preferably we should make situational aware test plans – but if your content isn’t as innovative I recently wrote a reply in a big contract tender for customer B reply that solved this in a good way.

For the contract, customer B requested a maximum of 4 pages – where we had to elaborate on what we would advise the customer and our experience in doing so. See we don’t need 100 pages to convey a quality narrative. And based on our reply they could still evaluate and score us on how well we understood their situation.

Continuous customer feedback succeeds when we show that we understand the customer.

Old Test Types Need Not Apply

I primarily work in situations that are less about application delivery and more about moving the whole system stacks, implementing a standard system, or similarly changing the organizational IT landscape. Some would list these projects as “staff projects” if that helps you. I often find the terms Regressiontest, SIT, and UAT to be misleading and not helpful to what kind of test is needed for my examples.

Everything and the kitchen sink
Everything and the kitchen sink

Example 1: We are rebuilding all the environments in the delivery stack. All the Dev-, Test-, Integration-, Preprod-, and Prod- environments, and their underlying databases, brokers, and websites. Every time we are constructing, an environment we will be testing the setup from a baseline version of the existing running systems. A baseline that we know is functional already. The classic software development testing types don’t really help us in this situation, as neither regression test, UAT, or SIT conveys the things we want to confirm and the learnings we want to explore.

Example 2: We are setting up a new expense platform for employee reimbursements to go live with a new branding of the company. It’s a SaaS system and we load it every month with data about the organization. So while it’s needed for various purposes – the risk is low and the mean time to repair is similarly low. The testing we will do will be a limited confirmation of an initial data load. Snow-plow style – not a full system-integration test and similar user-acceptance test. After all, this is a SaaS – not a custom solution. It’s OK to shift right.

SIT and UAT has become generic term that it has lost the strength to convey the needed quality narrative. If you do CI/CD (which you should) for your application development that such be sufficient. If you figure out, you need to do a “connection alive” test for your third-party integrations that you move from one environment stack to another that should be accepted with the acknowledgment that you actually considered the challenges ahead.

It’s all about the risks and the mitigations – less about testing everything to the dot. One tool to read the landscape is to communicate curiously about what the stakeholders value more – and value less. And on the other hand, consider the nature of the solution being proposed.

Example 3: Setting up a cloud-based Azure Active Directory – the solution comes with a given security level out of the box (OOTB). As with other OOTB and Software-as-a-service solutions you have little impact on the security features of the solution, besides some simple configurations. While you might think that all security requirements would require 100% acceptance testing coverage, what you want to accept is that they might be provided “by design” – or by a solution decision made long ago.

I would prefer that we can call things what they are and not blindly apply old testing types

Communicate Curiosity

A “testing mindset” of inquiry applies when you are a principal tester or even a senior advisor in testing. Asking open questions and approaching a challenge with exploration proves better than command and control. People are people – be mindful of where they are.

Currently, I’m involved in an extensive change program for a company. It’s not your usual agile feature factory but more about the technology stack needed to run a 3000 people company (payment, finances, etc). While testing professionals could be onboarded – their learning curve would not scale – not even being equipped with all the heuristics and testing vocabulary in the world. As it is a one-off, automation wouldn’t scale either.

Fortunately, most applications have a system owner (or product owner, or manager in charge) and usually a team around maintaining the application. To some degree, we could frame these teams as stream-aligned teams. The experts in testing the applications are the superuser – hence they do the testing.

Trust must be extended before it is given

Rachel Happe – @mastodon.social/@rhappe

One such team is the XYZ team anchored by a VP-level manager. A colleague and I had been trying to communicate with them to align on the testing needed but had met reluctance. They would not have time to help us. We discussed the matter and realized that we needed to approach them differently. Since the change program was such a fundamental shift for the team, they had already considered the implications. They were already testing and thinking about the impact. We just had to trust them – and build on what they already had instead of imposing specific ways of working.

Without requiring them to earn it first, tell everyone who works for you that you #trust them implicitly. From that point forward, just ask them to never give you reason to withhold it. (In my experience, most won’t).

Mark C. Crowley

Communicating curiosity can be a little thing like “Remind me, how does XYZ work” or simply “sitting on your hands” during meetings that you would previously fill with questions and requests. The tables of the great program test lead are turning, it’s about being an enabling function for others to succeed. Similar to the freedom you should give your growing up kids – it’s about leadership.

How Do you Eat a Mastodon?

… one piece of the time. Really. Don’t worry about it – it’s similar to eating an elephant.

While Discord and Slack sites can be places to go for communities with the faill of Twitter – they are closed systems. You need an invite and communication closed within the instance. This is good for closed discussions you want to gatekeep. Mastodon seems to be the place for serendipity and cross-collaboration across topics and communities. It’s not new, the framework started in 2016 gaining 1 million users in 2017. There are a number of great and extensive guides – one of them is fedi.tips. Another great guide is Mastodon Lessons Learned by Lisi Hocke (@lisihocke@mastodon.social).

Hi I’m Jesper🧷🌻🐐📕🗺🇩🇰 (@Jo2sn@mastodon.social)

Signup and Servers

For Mastodon, you create accounts on a server (or instance) in the Mastodon-based network. Through a shared protocol the people on the servers can communicate with posts, threads, likes, and resharing as known from other platforms. Mastodon is really the software framework (on GitHub) and the network called “Fediverse” – but people mostly call it “on Mastodon”.

Someone wrote that picking a Mastodon Server feels like selecting a character in a new video game. You don’t really know what you’re looking into and how you will play the game. Luckily you are not fixed to one account on the Mastodon network. You can have multiple and easily redirect from the old to the newer. I have tried jumping profiles myself, as the first one I made was on a server with too few general topics. One of the elements of Mastodon is that each server has a timeline – besides the ability to communicate across.  

Some servers are topic-based on Open-source software, others on art and LGBT, you can pick a specific server and expect the local content mostly on that topic. Or you can pick a general-purpose server – if one topic doesn’t define you. There are some tools, like debirdify that can help you in the process – but not yet a 1-click tool to move over. 

Two ways to select a server:

People can and will move to many places – there is yet to be a primary test community instance. 

Profile and Settings

  • Your username is text-based, but you can go crazy with emojis in the profile name field
  • Create your profile with a picture, banner, and profile text
  • Profile meta-data and hashtags will help people find you and recognize you
  • Add 2-factor authentication and save the password and keys
  • If English is not your native language, there are other languages available (German is rather prominent). Selecting a language seems to give you more content in that language.
  • Enabling the “slow mode” setting is recommended. It forces you manually to reload the feeds and not have the dreaded firehose. 
  • Make an #introduction post and write your favorite hashtags, that will help people find you.
  • Write your Mastodon username @user@server.name in your Twitter bio so that others can follow you

General Use

As there is no algorithm to drive engagement, so use hashtags and reshare content from others. Likes are only posted to the author – so reshare/retoot great posts. Similarly to following people you can follow hashtags like #MinistryOfTesting and #TwitterTestingPeople. The culture of Mastodon is to be inclusive. Use Content-Warnings and hide images as a common rule – and ALT-text on images as a general rule. 

Mastodon has free apps (Android, iOS) for casual use and some pay-to-use apps. Currently, it seems you can’t follow hashtags in the iOS app, so I tend to use the browser more. There is a range of choices – see what bests fit you. You can get a long way with just the primary site.

Servers are run by small teams of admins, so in peak periods there might be suboptimal performance. Most servers seem to be funded by patreons or similar fundraising. 

In theory, an instance could close down and take everything away. If you have content that is more permanent and you want to own, put it somewhere you control. The same challenge is true for ad-based social media (Facebook) and subscription-based platforms like Medium. 

If you feel like organizing the content coming your way, enabling the advanced web GUI setting will give you columns like “TweetDeck” – but the setting is global for your login. Sometimes I just want a quick read over the timelines, other times I want to monitor specific topics. To monitor topics I have set up columns in Sengi https://sengi.nicolas-constant.com/. It’s a per-browser setting, so it’s more of a topic radar that can be different across my devices. Personal lists are a feature too, but it seems based on people not so much on topics. 

Do notice that direct messages aren’t private messages. All the people you tag can read the messages. So don’t tag the people you want to rant about. 

The visibility of a toot/post follows the flowchart below. 

Via @geekyMary@Mastodon.social: https://mastodon.social/@Geekymary/109290888512800541

Writing a Book in 30 Evenings

As mentioned I have written a book. Looking through my notes I have around 30 versions, one for each significant session of working on it. I have been asked to share tips on writing a book, so here you go, Simon. The key lessons are:

  1. Set a Recurring Time and Space
  2. Unique Content – Reference the Rest
  3. Storyline and Flow
  4. The opposite of passive voice is not an aggressive voice
  5. LeanPub’bing

Set a Recurring Time and Space

One of my book’s themes was moving something from a hunch to a hard truth. And the same really applies here. When I finished my master’s degree while working (back in 2002), I had two slots a week at a “study office”. From that, I learned that not all study days are equally productive “on paper” – but that’s ok. Each session had its purpose.

Similarly for my book project. I set a weekly evening booking in my Calendar – family chores were arranged around it. After dinner, I work start working on the book and work for 2-2½ hours. I used my personal computer in my work-from-home setup.

Unique Content – Reference the Rest

There is so much great content out there already, my focus was on writing about my experience and my vision of better test strategies. But to do that I needed to stand on the shoulders of others to set the scene and describe the techniques I applied.

While writing I did want to bring in loads of existing content to elaborate and provide a foundation for my thinking. While editing I removed most of it, partly because I didn’t want to sound like a high school book report (thank you for that one Tristan). What I did leave in were quotes, recommendations, and listings of the work of others. The book is full of footnotes directly on the page (as compared to end notes) to highlight everyone in context.

Storyline and Flow

Initially, I outlined the chapters and subchapters and it was important to get the right “flow” and storyline into the content. My base model was inspired by “Situation, Complication, Question, Answer” from the guide “How to present to executives” (StaffEng Book) and similar lessons on taking the first steps.

  • This is the situation and challenges
  • These are the techniques, we can build on
  • These are the first steps, where the techniques are used

One thing I have worked a lot on is the flow of the text: Sections would be 4-6 lines long, with empty lines in between. I also worked to reduce “dangling lines”, so no two lines would linger into the next page. No pages should be text only, so quotes and illustrations are important for readability – as well as making the book content visible. Lastly, I worked a lot on having one section end with words that tap into the next section.

The below recommendation is cool, I could have done that too.

The opposite of passive voice is not an aggressive voice

Often I started my “book evening” by reading the book from the start all over again. As I’m not a native English writer, I installed Grammarly and paid for the premium version for a period to rewrite phrases that were in a passive form. Fun Fact: the opposite of passive voice, is not an aggressive voice but an active voice.

The tool you use to write your book isn’t that important. I used Google Docs with embedded Google Drawings, others might prefer Word or other editing platforms. The biggest challenge in publishing was getting the embedded pictures in a resolution that could be printed by the print shop. The book exists in a printed format – but it’s primarily an electronic book.

LeanPub’bing

Self-publishing on LeanPub is intuitive and easy, it also has tools for incremental versions, previews, and updates. Let people sign up for it in advance with price suggestions. That allows you to set a price on the book based on audience-based quotes.

They also have a great system for coupons. I have coupons for those that helped in preparing the book, those I have referenced, and those that participated in the virtual launch. Let me know with the phrase “LeanPubbing” if you want a coupon too.

On the Shoulders of 112 Giants

In my book “Goal-Aligned Test Strategies,” I draw extensively on the work of others. This is a reference post to further credit my reviewers and to list the 112 references in the book.

Special thank you to Lonnie for my writing evenings and to Simon Wardley for finding a path.

Reviewers and Feedback

Continue reading

Taking My Own Medicine

Recently I had the chance to apply my own templates to myself and my active project – as I had to mentor a new test manager. I was challenged in explaining how I read the upcoming IT environment project. After looking into resources for new test leads, I realized I could take my own medicine.

Photo by PhotoMIX Company on Pexels.com
Photo by PhotoMIX Company on Pexels.com

A year ago, I created a new test plan format – the Situational Aware Test Plan. While mind-maps and one-page test plan canvases exist, I wanted to elaborate using the evolution principles from Wardley mapping and stop writing test plan documents.

The table structure is there to provide guard rails for the elaboration. I will use the Darlings, Pets, Cattle, and GUID -mnemonic as headlines. Our strategic decisions emerge as we use the worksheet based on the current situation and state. The strategies will be the decisions to push a field in the grid to another state. 

Delivery and Situation

DarlingsPetsCattleGUID’s
New projectFixed date
Existing delivery speedScheduled
Quarterly
Test Environments, internalRepeatable
Test environments with integrationsCraftedSome existing know-how
Environment InfrastructureHosted data center practices
Test dataKnown but cumbersome

While this project introduces new test environments, there is an existing environment with a quarterly delivery pace. This is a classic example of the core chronic conflict of pursuing both: responding to the rapidly changing competitive landscape and providing stable, reliable, and secure services (DevOps handbook introduction xxv) as elaborated on Align your Test Strategy to your Business Strategy.

The test team allocated beside me and the new test lead is a new junior and senior tester. We are in the same team, and most are even in the same office. So collaboration will be collaborative and pervasive, with a focus on helping the new people grow.

The test team

DarlingsPetsCattleGUID’s
Test team collaborationGrowingPervasive
Test leadGrowing
MentoringEnabling
Domain know-howGetting there

Test tools and approach

DarlingsPetsCattleGUID’s
Test activityExplore integrationsConfirm internal requirements
Test casesExisting can be updated.
Test case reproCreate new repository

As mentioned in the blog post about visualization, we can now use the map to discuss why we need CT and ET for the project. Based on the project’s layout, I would advise having an expert exploration of the integrations and more standard scripts for the known construction of the internal environments.

Where I Have Changed

The Ministry of Testing Bloggers Club suggested that I write a post based on “In testing, I have changed my mind about ________”. As this blog dates back to 2012 with consistent (220) articles about testing, and my career in the field dates back to 2002 – it seems a 20-year experience should give me a few things. Testing is still not dead – and it’s still about the context (lower-case context, not CDT).

It’s not about: Testers being the only ones doing Testing

yeah, not so much these days. Testing is an act that any role can do in context. It’s about the testing – not so much the testers. And I have realized that even classic test management tasks can be done by someone else. Testing is not owned by the testers – it might be stewarded/facilitated by us, but it’s performed by a team member (who could be a tester).

It’s not about: Perfect Requirements

After decades in IT, it’s clear that even requirements are never perfect. When we look closer we see the business requirements can vary from a profound idea to a rudimentary feature of the system under test. Even in regulated industries requirements can be both about a specific configuration in a SaaS system or a loose idea of a relevant dashboard. Sometimes a requirement can be by design of an underlying commodity product – there doesn’t need to be a test case for everything.

The more rigor you add to the requirements management – the more fragile it becomes. It’s key to understand the risks and bets of the person paying for your solution. – in that lies the true borders of the delivery. Much can go informally along if it aligns with stakeholder values.

It’s not about: Defects

Back in the day defects needed to be accounted for, tracked, and distributed. Besides testing documents – defects were the only tangible delivery of the testers. The defects needed to be raised and closed. I recently wrote a guideline that stated that only observations that couldn’t be fixed within a day should be raised to the project manager for shared handling. In that context fixing things is within the same team. If it’s for another team to fix, defects are simply something communicated between the teams (check team topologies for team interactions). Sure you can still find a blocker or a P1 – what matters is how fast you can fix things.

It’s not about: Month-Long Testing Phases

The more time there is from idea to implementation – the more the requirement risks not addressing an up-to-date business objective. Timing is key. Some tools provide epics and user stories – but the structure is often misused to be a simple work aggregation – and not goal aligned.

The counter-intuitive trick is not to add formality, and more time between releases – but less. Less time between feedback between idea and implementation, and less time between implementation and test. Less time between the various forms of feedback adds to better results faster.

It still happens, I’m sure, that a business needs a month-long testing phase before a release; having a range of business staff to participate in testing the latest release of the enterprise ERP or CRM. More often the testing phase is one sprint behind the development activity. I have pondered this a lot.

At best testing is an integrated activity in the team and in the sprint. But if testing is a more separate activity – it can be both agile and context-relevant. So I have changed my mind about this anti-pattern.

It’s ok for testing to be in the next sprint –

if that adds consistency and less stress to the team*

* Seperate boards needed. Your Mileage Will Vary

Why Do We Fall, Master Bruce?

… So that we can learn to pick ourselves up, Alfred! I was recently reminded of this quote from watching the “Batman Begins” movie with my 16yo. I really needed that reminder. Then I read the two blog posts by Beren on “Those who Failed” and “Versus the Endboss“. Let’s put it out there that we fall – and fail to remember that we fall.

I had been part of a large project – but had read the culture all wrong and we had failed hard. For a number of reasons and maybe mostly for systemic reasons. The team expected one mindset and one way of tooling – we provided another one. Even with all my best intentions and know-how of change management, this crashed. As Hannes elegantly put it, we had cycled too far ahead of the team:

  • The team expected minutes of meetings and agendas, we worked for making things visible and shared
  • The team expected testing to be checking the requirements, we worked for testing to support critical decision making
  • The team worked political with back channels, we worked open and power-lifting
  • The team participants had agendas that didn’t align with the project’s purpose
  • The team expected all things equally important, we worked by priority and deadline
  • The team expected detailed test cases to be approved at all steps, we provided intentions and purpose
  • The team expected detailed handovers, we worked entrepreneurially to set things in motion
  • The team expected an error-free lead, we worked knowing I wouldn’t remember everything

At one point I was arguing that the team needed to read Hofstede’s cultural dimensions theory – to read up on the different cultures we would be interacting with (our customers). In retrospect, we should have used it on ourselves first of all.

Hofstede’s cultural dimensions theory

It’s a little more detailed than Westrum – and even Westrum might have helped. That is if we had been able to articulate the conflict well in advance. Perhaps a senior hire should have spotted the signals beforehand. As an outsider, I relied on people telling me things. I couldn’t hear or see the back-channel communications. This is a struggle for many staff people when switching roles:

Initially, no one from the operations organization and latest implementation opted for the leading the activity. As we had no playbook or project plan (only the produced artifacts) – I made a scrum-board-inspired work tracking system. Perhaps I should have used a Wardley map first of all as recommended by John Cutler in “TBM 18/52: We Need Someone Who Has Done “It” Before

What is Wardley Mapping doing for us here? It is letting us explore a more nuanced view of the problem space. Instead of treating things as one problem, we break the problem apart into a bunch of capabilities. When we do this exercise we typically find:

Not everything is an existing playbook. Not everything is a new playbook.

To solve new problems, we need a foundation of stable playbooks. For example, to solve that crazy new problem, the team might need a foundation of trustworthy data.

Yes, you can break things apart to see them better. But you’re also dealing with the whole thing.

But then again the team would probably have stalled over the very concept of a strategy map. People are weird. No matter how it looks at first, it’s always a people problem. And even if you do try to take the first steps – your steps could be in the wrong direction. Even Master Bruce will fall in that situation.