While my first book was on the more advanced topic of visual test strategies my next book will be an entry-level guide to people moving into test management and test leadership roles – primarily for all those without a background in testing. But also for testers moving into leadership, as Meg below.
Folks who moved into test leadership roles – what do you wish you could go back and tell yourself before you started?
@cactusflamingo asks: Folks who moved into test leadership roles – what do you wish you could go back and tell yourself before you started?
I have been playing with the idea of it being a guidebook to these destinations:
The Sea of Testing Know-How
The Community Deep
The Pearly Personalities
The Leadership Lands
Management meadows
Shallow Shores
Project roadways
Transition highways
Agile release trains
Artifact wetlands
Document pitfalls
The Certification Wastelands
Rivers of Tenacity
And the various tool towns scattered across the land
Which parts of the testing world would you need a guidebook for? Let’s see how it goes – the voice and style will more likely be similar to the first. Currently, the introduction starts with the following verse
Once upon a time,
our protagonist set out on a quest to lead a testing activity.
While they searched and searched across the world,
they found no clear map to guide their quest.
And there, my friend, is where this story begins.
TL;DR: What I learned from applying the research from the book Accelerate (2018) by Forsgren et.al. as a strategy legend on my Wardley Strategy Map. Spoiler alert – it’s more about people than technology.
This, of course, is not a map, as I have not considered the user need, but more given each element a score based on my initial reading of the project. It’s a new project team that hasn’t worked together before – building a new system, that does not yet exist similarly. That we know of – so it’s Cynefin Complex domain. The tech stack and testing challenges are well-known and can be mostly automated.
There is probably a natural order to the items above: It’s a new situation (for the customer too) and a release is needed. the delivery needs staffing – staffing needs collaboration and tools. Delivery (releases) needs environments and verification and verification needs test data. The table is neither a strategy as I have not considered which items to move – but I had an idea.
If we zoom in, we can see the prerequisites for Continuous Delivery to be successful:
Test Automation
Deployment Automation
Trunk-based Development
Shift Left on security
Loosely Coupled Architecture
Empowered teams
Continuous Integration
Version Control
Test Data management
Monitoring
Proactive system notifications
These items are just in the order from the book – but I can similarly score our ability and tools. Because of the platform, most are a commodity or best practices: Loosely Coupled Architecture, version control, CI/CD, trunk-based development, and deployment automation. A few require a bit of a push from possibility to active use (proactive system notifications, monitoring). The sore spots are (as detailed in 24 Key Capabilities to Drive Improvement in Software Delivery):
for the development team to own test automation: Test automation is a practice where software tests are run automatically (not manually) continuously throughout the development process. Effective test suites are reliableāthat is, tests find real failures and only pass releasable code. Note that developers should be primarily responsible for the creation and maintenance of automated test suites.Ā
establish test data management: Test data requires careful maintenance, and test data management is becoming an increasingly important part of automated testing. Effective practices include having adequate data to run your test suite, the ability to acquire necessary data on demand, the ability to condition your test data in your pipeline, and the data not limiting the number of tests you can run. We do caution, however, that teams should minimize, whenever possible, the amount of test data needed to run automated tests.Ā
Empowered teams: Research shows that teams that can choose which tools to use do better at continuous delivery and, in turn, drive better software development and delivery performance. No one knows better than practitioners what they need to be effective.
The Test Strategy
Hence my focus as a test manager would obviously be the first two. Let me be the first to move for test automation by the developers (the domain for the test is mostly API endpoints) and the first to ask for killing the test environment so that we can learn to bring it up again – and again.
But because I care – and because I have read in the book what is a requirement for the listed team capabilities above – I should be mindful to consider also empowering teams and encouraging transformational leadership: Vision, Inspirational Communication, Supportive Leadership, Intellectual Stimulation, and Personal Recognition.
Using the research from Accelerate as a mapping legend for the team’s Continuous Delivery Practices I can see that even with state-of-the-art tooling and a modern high-level stack – it’s a people problem first of all – and thus the test strategy is threefold:
Encourage testing by the developers (evolve test automation)
Prioritize the construction of stubs and drivers for test data (evolve test data)
Encourage collaboration, learning, and a supporting culture (evolve culture)
For reasons of regulations and data restrictions, we were recently doing a cloud project for a large customer of ours. Cloud resources had to be spun up, granted access to, utilized, and taken down (CRUD-like). But we had to do it without granting any of the customer’s employers access to the cloud controls. They couldn’t even have a cloud admin. So for various organizational reasons, we designed the solution below.
The user would order cloud activities in a “shop” and the requests sent first to the customer ticketing system, then to our ticketing system. And finally to the cloud pipelines and scripts. And there my friend is where the testing started.
Testing Input Sanity
The first script was for initializing the cloud services: Ressource Name. First of I found out the (cloud) developers had already tested the happy paths. I loaded Bugmagnet and gave some funny names, as the requirements for that field hadn’t really been that specific.
I quickly learned that the cloud provider was already sanitizing the input, so no Little Bobby Tables were allowed. It made the customer think, though, and she added a rule for the shop with regard to the Ressource names. This was coded as a regular expression (Start with a capital letter, then some letters, perhaps some numbers) – not that we or the cloud cared for that strict naming scheme, it was more of a business usage rule. Was that a bug? No mostly something that testing finds – not a bug.
I Found A Bug- Can I Go Home Now?
The next request could assign privileges to the cloud resources. Based on the resource name and Access Level, the cloud resources would be enabled. Among the access level options was “admin” rights. Which was directly opposite to the whole purpose of the project, and the customer risked breaching the regulatory inertia/controls.
And that was really the only formal bug we logged. And I don’t go home – I quickly reached out to the team, had the Accces level parameter removed and all was good. The intention was all along that only one contributor access level should be granted. That level was then hardcoded in the code repository.
And then we tested further around creating things already created, revoking resources that were not there, and race conditions for revoking the services. The only thing that really gave us a challenge was when a request failed (say you request something already there). Having the two ticking systems pick up the response and send the error message back to the users was more trick than expected as they were more REQUEST focussed – but we managed. Without creating a bug the team worked together and resolved the issues.
Key Learnings
As always this is a story retold from actual experience, there are details I can’t share and elements I preferred to have done differently – but due to the organizational culture had its own way.
A mindset toward input sanity and access control is key to supporting business goals
Bugs, issues, and observations will be found they are learnings for the team, not problems
If something as seemingly simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.
Once upon a time, there was a professional chocolate shop. It was an actual chocolate shop, full of products for many needs, and a staffed counter, where you could go and order. One fine day a customer arrived and asked if the shop could create a chocolate car. The shop chocolatier said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car”. “Ah, very nice chocolate car. But please could the wheels turn.”The shop chocolatier a little annoyingly said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn”. “Ah, very nice chocolate car. But please could the doors open.” The shop chocolatier annoyingly said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open”. “Ah, very nice chocolate car. But please could it have a sunroof.” The shop chocolatier a little more annoyingly said: “Come by tomorrow and I will have it for you”.
The next day the customer arrived and was given a demo of the product: “Here is your chocolate car, see the wheels turn, see the doors open, see the sunroof”. Just as the shop chocolatier was going to get even more annoyed, the customer said: “thank you, no more updates”. Baffled the shop owner forgot all about steering, ABS, and passenger seats, and asked: “How would you like it packaged”. And the customer said “ah! No need, I’ll eat it right away” And so he did. The End.
Despite all the product demos and product development, the customer consumed the product in a heart beat – or probably two.
The Morale, as There is Always a Morale
The origin of this story is a two person sketch from my local youth work[link in Danish] in the tradition of Abbott and Costello’s “Who is on first base” – so it’s probably from the 1960’es. The sketch is usually acted with elaborate gesturing building up to the absurd punchline on the final day.
I was reminded about it recently – in a project where external reviewers kept coming back for more. More requests for bells and whistles that where not originally stated. While we did share the test approach initially, the feedback from the customer is coming towards the end of the project delivery.
All the extra effort seems absurd in contrast to communicating goals and collaborating up front. While each party might be trustworthy, it only takes one of them to extend trust and improve the relationship significantly. Extending trust would obviously defuse the absurdity of the exchange, and there would be no sketch. And we need the sketch to provide the backdrop to the morale:
As a customer state your end goal clearly, don’t be a push-over
As a development team, stop and ask. If it’s outside your usual range it’s ok to say no.
If something as simple as ordering a chocolate car can be an absurd people problem, no wonder complex IT projects can be a calamity.
There are many ways for continuous customer feedback. In coding deliverables, you can demonstrate how the solution works so far and monitor the usage to get actual data on the end-user experience. It’s proven over and over again that faster feedback loops are essential to optimal deliverables – and to improve business performance. Go read Accelerate.. again! Here are some of my tricks for the non-coding deliverables as a staff-level testing consultant.
Provide Continuous Training
One task I have is to help others manage the testing activities. It’s usually people fresh out of Uni – as long as there is curiosity the rest can be learned. The way I usually do it is to have an intro session, break down the first few steps and then have recurring meetings to align and dig even further into the tasks. My outcome as a leader is to make them self-reliable in doing the tasks. The usual struggle is for me to sit on my hands and to pause more.
I’m currently leading a testing activity to deliver some IT services to company A. There’s a huge contract that stipulates all the mechanics of delivering these services. And for each testing activity, there is a good old-fashioned test plan document that needs to be delivered and approved by company A. (sigh)
The old-fashioned way would be for me to prepare the document in detail and forward it formally. If you have to write too much down and debate the documents over and over ā it might be an organizational maturity issue, but itās most likely a people problem and a trust issue.
Recently I have started stating that the test plan will be presented continuously. At my weekly meetings with the Customer A test responsible persons I present the current draft of the document. This way they see it being developed step-by-step and we can align – and the approval will be a formality in the end. I prefer this way of working – but have also failed where the customer expected detailed hand-overs and I preferred to work entrepreneurially to set things in motion.
Limiting Contracts Replies to a Few Pages
When working with large IT services contracts it’s unfortunately customary that there are requirements for the content and headlines of test plans and other test documents. These content requirements are often straight out of old templates listing testing deliverables, testing types, and physical conditions for the testing activities. No test documents will be approved if there are sections missing, as the list items are contractual minimum requirements. No one wants to be non-compliant with a contract based on a missing headline in a document somewhere.
The contract authors and their lawyers would probably argue that it provides business value – at least in the sense of comparing competing companies submitting for the contract. That best-practice templates are market-leading. Yet, the case is over and over again that best practices will only give you standard solutions that are not tailored to you.
In the year 2022, it’s so awkward that you have to address physical rooms and equipment as a required element. Sure if it’s special in this case, by all means. But usually, it probably isn’t that important. Preferably we should make situational aware test plans – but if your content isn’t as innovative I recently wrote a reply in a big contract tender for customer B reply that solved this in a good way.
For the contract, customer B requested a maximum of 4 pages – where we had to elaborate on what we would advise the customer and our experience in doing so. See we don’t need 100 pages to convey a quality narrative. And based on our reply they could still evaluate and score us on how well we understood their situation.
Continuous customer feedback succeeds when we show that we understand the customer.
I primarily work in situations that are less about application delivery and more about moving the whole system stacks, implementing a standard system, or similarly changing the organizational IT landscape. Some would list these projects as “staff projects” if that helps you. I often find the terms Regressiontest, SIT, and UAT to be misleading and not helpful to what kind of test is needed for my examples.
Everything and the kitchen sink
Example 1: We are rebuilding all the environments in the delivery stack. All the Dev-, Test-, Integration-, Preprod-, and Prod- environments, and their underlying databases, brokers, and websites. Every time we are constructing, an environment we will be testing the setup from a baseline version of the existing running systems. A baseline that we know is functional already. The classic software development testing types don’t really help us in this situation, as neither regression test, UAT, or SIT conveys the things we want to confirm and the learnings we want to explore.
Example 2: We are setting up a new expense platform for employee reimbursements to go live with a new branding of the company. It’s a SaaS system and we load it every month with data about the organization. So while it’s needed for various purposes – the risk is low and the mean time to repair is similarly low. The testing we will do will be a limited confirmation of an initial data load. Snow-plow style – not a full system-integration test and similar user-acceptance test. After all, this is a SaaS – not a custom solution. It’s OK to shift right.
SIT and UAT has become generic term that it has lost the strength to convey the needed quality narrative. If you do CI/CD (which you should) for your application development that such be sufficient. If you figure out, you need to do a “connection alive” test for your third-party integrations that you move from one environment stack to another that should be accepted with the acknowledgment that you actually considered the challenges ahead.
It’s all about the risks and the mitigations – less about testing everything to the dot. One tool to read the landscape is to communicate curiously about what the stakeholders value more – and value less. And on the other hand, consider the nature of the solution being proposed.
Example 3: Setting up a cloud-based Azure Active Directory – the solution comes with a given security level out of the box (OOTB). As with other OOTB and Software-as-a-service solutions you have little impact on the security features of the solution, besides some simple configurations. While you might think that all security requirements would require 100% acceptance testing coverage, what you want to accept is that they might be provided “by design” – or by a solution decision made long ago.
I would prefer that we can call things what they are and not blindly apply old testing types
A “testing mindset” of inquiry applies when you are a principal tester or even a senior advisor in testing. Asking open questions and approaching a challenge with exploration proves better than command and control. People are people – be mindful of where they are.
Currently, I’m involved in an extensive change program for a company. It’s not your usual agile feature factory but more about the technology stack needed to run a 3000 people company (payment, finances, etc). While testing professionals could be onboarded – their learning curve would not scale – not even being equipped with all the heuristics and testing vocabulary in the world. As it is a one-off, automation wouldn’t scale either.
Fortunately, most applications have a system owner (or product owner, or manager in charge) and usually a team around maintaining the application. To some degree, we could frame these teams as stream-aligned teams. The experts in testing the applications are the superuser – hence they do the testing.
One such team is the XYZ team anchored by a VP-level manager. A colleague and I had been trying to communicate with them to align on the testing needed but had met reluctance. They would not have time to help us. We discussed the matter and realized that we needed to approach them differently. Since the change program was such a fundamental shift for the team, they had already considered the implications. They were already testing and thinking about the impact. We just had to trust them – and build on what they already had instead of imposing specific ways of working.
Without requiring them to earn it first, tell everyone who works for you that you #trust them implicitly. From that point forward, just ask them to never give you reason to withhold it. (In my experience, most won’t).
Communicating curiosity can be a little thing like “Remind me, how does XYZ work” or simply “sitting on your hands” during meetings that you would previously fill with questions and requests. The tables of the great program test lead are turning, it’s about being an enabling function for others to succeed. Similar to the freedom you should give your growing up kids – it’s about leadership.
… one piece of the time. Really. Don’t worry about it – it’s similar to eating an elephant.
While Discord and Slack sites can be places to go for communities with the faill of Twitter – they are closed systems. You need an invite and communication closed within the instance. This is good for closed discussions you want to gatekeep. Mastodon seems to be the place for serendipity and cross-collaboration across topics and communities. Itās not new, the framework started in 2016 gaining 1 million users in 2017. There are a number of great and extensive guides – one of them is fedi.tips. Another great guide is Mastodon Lessons Learned by Lisi Hocke (@lisihocke@mastodon.social).
For Mastodon, you create accounts on a server (or instance) in the Mastodon-based network. Through a shared protocol the people on the servers can communicate with posts, threads, likes, and resharing as known from other platforms. Mastodon is really the software framework (on GitHub) and the network called āFediverseā – but people mostly call it āon Mastodonā.
Someone wrote that picking a Mastodon Server feels like selecting a character in a new video game. You donāt really know what youāre looking into and how you will play the game. Luckily you are not fixed to one account on the Mastodon network. You can have multiple and easily redirect from the old to the newer. I have tried jumping profiles myself, as the first one I made was on a server with too few general topics. One of the elements of Mastodon is that each server has a timeline – besides the ability to communicate across.
Some servers are topic-based on Open-source software, others on art and LGBT, you can pick a specific server and expect the local content mostly on that topic. Or you can pick a general-purpose server – if one topic doesnāt define you. There are some tools, like debirdify that can help you in the process – but not yet a 1-click tool to move over.
People can and will move to many places – there is yet to be a primary test community instance.Ā
Profile and Settings
Your username is text-based, but you can go crazy with emojis in the profile name field
Create your profile with a picture, banner, and profile text
Profile meta-data and hashtags will help people find you and recognize you
Add 2-factor authentication and save the password and keys
If English is not your native language, there are other languages available (German is rather prominent). Selecting a language seems to give you more content in that language.
Enabling the āslow modeā setting is recommended. It forces you manually to reload the feeds and not have the dreaded firehose.
Make an #introduction post and write your favorite hashtags, that will help people find you.
Write your Mastodon username @user@server.name in your Twitter bio so that others can follow you
General Use
As there is no algorithm to drive engagement, so use hashtags and reshare content from others. Likes are only posted to the author – so reshare/retoot great posts. Similarly to following people you can follow hashtags like #MinistryOfTesting and #TwitterTestingPeople. The culture of Mastodon is to be inclusive. Use Content-Warnings and hide images as a common rule – and ALT-text on images as a general rule.
Mastodon has free apps (Android, iOS) for casual use and some pay-to-use apps. Currently, it seems you canāt follow hashtags in the iOS app, so I tend to use the browser more. There is a range of choices – see what bests fit you. You can get a long way with just the primary site.
Servers are run by small teams of admins, so in peak periods there might be suboptimal performance. Most servers seem to be funded by patreons or similar fundraising.
In theory, an instance could close down and take everything away. If you have content that is more permanent and you want to own, put it somewhere you control. The same challenge is true for ad-based social media (Facebook) and subscription-based platforms like Medium.
If you feel like organizing the content coming your way, enabling the advanced web GUI setting will give you columns like āTweetDeckā – but the setting is global for your login. Sometimes I just want a quick read over the timelines, other times I want to monitor specific topics. To monitor topics I have set up columns in Sengi https://sengi.nicolas-constant.com/. Itās a per-browser setting, so itās more of a topic radar that can be different across my devices. Personal lists are a feature too, but it seems based on people not so much on topics.
Do notice that direct messages arenāt private messages. All the people you tag can read the messages. So donāt tag the people you want to rant about.
The visibility of a toot/post follows the flowchart below.Ā
In my book “Goal-Aligned Test Strategies,” I draw extensively on the work of others. This is a reference post to further credit my reviewers and to list the 112 references in the book.
Special thank you to Lonnie for my writing evenings and to Simon Wardley for finding a path.
Reviewers and Feedback
Tristan Lombard, a guardian angel for the project since the start in 2021.
Praise from Simon Prior, Head of Quality Engineering, EasyJet.
Complexity is a Matter of Perspective 