Testing across the IT landscape

Leave a comment

Good testing is much more than confirmation of explicit requirements. It’s figuring out the implicit requirements, what blocks the business and what drives the business. Businesses are not driven by SDLC’s but by decisions and strategies. IT road maps are just a representation of the business strategy, an engine to build business solutions on. The is much more to the business than the software solutions and related risk mitigation.

Very often the biggest business risks are outside the project scope. When we look at it this way we see that testers and testing activities has an opportunity outside the classic project life cycle. Testing is about experimenting with a IT solution, to evaluate if it fits the business requests. IT solutions that supports the business exists in many forms, I am certain that explicit testing (*) can add value in other parts of the IT landscape.

Here is a model of an enterprise IT landscape consisting of business ideas, solution development, operations and end user devices + support. Solution delivery is boring in the sense of well-known software creation and maintenance. What if the item under test and the requirements are around network, servers, end-user devices and IT support tickets. I am certain that it’s valuable to test business cases before the project is even formally assembled.

 

*: implicit testing in the form of critical evaluation always happen. .. similarly does checking.

Testing for Create, Maintain, Move and Close of Applications

1 Comment

Usually when we talk testing it’s about the road to production. It’s about getting requests from the customer/Product owner and shipping it. We tend to forget that there is more to the life cycle of application than adding to the pile. Inspired by the old CRUD I identify the following stages in the application life. Create, Maintain, Move and Close. Testing can add value all of the four modes, with twists for each one.

Create: “oh shiny” – Creating an application is usually novel, but the more times you have build similar applications it becomes routine. Some applications have to be build from scratch, others merely configured. It matters a lot if you are building a unique app – or if it yet another roll-out of a COTS application. The testing in “create” usually focuses on bugs to be fixed before go-live, and very little on what happens afterwards. Building a new  application is usually a strategic decision to the business that solves a problem or builds on a potential. Requests are numerous for new things.

Maintain: “ship, but don’t destroy production” – At some point the customer sends you more requests to the stuff already build than new features. Application maintenance is all about balancing new features and updates to existing features. Existing features are being used by the end users, and they will eventually request updates and bug fixes. Fragmentation, merging and branching becomes and issue – especially if you maintain the application as a solution for a range of customers. Customers might want to differentiate between their requests – as they won’t want to pay for bugs in previous releases, but rather want to pay for new additions.

Move:  “It has to work as before, just with a new team“. To many businesses maintaining an application is not their key area; They might be a public organisation with no need to have their own staff of developers. So “Application Outsourcing” becomes a thing for many applications, and with deals being won and lost – it will happen that  the development tasks moves from one supplier to another. Testing can make sure that processes are in place in the new location and that the state of the application is known in the new location. The testing during “move” doesn’t involve the functionality of the application, but rather the ability of the new team. Sometimes the hosting of the application stays they same,  in other cases it is the hosting of the application that changes and nothing else.

Close: test that it’s gone” Sometimes IT strategies and businesses decide to close down an application. Perhaps it’s being replaced, perhaps it’s redundant after a long time. Examples could be hospitals moving from one patient journal to another or whole systems being sunsetted. It could also be the closing of end-of-life applications (Windows servers, HPQC etc). The value to the business is knowing the application is gone, and the information in the old systems not trusted anymore.

It is very much possible to have testing in all modes of the application life cycle. Similarly it is very much possible for testing to add value in all stages of the software development life cycle. It’s a matter of perspective.

Getting Testing in Early

4 Comments

Even before there is an “system development life cycle” – testing in the form of thought experiments and  evaluation can take place and add valuable information to the context.

My test management tasks are often about the next thing coming up. Bids for outsourcing agreements and application development often comes with a large document of test activities to be answered and elaborated. In this role I am the the subject matter expert (in test), and while have to write the tender reply for my domain. Sometime down the line the bid materials becomes an actual project, but by then I’m onto the next thing.

Sometimes I draw an analogy to the Secret Service advance team arriving two weeks before the president, setting up protection and identifying gaps – while then moving on to the next location before the president even gets there.

Another example of advance work for test people, is where the organisation uses frequent releases of systems. While the majority of the test effort is put into the release currently being tested, some effort must go into looking into the frame of the coming release. In the coming release the test manager can look for headlines to test, review initial high level design and find flaws and conflicts in the release content.

Sometimes I draw the analogy to the blue and gold teams of US nuclear submarines. While one full crew is out sailing/delivering, the shore team prepares, trains for the next big push.

Testing early can also be in the form of running simulations on various business case scenarios. Business simulations is all about experimenting and evaluating. For novel solutions prototyping, wire-framing and user experience activities helps develop minimum solutions to be tested for viability by the customer.

In the article “Continuous Testing in Dev Ops…” we see testing happening during Plan and Branch. In the article “A Context-Driven Approach to Delivering Business Value” testing can help establish viable market, match to vision and in identifying business risks.

testing related to revenue generation may focus on functionality or regulatory compliance; testing related to revenue protection may focus on maintainability or legal defense; testing related to supporting revenue may focus on business process improvement or cost reduction.

Testing is a lot of things – also outside the SDLC.

 

3 Sessions of Security Testing

1 Comment

One way to collaborate in a team is to achieve shared knowledge together. An example of this is the online activity of “30 days of testing” that The Ministry Of Testing has been putting out to the online community to participate it. My test team has a “Work Group / Special Interest Group” with regards to security testing, so when a 30 day challenge for security testing came up, we scheduled sessions to learn from the topics provided (see below).

As we are testing consultants doing work for our customers, we scheduled 3 sessions – initially for an hour. At the start of the hour we picked 4-5 topics from the list, and worked our way through them in a prioritized order – within the time box of the hour. Come to think of it we might as well have used the Lean Coffee format. As we have team members two places in DK and one place in PH, it was a skype call using screen sharing. After the call I  summarized sending out a “link mail” to all in the testing group (DK and PH). Evaluating the sessions we extend our ordinary scheduled WG meetings to make room for collaboratively investigate additional security testing topics.

12 From the list: ZAP, Google Gruyere, threat models, HTTP proxies, posture assessments, tiger boxes, recent hacks (elaborated by Troy Hunt), OWASP top 10, OWASP SQL injections, adding data integrity testing into a test plan, share ideas for security testing internally and externally, discuss security testing with regards to EU GDPR compliance.

7 Not on the listNaughty Strings form GitHub, Bug Magnet plugin, How real persons names trick IT systems, how to be careful with custom license plates, DDoS attacks, IoT privacy failures, Chaos monkeys/Siamese army and little Bobby Tables:

exploits_of_a_mom

XKCD: Exploits of a mom

To sum up, we have learned about: what tools that can make testing easier, where to read about vulnerabilities and and simple exploits, understand how personal data and logins are used and stored, how to pitch security testing based on fear of breaches and safety concerns, testing the requirements for “by design” security.

30 Days of Security Testing

30 Days of Security Testing

Shift-Deliver, TestOps and Management of Changes

4 Comments

Shift-Deliver is a label I choose to put on the changes the roles and activities of the TEST MANAGER, when the test manager moves towards (also) being involved in the ITIL change requests, delivery management, configuration management and branch management that happens when the solution goes from the test phase to production. Another label could be “TestOps” as presented here, as the intersection of Testing and Operations. TestOps have been identified for along time. ….Interesting.  🙂

In my IT outsourcing context, this is less about software, and more about solutions. In at least two of my long term enterprise scale projects, half the job was test management (of operations) projects, half the job was regarding ITIL change management. My change management activities was mostly making sure that

  • the process was followed
  • that information was provided to the stakeholders
  • that testing happened
  • risk mitigation happened

I was hired as “the quality guy”, but expanded the role over the time I’ve been on the team to take ownership of all of our build and release infrastructure as well. Basically, I’m responsible for everything from the moment code is checked in, until it hits our production servers 

To use a quote by Alan Page. Again Alan is a representative of what happens with regards to trends in testing. He might be wrong, as well as I. I try to label the trends to understand them. These four trends that I have spotted are not mutually exclusive, neither do they all four need directions. Change is happening to the classic test manager rolle of going through the motions of test cases and documents. This is clear when looking into these posts:

Initially I discussed Shift-Deliver, Shift-RightShift-Left and Shift-Coach  at Nordic Testing Days 2016 during the talk “How to Test in IT operations“ and coined the labels on the EuroStar Test Huddle forum.

DSC03562

Shift-Right, you wild one!

3 Comments

The Shift-Right label is that more and more testing (and checking) can happen on the live application in production. Some call it monitoring, some call it Testing in the Wild. It is a very wild idea for some people and some contexts #YMMV. It may very well be the best way of testing in some contexts.

Once I consulted on a network stabilization and delivery optimization project for a consumer bank. They had many issues in their production environment… I strongly advocated that they did test controlled and structured in production on the network changes and other operational activities. (I have talked about “How to Test in IT operations“ at Nordic Testing Days 2016). More on testing during IT deliveries in Shift-Deliver.

Shift-Right is trend that people have covered well before me, here are some pointers:

The key is really as Alan puts it “testers should try to learn more from the product in use” and with that comes the tools of Google Canary builds, NetFlix Chaos Monkeys etc.

kabuum

This trend goes along with Shift-Coach, Shift-Left and Shift-Deliver discussed separately. Initially I considered shift-right to be regarding consulting, but after hearing Declan O’Riordan at DSTB 2016 I realized that shift-right was the right label for test in production, testing in the wild etc.

Similar posts regarding things in the wild: Bugs HappensThe Kcal bugTradition is a choice and Can you see beyond the visible.

The Shift-Coach Testing Trend

8 Comments

Shift-Coach is when testers and test managers trends towards being coaches and facilitators of the testing activities. Shift-Coach is more about leading the testing than leading the testers to paraphrase from @DevToTest Joe DeMeyers blog post.

The ground breaker for this trend, is to me, the talk “How I Lost My Job As a Test Manager” presented at Test Bash 2015 by Stephen Janaway. Stephen explains how reorganization of the test manager role forced him to be more a facilitator than embedded in the teams. Similarly many other great test managers talk more and more about people skills and coaching, especially in agile projects. I want to define shift-coach around the facilitation testing activities, and place testers that doubles as scrum masters in the Shift-Deliver trend.

In traditional (v-model) projects testing has often included people that were not professional testers; – in user acceptance tests this has often been business subject matter experts. The testing was done by someone with the best knowledge of the topic, and that may not have been the professional tester. That more and more projects do this – more and more, is a big challenge for many testing folks. But it is a significant trend in testing world of 2016.

Shift-Coach trend is visible when Alan Page  talks at Test Bash Philly 2016:

You’ve heard the rumors, and you’ve seen it happen. An organization or development team decides they don’t need testers, and you have big questions and massive concerns. Is quality not important anymore? Are they irresponsible or idiotic? Are their hats on too tight? Do testers still have jobs?

Alan Page is a career tester who has not only gone through the “no-tester” transition, he’s taking it head on and embracing it. Alan will share experiences, stories, strategies, and tactics (and failures) on how he’s taken everything he’s learned in over twenty years of software testing, and used those skills to have an impact on software engineering teams at Microsoft. Whether you’re going through this transition yourself, think it may be coming, or just want to tell someone what an absurd idea this is, this is the talk for you.

This trend goes along with Shift-Right, Shift-Left and Shift-Deliver discussed separately. I discussed these trend labels at Nordic Testing Days 2016 during the talk “How to Test in IT operations“ and coined the labels on the EuroStar Test Huddle forum.

legocoach

Drive the Testing – Coach!

Older Entries