Lessons Learned in Cloud Testing

For reasons of regulations and data restrictions, we were recently doing a cloud project for a large customer of ours. Cloud resources had to be spun up, granted access to, utilized, and taken down (CRUD-like). But we had to do it without granting any of the customer’s employers access to the cloud controls. They couldn’t even have a cloud admin. So for various organizational reasons, we designed the solution below.

The user would order cloud activities in a “shop” and the requests sent first to the customer ticketing system, then to our ticketing system. And finally to the cloud pipelines and scripts. And there my friend is where the testing started.

Testing Input Sanity

The first script was for initializing the cloud services: Ressource Name. First of I found out the (cloud) developers had already tested the happy paths. I loaded Bugmagnet and gave some funny names, as the requirements for that field hadn’t really been that specific.

https://bugmagnet.org/

I quickly learned that the cloud provider was already sanitizing the input, so no Little Bobby Tables were allowed. It made the customer think, though, and she added a rule for the shop with regard to the Ressource names. This was coded as a regular expression (Start with a capital letter, then some letters, perhaps some numbers) – not that we or the cloud cared for that strict naming scheme, it was more of a business usage rule. Was that a bug? No mostly something that testing finds – not a bug.

I Found A Bug- Can I Go Home Now?

The next request could assign privileges to the cloud resources. Based on the resource name and Access Level, the cloud resources would be enabled. Among the access level options was “admin” rights. Which was directly opposite to the whole purpose of the project, and the customer risked breaching the regulatory inertia/controls.

And that was really the only formal bug we logged. And I don’t go home – I quickly reached out to the team, had the Accces level parameter removed and all was good. The intention was all along that only one contributor access level should be granted. That level was then hardcoded in the code repository.

And then we tested further around creating things already created, revoking resources that were not there, and race conditions for revoking the services. The only thing that really gave us a challenge was when a request failed (say you request something already there). Having the two ticking systems pick up the response and send the error message back to the users was more trick than expected as they were more REQUEST focussed – but we managed. Without creating a bug the team worked together and resolved the issues.

Key Learnings

As always this is a story retold from actual experience, there are details I can’t share and elements I preferred to have done differently – but due to the organizational culture had its own way.

• A mindset toward input sanity and access control is key to supporting business goals
• Bugs, issues, and observations will be found they are learnings for the team, not problems
• Hold end-2-end rehearsals of the solution and play out the scenarios
• Don’t restrict the testing activities to software development, look at the whole solution