Cloud Migration Success Depends On A Comprehensive Cloud Testing Framework

Cloud Migrations continue to represent a large portion of enterprise IT transformation spend and the trend shows no signs of slowing over the next 3 to 5 years. A recent survey report from Deloitte this past June highlights this growth trend. https://www2.deloitte.com/content/dam/Deloitte/us/Documents/consulting/us-future-of-cloud-survey-report.pdf

Increased adoption of cloud platforms and services is underway with upwards of 45% year-over-year growth.

The respondents of this study overwhelmingly view industry cloud (the set of cloud native applications, code, and services) as an enabler and catalyst for their technology and business roadmaps.

Cloud migrations are moving beyond lift and shift (infrastructure services) and are increasingly taking advantage of native cloud services to enable the cloud "as a force multiplier". From the same study:

Cloud can be a “force multiplier” and cornerstone of digital strategy. Overall, 90% of respondents agree or completely agree with the statement that cloud combined with other technologies, such as AI, IoT, and analytics serve as “force multipliers” for their digital strategy

Testing In The Cloud

Given this rapid rate of adoption of cloud native services within many cloud migrations and digital transformations, ensuring the quality of those systems is paramount to the success of the overall migration strategy.

During cloud migrations, many legacy applications are being re-written to take full advantage of these services to enable the cloud cost/scale advantages of architectures like serverless. However, many quality assurance and testing organizations are ill prepared for this shift. Cloud native frameworks, such as serverless, require new technologies and approaches above and beyond the traditional set of tools, testing frameworks, and experience found in most QA organizations. It is very difficult for QA teams to wrap their heads around all of these services, not only because there are hundreds of them, as is the case with Amazon Web Services (AWS), but because the integration patterns of each are quite complex (many of these services have direct connections to each other which eliminates the need for code as glue). While this direct integration is a huge win for developers, it makes life quite difficult for QA engineers because without the ability to do a code walk-through with a developer, getting a clear vision of how to test the workflows between these service component becomes daunting.

There are hundreds of architectural application patterns that can develop out of this service palette. Let's discuss one AWS pattern that is quite popular: refactoring a SaaS application to use Serverless Patterns: API-first development with event driven microservice architecture and Lambda (Serverless functions).

Testing teams are accustomed to approaching API testing with an open and documented specification like Open API Specifications (OAS), and/or with collaboration tools like Postman. When specifications for API's are ready, tests can be written and automated to interrogate the end-to-end workflows. This specification and associated documentation provides an easy way to figure out which endpoints to call and what data to pass in to them to complete testing a business process end to end. In fact, many QA teams are well trained in basic REST API testing and have the benefit of using tools like Postman to enable that. However, things are very different when using cloud native services. The old ways of API testing are not applicable end to end. There are now native cloud services integrated within the API pattern that become a black box unless you know how to look inside. The simple pattern of a REST API calling a microservice is no longer the norm. Instead, API's can be used to trigger events that get placed on an event bus that call lambda functions. These lambda functions can call other native cloud services, other lambda functions, or can direct connect to a SQL or NoSQL database and then trigger other integrated services that then put even more messages on the same (or different) event bus to orchestrate complementary workflows. These patterns of connectivity sometimes use code as glue (like Lambda), but sometimes do not because these services can be directly connected to each other using configuration settings and templates, which further obfuscates the logical or physical system workflows that the QA team needs to understand to build their testing approach. The QA team must have the know-how and the tools to be able to trigger these services as they get built and to feed test data through these services to verify the intended behavior of the system.

Some may argue that QA teams do not need to understand these workflows in detail in order to test. I disagree wholeheartedly. It is imperative that API based microservice workflows are tested by the QA team - the sooner the better in today's shift-left agile environment. It is counter-productive to have development teams build API and microservice componentry critical to the operation of a SaaS application and not have the testing team actually test those services in parallel to provide near real-time feedback of quality. Testing a platform as a black box at the end of a long development cycle is counter-productive to meeting your digital transformation goals.

A Cloud Testing Framework: 3 key focus areas

These new cloud native architectural approaches require a cloud-specific testing framework. There are many moving parts to a comprehensive cloud testing strategy and as previously mentioned there are hundreds of services, thereby making it impossible to cover everything in an article like this, but I will cover 3 critical areas here that will get you moving in the right direction.

1. Cloud native testing patterns need to be defined and training around these patterns and tools needs to be planned for well in advance of the development efforts. Cloud specific training for the QA team around the native services that are being used as part of the development process should be the first order of business for the QA team. This training does not need to be as in-depth as developer training. Instead the basics of the operational components of the native service needs to be in focus. The QA engineers need to understand the service well enough to be able to do things like trigger an API properly that has direct integration with another service, such as AWS EventBridge. They have to know enough to be able to operate these services in a test environment and drive data through these services to effectively exercise the workflow paths. Another good example of a connected service that you will encounter in AWS is Lambda, which are serverless functions. While the QA person does not need to be an expert in a Lambda supported language like Node.js, they should have the ability to build a basic Lambda function to invoke a downstream service to be able to test test these services in isolation as they get developed. This includes understanding the main configuration attributes of a Lambda function, which include things like how to define IAM security roles for each service you are calling downstream. You do not want to have to rely on your developers to support every test component the QA team needs to stand up, because developers have more than enough of their own work to do.
2. Cloud testing stubs and helper functions that are QA specific need to be prepared. Native service specific tools need to be created to facilitate an efficient testing framework. Without these tools, the QA team will not be able to achieve a high level of test coverage in parallel with development efforts. Once there is a clear understanding of how a native service works, tools can be stood up to operate testing triggers for these services. In this case of API testing on AWS, a tool like Postman can be used to drive some manual and automated test cases. You can setup Postman scripts to call api's and pass in test payloads that trigger downstream workflows and services. However, when testing API Gateway connected to EventBridge you will need additional tooling that enables you to isolate the service integration testing. For example, when using EventBridge, you should create 'qa' specific targets from the workflow rules that allow you to send detailed information that can be used for debugging problems with workflow verification. So, when a rule is triggered, create a CloudWatch log group that writes the processed payload to a QA log. This provides critical visibility into the data that is getting passed through the workflow that can then be used to identify defects quickly. You do not need these 'qa' specific targets in Production, they are for testing and debugging purposes only, so if you have a separate test environment such as another AWS account or you use another AWS region, you can leave this target intact. If you are using the testing environment for production, just remember to remove the target so as not to incur any extra AWS costs.
3. Cloud testing logging tools need to be implemented. In AWS, CloudWatch will satisfy 95% of the logging you will need for all testing activity. However, you will need to be sure that the service specific logging is turned on. For example, in API Gateway, you must enable the log levels in your api's stage editor settings; be sure to select full request and response logs. For even more granular logging needs, such as when you are doing performance testing, you may need to enable AWS X-Ray which gives you an end-to-end view of an entire request which you can analyze for latency times in your APIs as well as their backend services. Don't forget to turn off these higher levels of logging once you go to production to avoid incurring extra AWS costs.

While these 3 components of the framework will get you started, be aware that an enterprise cloud testing framework for a large digital transformation/cloud migration will have many more considerations and many more moving parts.

Summary

Building a thorough testing strategy for cloud migrations is not to be taken likely; an informed strategy will need to be developed and vetted with the architecture and technology teams. A lot of preparation will need to go into this strategy, including training QA engineers, selecting the right testing tools where possible and creating your own tools where necessary. In addition, testing approaches for each cloud native service in your applications' architecture must be documented and planned for.

Ensuring cloud quality is critically important to succeeding at your cloud migration and digital transformation efforts. Be sure you have the right people, processes and tools in place with your QA team before attempting to take this on. Don't be afraid to ask for help or hire a QA partner that has a cloud testing framework ready to go with test engineering experts that can execute it well.