Shadow / Production testing in production environment

There are multiples articles exists on What & Why around topic "TESTING IN PRODUCTION", whereas limited articles exists to cover the How part. In this article will cover details on How to perform the testing in production.

In this article, will cover the design, workflow & practices for application based on SOA principles, in case your application is monolithic then details implementation details will be covered in upcoming articles.

What is SOA?

A service-oriented architecture (SOA) is an architectural design pattern, in which application components provide services to other components via a communications protocol, typically over a network.

The principles of service-orientation are independent of any product, vendor or technology.

Most of the enterprise applications are build on top of in-house build framework, that helps in reducing the boilerplate code & to have centralized access on outbound & inbound connections.

In case your applications also build on top of some in-house build framework, then shadow / Production testing is not a big challenge to implement. Below picture demonstrate how production / shadow testing will be performed end to end.

You have to implement the following steps to get it working.

Step 1: Write a wrapper that can send request & receive response to & from another component, in addition to persisting the request & response in the memory (Redis)

Step 2: Write a wrapper to publish the Root request & response to Kakfa

Step 3: Write an application that will perform following operations

• Kakfa Consumer : Consume the messages send using step 2, and replay the same on the application
• Mocked Responses: In this environment, application will talk to Shadow Service, instead of talking to real service, and this shadow service will look into the memory to return the response.
• Payloads comparison: Once root response in available in new environment, then compare the original root response with the newly generated response
• Kafka Publisher: Publish the required details to the Kakfa, such that analysis can be done on real time.

The above mentioned steps are bare minimal, which are required to get the shadow / production testing to work. But there are few factors those needs to be considered before rolling out the changes to production, like controlling the traffic for which production / shadow testing is required, as this design consumes the memory to persist the request & response Payloads.