The Importance of Distributed Performance Testing

What is distributed performance (load) testing strategy?

When an IT solution involves multiple disparate systems that operate independently on parts of an end-to-end workflow lifecycle, it introduces the need to track various scenarios and checkpoints along the lifecycle to detect if any of those checkpoints are causing delays, failing or producing errors. In this sense, distributed performance testing is often a custom solution to a unique problem because the systems involved will vary from one IT ecosystem to another.

For example, consider the lifecycle of a customer placing an order on a website. The systems involved may be a public-facing CMS, an ERP that keeps track of products and inventory, a logistics solution that orchestrates shipping, an internal sales support system that reflects orders to a customer support team, and finally some workflow or integration technology to drive messaging between all these systems. Now say there's a software release for the ERP, and suddenly all orders are failing, and you have no clue why. A distributed performance testing solution will predict that failure before the changes go live and ensure that the entire system behaves according to expectations—key criteria that must be maintained to keep the system stable. These are known as Service Level Agreements (SLAs). Basically, a primary goal of a distributed performance testing solution is to ensure those "agreements" are met.

Why should businesses care about this testing? Why is it critical?

It's critical for any business that has software facing the public. Any delay, failure, or disruption in a public-facing user interface will directly impact revenue because it will drive the user's next decision. With so many parts moving in the ether, that balance is delicate, and maintaining that balance requires insight and visibility and the ability to predict failures before they happen. Performance testing is how you achieve those things. As soon as you have multiple systems involved, your end-to-end lifecycle of a business process is complex enough to require more than just an automated performance test on one system. You need an elegant master solution that knows what's going on, when it's going on, and whether it finished at all points.

Can you briefly walk us through the steps your practice takes to execute a successful distributed performance test for a client?

First, we break down the architecture, looking at software, software versions, the integrations between software, what data lives in what software, and the network topology around the software. We then aim to understand the lifecycle checkpoints of crucial business processes. Once we have that, we can define what response time is expected at each checkpoint, at various load levels within/outside of peak times.

We ensure we know any external processes that may affect the systems or data within the lifecycle. Then, we look at what solutions already exist for individual system performance testing, what kind of traceability there is on the metrics we need insight into via logging or alerts in other systems, or results from other software executions. Then we attempt to sow the story together with the best possible solution for the client.

Some clients need a solution that involves custom development to gather data, while others prefer to pay for commercial software to meet those needs. In the end, the answer is not just one tool. It's always a solution with many parts, some of which may be commercial products, some not.

What are the advantages of distributed performance testing?

Advantages include the ability to predict problems before they happen, to remediate issues after they occur more quickly by knowing what failed and why. Distributed performance testing also assures that the system is stable—unless you're told it's not automatically—thus releasing a monitoring team of the duty of constantly watching the system with a microscope. Insight into opportunities for improvement (like speeding up one long-running process to improve the overall response time) is another benefit.

Where do you see the future of distributed performance testing headed?

I see it playing a significant role in e-commerce, as more customers are exposed to more complex features than they realize. I anticipate it becoming a large part of predicting failures in industries where one minor issue could have major, devastating effects, such as mining, manufacturing, or even the growing medical device industry.