Testing with More Cloud - for Less

There is so much cloud, but so few hours in each day.  By using more cloud, your load testing project can be finished in much less time, especially if it needs to validate the performance of multiple environment configurations.

This post is quite technical, but in summary it highlights new possibilities for shrinking  both the time required and the total cost of testing by testing in the cloud.  It is especially relevant to the tuning phase of a project, where the object is to maximise throughput and minimise response times.  It proposes that the old way of 'test - change - test - change - test - change....' is expensive and time consuming and could be replaced with 'deploy lots of instances and test them concurrently'.  The post provides a worked example where test licence costs are halved, test duration is reduced from 10 days to 2 days, total cost is reduced, and best of all, all of the test instances can be retained until the end of the test cycle, to allow deeper investigation into root cause of unexpected behaviour. 

The approach assumes that it is possible to deploy the entire application stack to the cloud, and that interfaces can be stubbed and configured to perform consistently (and independantly) across all instances.  It assumes that your cloud account is able to burst to the required number of instances.  It also allows for concurrent validation of different cloud venders and their offerings.  There is no reason why some of the tests executed could not be executed on multiple cloud platforms at the same time.

Based on recent Load Tests of applications deployed on AWS, it is now clear to me that executing multiple concurrent tests on multiple instances of an application can yield repeatable results and costs less than traditional ‘one change at a time’ test plans.  

My personal experience with ‘on premise’ virtualised environments or ‘private cloud’ offerings has highlighted significant differences in performance, based on slight differences in the underlying servers hosting the VMs that the applications under test were deployed on.  Some tests have showed a base response time variance of 10% in response times, with time of day anomalies generating even higher variance.  Such significant differences in performance of applications with the ‘same’ configuration make it very difficult to highlight the impact of planned differences in the configuration of the application stack.  

Enter ‘The Cloud’.  I have frequently verified that running the same test multiple times (using different instances of the same instance type) gives repeatable results with very little variance.  I have also confirmed that multiple tests can be executed at the same time, greatly reducing the time taken to run them all, while still delivering consistent results.  (I would still encourage any test plan to include a verification on this observation.  It does tend to hold for larger instances more than smaller ones.)

It is possible to take this approach with HP LoadRunner, but it means running multiple LoadRunner controllers and distributing the Licence between them all.  This is actually quite easy with VUDs (Virtual User Days) as they can be downloaded from HP in parts, but is easier with HP Performance Centre.
Lets say that you are using HP LoadRunner and have 10,000 VUDs and would like to run as many as 5 concurrent test runs to gather performance statistics on 9 different appliation configurations.

You would plan your tests by grouping a LoadRunner controller with an appropriate series of test runs.  To save on VUD licence usage, you should try to allocate test runs to controllers based on peak VUser usage, to minimise VUD consumption. You could then download the required VUDs to each of the LoadRunner controllers, which would take all of 10 minutes of effort.  If you have HP Performance Centre, then you just need to setup 5 controllers and allow ‘5 concurrent runs’ and it will look after the licences for you, but to conserve VUDs, you should still run some High User tests at the same time as Low Vuser tests, rather than running all of the High VUser tests at the same time.

An example ‘run sheet plan’ for using 5 controllers, to test 9 completely different configurations (9 distinct test instances each deployed on the cloud) for 4 different tests is shown below.  It would take 2 days, and consume 9,750 VUDs and require some overnight hours work for one night.  Sequential test execution would take 162 hours, which if each day included around 16 hours of test execution, would take 10 days to complete, and could consume nearly 20,000 VUDs, and would be dependant on timely configuration changes to the environment between each test run.

The biggest advantage of this Cloud Stacked load test approach is that it leaves all of the environments ‘in place’, so that they are not changed at any point in the test cycle.  This means that tests can be re-executed if they yield unexpected results to enable deeper investigation and analysis.

If a Design Of Experiments strategy is the key test driver to optimise a new technology stack (see https://www.dhirubhai.net/pulse/two-factors-solve-factor-problem-paul-mclean for more detailed discussion), then it is possible to configure 21 separate environments to screen 10 independent environment factors and run a series of tests concurrents on each environment.  

Without such an approach, it would be practically impossible to assess the performance implications of running the thousands of tests required to assess each factor with multiple levels (for example JVM heap settings for multiple JVMs, connection settings, provisioned IOPs, vertical/horizontal scale approaches….) especially where there is an interplay between some factors that can result in degraded performance or yield tremendous efficiency and resiliency.

Following the screening phase, many of the 21 instances could be terminated, and more detailed tuning and testing could continue with the most promising candidates.  

Each hour of Cloud usage is a fraction of the cost of the specialist resources required to administer and test the environment, so setting up concurrent stacked tests is a new way of "Testing with More Cloud - for Less".