Load Testing using Soap UI

In this blog, I’ll describe how to load test the web services using the soapUI. But first let me brush up your knowledge about the different types of Load testing which executed on the AUT.

Load testing or Performance testing, many times both terms used by testers interchangeably. If I can say one line about the performance testing in layman’s terms, I’d say “Performance testing is actually simulating multiple users load on the AUT and see how the application fares under such loads.”

Here is some performance testing types which we came across generally.

?? Load Testing – This testing is done by putting the application under various users load to see the application behavior. The number of users may vary with different sets of input condition and is generally defined in the performance requirements.

?? Stress Testing - This testing is done by increasing the load at the point where application breaks. This is generally sued to find the threshold or breaking point of the application.

?? Endurance Testing - This testing is done by running the AUT for extended period of time under load to make sure application is sustainable.

?? Baseline Testing – This testing is done by running the AUT under 1, 5, 10, 20 users to see the behavior. This test actually create the base on which we ran our performance test. It is very important to understand that running baseline test actually helps you identify most of the performance issues .

?? Volume Testing – This testing is done by running the application with huge amount of data and verifies the application behavior.

Now let’s start the SoapUI(Free version) and see how we can do load testing of web service with that.

Adding the Load Test to your existing test case à Open the soapUI and select any existing testcase. then right-click on it. Select “New Load Test” from the options (or press “Crtl + N”).It will show a dialogue box asking you to enter the name of load test.?

Once you click “OK” it will add this test and you’ll see the following screen.

As you can see there are 4 steps added under “Test Step” because the test case which I choose has 4 steps. Also notice that except the first step rest steps has marked as disabled because I disabled those in the test case. If you want to run all or some steps in your load test , you can simply enable/disable them in the test case by right clicking and selecting the appropriate option.

Now let’s check out the different buttons/options present in the window.

The top left corner has first “Play” button icon which uses as to run the load test. Second icon is used to stop the current execution. The next 4 icons are related to Statistics graphs. Then the option icon is there. Then the icon used to convert this test into loadUI test case and the last one is help.

Limit – It will allow you to set the limit in Seconds or Total Runs or Runs per Thread. So you can choose how you test will run.

The next Box shows the % of current test execution.

Threads – It allows us to set the number of virtual users we want to run this test. The default value is 5.

Strategy – This is how we want to run the load test. It should be chosen based on the requirements. Let’s see the different strategies and their significance.

1.????? Simple - It is the default strategy to be used. It run the test case with the number of visual user?? defined in the “Threads” and you can also define the delay in between each run.? “Test Delay” field let us define the delay in milliseconds. “Random” defines the order of randomness you want in the delay. “0” means no randomness and “1” means all random. This strategy is good for running the baseline tests.

2.????? Burst - In this strategy, it runs all the defined threads at once for the time specified into “Burst Duration” and then wait for the time defined into “Burst Delay”. After that it again burst all threads again. It’s good to test the recoverability of the application.

3.????? Variance - In this mode, the test runs with specified threads for the duration specified into “Interval” in seconds and the thread count varies ups and downs (like ECG readings) with the order defined into the “Variance”. So if we set “6” threads and Interval is 120 seconds and the variance is “0.5” then the test will run for 120 seconds with threads ranging from 3 to 9. This strategy is good for stress test and also runs to see the baseline test once we have done the “Simple” Strategy for baseline.

4.????? Thread – This strategy is used to observe the AUT behavior with the virtual users increment by 1 starting from the value set at “Start Threads” and goes till “End Threads” within the time specified into Limit. It usually used to find the breaking point.

When the test is running, the statistics collected and displayed under following headers:-

Load Test Assertions: - Assertions measure the actual values against the defined values for the test step and log the number of assertions errors under “err”. We can add assertions to the test steps by right click on the step and select “Add Assertions”. We have to choose the assertion type while adding the assertion from the given list. Following are the assertion types:

1.????? Step TPS: This assertion is used to verify the Transaction per second of the test step. When you select this type, a model window opens for this assertion having the following options—

o?? Name – Define the assertion name.

o?? Minimum Requests – This feature lets you define the minimum numbers of requests that you want to execute before the assertion has to be applied. It’s useful when you start the load test and don’t want to assert some initial requests.

o?? Minimum TPS – The desired value of the TPS. It will log an assertion error if the actual value is less than this.

o?? Max Errors – This is the value by which test should be cancelled. “-1” means that there is no such number.

o?? Test Step – Its shows the entire test step defined under the test case and let you select one on which you have to apply assertion. It also has value “Any” which lets you generalize this assertion for all the steps and “Total” which will assert the total of all the steps against the set value.???????????????????????????????????????????????????????????????

2.????? Max Errors: This assertion is used to verify the total errors of the test step should not exceed the specified value. When you select this type, a model window opens for this assertion having the following options (skipping the common options as they mentioned above) —

o?? Max Absolute Errors – This is the absolute maximum numbers of errors allowed for the test step.

o?? Max Relative Errors – This can have value between 0 to 1 . If the value is “0.4” is set that means at most 40% of the run of the specified test step should be result in error.

3.????? Step Average: This assertion is used to verify the averages of the test step should not exceed the specified value. When you select this type, a model window opens for this assertion having the following options (skipping the common options as they mentioned above) —

o?? Max Average - This checks that if the test step average is greater than the defined value then it should log as assert error.

o?? Sample Interval – The step count used to calculate the average.

4.????? Step Maximum: This assertion is used to verify the time taken by test step should not exceed the specified value. When you select this type, a model window opens for this assertion having the following options (skipping the common options as they mentioned above) —

o?? Max Time – The time that test step should not exceed else it should log as an assert error.

5.????? Step Status: This assertion is used to verify that test step is not failed with an error. When you select this type, a model window opens for this assertion having the following options (skipping the common options as they mentioned above) —

?There are Log sub windows into this window in which you can see the logs. They are pretty obvious by the name itself like memory log etc., hence not dwelling into the details.

A real Time example :? Let me walk you through a real time example which I used in my testing. My application lets me add a new entity by the create request and I can update that entity via an update request. I have 3 update steps so I’ve disabled 2 update steps as they are nothing different in terms of functionality.

I am just doing the baseline test with 1, 5, 10, 20, 50 users for 1 minute duration with simple strategy and delay is 1 second (I have not randomized the test).

Step 1: Execute with 1 thread and here is the result.

So we can see that create took average 2.5 second and update took around 2 second. This looks OK with 1user load and with the AUT running on the VM (This is not the actual load test environment).

Step 2: Execute with 5 threads and here is the result.

So we can see that create took average 5 second and update also took around 5 second. This is almost double as compare to 1 user result.

Step 3: Execute with 10 threads and here is the result.

Now the average time to create has gone to around 11 second and for update its 10 seconds. Also notice that the max time for create and update is gone up to 18 and 14 seconds respectively. Now we can see that even for 10 seconds the time is very high as compare to 1 and 5 users.

Step 4: Execute with 20 threads and here is the result.

Here the average time to create is around 16 seconds but the average time to update is gone up to 40 seconds. This gives us the glimpse that our application update function is not performing well under even 20 users load as compare to create function.

At this point we should not run the test for 50 users and we can see the performance degradation with 20 users for update. We can identify this kind of issues earlier with baseline test as shown above that really helps developers to fix the major performance issue at this stage itself.