Performance Testing using Neocortix LoadTest and JMeter

Disclaimer: This article is not sponsored, contains no product placement and the opinions are my own - as all my articles.

Service Introduction

Neocortix LoadTest is an infrastructure-as-a-service platform, using shared economy model - with real devices and real network. Users across the world leave their phones for charging for the night and you can utilize their processing power for various purposes, including load testing. Phone owners receive renumeration for usage of their hardware from the service owner.

How is this different from on-premise machines?

When using on-prem machines, you typically reserve a few big machines for load generation and a single master machine to aggregate the test data, create the reports and orchestrate the behavior of all the load generators. Architecture diagram looks like this:

Mobile phones have less computing power, this means you'll most likely need way more load generators if you want to achieve the same throughput as on a large server machine. This also means you'll have more JVMs so the static CPU and memory costs would also grow the more mobile machines you'll utilize in your test. To reduce the static cost per mobile, Neocortix picked a decentralized architecture, with no master node to manage the jmeter process and aggregate the logs on runtime - this is deferred until after the test is completed. This approach saves IO and CPU cycles on mobile devices and doesn't require having a big master node to collect the results on runtime. The test state is managed by a batchRunner with which you can stop the test at any time. Runtime test statistics are forwarded directly by each node to your backend of choice (in my example it's influx db)

What do I need to run my existing tests?

I ran a few test configurations using the tests i've prepared with the latest JMeter version, using standard plugins and they were all running successfully. Changes you'll most likely be required to make in your current scripts would be related to thread configuration and test data. Due to lower compute power per device, you won't be able to generate as many users per machine comparing to large servers - this will be compensated by increasing the number of machines. Since you'll be increasing the number of machines, you also need to make sure the test data required is evenly distributed between these hosts. One way to achieve that is to store your test data in a database, or a distributed cache like redis. Here you can find a good video with instructions how to set it up correctly with JMeter. If you don't have an option to use any of these tools, you have to split your data files to match the number of load generators you plan to use for your test case - this requirement may change often so you may consider automating the file split to save yourself some time for test maintenance.

Test Structure

The test I'm simulating represents a load created against a single application from different countries and it's important to me to know the response and connect times per region I'm generating. Structure of the test looks as follows:

Each country has its own dedicated worker directory and this part can be executed separately - this gives me possibility to split the configuration per country if anything changes in the application. Inside a worker directory I'm including a .jmx file, all the required plugins, user.properties file and the data sets required by each machine.

Plugin jars are automatically copied over to jmeter's lib directory, user.properties, jmx and data files land on the /bin directory. Each region can have unique properties, which gives you a lot flexibility and options to expand your reporting of the test. Properties for Europe look like this:

In the user.properties I include the test parameters to determine the test duration, number of threads and the ramp up- they're parametrized in the test definition. This approach allows you to re-use the same test scripts for all the test scenarios (following the DRY principle):

Properties like latitude and longitude (lat & lon) are artificial values I use for the report and it's not an accurate position of the device. Neocortix allows you also to use more accurate device position and it's in the default parameters, but in my case I only care about the general area and it's easy to aggregate the report if all of them share the same location. These properties are used in the InfluxDB backend listener, as tags. I found this feature by reviewing the JMeter soure code - it's not documented, at least not in the official documentation, but later in the article you'll see how this can be expanded:

neocortix.instanceNum is the sequential ID for each device in batch - this is helpful for many reasons, including unique data distribution. In the worker directory you've seen I have 5 files named users-{number}-{location}.csv. With injection of the instance number and location I can guarantee no other instance would access the same file during the test. This is useful if your application allows only one active session per user.

If you plan to use it, make sure you change the influxdb ip

As I don't actually have a running application to test and to allow you to use my work as an example, my test example consists of mock calls only - this way we won't be responsible for any DDOS attack. To simulate different response times from the device, calculate the connect time based on pseudo-distance from an imaginary server somewhere in the sunny California:

Test Execution

While you can use the UI to execute the tests, for a scenario requiring more complexity I find batchRunner to be more flexible and persistent. BatchRunner accepts a json file as an input for configuration and you can store your config for later use to assure the repeatability of your test. My configuration file looks like this:

{
	    "defaults": {
	        "filter": {
	            "dar": ">=70",
	            "storage": ">=2000000000"
	        },
	        "workerDir": "CommonWorker",
	        "jmxFile": "GlobalTestExample.jmx",
	        "planDuration": 600,
	

	        "rampStepDuration": 60,
	        "SLOResponseTimeMax": 2.5,
	        "SLODuration": 300
	    },
	    "batches": [
	        {
	            "regions": ["US"],
	            "nWorkers": 4,
	            "workerDir" : "USWorker"
	        },
	        {
	            "regions": ["Russia"],
	            "nWorkers": 2,
	            "workerDir" : "RussiaWorker"
	        },
	        {
	            "regions": ["Europe"],
	            "nWorkers": 2,
	            "workerDir" : "EuropeWorker"
	        }
	    ]
	}        

dar is an acronym for Device Availability Rating - this measures the % of the time the device for the past 48 hours, storage is the device storage given in bytes. List of batches represents a list of test cases you plan to execute concurrently - that's useful if you want to execute 2 separate jmx files at the same time and, like in my case, you want to execute the same script from different regions. WorkerDir represents a directory on my device, relative to the config file.

To copy my test configuration, you need to have an instance of influxdb accessible from external IPs, so make sure you use the latest version and use all the security recommendations to secure your infrastructure from potential attackers. Also update the influx IP in the properties file. I've created a grafana dashboard for this test you can import from here .

To start the test, you need to get the api token, register on Neocortix website and follow these steps , then, you need to clone the repository from my fork, cd to the jmeterTest directory and execute runMultibatchJMeter.py from jmeter directory. You can use the following commands:

git clone -b multibatch_test https://github.com/kbdering/ncscli.git
cd jmeter
../jmeter/runMultibatchJMeter.py --projDir globalTestWorker --outDataDir data/globalTestWorker        

Test Report

As the load architecture differs from the standard configuration, where only a single node is reporting the statistics, I had to prepare a custom grafana dashboard, covering the performance metrics aggregated from multiple devices. You can use the transaction and location dropdowns to select only the transactions you're interested in - this can be further expanded if you need to limit your report to certain devices:

An important aspect of this report, which is not something you'll likely seen on other implementations is the number of devices active. That's something worth tracking as you don't have any control over the availability of a reserved phone - they can be unplugged, loose the Internet connectivity or the user can just switch the application off. One of the great features that can be expanded further is the visibility of the load on the world map - this is supported by latitude and longitude.

After the test run finishes, the BatchRunner also generates a full html report for you, including a full JMeter report you're probably familiar with.

Who is the service for?

I can see Neocortix LoadTest a useful tool for e-commerce applications, especially with heavy focus on mobile device customers and mobile applications, especially targeting multiple geographical regions. The scale factor is also important here - if you only use 5 or 6 devices to generate your entire application load, your test results would vary a lot between the test runs, you need more devices from the same region to get the proper representation of the network conditions in a country. It's a great solution if you're looking for a sustainable infrastructure for your performance tests.

Things that are not there yet

One of the biggest risk for now is lack of disaster-recovery setup. If you lose a device, it doesn't recover and there is no option of to retrieve the data - this makes a InfluxDB setup mandatory. To minimize the risk you can request for the devices with availability rating over 99% - these are the devices plugged in all the time but there's no guarantee they'll be available for your soak tests.

I couldn't find any monitoring of the mobile hardware utilization, but this can also be expanded with some JMeter plugins and requires custom exports to influx if you want to see the live data. There can be some processes running in the background of the phones distrupting the results.

Security

That's a big topic, to test your application with LoadTest, you need to expose your test application to external access. Neocortix secures their VM from external access, and also doesn't let you access any data on the phone you're using, but the phone owner can still sniff the network traffic his/her phone generates. It's one of the reasons your application (as well as influxDB) have to use encrypted protocols - you'll be transferring your test user data there after all.

Go ahead, try it - the source code for the batch runner is open (I got to modify some of the functions and options myself, super-readable for pythonistas)