Instrumenting Load Tests with OpenTelemetry

The first and most pertinent question you might ask is: Why should you integrate telemetry with load testing? After all, you have reports from load testing tools and telemetry from the application itself. Surely, nothing slips through your analysis, right?

This assumption is only partially true. To understand this better, let's address more fundamental questions: What exactly is your load test, and what does it represent? The purpose of a load test is to generate a load against your application and infrastructure, thereby measuring the stability and efficiency of your system. This concept is well-understood among those experienced in load testing, as we often boast about throughput and latency values in stakeholder meetings. However, the data generated by load testing tools encompasses more than just latency measurements. If your test is properly defined, the latency values represent your end-user experience. This means that, in most cases, each virtual user generated in your test represents one of your actual customers – a fact that is often overlooked. This realization was the starting point for me to consider extracting telemetry from load tests. My load tests are the eyes of my customers, and I want to know exactly how they perceive my application.

Whenever I notice any anomalies in the test report, my immediate thought is: "How can I find this in my application?" End-to-end tracing answers this question within seconds.

Instrumenting load tests with OpenTelemetry provides significantly better visibility over the payload sent to the system. Instead of persisting the entire payload, you can add custom attributes to your spans, which greatly reduces root cause analysis (RCA) time.

Prerequisites

Before considering telemetry for your load test, you need an application to which you can propagate your load context. Presumably, you have an OpenTelemetry collector, exporter, and an analysis platform for checking your traces. For my proof of concept, I used Jaeger, and it worked exceptionally well. I'll assume you have these in place and can export your spans to a defined collector. Additionally, as I've included the JMeter logo in this article, you'll need a JMeter test to instrument. The version of JMeter is not critical, though newer versions are preferable.

You'll also need Maven installed to download the OpenTelemetry (OTel) dependencies. While it's possible to download them manually, I don't recommend it.

Installing dependencies

Save this POM file on anywhere on your drive and run the mvn command to download dependencies to your Jmeter lib directory (I"m assuming you have your JMETER_HOME defined in environment variables")

<project>
<groupId>com.kuba</groupId>
<artifactId>jmeter-otel</artifactId>
<modelVersion>4.0.0</modelVersion>
<version>0.0.1</version>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-bom</artifactId>
<version>1.32.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-metrics</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-logging</artifactId>
</dependency>
<dependency>
<!--  Not managed by opentelemetry-bom  -->
<groupId>io.opentelemetry.semconv</groupId>
<artifactId>opentelemetry-semconv</artifactId>
<version>1.23.1-alpha</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-extension-autoconfigure-spi</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-jaeger</artifactId>
</dependency>
</dependencies>
</project>        

mvn dependency:copy-dependencies -DoutputDirectory=$JMETER_HOME/lib        

This command should download all the dependencies to your jmeter lib directory and you can start instrumenting your test

OTEL Initialization

To start any instrumentation, you need to initialize your Opentelemetry context. The easiest way to do it is to use the auto configured opentelemetry SDK

Step 0:

Add environment variables required for instrumentation:

export OTEL_SERVICE_NAME=definitely-not-jmeter

export OTEL_TRACES_EXPORTER=jaeger

export OTEL_METRICS_EXPORTER=jaeger

and any other OTEL environment variables you find suitable for your configuration from here:

https://opentelemetry.io/docs/concepts/sdk-configuration/general-sdk-configuration/

Step 1:

Add a setUp Thread Group - make sure it runs only once

Add a JSR223 controller with this groovy script:

import io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdk;
import io.opentelemetry.sdk.trace.SdkTracerProvider;
import io.opentelemetry.api.GlobalOpenTelemetry;


try {
  AutoConfiguredOpenTelemetrySdk.initialize()
} catch (Exception e)
{
log.error("failed to initialize OpenTelemetry, error : " + e.getMessage());
    
}
def sdk = GlobalOpenTelemetry.get();

props.put("opentelemetry", sdk);
def tracer = sdk.getTracer("totally-not-jmeter", "0.0.1");

props.put("tracer", tracer);        

Note that it may log an error if you run the same test twice from GUI, but the Opentelemetry remains initialized so you can disregard this error when you re-run or debug your test

Step 2: Create a JSR223 preprocessor

This is the place where your span will start. Your current span will be stored in variables as "currentSpan" object and you can refer to it any time of duration of the span. The span name will be overwritten during the ending of the span.

import org.apache.jmeter.threads.JMeterVariables;
import io.opentelemetry.context.propagation.TextMapSetter;
import io.opentelemetry.context.Context;

def tracer = props.get("tracer");

def currentSpan = tracer.spanBuilder("http request").startSpan();
vars.putObject("currentSpan", currentSpan);

TextMapSetter<JMeterVariables> setter = new TextMapSetter<JMeterVariables>() {
    @Override
    public void set(JMeterVariables carrier, String key, String value){
            carrier.put(key, value);
    }
}

props.get("opentelemetry").getPropagators().getTextMapPropagator().inject(Context.current().with(currentSpan), vars, setter);        

Note that once you start your span, the propagator automatically injects the span context to JMeter variables - you can refer to them later in any protocol you want

Step 3: create a JSR223 postProcessor

This postprocessor ends your span and updates it's name. The span's name will be the JMeter sample that was invoked. I'm updating this name here because the sample name is dynamic in Jmeter and it can be overwritten with other preprocessors and variables. Once the span is ended, it will be sent to your opentelemetry collector.

import io.opentelemetry.context.Context;
import org.apache.jmeter.protocol.http.sampler.HTTPSampleResult;


def currentSpan = vars.getObject("currentSpan");
currentSpan.updateName(prev.getSampleLabel());

currentSpan.end();        

Step 4: Add HTTP Header Manager

Context propagation of HTTP calls happens via headers. The preprocessor injected your JMeter variables with a valid trace context, named ${traceparent}

This context will be then captured by your opentelemetry-instrumented application and you can trace your span from your load test context, down to your application stack. This is the core of the tracability magic.

Step 5: Run your test

No, seriously- that's it , once your test is instrumented, the Jmeter will start sending captured traces to your opentelemetry collector and you should see the whole e-2-e traces in your Otel application. As I don't have a working http server at hand, here's a screenshot of this implementation running with kafka protocol.

Instrumenting event-driven and message bus application is a bit harder though so I'll cover that in the next article.

Thank you for reading my articles, if you have any questions - feel free to reach out to me - I'm always happy to help!