Optimize Applications with Performance Configuration Testing

Since 2022 I have been talking about Secrets of Performance Tuning Java on Kubernetes (Devoxx Belgium 2023); with a shorter recording available from JavaZone 2022.

One idea that came through that research and experimentation was initially called “A/B Performance Testing”. I’ve decided to write down the idea in more details and offer you a practical example so you can try yourself. Let me know your thoughts in the comments .

Introduction to PCT

In the rapidly evolving field of software development, ensuring optimal application performance under real-world conditions is paramount. Whether through code enhancements, runtime adjustments, or infrastructure optimizations, effectively leveraging resources is key to delivering a seamless user experience and maintaining service quality. As the complexity of systems increases and user expectations rise, developers are continually challenged to explore and adopt innovative strategies that ensure their applications not only meet but exceed these demanding performance standards.

A novel approach, which I refer to as Performance Configuration Testing (PCT), is inspired by the principles of A/B testing, commonly used in user experience research, but adapts it for performance optimization in live production environments. This approach expands upon the concept of Configuration Testing[1][2][3] in software engineering, providing a more dynamic and empirically driven method.

The Concept of Performance Configuration Testing

Performance Configuration Testing involves deploying the same application in a production environment using two or more configuration variants while the business application and its functionality remains exactly the same. These configurations may differ in various system parameters such as hardware, replica set, memory allocation, CPU allocation, or even the use of different runtime and its configurations — e.g. JVM (Java) parameters such as garbage collector and other tuning options. The key is that all versions run concurrently, under the same external conditions, thereby allowing for a direct and fair comparison of performance metrics.

Expansion Over Traditional Configuration Testing

Configuration Testing traditionally involves testing an application’s setup in a controlled environment to determine the optimal configurations that meet the specified performance criteria. PCT expands this by moving the testing into the production environment, which introduces real-world conditions that are often difficult to replicate in simulated environments. This shift enhances the accuracy and relevance of the testing outcomes, ensuring that the configurations are optimized not just in theory but in actual user conditions.

Setting Up a Performance Configuration Testing

To implement PCT, you need to first identify the performance metrics that matter most to your system’s operations — whether it’s response time, throughput, or error rate. Next, configurations are set up as variants. These could be based on theoretical performance improvements, past issues, or suggested configurations from vendor updates. Using a load balancer, incoming traffic is then evenly distributed between the two variants, ensuring each configuration receives a comparable and randomized workload.

Monitoring and Analyzing Results

Once the test is running, real-time monitoring tools are crucial. These tools track the defined metrics and gather extensive data on how each configuration performs under identical workloads. Over a predetermined period, which must be long enough to gather significant data but short enough to remain relevant, performance data are collected and analyzed. The analysis involves looking at averages, variances, and the distribution of metrics to understand which configuration performs better and under what conditions.

Making Informed Decisions

After the testing period, the data are evaluated to determine which configuration yields the best overall performance. This decision might not always be straightforward; trade-offs between different metrics (e.g., faster response time vs. higher error rate) must be carefully considered. The winning configuration is then adopted as the new standard for the application, while insights gained from the performance of the lesser configuration can inform future tuning and development efforts.

Advantages and Future Outlook

By testing in the actual production environment, PCT eliminates the guesswork and inaccuracies of simulated tests and seasonal production monitoring data analysis. Moreover, it allows for continuous improvement and adaptation to changing real-world conditions. Looking ahead, as systems grow in complexity, PCT could become an essential part of performance strategy, enabling businesses to stay efficient and competitive by making data-driven decisions about their technology stacks.

This approach not only streamlines the process of performance tuning but also turns it into a precise and systematic discipline. I anticipate a shift towards more proactive and adaptive performance management strategies in production environments as companies start applying this novel approach.

Practical Implementation of Performance Configuration Testing with Kubernetes and Nginx

This practical example demonstrates the implementation of Performance Configuration Testing (PCT) using Kubernetes and Nginx. By deploying a sample application — PetCart — in two distinct configurations with varied resource allocations, this setup enables live, production-level performance comparison under actual user loads.

The primary objective of this example is to illustrate how organizations can use Kubernetes to deploy the same application with different configurations and how Nginx can be used as a load balancer to evenly distribute traffic between these deployments.

The example involves:

1. Creating Kubernetes Deployments: Two versions of the PetCart application are deployed. One configuration utilizes higher resource limits (2000m CPU and 2Gi memory) with 2 replicas, while the other uses lower limits (1000m CPU and 1Gi memory) but with 4 replicas.
2. Setting up Kubernetes Services: Each deployment is exposed through its service, which helps in managing the network traffic to the deployments.
3. Configuring Nginx as a Load Balancer: Nginx is set up to distribute incoming traffic between the two services based on predefined weights. This setup mimics a real-world scenario where traffic distribution can impact application performance.
4. Integrating Azure Application Insights: Both deployments are configured to send telemetry data to Azure Application Insights with distinct roles, enabling detailed analysis and comparison of performance metrics in real time. Other solutions can certainly be used.

Alternatively, the same exercise can be done to compare language runtime configurations such as garbage collector, heap memory allocation, and so on, as long as the goal is to compare your application runtime tuning flags. In this case, it is important that resource allocation (CPU, Memory, Replicas) is the same for all deployed runtime configuration variants.

Let’s dive in.

Step 1: Kubernetes Deployments for PetCart

Create one YAML file for the first deployment — deployment-high-spec.yaml— with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: petcart-high-spec
spec:
  replicas: 2
  selector:
    matchLabels:
      app: petcart
      type: high-spec
  template:
    metadata:
      labels:
        app: petcart
        type: high-spec
    spec:
      containers:
      - name: petcart
        image: image.petcart.io:latest
        resources:
          limits:
            cpu: "2000m"
            memory: "2Gi"
        env:
        - name: APPLICATIONINSIGHTS_CONNECTION_STRING
          value: "InstrumentationKey=your-instrumentation-key;IngestionEndpoint=https://your-region.in.applicationinsights.azure.com/"
        - name: APPLICATIONINSIGHTS_ROLE_NAME
          value: "petcart-high-spec"        

Create one YAML file for the second deployment — deployment-low-spec.yaml— with the following content:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: petcart-low-spec
spec:
  replicas: 4
  selector:
    matchLabels:
      app: petcart
      type: low-spec
  template:
    metadata:
      labels:
        app: petcart
        type: low-spec
    spec:
      containers:
      - name: petcart
        image: image.petcart.io:latest
        resources:
          limits:
            cpu: "1000m"
            memory: "1Gi"
        env:
        - name: APPLICATIONINSIGHTS_CONNECTION_STRING
          value: "InstrumentationKey=your-instrumentation-key;IngestionEndpoint=https://your-region.in.applicationinsights.azure.com/"
        - name: APPLICATIONINSIGHTS_ROLE_NAME
          value: "petcart-low-spec"        

Step 2: Kubernetes Services for Each Deployment

Create Services — services.yaml:

apiVersion: v1
kind: Service
metadata:
  name: petcart-high-spec-service
spec:
  type: ClusterIP
  selector:
    app: petcart
    type: high-spec
  ports:
    - port: 80
      targetPort: 8080

---
apiVersion: v1
kind: Service
metadata:
  name: petcart-low-spec-service
spec:
  type: ClusterIP
  selector:
    app: petcart
    type: low-spec
  ports:
    - port: 80
      targetPort: 8080        

Step 3: Create the ConfigMap for Nginx Configuration

We’ll now create a Kubernetes ConfigMap that contains the Nginx configuration. This approach allows the Nginx pod to use this configuration dynamically.

Create a file nginx-configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configmap
data:
  nginx.conf: |
    user nginx;
    worker_processes auto;

    http {     
        upstream petcart {
            server petcart-high-spec-service:80;
            server petcart-low-spec-service:80;
        }

        server {
            listen 80;

            location / {
                proxy_pass https://petcart;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
            }
        }
    }        

Create a file nginx-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-load-balancer
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx-load-balancer
  template:
    metadata:
      labels:
        app: nginx-load-balancer
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: config-volume
          mountPath: /etc/nginx/nginx.conf
          subPath: nginx.conf
      volumes:
      - name: config-volume
        configMap:
          name: nginx-configmap
          items:
          - key: nginx.conf
            path: nginx.conf        

Create a file nginx-deployment.yaml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-load-balancer-service
spec:
  type: LoadBalancer
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: nginx-load-balancer        

Step 4: Deploying Everything

kubectl apply -f deployment-low-spec.yaml
kubectl apply -f deployment-high-spec.yaml
kubectl apply -f services.yaml
kubectl apply -f nginx-configmap.yaml
kubectl apply -f nginx-deployment.yaml
kubectl apply -f nginx-service.yaml        

Now expose the application through the Nginx service IP, and monitor real time data.

For my own test, the image below from Azure Application Insights shows a comparison of 4 configuration variants and how you would see the roles assigned to each variant.

Above, I’ve compared “X by Y” where X stands for replicas, and Y stands for CPU and memory where “1” equals to 1000m CPU limit and 1Gi RAM.

Conclusion

This practical example serves as a comprehensive guide to implementing a dual configuration testing approach in a live environment. The insights gained from this exercise are expected to empower developers, DevOps engineers, and system administrators with the knowledge and tools required to optimize application performance through empirical data and real-world testing. By embracing these methodologies, organizations can enhance their capability to deliver high-performing, resilient applications that meet the demands of their users.