A Comparative Analysis of Gatling and Locust Through the Performance Testing Lifecycle

Why I wrote this Article ?

Couple of weeks back during a discussion some one mentioned how their customer preferred Gatling for performance testing , this got me to check-out about Gatling. For me , the first load testing tool I ever knew about was LoadRunner , but locust is what introduced me to the world of load testing. While I am by no means a performance testing expert , my day job gave me a good opportunity to explore Locust and build a solution and eco-system around it to help simplify and leverage the best out of locust for our performance test engineers.

I knew the existence of Gatling through a few people but never fully explored it. The last time it came up was when I was learning and customizing Karate test automation framework. I came to know that Karate provided some support to runs its script as Gatling tests , but did not fully explore it as by that time our locust eco-system was the de-facto preferred system. Now when my recent conversation brought up Gatling , I thought I needed to check it out and know if it got anything better to offer or if I could be inspired by any of its features.

Now while I did that combing through various links/articles(I have added a few of the references towards the last ) , it struck me that this would be a good article and an opportunity to flex my muscles with Gen AI for the comparison and content creation. Hope it is useful for you. Please do let me know your thoughts on this article as well as the performance testing tools you prefer and why.

Introduction

Performance testing is a crucial aspect of software quality assurance, as it ensures that the software can handle the expected load (in terms of users and transactions) and deliver the desired performance( response and the speed of response to an action/transaction). Performance testing tools are software solutions that enable testers to create, execute, and analyze performance tests. There are many performance testing tools available in the market, each with its own features, advantages, and disadvantages.

In the realm of performance testing, this article aims to contrast the capabilities of Gatling with those of Locust. We will begin by providing an overview of performance testing including its different phases and the key factors to consider when choosing a performance testing tool. We will then dive into a detailed comparison of Gatling and Locust, highlighting their strengths and weaknesses in each phase of performance testing.

By the end of this article, you will have a comprehensive understanding of performance testing and be able to choose the right tool for your needs

Gatling: Gatling is a open-source Scala based load testing tool that provides option to script using Scala/Kotlin/Java. It uses an event-driven, simulations-based approach to create and run performance tests. It is known for its high performance and detailed reporting capabilities. Locust: It is another open-source load testing tool that allows testers to write test scenarios in Python. It follows an imperative, code-based approach to define and execute performance tests. It supports distributed testing and real-time monitoring of performance metrics.

What is a performance test ?

Typically an end-user identifies performance in relation with how the application responds to an action or page/data/resource(image/video) load. This slowness can be attributed to multiple reasons like

• Server Overload: When too many requests overwhelm the server, causing a bottleneck in processing.
• Inefficient Code: Poorly optimized or complex code that takes longer to execute than necessary.
• Network Latency: Delays in data transmission over the network, affecting the speed of data exchange.
• Insufficient Resources: Lack of adequate CPU, memory, or storage to handle the application’s demands.
• High Traffic Volume: A large number of users accessing the application simultaneously can slow down performance.
• Slow Database Queries: Inefficient database queries that take a long time to fetch data.
• Unoptimized Content Delivery: Large files or unoptimized media that take longer to load, slowing down the application.

The intent of a performance test is to identify and resolve these factors. To help identify this , different tests are undertaken as part of performance tests.

• Load Testing: Evaluates the system’s behavior under an expected workload(a predefined volume of users and traffic/transactions).
• Stress Testing: Determines how the system behaves under an ever increasing amount of load and identifies its breaking point.
• Volume Testing: Focuses on the system’s ability to handle a large volume of data.
• Soak Testing (Endurance Testing): Checks the system’s performance over an extended period for a steady rate of traffic to identify potential issues like memory leaks.
• Spike Testing: Assesses the system’s reaction to sudden increases in load.
• Capacity Testing: Measures the maximum user load the system can handle.
• Scalability Testing: Tests the system’s ability to scale up or down with changes in load.
• Reliability Testing: Ensures the software consistently performs as expected under specific conditions.

From these different performance tests, one can expect to gain a variety of metrics that provide insights into the application’s behavior under test conditions. Some of the key metrics include:

• CPU Utilization: Measures the percentage of CPU capacity utilized while processing requests.
• Memory Utilization: Tracks the usage of the system’s primary memory during workload processing.
• Response Times: Records the total time taken from sending a request to receiving a response.
• Average Load Time: Calculates the time it takes for a webpage to fully load and become visible to the user.
• Throughput: Indicates the number of transactions processed by the application within a certain time frame.
• Concurrent Users: The number of users accessing the application simultaneously.
• Error Rate: The frequency of errors encountered during the test.
• Network Latency: The delay in data transmission across the network.

These metrics help in identifying areas of improvement and ensuring that the application meets performance expectations. They are typically obtained through a combination of sources, including:

• Application Logs: Detailed records of events that have occurred within the application.
• Monitoring Tools: Software that tracks and records various performance metrics in real-time.
• Load Testing Tools: Tools like Gatling and Locust that simulate user traffic and capture performance data.
• Profiling Tools: Utilities that help identify resource usage and bottlenecks within the application code.
• Network Analysis Tools: Tools that analyze network traffic and latency between the client and server.
• Browser Developer Tools: Built-in browser tools that provide insights into webpage loading and rendering times.

By aggregating data from these sources, a comprehensive view of the application’s performance can be constructed, enabling teams to make informed decisions about optimizations and improvements.

Choosing a performance Test Tool

When selecting a performance test tool, it’s important to consider various factors to ensure it aligns with your project’s requirements. Here are some key factors to consider:

1. Project Requirements: Understand the specific needs of your project. Consider the scope of the testing, and the issues you intend to solve with the tool.
2. Testing Objectives: Define what you want to achieve with performance testing. This could include evaluating response time, measuring throughput, or resource utilization. Both Gatling and Locust are capable of evaluating response time, throughput, and resource utilization, but Gatling offers detailed reports out of the box which can be beneficial for in-depth analysis, where as for Locust we would need to extract it.
3. Supported Protocols: Ensure the tool supports the protocols used by your application. Gatling supports a wider range of protocols out of the box, including HTTP, Websockets, Server-sent events, JMS, MQTT, AMQP, and ZeroMQ. Locust primarily supports HTTP/HTTPS protocols out of the box. However, it can be extended to test almost any system by wrapping the protocol library and triggering a request event after each call has completed. This allows Locust to be used for a variety of protocols as long as the libraries you use can be monkey-patched by gevent. Protocols such as WebSocket, MQTT, and gRPC are supported through custom Python functions or libraries.
4. Realistic Load Simulation : The tool should be able to simulate real-world load scenarios that your application will face. Gatling provides an event-driven, simulations-based approach with a DSL for writing simulations, allowing precise control over the test flow. Locust allows defining the test flow directly in Python code, which can be straightforward for Python developers. In many a case the load simulation requires a distributed environment . While both Locust and Gatling support this. Gatling supports distributed testing with it's enterprise edition, while it is also feasible with the open source edition , to achieve it is a little complicated and required additional setup. With Locust this is easily achievable out of the box.
5. Reporting and Analysis: Look for tools that provide comprehensive reporting and analysis capabilities to help you understand performance bottlenecks. Gatling offers comprehensive real-time monitoring and reporting features, including detailed graphical representations with Gatling Enterprise. Locust provides basic monitoring and reporting, which may not be as detailed as Gatling’s but we can extract and build comprehensive reports from it.
6. Integrations: The tool should integrate well with other tools and systems in your SDLC. Gatling’s modular and extensible architecture allows for creating custom components and plugins. Locust’s is also modular and extensible. Additionally it is customizable with Python and integrates well with other tools and systems. (https://github.com/SvenskaSpel/locust-plugins is one among the many widely used locust functionality enhancer.)
7. Cost and Licensing: Consider the total cost of ownership, including licensing fees and the resources required to operate the tool effectively. Both Gatling and Locust are open-source; Gatling also offers a commercial version which provides the most of its premium features. Locust is licensed under the MIT license, and Gatling under the Apache License v2.01.Gatling Enterprise is Gatling's commercial version. While Gatling is the open-source load testing tool for web applications , Gatling Enterprise is a management platform for your Gatling load tests. It includes advanced features for reporting (live reporting, TCP connections metrics, bandwidth usage, injectors' monitoring) and automation (public APIs, new Continuous Integration plugins, clustering mode).
8. User-Friendliness: A tool that is easy to use can significantly reduce the learning curve and improve efficiency. Gatling offers a user-friendly and intuitive Domain-Specific Language (DSL) for creating test scenarios, which can be written in a concise and readable manner, but it requires a learning period especially those who are new to Performance testing .To counter this Gatling provides a recorder feature with its open source version and a no-code quickstart feature with its Enterprise version. Locust defines user behavior and test scenarios using straightforward Python code, which is simpler than learning a new domain-specific language. It also offers a web interface that helps control the test execution and provides real-time monitoring of load test results, helping one quickly identify performance issues. When we take into consideration the ease of setting up and configuring the tool , one would find locust easy to install with pip and configurations straight forward due to its python nature compared to Gatling that requires a JVM setup and it's detailed documentation to aid in its configuration.

Performance Test Phases

A typical performance test follows these phases:

1. Risk Assessment: Evaluating the components for performance testing based on risk scores.
2. Requirement Gathering & Analysis: Understanding client expectations and concluding requirements.
3. Performance Test Planning: Outlining the approach, resources, and timeline for the testing process.
4. Performance Test Design (Scripting): Creating the tests that will simulate the load on the system.
5. Workload Modelling: Defining the types of load and user behavior patterns to be simulated.
6. Performance Test Execution & Result Analysis: Running the tests and analyzing the results to identify bottlenecks.
7. Reporting and Recommendation: Documenting the findings and suggesting improvements or changes.

These phases ensure a structured approach to identifying and addressing performance issues within a system. While the role of locust and Grafana holds prominence only from the execution phase , we will still outline the other phases to help grasp their importance.

Phase 1 : Risk Assessment

Risk Assessment in performance testing is a process used to determine the need for performance testing of each component within a software system. It involves evaluating the potential risks associated with the system’s performance, such as response times, reliability, and scalability. The goal is to prioritize the testing scope and scenarios based on the criticality of the components and their impact on the business in terms of user experience, revenue, and potential application outages. It requires:

• Determining Component Criticality: Assessing each component’s importance and potential impact on overall system performance.
• Calculating Risk Scores: Assigning scores to components based on their criticality, which helps in prioritizing testing efforts.
• Identifying High-Risk Areas: Focusing on parts of the system that, if failed, could significantly affect user experience and business revenue.
• Scope Definition: Using risk assessment to define the scope of performance testing, including in-scope and out-of-scope components.
• Impact Analysis: Understanding the impact of new features or changes on existing functionality from a performance perspective.

Phase 2 : Requirement Gathering and Analysis

Requirement Gathering & Analysis is a crucial phase in the Performance Testing Life Cycle where the performance testing team collects and analyzes the application’s performance requirements. This involves working closely with stakeholders such as product owners, business analysts, and developers to understand the expected user load, usage patterns, and performance goals. Here’s an expanded explanation:

• Identifying Performance Goals: It involves defining specific performance targets such as response times, throughput, and resource utilization.
• Understanding User Load: Analyzing expected user traffic patterns to ensure the system can handle peak loads.
• Collaborating with Stakeholders: Engaging with product owners, business analysts, and developers to align on performance objectives.
• Documenting Non-Functional Requirements: Recording requirements that describe the system’s operational characteristics and constraints.
• Analyzing Business Processes: Evaluating critical business functions that will influence performance testing scenarios.
• Establishing Benchmarks: Setting clear benchmarks for acceptable performance levels based on industry standards or business needs.
• Prioritizing Test Scenarios: Prioritizing which aspects of the system to test based on risk and importance.

Phase 3 : Performance Test Planning

Performance Test Planning is a strategic phase in the Performance Testing Life Cycle that outlines a roadmap for conducting successful performance tests. This ensures:

• Defining Objectives: Establishing clear goals and objectives for what the performance testing should achieve.
• Scope Identification: Determining the extent of the testing, including which systems and components will be evaluated.
• Strategy Development: Crafting a detailed approach for how the testing will be conducted, including the selection of tools and methodologies.
• Resource Allocation: Allocating the necessary resources, such as hardware, software, and personnel, required to carry out the tests.
• Scheduling: Setting a timeline for the execution of performance tests, ensuring alignment with project deadlines.
• Test Data Management: Planning for the creation, management, and maintenance of test data used during performance testing
• Risk Management: Identifying potential risks and issues that may arise during testing and documenting mitigation strategies.
• Documentation: Creating a performance test plan document that records all the details of the planning phase for reference and accountability.
• Stakeholder Communication: Ensuring clear communication with stakeholders about the plan, obtaining approvals, and setting expectations.

Phase 4 : Performance Test Design (Scripting)

Performance Test Design (Scripting) is an intricate phase in performance testing where the test scenarios are meticulously crafted to emulate real-world user interactions with the system. This involves:

• Understanding User Behavior: It begins with analyzing how users interact with the application to create realistic usage patterns.
• Scripting Test Scenarios: Testers write scripts that simulate a variety of user actions and transactions to stress the system.
• Incorporating Business Processes: Critical business functions are identified and included to ensure the test covers all essential aspects of the application.
• Setting Performance Benchmarks: The scripts are designed to measure the system’s performance against specific criteria and thresholds.
• Data Preparation: Test data that reflects real-world scenarios is prepared and used within the scripts for authenticity.
• Environment Configuration: The test environment is configured to closely resemble the production setting for accurate results.
• Test Execution Strategy: A strategy is developed for executing the scripts, including the order and timing of each test.
• Verification and Validation: Scripts include checkpoints to verify that the system responds correctly under various conditions.
• Iterative Refinement: The scripts are continuously refined based on feedback and results to enhance their effectiveness.

Apart from the language difference of Locust and Gatling , the very nature of the language provides considerable benefits. Locust uses an imperative, code-based approach, where users define the test flow directly in Python code and execute them using the Locust framework. This approach allows for more flexibility and simplicity in defining the test flow and the user behavior, as well as easier customization and extensibility. Gatling uses an event-driven, simulations-based approach, where users define scenarios using the DSL (Domain-Specific Language - which can be more expressive for complex user behaviors) and execute them using the Gatling engine. This approach allows for more precise control over the test flow and the user behavior, as well as better performance and scalability. Gatling can be advantageous for teams familiar with the JVM ecosystem. It offers a powerful DSL (Domain-Specific Language) for writing test scenarios, which can be more expressive for complex user behaviors. One clear advantage Gatling has over locust is its Recorder Feature that is available with the open-source edition and No code quickstart that is available with Gatling Enterprise. These features help ease the test design phase. Both Gatling and Locust provides a modular and extensible architecture, . They allow users to organize and structure their test components into different files and folders, as well as reuse and import them as needed. Gatling and Locust both provide high maintainability, as they allows users to update and modify their test components or scripts easily and efficiently. They also provides good documentation and support, as well as a large and active community that contributes to its development and improvement.

A Simple Gatling Script Sample:

package computerdatabase;

import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;

import io.gatling.javaapi.core.*;
import io.gatling.javaapi.http.*;

public class ComputerDatabaseSimulation extends Simulation {

  HttpProtocolBuilder httpProtocol =
    http.baseUrl("https://computer-database.gatling.io")
      .acceptHeader("application/json")
      .contentTypeHeader("application/json");

  ScenarioBuilder myFirstScenario = scenario("My First Scenario")
    .exec(http("Request 1")
      .get("/computers/"));

  {
    setUp(
      myFirstScenario.injectOpen(constantUsersPerSec(2).during(60))
    ).protocols(httpProtocol);
  }
}        

A Simple Locust Script Sample:

from locust import HttpUser, between, task


class WebsiteUser(HttpUser):
    wait_time = between(5, 15)
    
    def on_start(self):
        self.client.post("/login", {
            "username": "test_user",
            "password": ""
        })
    
    @task
    def index(self):
        self.client.get("/")
        self.client.get("/static/assets.js")
        
    @task
    def about(self):
        self.client.get("/about/")        

A Locust Script Sample with HTML Parsing:

# This locust test script example will simulate a user 
# browsing the Locust documentation on https://docs.locust.io

import random
from locust import HttpUser, between, task
from pyquery import PyQuery


class AwesomeUser(HttpUser):
    host = "https://docs.locust.io/en/latest/"
    
    # we assume someone who is browsing the Locust docs, 
    # generally has a quite long waiting time (between 
    # 10 and 600 seconds), since there's a bunch of text 
    # on each page
    wait_time = between(10, 600)
    
    def on_start(self):
        # start by waiting so that the simulated users 
        # won't all arrive at the same time
        self.wait()
        # assume all users arrive at the index page
        self.index_page()
        self.urls_on_current_page = self.toc_urls
    
    @task(10)
    def index_page(self):
        r = self.client.get("")
        pq = PyQuery(r.content)
        link_elements = pq(".toctree-wrapper a.internal")
        self.toc_urls = [
            l.attrib["href"] for l in link_elements
        ]
    
    @task(50)
    def load_page(self):
        url = random.choice(self.toc_urls)
        r = self.client.get(url)
        pq = PyQuery(r.content)
        link_elements = pq("a.internal")
        self.urls_on_current_page = [
            l.attrib["href"] for l in link_elements
        ]
    
    @task(30)
    def load_sub_page(self):
        url = random.choice(self.urls_on_current_page)
        r = self.client.get(url)        

Phase 5 : Workload Modelling

Workload Modelling is a fundamental aspect of performance testing that involves creating a test scenario to simulate real-world usage of an application. It requires:

• Defining User Behavior: Workload Modelling starts with understanding how real users interact with the application, including peak times, average use, and specific actions taken.
• Simulating Real Conditions: The model aims to replicate actual production conditions or to recreate issues detected in the live environment for troubleshooting purposes.
• Incorporating Non-Functional Requirements: It uses the gathered non-functional requirements to shape the workload, ensuring the model aligns with expected performance outcomes.
• Creating Predictable Scenarios: The model helps in designing scenarios that are predictable for stress tests, ensuring consistent replication of conditions.
• Ensuring Repeatability: It allows for repeatable testing, such as regression load tests, to verify that performance remains stable over time.
• Facilitating Scalability: Workload Modelling assists in creating scalable scenarios to test how well the application performs when the load increases or decreases.
• Supporting Sustainability: It ensures that scenarios are sustainable, as seen in endurance tests, to check long-term performance stability.
• Guiding Resource Allocation: By predicting the load, it helps in planning the necessary resources to handle the expected user traffic.
• Aiding in Performance Optimization: The insights gained from Workload Modelling guide developers in optimizing the application for better performance.
• Providing a Basis for Analysis: Finally, it serves as the foundation for analyzing the performance data collected during testing, leading to informed decisions about improvements.

While both Gatling and Locust can simulate user behavior and generate traffic to test the application. Gatling is considered suitable for creating complex user behavior simulations. Apart from the code simulations , Locust's GUI enables user to control the traffic , this feature is particularly useful to anyone new to performance tests especially during the design and workload modelling.

Phase 6 : Performance Test Execution & Result Analysis

Performance Test Execution and Result Analysis is a pivotal phase in the performance testing lifecycle where the designed tests are run, and the resulting data is scrutinized to assess the system’s behavior under simulated conditions. This stage is essential for:

• Executing the Designed Tests: Running the tests as per the scenarios outlined in the test plan to generate performance data.
• Monitoring System Behavior: Observing the system’s response in real-time to identify any immediate issues.
• Gathering Performance Metrics: Collecting data on response times, throughput, error rates, and resource utilization.
• Analyzing Test Results: Using the collected data to evaluate the system’s performance against predefined benchmarks and requirements.
• Identifying Bottlenecks: Pinpointing areas where performance lags, such as slow database queries or inadequate server resources.
• Assessing System Stability: Determining if the system remains stable under varying loads or if it crashes under peak conditions.
• Evaluating Scalability: Checking if the system can handle increased load by adding more resources without a drop in performance.

During test execution, Locust can generate and handle large and realistic user loads with minimal resource consumption. Locust can also distribute and coordinate the test execution across multiple machines or nodes using the Locust master-slave architecture, which is an open source feature of Locust that allows users to run multiple Locust instances in parallel. Gatling Enterprise provides distributed setup as a premium feature, though the same can also be achieved in the open source solution through a little complex steps. Gatling's test execution is known for generating detailed reports that include various performance metrics like response times, throughput, and error rates. It provides real-time reporting and interactive charts during the test execution. It also allows for extensive data analysis post-execution, with the ability to parse responses and verify request outcomes using assertions. Gatling’s reports can be generated in HTML or JSON formats, and can be integrated with other tools, such as Grafana or InfluxDB, for further analysis and monitoring. For result analysis, Locust provides real-time statistics and downloadable HTML graphs through its web interface, and it allows exporting the test results to CSV or JSON files for further analysis using third-party tools. Locust with its API's can also be extended and integrated with other tools like Grafana or InfluxDB. Gatling does not have a built-in API for invoking and controlling test execution, but it provides a way to do so by using its Java APIs.?For example, you can use the Gatling Runner class to programmatically run a simulation from a Java application.?You can also use the Gatling Maven plugin to run a simulation from the command line or a CI/CD tool. Locust has a built-in API for invoking and controlling test execution, which is exposed as a RESTful web service.?For example, you can use the /swarm endpoint to start a test with a given number of users and hatch rate.?You can also use the /stop endpoint to stop a running test.?You can also use the /stats/requests endpoint to get the current statistics of the test.

Below screenshots give a quick snapshot of Gatling and Locust test statistics.

Phase 7 : Reporting and Recommendation

The Reporting and Recommendation phase in performance testing is a critical juncture where the gathered data is analyzed, and a comprehensive report is generated. This report encapsulates the overall test results, observations, findings, and crucially, recommendations for performance enhancements. It serves as a decisive document for stakeholders, providing the necessary insights to make an informed GO/NO-GO decision on the application’s launch.

The report typically includes a detailed description of the performance test outcomes, a clear indication of the GO/NO-GO status, and a rationale for the decision. It also assesses whether the application meets the predefined Non-Functional Requirements (NFRs) and documents the status of identified defects.

Moreover, the report highlights any performance risks, attaches relevant artifacts, and offers actionable recommendations to address any issues.

The choice between Locust and Gatling during this phase hinges on the specific requirements of the test, the protocols involved, and the team’s proficiency with Python or Scala/Java. Gatling’s detailed graphical reports aid in a clearer understanding of performance bottlenecks for the stakeholders out of box, while Locust’s customizability allows for extensive tailoring and integrations.

Conclusion

In conclusion, Gatling and Locust are two popular open source performance testing tools that have their own features, advantages, and disadvantages. Gatling is a Scala-based tool that uses an event-driven, simulations-based approach to create and run performance tests. Locust is a Python-based tool that uses an imperative, code-based approach to define and execute performance tests. They have a few differences but have one common objective to help you with your performance test.

Based on our comparison, we can recommend the following:

• If you are looking for a tool that provides more precise control, better performance, and comprehensive reporting, you should choose Gatling.
• If you are looking for a tool that provides more flexibility, simplicity, and customization, you should choose Locust.
• If you are looking for a tool that supports Scala, Java, Kotlin or the DSL, you should choose Gatling.
• If you are looking for a tool that supports Python or other custom protocols or functions, you should choose Locust.
• If you are looking for a tool that has a dedicated support team and enterprise support services, you should choose Gatling.
• If you are looking for a tool that has a rapidly growing community and open source features, you should choose Locust.
• If you are a beginner to Performance Test Automation , Locust can quickly help you get your performance test running.

However, these recommendations are not absolute, as the best tool for your performance testing needs depends on your specific context and scenario. Therefore, you should evaluate both tools based on your own criteria and preferences, and choose the one that suits you best.

We hope this article has helped you understand the differences and similarities between Gatling and Locust, and how to choose the best tool for your performance testing needs. For more information and resources on these tools, you can visit their official websites or follow the links in the search results. Thank you for reading! ??

References

Gatling vs. Locust: What are the Differences? - Blazemeter

Gatling vs Locust | What are the differences? - StackShare

Awesome Gatling | Curated collection of resources on Gatling load …

https://gatling.io/blog/my-first-gatling-test-report

https://locust.io/

https://gatling.io/

https://www.james-willett.com/gatling-load-testing-complete-guide/

?