How to execute your test automation in parallel

Almost every company has seen the benefits of test automation and has bought into the idea of test automation throughout their entire stack. While automation has significantly improved delivery times and allowed us to deliver software at a quicker rate – a common problem I’ve seen with many automation frameworks is that they do not do a very good job of handling execution in parallel.

Benefits of Parallel Test Execution

Running test automation in parallel offers several significant benefits, particularly in terms of efficiency, speed, and resource utilization. Here are the key reasons why parallel execution of test automation is advantageous:

Reduced Test Execution Time

Faster Feedback Loop: Parallel execution significantly reduces the total time required to run your entire test suite. This is especially important in continuous integration and continuous deployment (CI/CD) pipelines where quick feedback is crucial.

Efficiency: By running tests concurrently, you make better use of available resources, thereby reducing idle time and improving the overall efficiency of the testing process.

Increased Test Coverage

Scalability: Parallel testing allows you to run a large number of tests across multiple environments and configurations simultaneously, increasing test coverage and ensuring that your application works correctly under various conditions.

Optimized Resource Utilization

Better Use of Hardware: Utilizing multiple cores or machines to run tests in parallel ensures that the available hardware is used more effectively, leading to improved performance and throughput.

Cost-Effectiveness: For cloud-based testing environments, running tests in parallel can be more cost-effective as it reduces the total time and resources required, potentially lowering the cost of cloud computing services.

Enhanced Reliability and Stability

Isolation of Tests: Running tests in parallel can help identify flaky tests (tests that sometimes pass and sometimes fail) and resource contention issues, as tests are run independently in isolated environments.

Error Detection: By testing in parallel, you can detect concurrency issues, race conditions, and other problems that might not be apparent in a sequential execution.

Improved Developer Productivity

Quick Turnaround: Developers receive faster feedback on their changes, enabling them to address issues promptly and maintain a high development velocity.

Concurrent Development and Testing: Parallel testing allows for concurrent development and testing activities, which can lead to a more streamlined and efficient development process.

Example Use Cases

• Continuous Integration/Continuous Deployment (CI/CD): In CI/CD pipelines, it’s essential to get quick feedback on code changes. Running tests in parallel helps ensure that builds and deployments are not delayed by lengthy test executions.
• Cross-Browser Testing: To ensure that a web application works correctly across different browsers and browser versions, parallel testing can be employed to run tests on multiple browser instances simultaneously.
• API Testing: For microservices architectures, running API tests in parallel can validate multiple endpoints and services concurrently, speeding up the overall testing process.

Practical Implementation

So, we can easily see the benefits of executing automation tests in parallel, but he real challenge for many might be in how best to implement this into their respective frameworks. We are now going to have a look at different approaches with different tools to showcase how this can be done.

There are many others ways of doing it, but the below approaches should be able to suit the needs of most teams looking to achieve parallel test execution with their different tools. I have also limited the tools here to just a few of the most popular ones with a popular programming language too. These framework approaches can be easily adapted to different programming languages. ?

Selenium with TestNG

TestNG is a popular testing framework for Java, which supports parallel test execution out-of-the-box. Here’s how you can configure it, with Java used asa tge core programming language in these examples:

Add dependencies: Ensure you have the necessary dependencies in your pom.xml (if you’re using Maven).

<dependencies>
??? <dependency>
??????? <groupId>org.testng</groupId>
??????? <artifactId>testng</artifactId>
??????? <version>7.4.0</version>
??????? <scope>test</scope>
??? </dependency>
??? <dependency>
??????? <groupId>org.seleniumhq.selenium</groupId>
??????? <artifactId>selenium-java</artifactId>
??????? <version>4.1.0</version>
??? </dependency>
</dependencies>        

Create Test Classes: Create test classes annotated with @Test from TestNG.

import org.testng.annotations.Test;

public class TestClass1 {
??? @Test
??? public void testMethod1() {
??????? System.out.println("Test Method 1 - " + Thread.currentThread().getId());
??? }
}
?
public class TestClass2 {
??? @Test
??? public void testMethod2() {
??????? System.out.println("Test Method 2 - " + Thread.currentThread().getId());
??? }
}        

Configure TestNG XML for Parallel Execution: Create a testng.xml file to configure parallel execution.

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="ParallelTestsSuite" parallel="tests" thread-count="2">
??? <test name="Test1">
??????? <classes>
??????????? <class name="TestClass1"/>
??????? </classes>
??? </test>
??? <test name="Test2">
??????? <classes>
??????????? <class name="TestClass2"/>
??????? </classes>
??? </test>
</suite>        

Run TestNG Suite: Execute the testng.xml suite in shell script.

mvn test -DsuiteXmlFile=testng.xml        

Selenium with JUnit

JUnit, particularly JUnit 5, can also run tests in parallel using a few configurations.

Add dependencies: Ensure you have the necessary dependencies in your build.gradle (if you’re using Gradle).

dependencies {
??? testImplementation 'org.junit.jupiter:junit-jupiter:5.7.0'
??? testImplementation 'org.seleniumhq.selenium:selenium-java:4.1.0'
}        

Create Test Classes: Create test classes annotated with @Test from JUnit.

import org.junit.jupiter.api.Test;

public class TestClass1 {
??? @Test
??? void testMethod1() {
??????? System.out.println("Test Method 1 - " + Thread.currentThread().getId());
??? }
}
?
public class TestClass2 {
??? @Test
??? void testMethod2() {
??????? System.out.println("Test Method 2 - " + Thread.currentThread().getId());
??? }
}        

Configure JUnit for Parallel Execution: Create a junit-platform.properties file in src/test/resources.

junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = concurrent
junit.jupiter.execution.parallel.config.strategy = dynamic        

Run Tests: Execute the tests using Gradle.

./gradlew test        

Using Python with pytest

Pytest is a popular testing framework in Python that supports parallel execution via the pytest-xdist plugin.

Install dependencies: Install pytest and pytest-xdist.

pip install pytest pytest-xdist selenium        

Create Test Files: Create test files with test functions.

import pytest

def test_method1():
??? print(f"Test Method 1 - {pytest.current.thread().id}")
?
def test_method2():
??? print(f"Test Method 2 - {pytest.current.thread().id}")        

Run Tests in Parallel: Use the -n flag to specify the number of parallel workers.

pytest -n 2        

Using C# with NUnit

NUnit is a widely used testing framework for C# and supports parallel test execution.

Add dependencies: Ensure you have the necessary NuGet packages.

dotnet add package NUnit
dotnet add package NUnit3TestAdapter
dotnet add package Microsoft.NET.Test.Sdk
dotnet add package Selenium.WebDriver        

Create Test Classes: Create test classes annotated with [Test].

using NUnit.Framework;
?
[TestFixture]
public class TestClass1 {
??? [Test]
??? public void TestMethod1() {
??????? Console.WriteLine($"Test Method 1 - {Thread.CurrentThread.ManagedThreadId}");
??? }
}
?
[TestFixture]
public class TestClass2 {
??? [Test]
??? public void TestMethod2() {
??????? Console.WriteLine($"Test Method 2 - {Thread.CurrentThread.ManagedThreadId}");
??? }
}        

Configure Parallel Execution: You can configure parallel execution in the .runsettings file or programmatically.

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
? <RunConfiguration>
??? <MaxCpuCount>2</MaxCpuCount>
? </RunConfiguration>
</RunSettings>        

Run Tests: Use the dotnet test command.

dotnet test --settings .runsettings        

Cypress

To run tests in parallel with Cypress, you can use the Cypress Dashboard Service, which provides built-in support for parallelization. Here’s a step-by-step guide:

Step 1: Install Cypress

First, ensure you have Cypress installed in your project:

npm install cypress --save-dev        

Step 2: Configure Cypress for Parallel Testing

You need to set up your project to use the Cypress Dashboard. If you don’t have a Cypress Dashboard project set up, you can create one by following the instructions on the Cypress Dashboard.

Step 3: Add Environment Variables

Add your CYPRESS_RECORD_KEY (which you get from the Cypress Dashboard) to your CI environment. For example, in a GitHub Actions workflow, you might add it like this:

env:
? CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}        

Step 4: Configure CI for Parallel Execution

Modify your CI configuration to run Cypress tests in parallel. Here’s an example using GitHub Actions:

name: Cypress Tests
?
on: [push, pull_request]
?
jobs:
? cypress-run:
??? runs-on: ubuntu-latest
??? strategy:
????? matrix:
??????? containers: [1, 2, 3, 4]
??? steps:
??? - uses: actions/checkout@v2
??? - name: Use Node.js
????? uses: actions/setup-node@v2
????? with:
??????? node-version: '14'
??? - name: Install Dependencies
????? run: npm install
??? - name: Run Cypress Tests
????? run: npx cypress run --record --parallel --group ${{ matrix.containers }}
????? env:
??????? CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}        

This configuration will split your Cypress tests across four parallel containers.

Playwright

Playwright also supports parallel test execution out of the box. We are using Typescript as a language in these examples. Here’s how you can set it up:

Step 1: Install Playwright

First, ensure you have Playwright installed in your project:

npm install @playwright/test --save-dev        

Step 2: Configure Playwright for Parallel Testing

Playwright runs tests in parallel by default. You can configure the level of parallelism in your playwright.config.ts file.

import { defineConfig } from '@playwright/test';
?
export default defineConfig({
? // Set the maximum number of parallel workers
? workers: 4,
? use: {
??? // Browser settings
??? browserName: 'chromium',
??? headless: true,
? },
? projects: [
??? {
????? name: 'chromium',
????? use: { ...devices['Desktop Chrome'] },
??? },
??? {
????? name: 'firefox',
????? use: { ...devices['Desktop Firefox'] },
??? },
??? {
????? name: 'webkit',
????? use: { ...devices['Desktop Safari'] },
??? },
? ],
});        

Step 3: Run Playwright Tests

Run the tests using the Playwright test runner:

npx playwright test        

This command will automatically use the parallelism level set in the configuration file.

Step 4: Configure CI for Playwright

Here’s an example of a GitHub Actions workflow for running Playwright tests in parallel:

name: Playwright Tests

on: [push, pull_request]
?
jobs:
? playwright-run:
??? runs-on: ubuntu-latest
??? strategy:
????? matrix:
??????? node-version: [14.x]
??? steps:
??? - uses: actions/checkout@v2
??? - name: Use Node.js
????? uses: actions/setup-node@v2
????? with:
??????? node-version: ${{ matrix.node-version }}
??? - name: Install Dependencies
????? run: npm install
??? - name: Run Playwright Tests
????? run: npx playwright test        

Summary

Parallel test execution is one of the fundamental tenets of test automation effectiveness and yet one that is rarely executed well. Hopefully, with some of the tips gained in this blog, you will be able to work with your existing automation tools and adjust them to better capitalize on the benefits of parallel test execution.