Mutation Testing: The Key to Flawless Code

“Let’s test our tests and catch the bugs before they catch us”

Preface

Imagine a world where every line of code you write not only performs flawlessly but also stands strong against the unexpected. Picture deploying your software with unwavering confidence, knowing it has been crafted with precision and tested with the utmost rigour. Ever wondered if your tests are truly up to the challenge? You’ve written them, and everything seems perfect. But what if there’s a sneaky bug lurking in the shadows? Welcome to the transformative realm of Mutation Testing, which supercharges your testing process. Mutation Testing acts like a detective, uncovering these hidden flaws and ensuring your code is bugproof.

Before moving forward, let’s go through the overview of Mutation Testing to understand the solution better.

Mutation testing, also referred to as code mutation testing, is a type of white box testing that entails modifying certain elements of an application's code to verify whether a software test suite can detect the modifications. The modifications made to the software are designed to induce errors in the program.

The code is mutated by introducing small changes to simulate potential bugs. For example:

1. Conditional Boundary Mutation: if (a > b) is changed to if (a >= b).
2. Math Operator Mutation: a + b is changed to a - b.
3. Boolean Return Mutation: return true; is changed to return false;.

These mutations test whether your test cases can detect subtle changes. Learn more about mutators here. The effectiveness of your tests is measured using the mutation score, calculated as:

Mutation score = (Number of killed mutants /Total number of mutants) x 100        

At Atlassian, our proactive approach has proven to be highly effective. By implementing mutation testing, we have improved the quality of our tests. In retrospect, we found that better Pitest coverage could have prevented a few incidents, demonstrating its effectiveness in maintaining high standards of code quality.

What is PITest? PITest, often just called PIT, is a clever tool that helps you check how good your unit tests really are. It does this by introducing tiny changes, or mutations, into your code. The aim? To see if your tests can catch these changes. If they do and fail, that’s a win! It means your tests are doing their job. But if they miss the changes and pass, it’s a hint that your tests might need some work.

Without Mutation Testing

You write tests; they test/consume production code that you have written.

With Mutation Testing

You write tests; they test/consume production code that you have written; they also test/consume mutants of the production code that are automatically generated.

How PITest Works: A Simple Walkthrough

1. Analyze Your Code: PITest starts by taking a close look at your code. It figures out where it can make changes to mimic common mistakes or logical errors.
2. Generate Mutants: Next, PITest tweaks your code in small ways to create "mutants". These are slightly altered versions of your code that still compile and run without errors.
3. Run Your Tests: PITest then runs all your unit tests against each mutant. The mission? To see if your tests can spot these sneaky changes. Every mutant is tested separately to make sure your tests are as sharp as they can be.
4. Evaluate Results: Once all the mutants are tested, PITest checks the outcomes. Here’s what could happen with each mutant:
5. Report Generation: Finally, PITest creates a report showing how many mutants were killed, survived, and more. This report gives you a clear picture of how well your tests are working and where you might need to improve.

In a nutshell, PITest mutates your code and uses your tests to hunt down these changes. This smart process helps you see how effective your tests are at catching mistakes and spots where your test coverage could be better. By focusing on the areas where mutants survive, you can enhance your test suite, leading to stronger and more reliable software.

Fine, but how can I integrate it with my project?

Step 1: Apply the pitest and arcmutate Plugin to your Gradle Project

NOTE: PITest is an open-source tool that provides powerful mutation testing capabilities. The Arcmutate plugin, on the other hand, is the commercial extension of PITest, offering additional features, especially for Kotlin or frameworks like Spring. It’s important to highlight that Arcmutate requires a valid license to use its features. For straightforward Java projects, the standalone PITest plugin may be sufficient and does not require a license.

1. Open your build.gradle file.
2. Add the pitest, arcmutate and downloadplugin to the plugins block:

plugins { 
  id 'info.solidsoft.pitest' version '1.15.0' //pitest plugin
  id "de.undercouch.download" version "5.6.0" // Download plugin
  id "com.arcmutate.bitbucket.cloud" version "1.1.1" //arcmutate plugin
  [id "com.arcmutate.github.cloud" version "1.1.1" //arcmutate plugin for github users]
}

// The above plugin should be added at the repository level for multi-module project.        

Configure the plugin as needed:

• Add the PITest Gradle Plugin and configure it as needed.
• For Maven users, please refer to the PITest Maven Plugin Documentation.

pitest {
    def base = System.getenv("BASE") ?: 'main'
    junit5PluginVersion = '1.2.1'
    features.add("+git_mixed(from[$base])", "+auto_threads")
    targetClasses = ['com.<project-path>.*']
    targetTests = ['com.<project-path>.*']
    excludedClasses = ['com.<project-path>.proto.*']
    outputFormats = ["XML", "HTML"]
    threads = 4
    mutationThreshold = 80 
    verbose = true
    avoidCallsTo = []
}        

Step 2: Add Plugins for Junit Accelerator, Kotlin, Spring, Git, License Management, and Bitbucket Integration [optional]

To further enhance your project with Git integration, license management, and Bitbucket integration, you can add corresponding plugins:

1. Junit Accelerator: The plugin improves mutation test performance by reducing the overhead in running tests that use a limited set of JUnit Jupiter features. Tests that use only these features will run more quickly, tests that use features outside this set will be executed as normal.
2. Kotlin Plugin: Although Kotlin compiles to normal java bytecode, some language features require compiler generated constructs that do not map back to the source code. This results in confusing mutants that are hard to interpret, and junk mutations which cannot be reproduced by mistakes in the source code. The kotlin plugin filters out these junk mutations and removes confusing noise from mutant descriptions.
3. Spring Plugin: Teams using pitest with spring face two issues
4. Git Integration: You might use the org.ajoberstar.grgit plugin for Git operations, but note that Gradle doesn't have a built-in plugin for Git integration. You typically handle Git operations outside of Gradle or through custom tasks that execute Git commands.

plugins { 
    id 'com.arcmutate:pitest-accelerator-junit5:1.1.0'
    id 'com.arcmutate:pitest-kotlin-plugin:1.2.2' 
    id 'com.arcmutate:arcmutate-spring:1.0.0'
    id 'com.arcmutate:pitest-git-plugin:1.2.0'
}
 // Refer https://docs.arcmutate.com/ for each plugin details        

5. License Management: Licenses are plain text files tied to one or more root packages. They will work for any code in those packages until the date on which they expire.

For gradle builds, we suggest using the gradle download task for downloading the latest and greatest version of the license.

tasks.register('arcmutateLicence', Download) {
    src 'https://subscriptions.arcmutate.com/AAFFDEMOAAFF/arcmutate-licence.txt'
    dest "$buildDir/reports/pitest/arcmutate-licences/arcmutate-licence.txt"
    onlyIfModified true
    overwrite true
}        

To work correctly with gradle’s caching mechanism, the pitest task must depend upon the licence task by adding the following line to build.gradle.

tasks.getByName('pitest').dependsOn('arcmutateLicence')        

6. Bitbucket Integration: The com.arcmutate.bitbucket.cloud plugin integrates with Bitbucket cloud and provides two tasks

• pitest-bitbucket
• pitest-bitbucket-upload

The pitest-bitbucket task runs pitest against all modules for which it has been configured, then updates the current PR.

The pitest-bitbucket-upload task updates the PR based on previously generated gitci json files.

Parameters can be configured via a pitestBitbucket extension block

pitestBitbucket {
  mutantEmoji = ':vomiting_face:'
  trailingText = 'Happy mutant hunting'  
}        

Step 3: Generate the Report

Here you go ?? Execute the command below to generate the report in both HTML and XML formats.

- ./gradlew pitest        

Enough Talk, Let’s Do Some Hands-On!

It's time to place your fingers on the keyboard and start coding!

In this section, we'll dive into a practical demo of mutation testing using a simple Calculator class. This example will demonstrate how achieving 100% line coverage doesn’t always mean your tests are effective. By optimizing the test suite, we’ll improve mutation coverage and ensure all mutants are caught. Let’s get started!

Unoptimized Code and Test Suite

• The Code: The Calculator class includes three basic methods:
• Initial Tests: The unoptimized test suite achieves 100% line coverage because all lines of code are executed. However, it fails to account for edge cases and logical variations.

• Mutation Coverage: 71% : Some mutants survive because the tests don’t detect subtle bugs introduced by mutators.

Unoptimised Report Analysis

Here’s an example of survived mutants and what they mean:

1. Replaced boolean return with true for Calculator::isPositive → SURVIVED: Mutation: The method always returns true, regardless of the input. Why? The test only checks one positive input and doesn’t consider negative or zero values.
2. Changed conditional boundary for Calculator::isPositive → SURVIVED: Mutation: The condition number > 0 is changed to number >= 0. Why? The test doesn’t check the edge case of 0, so the mutant survives.

Optimized Test Suite

After reviewing the mutation report, the test suite is updated to cover edge cases and additional scenarios:

1. add and subtract now test: Positive, negative, and mixed signs, Zero values.
2. isPositive now tests: Positive, negative, and zero inputs.

Optimized Report Analysis:

• Mutations like Replaced boolean return with true and Changed conditional boundary are now detected by the improved test cases.
• The mutation coverage report shows all mutants killed, confirming strong test effectiveness.

FAQ

1. How does PITest decide where to introduce mutations? PITest analyzes your bytecode, identifying potential spots for mutations based on common programming errors and logical changes, ensuring a wide coverage of possible faults.
2. Mutation testing vs. other methods? Mutation testing evaluates the quality of your tests by introducing small, deliberate faults (mutations) into your code to see if your tests catch them. It's different because it tests the tests themselves, unlike other methods, which directly test the code's functionality.
3. Integration with CI/CD pipeline? Yes, PITest can be integrated into CI/CD pipelines through plugins for common build tools like Maven and Gradle, allowing automated mutation testing in your development process.
4. Execution time for PITest? The time depends on the size of your codebase and the number of generated mutants. PITest is designed to be efficient, but mutation testing is inherently resource-intensive. Parallel execution and targeted testing can reduce time.