Mutation Testing with PITest – 用PITest进行突变测试

最后修改: 2016年 7月 14日

1. Overview


Software testing refers to the techniques used to assess the functionality of a software application. In this article, we’re going to discuss some of the metrics used in the software testing industry, such as code coverage and mutation testing, with peculiar interest on how to perform a mutation test using the PITest library.


For the sake of simplicity, we’re going to base this demonstration on a basic palindrome function – Note that a palindrome is a string that reads the same backward and forward.


2. Maven Dependencies


As you can see in the Maven dependencies configuration, we will use JUnit to run our tests and the PITest library to introduce mutants into our code – don’t worry, we will see in a second what a mutant is. You can always look up the latest dependency version against the maven central repository by following this link.

正如你在Maven依赖关系配置中看到的,我们将使用JUnit来运行我们的测试,并使用PITest 库将突变体引入我们的代码中–别担心,我们一会儿就会看到什么是突变体。你可以随时根据这个链接,在maven中央仓库中查找最新的依赖版本。


In order to have the PITest library up and running, we also need to include the pitest-maven plugin in our pom.xml configuration file:



3. Project Setup


Now that we have our Maven dependencies configured, let’s have a look at this self-explanatory palindrome function:


public boolean isPalindrome(String inputString) {
    if (inputString.length() == 0) {
        return true;
    } else {
        char firstChar = inputString.charAt(0);
        char lastChar = inputString.charAt(inputString.length() - 1);
        String mid = inputString.substring(1, inputString.length() - 1);
        return (firstChar == lastChar) && isPalindrome(mid);

All we need now is a simple JUnit test to make sure that our implementation works in the desired way:


public void whenPalindrom_thenAccept() {
    Palindrome palindromeTester = new Palindrome();

So far so good, we are ready to run our test case successfully as a JUnit test.


Next, in this article, we are going to focus on code and mutation coverage using the PITest library.


4. Code Coverage


Code coverage has been used extensively in the software industry, to measure what percent of the execution paths has been exercised during automated tests.


We can measure the effective code coverage based on execution paths using tools like Eclemma available on Eclipse IDE.

我们可以使用Eclipse IDE上的Eclemma等工具,根据执行路径测量有效的代码覆盖率。

After running TestPalindrome with code coverage we can easily achieve a 100% coverage score – Note that isPalindrome is recursive, so it’s pretty obvious that empty input length check will be covered anyway.

在运行TestPalindrome与代码覆盖率后,我们可以很容易地达到100%的覆盖率 – 注意isPalindrome是递归的,所以很明显,空输入长度检查无论如何都会被覆盖。

Unfortunately, code coverage metrics can sometimes be quite ineffective, because a 100% code coverage score only means that all lines were exercised at least once, but it says nothing about tests accuracy or use-cases completeness, and that’s why mutation testing actually matters.


5. Mutation Coverage


Mutation testing is a testing technique used to improve the adequacy of tests and identify defects in code. The idea is to change the production code dynamically and cause the tests to fail.


Good tests shall fail


Each change in the code is called a mutant, and it results in an altered version of the program, called a mutation.


We say that the mutation is killed if it can cause a fail in the tests. We also say that the mutation survived if the mutant couldn’t affect the behavior of the tests.


Now let’s run the test using Maven, with the goal option set to: org.pitest:pitest-maven:mutationCoverage.


We can check the reports in HTML format in the target/pit-test/YYYYMMDDHHMI directory:


  • 100% line coverage: 7/7
  • 63% mutation coverage: 5/8

Clearly, our test sweeps across all the execution paths, thus, the line coverage score is 100%. In the other hand, the PITest library introduced 8 mutants, 5 of them were killed – Caused a fail – but 3 survived.


We can check the com.baeldung.testing.mutation/ report for more details about the mutants created:




These are the mutators active by default when running a mutation coverage test:



For more details about the PITest mutators, you can check the official documentation page link.


Our mutation coverage score reflects the lack of test cases, as we can’t make sure that our palindrome function rejects non-palindromic and near-palindromic string inputs.


6. Improve the Mutation Score


Now that we know what a mutation is, we need to improve our mutation score by killing the surviving mutants.


Let’s take the first mutation – negated conditional – on line 6 as an example. The mutant survived because even if we change the code snippet:


if (inputString.length() == 0) {
    return true;



if (inputString.length() != 0) {
    return true;

The test will pass, and that’s why the mutation survived. The idea is to implement a new test that will fail, in case the mutant is introduced. The same can be done for the remaining mutants.


public void whenNotPalindrom_thanReject() {
    Palindrome palindromeTester = new Palindrome();
public void whenNearPalindrom_thanReject() {
    Palindrome palindromeTester = new Palindrome();

Now we can run our tests using the mutation coverage plugin, to make sure that all the mutations were killed, as we can see in the PITest report generated in the target directory.


  • 100% line coverage: 7/7
  • 100% mutation coverage: 8/8

7. PITest Tests Configuration


Mutation testing may be resources-extensive sometimes, so we need to put proper configuration in place to improve tests effectiveness. We can make use of the targetClasses tag, to define the list of classes to be mutated. Mutation testing cannot be applied to all classes in a real world project, as it will be time-consuming, and resource critical.


It is also important to define the mutators you plan to use during mutation testing, in order to minimize the computing resources needed to perform the tests:



Moreover, the PITest library offers a variety of options available to customize your testing strategies, you can specify the maximum number of mutants introduced by class using the maxMutationsPerClass option for example. More details about PITest options in the official Maven quickstart guide.


8. Conclusion


Note that code coverage is still an important metric, but sometimes it is not sufficient enough to guarantee a well-tested code. So in this article we’ve walked through mutation testing as a more sophisticated way to ensure tests quality and endorse test cases, using the PITest library.


We have also seen how to analyze a basic PITest reports while improving the mutation coverage score.


Even though mutation testing reveals defects in code, it should be used wisely, because it is an extremely costly and time-consuming process.


You can check out the examples provided in this article in the linked GitHub project.