Mutation Testing with PITest – 用PITest进行突变测试

最后修改: 2016年 7月 14日

中文/混合/英文(键盘快捷键：t)

1. Overview

1.概述

Software testing refers to the techniques used to assess the functionality of a software application. In this article, we’re going to discuss some of the metrics used in the software testing industry, such as code coverage and mutation testing, with peculiar interest on how to perform a mutation test using the PITest library.

软件测试是指用于评估软件应用程序功能的技术。在这篇文章中，我们将讨论软件测试行业中使用的一些指标，如代码覆盖率和突变测试，特别关注如何使用PITest库进行突变测试。

For the sake of simplicity, we’re going to base this demonstration on a basic palindrome function – Note that a palindrome is a string that reads the same backward and forward.

为了简单起见，我们将以一个基本的回文函数为基础进行演示–注意，回文是一个前后读数相同的字符串。

2. Maven Dependencies

2.Maven的依赖性

As you can see in the Maven dependencies configuration, we will use JUnit to run our tests and the PITest library to introduce mutants into our code – don’t worry, we will see in a second what a mutant is. You can always look up the latest dependency version against the maven central repository by following this link.

正如你在Maven依赖关系配置中看到的，我们将使用JUnit来运行我们的测试，并使用PITest 库将突变体引入我们的代码中–别担心，我们一会儿就会看到什么是突变体。你可以随时根据这个链接，在maven中央仓库中查找最新的依赖版本。

<dependency>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-parent</artifactId>
    <version>1.1.10</version>
    <type>pom</type>
</dependency>

In order to have the PITest library up and running, we also need to include the pitest-maven plugin in our pom.xml configuration file:

为了让PITest库正常运行，我们还需要在pitest-maven配置文件中加入pom.xml插件。

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.1.10</version>
    <configuration>
        <targetClasses>
            <param>com.baeldung.testing.mutation.*</param>
        </targetClasses>
        <targetTests>
            <param>com.baeldung.mutation.test.*</param>
	</targetTests>
     </configuration>
</plugin>

3. Project Setup

3.项目设置

Now that we have our Maven dependencies configured, let’s have a look at this self-explanatory palindrome function:

现在我们已经配置好了Maven的依赖性，让我们来看看这个不言自明的调色板函数。

public boolean isPalindrome(String inputString) {
    if (inputString.length() == 0) {
        return true;
    } else {
        char firstChar = inputString.charAt(0);
        char lastChar = inputString.charAt(inputString.length() - 1);
        String mid = inputString.substring(1, inputString.length() - 1);
        return (firstChar == lastChar) && isPalindrome(mid);
    }
}

All we need now is a simple JUnit test to make sure that our implementation works in the desired way:

现在我们需要的是一个简单的JUnit测试，以确保我们的实现以理想的方式工作。

@Test
public void whenPalindrom_thenAccept() {
    Palindrome palindromeTester = new Palindrome();
    assertTrue(palindromeTester.isPalindrome("noon"));
}

So far so good, we are ready to run our test case successfully as a JUnit test.

到目前为止还不错，我们已经准备好作为JUnit测试成功运行我们的测试案例。

Next, in this article, we are going to focus on code and mutation coverage using the PITest library.

接下来，在这篇文章中，我们将重点讨论使用PITest库的代码和突变覆盖率。

4. Code Coverage

4.代码覆盖率

Code coverage has been used extensively in the software industry, to measure what percent of the execution paths has been exercised during automated tests.

代码覆盖率在软件行业中被广泛使用，以衡量在自动测试期间有多少百分比的执行路径被行使过。

We can measure the effective code coverage based on execution paths using tools like Eclemma available on Eclipse IDE.

我们可以使用Eclipse IDE上的Eclemma等工具，根据执行路径测量有效的代码覆盖率。

After running TestPalindrome with code coverage we can easily achieve a 100% coverage score – Note that isPalindrome is recursive, so it’s pretty obvious that empty input length check will be covered anyway.

在运行TestPalindrome与代码覆盖率后，我们可以很容易地达到100%的覆盖率 – 注意isPalindrome是递归的，所以很明显，空输入长度检查无论如何都会被覆盖。

Unfortunately, code coverage metrics can sometimes be quite ineffective, because a 100% code coverage score only means that all lines were exercised at least once, but it says nothing about tests accuracy or use-cases completeness, and that’s why mutation testing actually matters.

不幸的是，代码覆盖率指标有时是相当无效的，因为100%的代码覆盖率得分只意味着所有行都至少被锻炼了一次，但它没有说明测试的准确性或用例的完整性，这就是为什么突变测试实际上很重要。

5. Mutation Coverage

5.变异覆盖率

Mutation testing is a testing technique used to improve the adequacy of tests and identify defects in code. The idea is to change the production code dynamically and cause the tests to fail.

突变测试是一种测试技术，用于提高测试的充分性和识别代码的缺陷。这个想法是动态地改变生产代码并导致测试失败。

Good tests shall fail

良好的测试应失败

Each change in the code is called a mutant, and it results in an altered version of the program, called a mutation.

代码中的每一个变化都被称为突变体，它导致程序的改变，称为突变。

We say that the mutation is killed if it can cause a fail in the tests. We also say that the mutation survived if the mutant couldn’t affect the behavior of the tests.

如果突变体能导致测试失败，我们就说突变体被杀死了。如果突变体不能影响测试的行为，我们也说突变体存活。

Now let’s run the test using Maven, with the goal option set to: org.pitest:pitest-maven:mutationCoverage.

现在让我们用Maven运行测试，目标选项设置为。org.pitest:pitest-maven:mutationCoverage。

We can check the reports in HTML format in the target/pit-test/YYYYMMDDHHMI directory:

我们可以在target/pit-test/YYYYMMDDHHMI目录中查看HTML格式的报告。

  • 100% line coverage: 7/7
  • 63% mutation coverage: 5/8

Clearly, our test sweeps across all the execution paths, thus, the line coverage score is 100%. In the other hand, the PITest library introduced 8 mutants, 5 of them were killed – Caused a fail – but 3 survived.

很明显，我们的测试横扫了所有的执行路径，因此，行覆盖率是100%。另一方面，PITest库引入了8个突变体，其中5个被杀死–导致失败–但3个幸存下来。

We can check the com.baeldung.testing.mutation/Palindrome.java.html report for more details about the mutants created:

我们可以查看com.baeldung.testing.mutation/Palindrome.java.html报告，了解更多关于创建的突变体的细节。

mutations

 

These are the mutators active by default when running a mutation coverage test:

这些是运行突变覆盖率测试时默认激活的突变器。

  • INCREMENTS_MUTATOR
  • VOID_METHOD_CALL_MUTATOR
  • RETURN_VALS_MUTATOR
  • MATH_MUTATOR
  • NEGATE_CONDITIONALS_MUTATOR
  • INVERT_NEGS_MUTATOR
  • CONDITIONALS_BOUNDARY_MUTATOR

For more details about the PITest mutators, you can check the official documentation page link.

关于PITest突变器的更多细节，你可以查看官方的文档页链接。

Our mutation coverage score reflects the lack of test cases, as we can’t make sure that our palindrome function rejects non-palindromic and near-palindromic string inputs.

我们的突变覆盖率得分反映了测试案例的缺乏，因为我们无法确保我们的宫格函数拒绝非宫格和接近宫格的字符串输入。

6. Improve the Mutation Score

6.提高突变分数

Now that we know what a mutation is, we need to improve our mutation score by killing the surviving mutants.

现在我们知道什么是突变，我们需要通过杀死幸存的突变体来提高我们的突变分数。

Let’s take the first mutation – negated conditional – on line 6 as an example. The mutant survived because even if we change the code snippet:

让我们以第6行的第一个突变–否定条件–为例。这个突变体存活了下来，因为即使我们改变代码片段。

if (inputString.length() == 0) {
    return true;
}

To:

敬请关注。

if (inputString.length() != 0) {
    return true;
}

The test will pass, and that’s why the mutation survived. The idea is to implement a new test that will fail, in case the mutant is introduced. The same can be done for the remaining mutants.

测试将通过，这就是突变体存活的原因。我们的想法是实现一个新的测试，将失败，以防突变体被引入。对其余的突变体也可以这样做。

@Test
public void whenNotPalindrom_thanReject() {
    Palindrome palindromeTester = new Palindrome();
    assertFalse(palindromeTester.isPalindrome("box"));
}
@Test
public void whenNearPalindrom_thanReject() {
    Palindrome palindromeTester = new Palindrome();
    assertFalse(palindromeTester.isPalindrome("neon"));
}

Now we can run our tests using the mutation coverage plugin, to make sure that all the mutations were killed, as we can see in the PITest report generated in the target directory.

现在我们可以使用突变覆盖率插件来运行我们的测试，以确保所有的突变都被杀死了，我们可以在目标目录下生成的PITest报告中看到这一点。

  • 100% line coverage: 7/7
  • 100% mutation coverage: 8/8

7. PITest Tests Configuration

7.PITest测试配置

Mutation testing may be resources-extensive sometimes, so we need to put proper configuration in place to improve tests effectiveness. We can make use of the targetClasses tag, to define the list of classes to be mutated. Mutation testing cannot be applied to all classes in a real world project, as it will be time-consuming, and resource critical.

突变测试有时可能需要大量的资源，所以我们需要进行适当的配置以提高测试的有效性。我们可以使用targetClasses标签，来定义要变异的类的列表。突变测试不能应用于现实世界项目中的所有类，因为它将会很耗时，而且对资源要求很高。

It is also important to define the mutators you plan to use during mutation testing, in order to minimize the computing resources needed to perform the tests:

定义你计划在突变测试中使用的突变器也很重要，以便最大限度地减少执行测试所需的计算资源。

<configuration>
    <targetClasses>
        <param>com.baeldung.testing.mutation.*</param>
    </targetClasses>
    <targetTests>
        <param>com.baeldung.mutation.test.*</param>
    </targetTests>
    <mutators>
        <mutator>CONSTRUCTOR_CALLS</mutator>
        <mutator>VOID_METHOD_CALLS</mutator>
        <mutator>RETURN_VALS</mutator>
        <mutator>NON_VOID_METHOD_CALLS</mutator>
    </mutators>
</configuration>

Moreover, the PITest library offers a variety of options available to customize your testing strategies, you can specify the maximum number of mutants introduced by class using the maxMutationsPerClass option for example. More details about PITest options in the official Maven quickstart guide.

此外，PITest库提供了多种可用选项，以定制您的测试策略，例如，您可以使用maxMutationsPerClass选项指定每个类引入的最大突变体数量。关于PITest选项的更多细节，请参见官方Maven快速入门指南。

8. Conclusion

8.结论

Note that code coverage is still an important metric, but sometimes it is not sufficient enough to guarantee a well-tested code. So in this article we’ve walked through mutation testing as a more sophisticated way to ensure tests quality and endorse test cases, using the PITest library.

请注意，代码覆盖率仍然是一个重要的指标，但有时它并不足以保证一个经过良好测试的代码。因此，在这篇文章中，我们通过突变测试作为一种更复杂的方法来确保测试质量和认可测试案例，使用PITest库。

We have also seen how to analyze a basic PITest reports while improving the mutation coverage score.

我们还看到了如何分析一个基本的PITest报告，同时提高突变覆盖率得分。

Even though mutation testing reveals defects in code, it should be used wisely, because it is an extremely costly and time-consuming process.

即使突变测试揭示了代码中的缺陷，也应该明智地使用它，因为它是一个极其昂贵和耗时的过程。

You can check out the examples provided in this article in the linked GitHub project.

你可以在链接的GitHub项目中查看本文提供的例子。