How to create Mutation tests with Pitest

Pitest is a tool that provides state-of-the-art mutation testing. It was written to run on Java applications or on applications that run on top of a JVM.

How it works?

Pitest works by generating mutant codes that will make changes to the bytecodes in the source code, changing logic, removing and even changing method return values. Thus, it's possible to evaluate the code in a more complete way, looking for failures and making the code reliable.

Basic example of mutation

See the code below:

if(name == ""){
   return "Empty name";
}else{
    return name;
}

Running Pitest on the code above, it's able to modify the bytecodes by changing the code above to:

if(name != ""){
   return name;
}else{
    return "Empty name";
}

Pitest enables you identifying possible points of failure and guide you to create tests based on these points.

Hands on time

Maven

<dependencies>
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.pitest</groupId>
        <artifactId>pitest</artifactId>
        <version>1.6.2</version>
    </dependency>
</dependencies>

Let's create a simple class representing a bank account called BankAccount and to make it simple we'll implement the logic inside the entity class itself.

In this class, we have a method called userValid that checks if the user is valid.

package com.bank.entity;

public class BankAccount {

    public String bankName;
    public String user;
    public Double balance;

    public Boolean userValid(BankAccount bankAccount){
        if(bankAccount.user != ""){
            return true;
        }else{
            return false;
        }
    }
}

Without creating any test classes, run this command via Maven:

mvn org.pitest:pitest-maven:mutationCoverage

See that a report was generated in the console with the mutations created based on the conditions that must be tested.

Looking at the console above, we have 3 mutations or point of failures, which means we need to eliminate these mutations from our code.

Pitest provides another way to validate these mutations through a report generated by an HTML page. You can access these generated reports through /target/pit-reports/ folder

In the image below we can see a report that summarizes the coverage percentage. Note that we have 0% of coverage.

In the BankAccount.java.html file shows details about uncovered code parts.

Note that in the image above there is a list of Active mutators that are mutations based on the code we created.

Removing these mutations

Based on the reports, let's create some tests to reach out 100% of coverage. First of all, we need to create a class test. For this example, we're going to create a class test called BankAccountTest.java. To do this, there's an attention point.

You'll have to create this class with the same package's name used for the BankAccount.java. Thus, create inside src/test/java a package called com.bank.entity and finally BankAccountTest.java.

package com.bank.entity;

import org.junit.Assert;
import org.junit.Test;

public class BankAccountTest {


    @Test
    public void userValid_NegateConditionalsMutator_Test(){

        BankAccount bankAccount = new BankAccount(
                "Chase","Jonas", 2000.00);

        Boolean actual = bankAccount.userValid(bankAccount);
        Boolean expected = true;
        Assert.assertEquals(expected, actual);
    }

    @Test
    public void userValid_BooleanFalseReturnValsMutator_Test(){

        BankAccount bankAccount = new BankAccount(
                "Chase","", 2000.00);

        Boolean actual = bankAccount.userValid(bankAccount);
        Boolean expected = false;
        Assert.assertEquals(expected, actual);
    }

}

We have created only 2 test method for our coverage:

1.userValid_NegateConditionalsMutator_Test(): Covers filled-in usernames. The constructor passed the "Chase" argument representing the username so that the method's condition is validated. And finally validates the condition of the method to be returned as TRUE.

To this case, we already eliminated two mutations according to statistic report.

1. BooleanTrueReturnValsMutator

2. NegateConditionalsMutator

2. userValid_BooleanFalseReturnValsMutator_Test(): Validates the possibility of the method returning the Boolean value FALSE.

Rerun the Maven command below:

mvn org.pitest:pitest-maven:mutationCoverage

Application console

Note that for every mutation generated, 1 was eliminated. Covering 100% coverage in tests.

References: https://pitest.org

---

Books to study and read

If you want to learn more about and reach a high level of knowledge, I strongly recommend reading the following book(s):

Unit Testing Principles, Practices, and Patterns: Effective Testing Styles, Patterns, and Reliable Automation for Unit Testing, Mocking, and Integration Testing with Examples in C# is a book that covers Unit Testing Principles, Patterns and Practices teaches you to design and write tests that target key areas of your code including the domain model. In this clearly written guide, you learn to develop professional-quality tests and test suites and integrate testing throughout the application life cycle.

Mastering Unit Testing Using Mockito and JUnit is a book that covers JUnit practices using one of the most famous testing libraries called Mockito. This book teaches how to create and maintain automated unit tests using advanced features of JUnit with the Mockito framework, continuous integration practices (famous CI) using market tools like Jenkins along with one of the largest dependency managers in Java projects, Maven. For you who are starting in this world, it is an excellent choice.

Well that’s it, I hope you enjoyed it!