What Is Mutation Testing: Tutorial With Examples

This tutorial explains what is Mutation Testing, how to perform it, and Mutation Testing types with examples:

What is Mutation Testing?

Mutation testing is a fault-based testing technique where variations of a software program are subjected to the test dataset. This is done to determine the effectiveness of the test set in isolating the deviations.

It sounds a little complicated, isn’t it? 

Mutation Testing

Mutation Testing (MT) goes a long way, back to the 70s where it was first proposed as a school project. It was written off as it was very resource-intensive. However, as humans continued to develop more advanced computers, it slowly made a comeback and is now one of the most popular testing techniques.

Mutation testing definition

MT is also known as fault-based testing, program mutation, error-based testing, or mutation analysis.

As the name suggests, mutation testing is a software testing type that is based on changes or mutations. Miniscule changes are introduced into the source code to check whether the defined test cases can detect errors in the code.

The ideal case is that none of the test cases should pass. If the test passes, then it means that there is an error in the code. We say that the mutant (the modified version of our code) lived. If the test fails, then there is no error in the code, and the mutant was killed. Our goal is to kill all mutants.

Mutation testing also helps to test the quality of the defined test cases or the test suites with a bid to write more effective test cases. The more mutants we can kill, the higher the quality of our tests.

Mutation Testing Concepts

Before we discuss mutation testing further, let us explore some concepts that we will come across.

#1) Mutants: It is simply the mutated version of the source code. It is the code that contains minute changes. When the test data is run through the mutant, it should ideally give us different results from the original source code. Mutants are also called mutant programs.

There are different types of mutants. These are as follows:

• Survived Mutants: As we have mentioned, these are the mutants that are still alive after running test data through the original and mutated variants of the source code. These must be killed. They are also known as live mutants.
• Killed Mutants: These are mutants that are killed after mutation testing. We get these when we get different results from the original and mutated versions of the source code.
• Equivalent Mutants: These are closely related to live mutants, in that, they are ‘alive’ even after running test data through them. What differentiates them from others is that they have the same meaning as the original source code, even though they may have different syntax.

#2) Mutators/mutation operators: These are what makes mutations possible, they are on the ‘driver’s seat’. They basically define the kind of alteration or change to make to the source code to have a mutant version. They can be referred to as faults or mutation rules.

#3) Mutation score: This is a score based on the number of mutants.

It is calculated using the below formula:

Note that, equivalent mutants are not considered when calculating the mutation score. Mutation score is also known as mutation adequacy. Our aim should be to achieve a high mutation score.

How To Do Mutation Testing

Step #1: Let’s write our Jasmine unit test.

Test Suite (Jasmine)

describe("User", function() {
  it("should compare the two numbers from user input", function(){
   expect(20).toBeGreaterThan(5);
 })
});

Step #2: Write the original code.

Original code (Javascript)

const user_info = () => {
   mother_age = parseInt(prompt("Enter mother's age"))
   daughter_age = parseInt(prompt("Enter daughter's age"))
 	
   if (mother_age > daughter_age) {
   	alert(`Daughter's age is ${daughter_age}. Mother's age is
   	${mother_age}. Welcome to the Mother-Daughter program`)
 
   } else {   
   	alert(`Daughter's age: ${daughter_age}, is more than mother's age: ${mother_age}.
   	Please enter correct ages`) 	 
   }
}
user_info();

Step #3: We will then run the test through the original code to ensure that we don’t have failed tests from the get-go. We should have some output that communicates that we have indeed written a test with zero failures.

For example:

finished in 0.019s 1 spec, 0 failures, randomized with seed 31435

Original code’s result:

Daughter's age is 5. Mother's age is 20. Welcome to the Mother-Daughter program

Step #4: Introduce the mutant. In this case, we change the greater-than operator (mother_age > daughter_age) to a lesser than operator (mother_age < daughter_age)

Mutant code (Javascript)

const user_info = () =>{
   mother_age = parseInt(prompt("Enter mother's age"))
   daughter_age = parseInt(prompt("Enter daughter's age"))
 	
   if (mother_age < daughter_age) {
   	alert(`Daughter's age is ${daughter_age}. Mother's age is
   	${mother_age}. Welcome to the Mother-Daughter program`)
 	
   } else {   
   	alert(`Daughter's age: ${daughter_age}, is more than mother's age: ${mother_age}.
   	Please enter correct ages`) 	 
   }
}
 
user_info();

Step #5: We then run the test through the mutant code.

Here are the test results:

finished in 0.017s 1 spec, 0 failures, randomized with seed 89555

Mutant code’s result:

Daughter's age: 5, is more than mother's age: 20. Please enter correct ages

Step #6: Compare the results from the original version and the mutant version. In this case, they are different, even if the same test suite was used. We have therefore killed our mutant. Our test suite is therefore good to go.

Mutation Testing types

There are several types of mutations. These are explained below.

#1) Value Mutation

Here, we introduce a mutation by changing the parameter and/or constant values, usually by +/- 1.

Example: Original code (Javascript)

let arr = [2,3,4,5]
for(let i=0; i<arr.length; i++){
   if(i%2===0){
   	console.log(i*2)
   }		
}

If the above code was meant to multiply the even numbers where i<4, then value mutation would mean changing the initialization to let i=1.

Example: Mutant code (Javascript)

let arr = [2,3,4,5]
for(let i=1; i<arr.length; i++){
   if(i%2===0){
   	console.log(i*2)
   }	
}

#2) Statement Mutation

Here, we delete or duplicate a statement in a code block. We could also rearrange statements in a code block.

In an if-else block, for example, we could delete the else part or even the entire if-else block.

Example: Original code (Javascript)

let arr = [2,3,4,5]
for(let i=0; i<arr.length; i++){
   if(i%2===0){
   	console.log(i*2)  	 
   }	
}

Example: Mutant code (Javascript)

let arr = [2,3,4,5]
for(let i=0; i<arr.length; i++){
   if(i%2===0){
   	console.log(i*2)
   	console.log(i*2)
   }
}	

#3) Decision Mutation

The target here is the code that makes decisions, for example, value comparisons. We can change > to < as in our Mother-Daughter program example.

Other operators that we can switch include the following:

Advantages of Mutation Testing(MT) includes:

• MT covers a large part of the code’s logic.
• We get to test specific parts of the code, and not just paths, branches, or statements.
• With MT, we can catch even errors that are easy to bypass.
• It helps us to evaluate the quality of our test suite and adjust accordingly.

Disadvantages of Mutation Testing (MT) includes:

• It is resource-intensive as there is a huge number of mutants that are usually generated for every code block.
• We need to counter check the surviving mutants, as some are invalid.

Mutation Testing Tools

Tools come in handy to speed up the process of mutant generation. Here are some tools that we can use in MT: Stryker, Jumble, PIT, and Insure++.

More on Mutation Testing

Let’s learn from an example: 

Say, there is a hospital site that lets new users register. It reads the Date of birth or age of the patient. If it is greater than 14, assigns a general physician as their main doctor.  To do so, it invokes the ‘general physician’ function that finds the available doctor.

Now, there might be other functionality. Maybe, patients below 13 get assigned to a pediatrician and so on. But we will only take the age-over-14 case.

This is what the code might look like:

1) Read Age
2) If age>14
3) Doctor= General Physician()
4) End if

Please note that the above lines of code are not specific to any programming language and won’t run. It is just hypothetical.

As a tester, if my data-set is 14, 15, 0, 13 – some random numbers.

The target is to check if the data-set of the 4 values (14, 15, 0, and 3) is adequate to identify all possible problems with this code.

Also read => Tips to design test data before executing your test cases

How does Mutation Testing achieve this?

First and foremost, you create mutants- variations of the program. A mutant is nothing but a program that is written as a deviation. It contains a self-seeded fault.

Examples are:

• Arithmetic operator replacement
• Logical connector replacement
• Statement removal
• Relational operator replacement
• Absolute value insertion, etc.

These replacements are also called ‘Mutation Operators.’

Let me show you examples:

Mutant #1: Relational operator replacement

1) Read Age
2) If age<14 ‘Changing the > with <’
3) Doctor= General Physician()
4) End if

Mutant #2:

1) Read Age
2) If age=14 ‘Changing the > with =’
3) Doctor= General Physician()
4) End if

Mutant #3:

1) Read Age
2) If age>=14 ‘Changing the > with >=’
3) Doctor= General Physician()
4) End if

Mutant #4:

1) Read Age
2) If age<=14 ‘Changing the > with <=’
3) Doctor= General Physician()
4) End if

Mutant #5: Statement Removal

1) Read Age
2) If age=14
3) ‘remove the doctor assignment statement’
4) End if

Mutant #6: Absolute Value Insertion

1) Read Age
2) If age>14
3) Doctor= Mr.X (Absolute value insertion- let’s say X is a pediatrician)
4) End if

Mutant #7: Incorrect syntax

1) Read Age
2) If age%%14 (incorrect syntax)
3) Doctor=General Physician()
4) End if

Mutant #8: Does the same thing as the original test

1) Read Age
2) If age> 14 & age>14 ‘means the same thing as age>14’
3) Doctor= General Physician()
4) End if

Once, all the mutants are created. They are subjected to the test data-set. Our set is 14, 15, 0 and 13. Which of these mutants will our data-set find?

Find out in the below table:

(Click on image for an enlarged view)

As you can see our data value 14 finds failures when it runs against, Mutant 2, 3 and 4. Or, 14 kills mutants 2, 3 & 4. But, it is ineffective against, 1, 6 and 8.

If your data-set kills all mutants, it is effective. Otherwise, include more or better test data. It is not necessary for the each value in the data-set to kill all mutants. But together, they should kill all. For example: 14 kills 2, 3 and 4. 15 kills 1, 2 and 4. And, so on.

What about 5, 7, and 8?

Mutant #5 – is the program instance that will fail irrespective of any data value you give. This is because it will not do any programming for both valid and invalid values.

Mutant #7 – will be a compile error. Or in the case of a scripting language an error that will prevent execution.

Mutant #8 – is the same thing as the main program.

As you can see, the above mutants are not useful at all.

Therefore, mutants to avoid are:

• Syntactically incorrect/‘Still-Born’ mutants. : You need syntactically correct mutants ONLY. Example: Mutant 7
• Equivalent Mutants: The ones that do the exact same thing as the original program. Example: Mutant 8.
• Trivial Mutant: Can be killed by any data-set. Example: Mutant 5

Points To Note

• The number of mutants, even for a small program, can be many. It is a finite number, but still very large. Due to this, a subset of mutants is usually used. It is common to choose mutants randomly.
• The mutation operators listed above is not an exhaustive list. There can be many other variations. I have oversimplified the concept for easy understanding.
• The mutation operators also differ with programming languages, design, and specifications of the application.
• If there are some mutants alive at the end of the test. It means either it is an invalid mutant (like 5, 7, and 8) or the data-set was inadequate. If it is the later one, go back and change it.
• Mutation Test is a structural, white-box, and unit testing method. It uses fault-injection or fault-seeding to generate its mutants.
• There are many unit testing frameworks and tools that aid in automatic mutation testing. Some of them are:
  • Jester for JUnit.
  • Pester for Python
  • MuClipse for Eclipse, etc.

Are you thinking, if it takes this much effort, what is going to happen when I have to test large samples of code?

Mutation testing relies on two things:

• Competent Programmer Assumption: If a programmer is competent enough to write and test a small piece of code, he will be good at writing larger programs too.
• Coupling effect Assumption: If 2 units are combined to form a program and each one is good in itself, then the combination is going to be good too.

So, it focuses on the smallest unit of code and places its faith in the programmer’s skill to scale mutation testing to larger programs.

Conclusion

This tutorial covered Mutation testing definition, types, and steps to perform this testing in detail with examples. We hope you have enjoyed reading and learning about this interesting testing technique- Mutation Testing.

About the author: This article is written by STH team member Swati S.

Please share your comments, questions, and thoughts below.