Mutation testing

Mutation testing tests your tests. A tool makes many copies of the code, each with one small change, such as >= turned into >, + into -, or a method body replaced with return null. Each changed copy is a mutant. The test suite runs against every mutant. If at least one test fails, the mutant is killed. If all tests still pass, the mutant survived, which means the tests would not notice that bug.

Mutation score = killed mutants / total mutants (excluding equivalent mutants).

Example

boolean isAdult(int age) {
    return age >= 18;
}

Tests: isAdult(30) is true, isAdult(10) is false. Line and branch coverage are both 100%.

The tool creates a mutant with age > 18. Both tests still pass, so the mutant survives. That exposes a missing boundary test: nobody checked isAdult(18). Adding it kills the mutant. Coverage could not tell you this; mutation testing can.

Tools

• Java: PIT (pitest), with Maven and Gradle plugins and JUnit 5 support.
• JavaScript/TypeScript: StrykerJS, which also has .NET and Scala versions.
• Python: mutmut and Cosmic Ray.

Practical limits

• Cost. Every mutant is a test run. Tools reduce this by running only the tests that cover the mutated line and running incrementally on changed code, but it is still much slower than a normal run. Most teams run it nightly or on changed files in a pull request, not on every commit for the whole codebase.
• Equivalent mutants. Some changes do not alter behavior at all, for example turning i < n into i != n in a loop where i always counts up by one from below n. No test can kill them, so they have to be reviewed and excluded by hand.
• Best at unit level. Mutation testing is designed for fast, deterministic unit tests. Running it against slow UI tests is rarely practical.

Use surviving mutants as a to-do list for missing assertions, especially in business logic, rather than chasing a perfect score.