PIT (also called Pitest) is the usual mutation testing tool for the JVM. It generates mutants for you, using a set of built-in mutators that cover things like boundary changes, negated conditions, arithmetic operators and return values. It plugs into Maven and Gradle, and its source lives in the project repository, whose latest release when I checked was 1.30.0.
Now for an honest note about this article. I could not run PIT itself. The sandbox I used for the measurements cannot reach Maven Central, where PIT’s jars are published, and the GitHub release only ships source code. So I did the next best thing: I wrote a small script that applies PIT-style changes by hand, one at a time, and runs the tests each time. The coverage numbers below come from real JaCoCo, but the mutation numbers are from my own harness. PIT would generate its own list of mutants, and its counts could differ from mine.
The experiment: 100% coverage, zero mutants killed
The code under test
I wrote a small shipping-cost rule, the kind of logic that lives in every e-commerce codebase. The base fee is 500 cents. Parcels over 1,000 grams pay 20 cents per extra 100 grams. Express doubles the cost. Orders of 10,000 cents or more ship free.
public class ShippingCost {
private ShippingCost() { }
public static int cents(int orderCents, int weightGrams, boolean express) {
int cost = 500;
if (weightGrams > 1000) {
cost += (weightGrams - 1000) / 100 * 20;
}
if (express) {
cost = cost * 2;
}
if (orderCents >= 10_000) {
return 0;
}
return cost;
}
}
Two test suites
The first suite is the kind many of us have written on a tired Friday. It touches every branch and asserts only that the answer is sane.
public class WeakTest {
@Test public void lightStandard() { assertTrue(ShippingCost.cents(2000, 500, false) >= 0); }
@Test public void heavyExpress() { assertNotNull(ShippingCost.cents(2000, 2500, true)); }
@Test public void bigOrder() { assertTrue(ShippingCost.cents(20000, 500, false) >= 0); }
}
The second suite checks exact values, including the awkward edges of each rule.
public class StrongTest {
@Test public void lightStandard() { assertEquals(500, ShippingCost.cents(2000, 500, false)); }
@Test public void exactlyAtWeightLimit(){ assertEquals(500, ShippingCost.cents(2000, 1000, false)); }
@Test public void justOverWeightLimit() { assertEquals(520, ShippingCost.cents(2000, 1100, false)); }
@Test public void heavyExpress() { assertEquals(1600, ShippingCost.cents(2000, 2500, true)); }
@Test public void justBelowFreeLimit() { assertEquals(500, ShippingCost.cents(9999, 500, false)); }
@Test public void exactlyAtFreeLimit() { assertEquals(0, ShippingCost.cents(10000, 500, false)); }
@Test public void freeExpress() { assertEquals(0, ShippingCost.cents(20000, 500, true)); }
}
What coverage says
I ran each suite under JaCoCo 0.8.15 on JDK 25 (Temurin 25+36) with JUnit 4.13.2. Both suites pass, and both get the same report:
| Metric | Weak suite | Strong suite |
|---|---|---|
| Instructions covered | 28 of 28 | 28 of 28 |
| Branches covered | 6 of 6 | 6 of 6 |
| Lines covered | 8 of 8 | 8 of 8 |
| Methods covered | 1 of 1 | 1 of 1 |
Read that table again. By coverage alone, the two suites are identical. A dashboard would give both a gold star.
What mutation testing says
Next I applied twelve single-change mutants, one at a time, and ran both suites against each. Every mutant compiled cleanly, and I checked that each kill came from failing assertions, not from a build error.
| # | Mutator style | Change | Weak suite | Strong suite |
|---|---|---|---|---|
| 1 | Conditional boundary | weightGrams > 1000 to >= 1000 | survived | survived |
| 2 | Negated conditional | weightGrams > 1000 to <= 1000 | survived | killed |
| 3 | Math | weightGrams - 1000 to + 1000 | survived | killed |
| 4 | Math | / 100 * 20 to * 100 * 20 | survived | killed |
| 5 | Math | / 100 * 20 to / 100 / 20 | survived | killed |
| 6 | Math | cost += to cost -= | survived | killed |
| 7 | Negated conditional | if (express) to if (!express) | survived | killed |
| 8 | Math | cost * 2 to cost / 2 | survived | killed |
| 9 | Conditional boundary | orderCents >= 10_000 to > | survived | killed |
| 10 | Negated conditional | orderCents >= 10_000 to < | survived | killed |
| 11 | Return value | return 0; to return 1; | survived | killed |
| 12 | Return value | return cost; to return 0; | survived | killed |
The weak suite killed 0 of 12. The strong suite killed 11 of 12. Here is the same story as a picture.
The mutant that nobody can kill
Did you spot mutant 1? It survived even the strong suite. Is that a gap in the tests?
No, and this is a lesson worth learning early. Changing > to >= on the weight check only matters at exactly 1,000 grams. At that weight the surcharge works out to (1000 - 1000) / 100 * 20, which is zero, so the output is identical either way. No test could ever tell the two versions apart. This is called an equivalent mutant, and every mutation tool produces some of them. They are noise you learn to recognize and dismiss.
Why this keeps happening in real projects
The weak suite was not written by a villain. Look at its assertions: not negative, not null. Both are true for almost every wrong answer too. A mutant that returns 0, 1, or a negated cost sails straight through.
A few habits make this easy to fall into:
- Chasing a coverage percentage, so tests are written to execute lines rather than to pin down results.
- Asserting on the shape of a result (it exists, it is positive) instead of its value.
- Skipping boundary values, which is exactly where
>versus>=bugs live.
Notice that all three are failures of assertions, not of effort. Coverage cannot see assertions at all, which is why it stays calm while the safety net has holes.
Using PIT without drowning in reports
Mutation testing has a reputation for being slow and noisy, and that reputation is partly earned. Each mutant means running tests again. As far as I know PIT reduces the cost by using coverage data to run only the tests that actually reach the mutated line, but I have not benchmarked that myself, so measure it on your own codebase.
A sensible way to start:
- Run it on one package or one recent change first, not the whole project.
- Read the survivors, not the score. Each surviving mutant points at a specific line and a specific missing assertion.
- Dismiss equivalent mutants quickly and move on.
- Do not chase 100% mutation score. Treat it as a smoke detector, not a target.
Notice how that last point mirrors the original problem. Any number becomes useless once people optimize the number instead of the behavior it was meant to represent.
Conclusion
Here is what the experiment showed. Two test suites, both green, both reporting 100% instruction, branch and line coverage on the same class. One of them killed none of twelve hand-made mutants, and the other killed eleven, with the twelfth being an equivalent mutant that no test could catch.
Coverage tells you where tests have been. Mutation testing tells you whether they would notice a problem. They answer different questions, and only the second one is about whether your safety net holds.
So the next time a report says 100%, ask a follow-up: if I broke this line on purpose, would anything fail? Then let a tool like PIT answer it for you. Remember, my mutation numbers came from a hand-built harness, so run PIT on your own code and trust its report over my table.
