Learn test quality metrics vocabulary: mutation testing, mutation score, killed mutants, branch coverage vs. line coverage, test coverage as vanity metric — the language of meaningful test quality measurement.
0 / 5 completed
1 / 5
The lead developer says: 'Test coverage is a vanity metric — 90% line coverage tells us nothing about test quality.' Why might this be true?
Line coverage (or statement coverage) measures what percentage of code lines are executed by the test suite. A test can execute a line without asserting the correct outcome. Therefore, high line coverage can give a false sense of security — the tests 'ran the code' but would not catch many real bugs. This is why experienced engineers call line coverage a vanity metric when used in isolation.
2 / 5
Your colleague explains: 'We use mutation testing to get a better measure of test quality.' What is mutation testing?
Mutation testing tools (e.g., Pitest for Java, Stryker for JavaScript/TypeScript) automatically create slightly modified versions of the source code (mutants) — for example, changing a '+' to '-', or flipping a boolean condition. They then run the test suite against each mutant. If the tests fail, the mutant is 'killed' (good). If the tests pass, the mutant 'survived' — indicating a gap in test quality.
3 / 5
The quality report shows: 'Mutation score: 78%. Killed mutant rate: 82%.' What does the mutation score represent?
The mutation score (killed mutant rate) is the percentage of generated mutants that were detected by the test suite (i.e., caused at least one test to fail). A score of 78% means 78% of the artificial code mutations were caught by the tests. The remaining 22% survived — meaning those code changes were not detected, indicating gaps in test assertions. A higher mutation score indicates a more effective test suite.
4 / 5
The tech lead says: 'We track branch coverage, not just line coverage.' What is branch coverage and why is it more meaningful?
Branch coverage (also called decision coverage) measures whether every possible branch from each conditional statement (if/else, switch/case, ternary) has been executed by at least one test. This is more meaningful than line coverage because a line containing an if/else only needs to be executed once to count toward line coverage, but branch coverage requires both the true and false paths to be tested.
5 / 5
During a code review, the developer says: 'The killed mutant rate is 82% — the remaining 18% are equivalent mutants or unkilled.' What is an 'equivalent mutant'?
An equivalent mutant is a code mutation that is syntactically different from the original but semantically identical — it does not change the observable behaviour of the program. For example, changing 'i = i + 1' to 'i += 1' produces an equivalent mutant. No test can kill an equivalent mutant because the behaviour is unchanged. Equivalent mutants are a known limitation of mutation testing and inflate the apparent 'unkilled' percentage.