Is code coverage a good measure of test quality? How would you measure the effectiveness of a test suite?

Code coverage is a useful negative indicator and a poor positive one. Low coverage reliably tells you something is untested. High coverage does not tell you the tests are good, because coverage measures which code executed during the tests, not whether any assertion would fail if that code were wrong. To judge effectiveness I combine coverage with mutation testing and with outcome metrics such as escaped defects, flakiness and feedback time.

What coverage actually measures

• Line/statement coverage: was this line run?
• Branch coverage: was each side of each decision taken?
• Condition / MC/DC coverage: did each boolean sub-condition independently affect the outcome? (Required in some safety-critical domains.)

A test with no assertions can reach 100% line coverage:

// 100% line coverage of applyDiscount, zero verification
test('applies discount', () => {
  applyDiscount({ total: 100 }, 'SAVE10');
});

Even branch coverage cannot tell you that total > 100 should have been total >= 100 if no test uses the value 100.

Mutation testing: testing the tests

Mutation tools make small changes (mutants) to production code, such as flipping > to >= or replacing a return value, then run the suite. If no test fails, the mutant “survived”, meaning your tests would not notice that bug. The mutation score (killed / total mutants) is a much better proxy for assertion strength than coverage.

• Java: PIT (pitest)
• JavaScript/TypeScript: StrykerJS
• Python: mutmut, cosmic-ray

It is computationally expensive, so teams usually run it on changed files in PRs or on critical modules nightly, not on the whole codebase every commit.

Outcome metrics that matter

• Escaped defects: bugs found in production or late stages, and at which layer they should have been caught. This is the most honest signal, analysed per incident.
• Flakiness rate: share of failures that pass on retry. A flaky suite trains people to ignore red builds.
• Feedback time: how long from push to a trustworthy result. Slow suites get skipped.
• Failure diagnosability: how long it takes to understand why a test failed.
• Risk and requirement coverage: are the top business risks and acceptance criteria actually tested, at a sensible layer?

Metrics as targets

Goodhart’s law applies: when a measure becomes a target, it stops being a good measure. Mandating “80% coverage” leads to assertion-free tests over getters and setters. I prefer using coverage as a diff-level check (“new code in this PR has tests”) and as a map for finding gaps, combined with review of what the tests assert.

What interviewers look for

• The execution-versus-verification distinction, ideally with an example.
• Knowledge of mutation testing and its cost.
• Outcome-based metrics, especially escaped defects and flakiness.
• Maturity about metrics being misused as targets.

Follow-up questions

• What coverage threshold would you set? (Usually a ratchet or a diff-coverage rule rather than an absolute number, decided with the team.)
• How do you track flakiness? (CI history, retries reported separately, quarantine with an owner and deadline.)