Test oracle

A test oracle is whatever tells you what the right answer is. Running a test produces an output; the oracle is the mechanism that judges that output as pass or fail. Usually it is the expected result written in the test case, but for many systems working out the expected result is the hardest part of testing. This difficulty is known as the oracle problem.

Types of oracle

• Specified result. A requirement says a $200 order with 10% off totals $180. The assertion compares against that value.
• Reference implementation. Compare against a trusted alternative: a previous release, a competitor’s calculation, or a simple but slow algorithm checking a fast optimized one.
• Consistency and properties. You may not know the exact output, but you know rules it must obey. Sorting must return the same elements in non-decreasing order; encoding then decoding must return the original input.
• Stored baseline. Snapshot and visual regression tests treat an approved earlier output as correct.
• Human judgement. A tester decides whether a layout or a generated text is acceptable. Flexible, but slow and inconsistent.

Example: a property-based oracle

You cannot hand-write expected output for every possible input to a sort function, but you can check properties that any correct result has:

function checkSorted(input: number[], output: number[]): boolean {
  const sameLength = input.length === output.length;
  const ordered = output.every((v, i) => i === 0 || output[i - 1] <= v);
  const sameItems =
    [...input].sort((a, b) => a - b).join(',') === output.join(',');
  return sameLength && ordered && sameItems;
}

Why it matters

A test is only as good as its oracle. A test that clicks through checkout and asserts only that no error appeared has a very weak oracle: it will pass while charging the wrong amount. Weak oracles are a major reason that high code coverage does not guarantee a good suite. The oracle problem is also central to testing AI systems, where outputs vary and teams rely on golden datasets, rubrics or a second model acting as judge.