LLM Evaluation

Testing LLM Applications introduced eval suites as a testing technique. This lesson goes deeper into the measurement itself: what to measure, how to build datasets that mean something, how to use a model as a judge without being misled by it, and how to tell a real improvement from noise.

Start from the task, not the metric

There is no single “quality score” for an LLM feature. Decide which dimensions matter for this task, and for each, how you will measure it:

Dimension	Question	Typical measurement
Correctness	Is the answer right?	Exact match, reference comparison, expert review
Faithfulness (groundedness)	Is every claim supported by the provided sources?	LLM-as-judge against the context, human review
Relevance	Does it address the question asked?	LLM-as-judge, human review
Completeness	Does it cover everything required?	Checklist assertions, judge with rubric
Format	Is it valid JSON, within length, in the right language?	Deterministic assertions
Safety	Does it refuse what it should and avoid leaks?	Assertions on adversarial cases, scanners
Tone and style	Does it follow the style guide?	Judge with rubric, human review
Cost and latency	Is it affordable and fast enough?	Token counts, timings

A classifier that routes support tickets needs accuracy per class. A summariser needs faithfulness and completeness. A chatbot needs most of the table. Write the dimensions down with the product owner before choosing tools.

Building evaluation datasets

Coverage and slices

A dataset should represent real usage and known risks. Tag each case with slices (topic, language, input length, user type, adversarial or not) so you can report results per slice. An overall score can hide a feature that works well in English and badly in Spanish.

Size and honesty about uncertainty

Small datasets give coarse, uncertain numbers. With 40 cases, each case is 2.5 percentage points, so a move from 85% to 87.5% is a single case and could easily be sampling noise. Grow the dataset in the areas that matter, run multiple samples, and look at which specific cases changed rather than at the headline number alone.

Hold-out sets

If you tune prompts by looking at failures on a dataset, you will eventually overfit to it: the prompt gets good at those exact cases. Keep a held-out set that you do not look at while iterating, and use it to confirm improvements generalise.

Versioning

Store datasets in version control with the eval code. Record which dataset version, prompt version and model version produced each result; otherwise results from different weeks are not comparable.

Reference-based metrics

When you have a reference answer:

• Exact match and accuracy suit classification, extraction and short factual answers. Normalise first (case, whitespace, punctuation) so formatting differences do not count as errors.
• Precision, recall and F1 suit extraction of multiple items (“list all products mentioned”).
• Overlap metrics such as BLEU and ROUGE count shared words or sequences with the reference. They were designed for translation and summarisation research and are cheap, but they reward surface similarity: an answer can share most of its words with the reference and still be wrong (“the refund window is not 30 days”).
• Embedding similarity compares meaning vectors. It is better at paraphrase, but it is also weak at catching negation and wrong numbers, and the threshold for “similar enough” needs calibration.

Use reference metrics as one signal, and check a sample of their judgments by hand before trusting them.

LLM-as-judge

An LLM-as-judge is a model prompted to grade an output against criteria. It can assess qualities rules cannot, such as faithfulness or helpfulness, and it scales better than human review. It is now an established technique, but only when its known weaknesses are managed.

Known biases

• Position bias: in pairwise comparisons (“which answer is better, A or B?”), judges tend to favour one position, often the first.
• Verbosity bias: longer, more detailed-looking answers tend to be rated higher even when they are not better.
• Self-preference bias: a model tends to rate outputs from itself, or from similar models, more favourably.

Judges can also be inconsistent across runs and swayed by confident wording.

Designing a judge

• Specific rubric: “Score 1 if every factual claim in the answer is supported by the context, otherwise 0” beats “Rate the quality from 1 to 10”.
• Small scales: binary or 3-point scales are more consistent than 10-point ones.
• One dimension per call: judge faithfulness and tone separately.
• Reasoning before verdict: ask the judge to quote the evidence before giving a score, and keep the explanation for review.
• Swap positions in pairwise comparisons and only count consistent verdicts.
• Control length: instruct the judge not to reward length, and track output length alongside scores.
• Use a different model (ideally a different model family) from the one being evaluated where possible, to reduce self-preference.

A pairwise judge with position swapping

// Stub: connect to your judge model.
async function callModel(prompt: string): Promise<string> {
  throw new Error('callModel not implemented');
}

type Verdict = 'A' | 'B' | 'tie';

async function judgeOnce(question: string, first: string, second: string): Promise<Verdict> {
  const prompt = `You compare two answers to a customer question.
Criterion: which answer is more accurate and directly useful? Do not prefer an answer
because it is longer. If they are equally good, answer "tie".

Question: ${question}
Answer 1: ${first}
Answer 2: ${second}

Reply with exactly one word: 1, 2 or tie.`;
  const raw = (await callModel(prompt)).trim().toLowerCase();
  if (raw.startsWith('1')) return 'A';
  if (raw.startsWith('2')) return 'B';
  return 'tie';
}

export async function judgePair(question: string, a: string, b: string): Promise<Verdict | 'inconsistent'> {
  const forward = await judgeOnce(question, a, b);
  const backward = await judgeOnce(question, b, a);
  // In the backward call, "A" means the first shown answer, which is b.
  const backwardMapped: Verdict = backward === 'A' ? 'B' : backward === 'B' ? 'A' : 'tie';
  return forward === backwardMapped ? forward : 'inconsistent';
}

Report the share of inconsistent verdicts. A high rate means the judge cannot reliably tell the answers apart on this criterion, which is useful information in itself.

Calibrate against humans

A judge is a measuring instrument; check it before relying on it. Have people label a sample of outputs using the same rubric, run the judge on the same sample and measure agreement:

export function agreement(human: number[], judge: number[]) {
  if (human.length !== judge.length) throw new Error('length mismatch');
  const agree = human.filter((h, i) => h === judge[i]).length;
  return agree / human.length;
}

Look beyond raw agreement: where do they disagree, and in which direction? A judge that passes answers humans fail is more dangerous than one that is overly strict. For a more rigorous measure that accounts for chance agreement, statistics such as Cohen’s kappa are commonly used. Re-calibrate when you change the judge prompt or model.

Evaluating retrieval-augmented generation (RAG)

RAG systems retrieve documents and pass them to the model. Failures can come from either half, so measure both:

• Retrieval quality: did the retriever find the relevant documents (recall), and was what it returned mostly relevant (precision)? Ragas calls metrics in this family context precision and context recall.
• Generation quality: is the answer supported by the retrieved context (faithfulness), and does it address the question (answer relevancy)?

Separating them tells you where to fix: a faithful answer built on the wrong documents is a retrieval problem; a hallucinated answer despite good documents is a generation problem. Ragas implements metrics along these lines, and DeepEval and promptfoo offer comparable checks; read each tool’s documentation for exactly how a metric is computed before comparing numbers across tools.

Comparing versions

When a prompt, model or retriever changes:

1. Run old and new on the same dataset version with the same sample count.
2. Compare per case: list cases that went from pass to fail and fail to pass. Read the regressions.
3. Compare per slice: an overall gain can hide a loss in one language or topic.
4. Check cost and latency alongside quality.
5. Confirm on the held-out set.
6. For high-stakes changes, add human review of a sample of changed outputs.

Practice exercise

A team reports: “We switched our summariser prompt and the LLM-as-judge quality score went from 3.9 to 4.3 out of 5 on our 25-case dataset. The new summaries are also about twice as long. Ship it?” Write the questions and checks you would require before agreeing.

Show a model answer

1. Verbosity bias: the summaries doubled in length and the judge may be rewarding length. Re-run with a judge prompt that explicitly ignores length, and compare at matched lengths or add a length limit check.
2. Self-preference: is the judge the same model as the summariser? If so, repeat with a different model family.
3. Rubric: what does a “4” mean? Replace the 1 to 5 overall score with specific binary criteria: faithfulness (every claim supported), completeness (key points from a checklist present), and length within limit.
4. Sample size and noise: 25 cases and a single run each is a small sample. Run several samples per case, show per-case changes and read every regression.
5. Calibration: has the judge been checked against human ratings for this task? Have two people blind-rate a sample of old versus new summaries.
6. Held-out data: was the prompt tuned on these same 25 cases? Confirm on cases not used during iteration.
7. Cost and latency: twice the length means more output tokens and slower responses; is that acceptable?

Ship only if faithfulness does not regress, completeness improves on held-out data and humans prefer the new outputs at an acceptable cost.

Key takeaways

• Choose quality dimensions from the task, then pick metrics for each; there is no single quality score.
• Build sliced, versioned datasets with a held-out set, and be honest about uncertainty on small datasets.
• Reference metrics are cheap but reward surface similarity; validate them by hand.
• LLM-as-judge needs specific rubrics, small scales, position swapping, length control and a different model where possible.
• Calibrate judges against human labels and re-calibrate when they change.
• Evaluate RAG retrieval and generation separately, and compare versions per case and per slice.