flowchart LR
TRACE["Production traces<br/>(successes and failures)"] -. "sample · redact · dedupe" .-> REVIEW
DESIGN["Designed coverage<br/>(requirements · threats)"] --> REVIEW["Human review<br/>+ dataset version"]
REVIEW --> SET[("Versioned eval set")]
SET --> RUN["Isolated baseline<br/>and candidate trials"]
RUN --> GATE{"Release rule met?"}
GATE -->|no| REVIEW
GATE -->|yes| SHIP["Eligible to ship"]
22 Evaluation: From Traces to a Release Gate
A support-agent team changes a prompt and a tool schema, and the numbers look like a promotion. The candidate passes 41 of 48 trials; the shipped baseline passes 35. A model judge agrees with adjudicated human labels 89 percent of the time. Every language slice improves or holds. Someone opens a pull request titled “ship it.”
It should not ship. One task came from a real incident: refund an eligible order, but only after looking it up. On that task the baseline succeeds four times out of four; the candidate succeeds twice, performs one refund with no lookup, and writes one poor explanation. Across the twelve independent tasks the candidate’s mean improvement is 12.5 points — but a task-paired 95 percent interval on that number runs from about -2 to +23 points, and the team’s predeclared tolerance for regression was 2 points. So the candidate fails a must-not-break case and, separately, fails the uncertainty rule. A dashboard showing only the aggregate would have hidden both facts. This is the whole subject of the chapter in one sentence: a metric is evidence, not a decision.
We are now ready to build the machine that turns traces into a defensible decision, in bare Python, over a shipped fixture of replayed support-agent runs. The parts are (i) the traces-to-dataset flywheel that promotes a real run into a versioned, isolated task; (ii) a grader that scores each trial as a strict conjunction of outcome, schema, policy, and quality — so an unsafe success cannot pass; (iii) a model judge we calibrate with Cohen’s kappa before we trust it, then probe for position and verbosity bias; (iv) the pass@k / pass^k split between best-of-k capability and repeated-run reliability; (v) a paired cluster bootstrap that puts an honest interval and a power estimate on the delta; (vi) trajectory, disparate-error-burden, and error-analysis measurements that survive aggregation; and (vii) a single release report that predeclares its rule and exits non-zero in CI. Everything runs offline from committed fixtures; where a synthetic fixture stands in for real production data, we say so and we never dress it up as a recorded run.
22.1 From traces to a versioned evaluation set
An evaluation set should begin with designed coverage and then learn from real failures. Start only from production traces and you miss behaviors that have not happened yet, including the rare severe ones. Start only from invented prompts and you miss the distribution, ambiguity, and dependency structure of real use. You want both, and you want a controlled door between them.
Four words will recur, so we pin them now. A trace is the immutable record of one run: input, messages, tool calls and results, approvals, errors, timing, output. A task is the versioned specification that says how a run begins and how it is graded. A trial is one attempt at a task. An outcome is the authoritative state after the attempt — database rows, files, a passing test suite, a receipt — owned by the environment, not by the transcript. The transcript explains behavior; the outcome establishes what happened.
The flywheel is a promotion process, not log ingestion. You sample failures and successes, strip or tokenize private data, group near-duplicates, write the intended initial state and the graders, have a human review the result, and stamp a dataset version. Figure 22.1 draws the one invariant that matters: nothing enters the decision set without review and a version.
We work from a frozen snapshot of that process: twelve promoted support tasks, replayed rather than executed, so the chapter is deterministic. Each task carries the initial state, the reference tool sequence, the expected final state, a language slice, and a golden flag for incidents that must never regress. Loading the file and reading one record shows exactly what a human added when promoting the trace.
# @save
from __future__ import annotations
import json
import math
import random
from collections import Counter, defaultdict
from pathlib import Path
from statistics import NormalDist, fmean, stdev
from typing import Any
FIXTURES = Path("code/ch22/fixtures")
def load_jsonl(path: Path) -> list[dict[str, Any]]:
"""Load one JSON object per non-blank line of a JSON Lines file.
Args:
path: The ``.jsonl`` fixture to read.
Returns:
The parsed objects, in file order.
Raises:
ValueError: If the file is empty or any line is not a JSON object,
which we treat as a corrupt eval set rather than a silent skip.
"""
rows = [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines() if line.strip()]
if not rows or any(not isinstance(row, dict) for row in rows):
raise ValueError(f"{path} must contain one JSON object per line")
return rowstasks = load_jsonl(FIXTURES / "traces.jsonl")
print(f"{len(tasks)} promoted tasks")
first = tasks[0]
print("task_id:", first["task_id"], "| slice:", first["slice"], "| golden:", first["golden"])
print("expected final state:", first["expected_state"])
print("reference tool path :", first["expected_tools"])
print("one candidate trial :", first["runs"]["candidate"][2])12 promoted tasks
task_id: refund-en | slice: locale-en | golden: True
expected final state: done
reference tool path : ['lookup_order', 'refund_order']
one candidate trial : {'final_state': 'done', 'schema_valid': True, 'policy_ok': False, 'judge_label': 'PASS', 'tools': ['refund_order']}The expected_state and expected_tools fields are the human contribution: a raw trace does not know what should have happened, and the promotion step is where a reviewer writes it down. Notice the third candidate trial already looks suspicious — it reaches final_state: done but its tool list skips lookup_order. We return to that trial in the next section.
Two disciplines make an eval set trustworthy, and both are easy to skip. The first is isolation. Suppose trial 1 refunds order O-41, and trial 2 runs a broken policy that merely claims “refund completed.” If the two share a database, trial 2 passes on the state trial 1 left behind. The same leak hides in caches, queues, browser profiles, clocks, and rate-limit counters. Every trial needs its own clean namespace, and the harness should refuse to score a run whose environment identity was reused. The second is an eval of the eval: run a known-good policy, a known-bad policy, and a contamination policy that only claims success, and mutate one required fact, argument, or final-state field to confirm the matching grader actually fails. A benchmark with invalid tasks is not repaired by a larger sample.
22.2 The cheapest grader that captures the requirement
A grader maps the evidence from one trial to a score and a reason. The rule is to use the least subjective grader that still measures the requirement — not only to save money, but because deterministic graders are reproducible, debuggable, fast enough to run on every change, and hard for fluent output to talk around. Figure 22.2 lays the options on the two axes that decide between them.
Show the code that draws this figure
import matplotlib.pyplot as plt
from matplotlib_inline.backend_inline import set_matplotlib_formats
set_matplotlib_formats("svg", metadata={"Date": None})
plt.rcParams["svg.hashsalt"] = "chapter-22"
rungs = [
("exact / normalized match", 1, 1),
("schema + structural check", 2, 2),
("environment / invariant check", 3, 2.4),
("reference-guided score", 4, 3.6),
("calibrated model judge", 5, 4.5),
("human review", 6, 5.2),
]
fig, ax = plt.subplots(figsize=(6.4, 3.8))
xs = [r[1] for r in rungs]
ys = [r[2] for r in rungs]
ax.plot(xs, ys, "-", color="0.7", zorder=1)
ax.scatter(xs, ys, s=60, color="#2a7f9e", zorder=2)
for label, x, y in rungs:
ax.annotate(label, (x, y), textcoords="offset points", xytext=(8, -2), fontsize=8)
ax.set_xlabel("cost per graded trial ->")
ax.set_ylabel("subjectivity / manipulability ->")
ax.set_xlim(0.5, 8.8)
ax.set_ylim(0.5, 6.0)
ax.grid(alpha=0.25)
fig.tight_layout()
plt.show()
The mistake is to collapse “good support answer” into one vague judge prompt. Decompose it instead: the final state matches the goal; the action obeyed policy; the structured response parses; and the explanation clears a human-authored rubric. Our fixture records exactly those four component signals per trial, and the release success is their conjunction. We grade one trial first, in the open, so the conjunction is concrete before we run it 96 times.
# @save
def trajectory_f1(actual: list[str], expected: list[str]) -> float:
"""Multiset F1 between the tools a trial used and the reference tool set.
This is a *diagnostic*, not a verdict: order-insensitive overlap between
the tools called and the reference path. A high value does not prove
success and a low value does not prove failure; it flags trajectories
worth inspecting.
Args:
actual: Tool names the trial called, in order.
expected: The reference tool names for the task.
Returns:
The multiset F1 in ``[0, 1]``; ``0.0`` when both sides are empty of
overlap, ``1.0`` for an exact multiset match.
"""
overlap = sum((Counter(actual) & Counter(expected)).values())
precision = overlap / len(actual) if actual else 0.0
recall = overlap / len(expected) if expected else 1.0
if precision + recall == 0:
return 0.0
return 2 * precision * recall / (precision + recall)
def grade_trial(run: dict[str, Any], task: dict[str, Any]) -> dict[str, Any]:
"""Score one trial against the task's four component graders.
Release ``success`` is the conjunction of four independent checks:
the authoritative final state matches, the structured output is schema
valid, the recorded action obeyed policy, and the calibrated judge passed.
Component scores are kept even when the conjunction is false, because they
are the raw material of error analysis.
Args:
run: One replayed trial: final state, schema flag, policy flag, judge
label, and the tools it called.
task: The versioned task, supplying the expected state and tool path.
Returns:
A dict of the four component booleans, the combined ``success``, and
the diagnostic ``trajectory_f1``.
"""
state_pass = run["final_state"] == task["expected_state"]
schema_pass = bool(run["schema_valid"])
policy_pass = bool(run["policy_ok"])
judge_pass = run["judge_label"] == "PASS"
return {
"state_pass": state_pass,
"schema_pass": schema_pass,
"policy_pass": policy_pass,
"judge_pass": judge_pass,
"success": state_pass and schema_pass and policy_pass and judge_pass,
"trajectory_f1": trajectory_f1(list(run["tools"]), list(task["expected_tools"])),
}Now the suspicious trial from the last section, next to a clean one from the same task.
refund = tasks[0] # the golden "refund-en" incident task
clean = grade_trial(refund["runs"]["candidate"][0], refund)
unsafe = grade_trial(refund["runs"]["candidate"][2], refund)
for name, g in [("clean trial", clean), ("skipped-lookup trial", unsafe)]:
print(f"{name:20} state={g['state_pass']} schema={g['schema_pass']} "
f"policy={g['policy_pass']} judge={g['judge_pass']} -> success={g['success']}")clean trial state=True schema=True policy=True judge=True -> success=True
skipped-lookup trial state=True schema=True policy=False judge=True -> success=FalseThe second trial reaches the requested refund state and passes schema and the judge, but it violated policy by refunding without the required lookup. It is an unsafe success: outcome-only grading would count it, and the release construct must not. This is the whole argument for a conjunction rather than an average — a policy failure cannot be outvoted by three components that happened to look fine. The full loader applies grade_trial to every trial and, critically, refuses the fixture if two trials ever claim the same environment identity.
# @save
def grade_traces(path: Path = FIXTURES / "traces.jsonl") -> list[dict[str, Any]]:
"""Expand every task's repeated trials into graded, isolated records.
Each trial is assigned a unique ``environment_id`` of
``snapshot:system:trial``; a repeat raises, because a shared environment
lets a later trial pass on state an earlier one left behind. Every task
must supply at least two trials per system so repeated-run reliability is
estimable.
Args:
path: The JSON Lines task fixture.
Returns:
One graded record per trial, carrying task identity, slice, the golden
flag, the four component scores, ``success``, and trajectory diagnostics.
Raises:
ValueError: On a duplicate task id, a reused environment identity, or a
system with fewer than two trials.
"""
rows: list[dict[str, Any]] = []
task_ids: set[str] = set()
environments: set[str] = set()
for task in load_jsonl(path):
task_id = str(task["task_id"])
if task_id in task_ids:
raise ValueError(f"duplicate task_id: {task_id}")
task_ids.add(task_id)
for system in ("baseline", "candidate"):
runs = task["runs"][system]
if len(runs) < 2:
raise ValueError(f"{task_id}/{system} needs repeated trials")
for trial, run in enumerate(runs):
environment_id = f"{task['snapshot']}:{system}:{trial}"
if environment_id in environments:
raise ValueError(f"environment reused: {environment_id}")
environments.add(environment_id)
rows.append({
"task_id": task_id,
"slice": task["slice"],
"golden": bool(task["golden"]),
"system": system,
"trial": trial,
"environment_id": environment_id,
**grade_trial(run, task),
})
return rowsrows = grade_traces()
print(f"{len(rows)} graded trials across "
f"{len({r['environment_id'] for r in rows})} isolated environments")
unsafe = [r for r in rows if r["state_pass"] and not r["policy_pass"]]
print(f"unsafe successes (goal state reached, policy violated): {len(unsafe)}")
for r in unsafe:
print(f" {r['task_id']:12} {r['system']:9} trial {r['trial']}")96 graded trials across 96 isolated environments
unsafe successes (goal state reached, policy violated): 2
refund-en candidate trial 2
cancel-es baseline trial 3Twelve tasks, two systems, four trials each: 96 records, 96 distinct environments. The two unsafe successes are exactly the trials outcome-only grading would have miscounted. One last distinction the ladder hides: a grader is not a guard. Our policy grader detects that a recorded action was disallowed; a runtime authorization gate — the loop from Chapter 16, hardened in Chapter 24 — prevents it. Evaluation measures whether the complete system, gate included, behaved correctly.
22.3 Calibrating a model judge before trusting it
The judge label above came from an LLM-as-judge: a model grading another model’s output. Treat it as an unvalidated classifier, never an oracle. It has a prompt, a parser, false positives, false negatives, and failure slices, and every change to those inputs produces a new judge that must earn trust again. The first thing to establish is that its accuracy against human labels tells you almost nothing on its own.
Suppose 90 percent of a calibration set genuinely deserves PASS, and we deploy the laziest judge imaginable: one that says PASS every time. It agrees with the humans 90 percent of the time. Reported as accuracy, that judge looks excellent; as a discriminator it is worthless, because it never once identified a failure. Cohen’s kappa is the fix: it corrects the observed agreement p_o by the agreement p_e two raters would reach by chance given their label frequencies,
\kappa = \frac{p_o - p_e}{1 - p_e}, \qquad p_e = \sum_{c \in \mathcal{C}} p^{(H)}_c\, p^{(J)}_c, \tag{22.1}
where \mathcal{C} is the label set, p^{(H)}_c the human frequency of class c, and p^{(J)}_c the judge frequency (Cohen 1960). Kappa is 1 for perfect agreement, 0 when agreement is exactly what the marginals predict, and can go negative. We implement it and point it straight at the always-PASS judge.
# @save
def cohen_kappa(first: list[str], second: list[str]) -> float:
"""Chance-corrected agreement between two categorical raters.
Implements @eq-ch22-kappa: observed agreement discounted by the agreement
expected from the two raters' marginal label frequencies. A judge that
always emits the majority label scores near zero however high its raw
accuracy, which is exactly why kappa, not accuracy, is the calibration
metric. Kappa can be negative when raters agree less than chance.
Args:
first: One rater's labels (for example, adjudicated human labels).
second: The other rater's labels, index-aligned with ``first``.
Returns:
Cohen's kappa in ``[-1, 1]``; ``1.0`` only when both raters are
perfectly and non-trivially in agreement.
Raises:
ValueError: If the label lists are empty or of unequal length.
"""
if not first or len(first) != len(second):
raise ValueError("raters need equal, non-empty label lists")
labels = set(first) | set(second)
observed = sum(a == b for a, b in zip(first, second)) / len(first)
ca, cb = Counter(first), Counter(second)
expected = sum((ca[label] / len(first)) * (cb[label] / len(first)) for label in labels)
return 1.0 if expected == 1.0 and observed == 1.0 else (observed - expected) / (1.0 - expected)human = ["PASS"] * 90 + ["FAIL"] * 10
always_pass = ["PASS"] * 100
agreement = fmean(h == j for h, j in zip(human, always_pass))
print(f"always-PASS judge: raw agreement = {agreement:.2f} kappa = {cohen_kappa(human, always_pass):.2f}")always-PASS judge: raw agreement = 0.90 kappa = 0.00Ninety percent agreement, kappa exactly zero. The raw number was lying, and kappa says so in one figure. Now the judge that actually ships with our fixture. Its 100 calibration labels are stored as compressed contingency cells; judge_report expands them, computes kappa against the human labels, and — because a global kappa can still hide a blind spot in the rare severe class — separately reports failure recall (of the runs a human failed, how many did the judge catch?), the per-slice agreements, and a pairwise position-flip rate we explain in a moment.
# @save
def judge_report(path: Path = FIXTURES / "judge_calibration.json") -> dict[str, Any]:
"""Summarize pointwise calibration and pairwise position consistency.
Expands the compressed contingency cells into aligned human/judge label
lists, then reports the four numbers a release rule actually consults:
chance-corrected agreement (kappa), recall on the human-labeled failure
class, per-slice agreement, and the pairwise position-flip rate. Failure
recall and slice agreement are reported separately because a healthy global
kappa can still hide a judge that misses the rare severe class or fails in
one language.
Args:
path: The judge-calibration JSON fixture.
Returns:
A report dict with ``n``, ``agreement``, ``kappa``, ``fail_recall``,
``position_flip_rate``, and per-slice ``slice_agreement``.
"""
payload = json.loads(path.read_text(encoding="utf-8"))
human: list[str] = []
judge: list[str] = []
by_slice: dict[str, list[bool]] = defaultdict(list)
for cell in payload["pointwise_cells"]:
count = int(cell["count"])
human.extend([cell["human"]] * count)
judge.extend([cell["judge"]] * count)
by_slice[cell["slice"]].extend([cell["human"] == cell["judge"]] * count)
fail_total = sum(label == "FAIL" for label in human)
fail_caught = sum(a == b == "FAIL" for a, b in zip(human, judge))
pair_total = sum(int(cell["count"]) for cell in payload["pairwise_cells"])
pair_flips = sum(int(cell["count"]) for cell in payload["pairwise_cells"]
if cell["ab"] != cell["ba_normalized"])
return {
"n": len(human),
"agreement": fmean(a == b for a, b in zip(human, judge)),
"kappa": cohen_kappa(human, judge),
"fail_recall": fail_caught / fail_total,
"position_flip_rate": pair_flips / pair_total,
"slice_agreement": {name: fmean(v) for name, v in sorted(by_slice.items())},
}judge = judge_report()
print(f"n = {judge['n']} agreement = {judge['agreement']:.2f} kappa = {judge['kappa']:.3f}")
print(f"failure recall = {judge['fail_recall']:.2f} position-flip rate = {judge['position_flip_rate']:.2f}")
print("per-slice agreement:", {k: round(v, 3) for k, v in judge["slice_agreement"].items()})n = 100 agreement = 0.89 kappa = 0.736
failure recall = 0.80 position-flip rate = 0.12
per-slice agreement: {'locale-en': 0.9, 'locale-es': 0.867, 'mixed-language': 0.9}The contrast between the two judges is the point of the section, so we draw it.
Show the code that draws this figure
labels = ["raw agreement", "kappa", "failure recall"]
always = [0.90, 0.00, 0.00]
fixture = [judge["agreement"], judge["kappa"], judge["fail_recall"]]
x = range(len(labels))
fig, ax = plt.subplots(figsize=(6.2, 3.4))
ax.bar([i - 0.19 for i in x], always, 0.38, label="always-PASS judge",
color="white", edgecolor="black", hatch="//")
ax.bar([i + 0.19 for i in x], fixture, 0.38, label="fixture judge", color="#2a7f9e")
ax.set_xticks(list(x))
ax.set_xticklabels(labels)
ax.set_ylim(0, 1.05)
ax.set_ylabel("value")
ax.legend()
fig.tight_layout()
plt.show()
A judge that compares two answers — a pairwise judge, which often matches a release comparison — introduces a bias the pointwise metrics cannot see: it may favor whichever answer sits in position A regardless of content. We detect it by presenting both answer orders on a calibration subset and normalizing the winner back to answer identity; a verdict that changes when only the order changed is a position flip. The fixture stores that pairwise experiment directly.
pairwise = json.loads((FIXTURES / "judge_calibration.json").read_text(encoding="utf-8"))["pairwise_cells"]
pair_total = sum(c["count"] for c in pairwise)
pair_flips = sum(c["count"] for c in pairwise if c["ab"] != c["ba_normalized"])
print(f"pairwise items: {pair_total} order-sensitive verdicts: {pair_flips} "
f"position-flip rate: {pair_flips / pair_total:.2f}")pairwise items: 25 order-sensitive verdicts: 3 position-flip rate: 0.12Randomizing answer order in production prevents a fixed system from always sitting in the favored seat, and double-judging measures consistency — but neither trick removes the biases that are about content rather than position. The most common is verbosity: a judge that mistakes length for quality (Zheng et al. 2023). Swapping order will never catch it, because the longer answer is longer in both orders. You catch it with a length-matched control — pairs where the longer answer is deliberately the worse one — and measure how often the judge still prefers length. Here is that control on a small representative set.
# representative length-matched control: in every pair the longer answer is the
# WRONG one, so a length-blind judge should never pick "long".
control = [("long", "short"), ("short", "short"), ("long", "short"),
("short", "short"), ("short", "short")]
verbosity_rate = fmean(pick == "long" for pick, _correct in control)
print(f"judge prefers the longer (wrong) answer on {verbosity_rate:.0%} of length-matched pairs")judge prefers the longer (wrong) answer on 40% of length-matched pairsForty percent is a verbosity problem that a position-flip audit would have scored clean. The general rule: name each bias you care about — position, verbosity, self-preference, style — and build a targeted control for it, because a single agreement number averages all of them into invisibility. And recalibrate whenever the judge model, prompt, or task distribution moves materially. “We switched to a stronger model” is a reason to recalibrate, not evidence that you may skip it.
22.4 Capability is not reliability: pass@k and pass^k
Sampling the same task repeatedly answers two opposite questions, and confusing them is how a system ships that demos beautifully and fails in production. pass@k asks whether at least one of k attempts succeeds — accessible capability, the right metric when the system really will draw k candidates and a trustworthy verifier can pick a good one. pass^k (“pass to the k”) asks whether every one of k attempts succeeds — reliability, the right metric when a user expects the same policy-following behavior on every ordinary interaction. If each run succeeds independently with probability p,
\operatorname{pass@}k = 1 - (1 - p)^k, \qquad \operatorname{pass}^{k} = p^{k}. \tag{22.2}
One rises with k, the other falls. At p = 0.8, pass@8 is about 0.83’s complement away from 1 — essentially certain — while pass^8 is about 0.17. Same system, same p: highly capable under best-of-eight selection, and unreliable across eight plain uses. On a finite eval set we never know p; we observe c successes in n runs of a task and estimate without replacement,
\widehat{\operatorname{pass@}k} = 1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}, \qquad \widehat{\operatorname{pass}^{k}} = \frac{\binom{c}{k}}{\binom{n}{k}}. \tag{22.3}
The first is the chance a size-k subset of the observed runs contains a success; the second is the chance all k drawn runs are successes (Chen et al. 2021; Yao et al. 2024). Both are per-task quantities you average across tasks — never pool all successes first, or an easy task with many runs dominates the mean.
# @save
def pass_at_k_estimate(n: int, successes: int, k: int) -> float:
"""Finite-sample pass@k: the chance a size-k draw contains a success.
Implements the without-replacement estimator of @eq-ch22-passk-estimate.
It answers "if we drew k of these n observed runs, would at least one
succeed?" and measures accessible capability, appropriate only when the
system truly produces k candidates and a trustworthy verifier selects one.
Average per-task values across tasks; do not pool successes first.
Args:
n: Number of observed runs of the task.
successes: How many of those runs succeeded.
k: Draw size, ``1 <= k <= n``.
Returns:
The estimated pass@k in ``[0, 1]``.
Raises:
ValueError: If ``k`` or ``successes`` fall outside their valid range.
"""
if not 1 <= k <= n or not 0 <= successes <= n:
raise ValueError("require 1 <= k <= n and 0 <= successes <= n")
failures = n - successes
return 1.0 if failures < k else 1.0 - math.comb(failures, k) / math.comb(n, k)
def pass_pow_k_estimate(n: int, successes: int, k: int) -> float:
"""Finite-sample pass^k: the chance all k drawn runs succeed.
Implements the without-replacement estimator of @eq-ch22-passk-estimate.
It answers "if we drew k of these n observed runs, would every one
succeed?" and measures repeated-run reliability, the right metric when a
user expects the same behavior on every interaction rather than the best of
several.
Args:
n: Number of observed runs of the task.
successes: How many of those runs succeeded.
k: Draw size, ``1 <= k <= n``.
Returns:
The estimated pass^k in ``[0, 1]``; ``0.0`` when fewer than ``k`` runs
succeeded.
Raises:
ValueError: If ``k`` or ``successes`` fall outside their valid range.
"""
if not 1 <= k <= n or not 0 <= successes <= n:
raise ValueError("require 1 <= k <= n and 0 <= successes <= n")
return 0.0 if successes < k else math.comb(successes, k) / math.comb(n, k)With four trials per task, the two estimators tell visibly different stories about the same runs.
print(f"{'c/n':>5} {'pass_rate':>10} {'pass@2':>8} {'pass^2':>8}")
for c in (4, 3, 2):
print(f"{c}/4 {c / 4:>10.3f} {pass_at_k_estimate(4, c, 2):>8.3f} {pass_pow_k_estimate(4, c, 2):>8.3f}") c/n pass_rate pass@2 pass^2
4/4 1.000 1.000 1.000
3/4 0.750 1.000 0.500
2/4 0.500 0.833 0.167Three successes out of four reads as pass@2 = 1.0 (you can almost always find a good one in a pair) but pass^2 = 0.5 (half of all pairs contain a failure). The analytic curves in Figure 22.4 make the divergence unmistakable across k.
Show the code that draws this figure
ks = list(range(1, 9))
fig, axes = plt.subplots(1, 2, figsize=(9, 3.4), sharey=True)
for p, marker in ((0.60, "o"), (0.80, "s"), (0.95, "^")):
axes[0].plot(ks, [1 - (1 - p) ** k for k in ks], marker=marker, label=f"p={p:.2f}")
axes[1].plot(ks, [p ** k for k in ks], marker=marker, label=f"p={p:.2f}")
axes[0].set_title("pass@k: at least one succeeds")
axes[1].set_title("pass$^k$: every run succeeds")
for axis in axes:
axis.set(xlabel="k repeated runs", ylabel="probability", ylim=(0, 1.02))
axis.grid(alpha=0.25)
axes[1].legend(title="single-run rate")
fig.tight_layout()
plt.show()
So the reporting rule is not cosmetic. State k, the sampling policy, the verifier, and whether you averaged per task — and pick the metric that matches how the system is actually served. Independence is an approximation the equations make and reality breaks: a provider outage, an ambiguous task, or a shared retrieval defect makes failures cluster, so a hundred samples against one broken database snapshot are not a hundred independent pieces of deployment evidence. To aggregate honestly we need one more object: the per-task summary.
# @save
def task_metrics(rows: list[dict[str, Any]], system: str, k: int = 2) -> dict[str, dict[str, float]]:
"""Aggregate a system's repeated trials within each independent task.
The task, not the trial, is the unit of independence, so we collapse each
task's repeats into one cluster carrying its pass rate, the finite-sample
pass@k and pass^k estimates, and mean trajectory F1. ``k`` is fixed at 2 to
suit the four-trial fixture; a production suite sizes ``k`` to its own trial
budget.
Args:
rows: Graded trial records from :func:`grade_traces`.
system: Which system's rows to aggregate (``"baseline"`` or ``"candidate"``).
k: Draw size for the repeated-run estimators.
Returns:
A mapping from task id to its per-task metrics.
"""
grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
for row in rows:
if row["system"] == system:
grouped[row["task_id"]].append(row)
result: dict[str, dict[str, float]] = {}
for task_id, trials in sorted(grouped.items()):
n = len(trials)
successes = sum(row["success"] for row in trials)
result[task_id] = {
"pass_rate": successes / n,
"pass_at_k": pass_at_k_estimate(n, successes, k),
"pass_pow_k": pass_pow_k_estimate(n, successes, k),
"trajectory_f1": fmean(row["trajectory_f1"] for row in trials),
}
return result22.5 Uncertainty lives at the task level
“Candidate +12.5 points” is a point estimate, and a point estimate is not a decision. The observed tasks are a sample from an intended population, each run may be stochastic, and the judge adds label noise; the interval has to respect the structure that produced the numbers. Two moves do most of the work. First, pair: score each task under both systems and take the difference, so task difficulty cancels out of the contrast. Second, cluster: resample the task, keeping its repeats together, because the task is the independent unit and the trial is not. For task i with per-system scores s_i^{(B)} and s_i^{(C)},
d_i = s_i^{(C)} - s_i^{(B)}, \qquad \bar d = \frac{1}{N} \sum_{i=1}^{N} d_i, \qquad \operatorname{SE}_{\text{cluster}} = \frac{s_d}{\sqrt{N}}, \tag{22.4}
where s_d is the standard deviation of the N task deltas. The paired cluster bootstrap resamples task ids with replacement and reads percentiles off the distribution of \bar d.
# @save
def percentile(values: list[float], q: float) -> float:
"""Linearly interpolated ``q``-quantile of a list of numbers.
Args:
values: The sample to summarize; need not be sorted.
q: The quantile in ``[0, 1]`` (``0.025`` for a 95% lower bound).
Returns:
The interpolated quantile, matching the percentile convention used by
the bootstrap interval so the reported bounds are reproducible.
"""
ordered = sorted(values)
position = (len(ordered) - 1) * q
low = int(position)
high = min(low + 1, len(ordered) - 1)
return ordered[low] + (ordered[high] - ordered[low]) * (position - low)
def paired_cluster_uncertainty(
baseline: dict[str, dict[str, float]],
candidate: dict[str, dict[str, float]],
*,
resamples: int = 10_000,
seed: int = 22,
) -> dict[str, Any]:
"""Bootstrap the paired task delta and report an 80%-power MDE.
Pairs the two systems on shared task ids, resamples *task ids* with
replacement (keeping each task's repeats together, so the unit of
resampling is the unit of independence), and reads a 95% interval off the
bootstrap distribution of the mean delta. The cluster standard error of
@eq-ch22-cluster-se gives a scale check and an approximate minimum
detectable effect at 80% power. The observed tasks must be an exchangeable
sample from the contract's population for the interval to mean anything; a
hand-curated threat suite is not, and its failures are reported separately.
Args:
baseline: Per-task metrics for the baseline system.
candidate: Per-task metrics for the candidate, over the same task ids.
resamples: Bootstrap resample count.
seed: Seed making the bootstrap reproducible.
Returns:
A dict with the point delta, the 95% ``low``/``high`` bounds, the
cluster SE, the 80%-power ``mde_80``, and the per-task ``task_deltas``.
Raises:
ValueError: If the two systems do not cover the same two-or-more tasks.
"""
if set(baseline) != set(candidate) or len(baseline) < 2:
raise ValueError("paired systems need the same two or more task ids")
deltas = [candidate[key]["pass_rate"] - baseline[key]["pass_rate"] for key in sorted(baseline)]
rng = random.Random(seed)
boot = [fmean(rng.choice(deltas) for _ in deltas) for _ in range(resamples)]
cluster_se = stdev(deltas) / math.sqrt(len(deltas))
z_alpha = NormalDist().inv_cdf(0.975)
z_power = NormalDist().inv_cdf(0.80)
return {
"point": fmean(deltas),
"low": percentile(boot, 0.025),
"high": percentile(boot, 0.975),
"cluster_se": cluster_se,
"mde_80": (z_alpha + z_power) * cluster_se,
"task_deltas": dict(zip(sorted(baseline), deltas)),
}baseline = task_metrics(rows, "baseline")
candidate = task_metrics(rows, "candidate")
unc = paired_cluster_uncertainty(baseline, candidate)
print(f"paired point delta : {unc['point']:+.3f}")
print(f"95% cluster bootstrap CI : [{unc['low']:+.4f}, {unc['high']:+.4f}]")
print(f"cluster standard error : {unc['cluster_se']:.4f}")
print(f"80%-power MDE : {unc['mde_80']:.3f}")paired point delta : +0.125
95% cluster bootstrap CI : [-0.0208, +0.2292]
cluster standard error : 0.0653
80%-power MDE : 0.183The point estimate is +0.125, but the 95 percent interval reaches from below zero to about +0.23. The minimum detectable effect — the smallest true improvement this design could reliably distinguish at 80 percent power — is about 0.183, larger than the effect we are trying to claim. The suite is underpowered on purpose: with twelve tasks, “not significant” would mean “this sample cannot tell a 12-point candidate from its neighbors,” not “the systems are equal.”
Why insist on the task as the resampling unit? Because choosing the wrong unit is the most common way an eval manufactures false confidence. If we treated each of the 48 trials per system as an independent observation, we would divide s_d by \sqrt{48} instead of \sqrt{12} — halving the standard error and the interval without adding a single independent piece of evidence.
z = NormalDist().inv_cdf(0.975)
deltas = list(unc["task_deltas"].values())
s_d = stdev(deltas)
point = fmean(deltas)
honest_se = s_d / math.sqrt(len(deltas)) # 12 tasks: the correct unit
inflated_se = s_d / math.sqrt(len(deltas) * 4) # 48 trials: pretending repeats are independent
print(f"honest (N=12 tasks): SE={honest_se:.4f} 95% CI=[{point - z * honest_se:+.3f}, {point + z * honest_se:+.3f}]")
print(f"inflated (N=48 trials): SE={inflated_se:.4f} 95% CI=[{point - z * inflated_se:+.3f}, {point + z * inflated_se:+.3f}]")honest (N=12 tasks): SE=0.0653 95% CI=[-0.003, +0.253]
inflated (N=48 trials): SE=0.0326 95% CI=[+0.061, +0.189]Show the code that draws this figure
fig, ax = plt.subplots(figsize=(6.6, 2.6))
bars = [("honest (N=12 tasks)", honest_se, "#2a7f9e"),
("inflated (N=48 trials)", inflated_se, "#b13f3f")]
for i, (label, se, color) in enumerate(bars):
ax.errorbar(point, i, xerr=z * se, fmt="o", color=color, capsize=5, lw=2)
ax.text(point, i + 0.16, label, ha="center", fontsize=8, color=color)
ax.axvline(0.0, color="black", lw=1)
ax.axvline(-0.02, color="0.5", ls="--", lw=1)
ax.text(-0.02, -0.55, "margin -0.02", fontsize=7, color="0.4", ha="center")
ax.set_ylim(-0.7, 1.7)
ax.set_yticks([])
ax.set_xlabel("candidate minus baseline (pass-rate delta)")
fig.tight_layout()
plt.show()
The inflated interval sits entirely above zero and would license a confident “significantly better.” It is arithmetically clean and completely wrong, because 20 turns from one conversation, or four repeats of one task, are not 20 or four independent facts about the population. The contract must also predeclare what the interval decides: superiority requires the lower bound above zero; non-inferiority requires it above -\Delta for a margin \Delta chosen from product consequences before the run. And a severe deterministic failure stays a blocker regardless of any interval — which is where we turn next.
22.6 Outcome, policy, and process — and who bears the failures
A correct final state is necessary but not sufficient for an agent. It can reach the goal after reading the wrong customer’s record, retrying a paid API twelve times, skipping a required disclosure, or taking an unauthorized action; conversely, a strict match to one reference path can reject a perfectly safe alternative. So we grade outcome, policy, and process as separate layers and never let one impersonate another. Figure 22.6 locates them.
flowchart LR
TRIAL["One isolated trial"] --> STATE[("Authoritative final state")]
TRIAL --> TRACE["Transcript"]
STATE --> G1["Outcome grader<br/>(goal state · receipts)"]
TRACE --> G2["Policy grader<br/>(allowed action + args)"]
TRACE --> G3["Trajectory grader<br/>(tools · order · loops)"]
TRACE --> G4["Calibrated judge<br/>(open-ended quality)"]
G1 --> R["Component scores<br/>+ failure category"]
G2 --> R
G3 --> R
G4 --> R
Our trajectory_f1 is a diagnostic on that middle layer, and it is instructive precisely because it disagrees with the verdict. The candidate’s mean trajectory F1 is higher than the baseline’s — it uses tools more efficiently on average — yet its release verdict will be BLOCK.
traj_base = fmean(v["trajectory_f1"] for v in baseline.values())
traj_cand = fmean(v["trajectory_f1"] for v in candidate.values())
print(f"mean trajectory F1 baseline={traj_base:.3f} candidate={traj_cand:.3f}")
print("a more efficient trajectory does not make the candidate safe to ship")mean trajectory F1 baseline=0.947 candidate=0.975
a more efficient trajectory does not make the candidate safe to shipFairness evaluation applies the same anti-aggregation discipline to the people who bear the errors. A slice is a predeclared cohort you report separately — language, dialect, assistive modality, geography, device. The first thing to report is per-slice success.
# @save
def slice_rates(rows: list[dict[str, Any]], system: str) -> dict[str, float]:
"""Trial success rate for each predeclared workload slice.
Args:
rows: Graded trial records.
system: Which system's rows to group.
Returns:
A mapping from slice name to mean trial success in ``[0, 1]``.
"""
grouped: dict[str, list[bool]] = defaultdict(list)
for row in rows:
if row["system"] == system:
grouped[row["slice"]].append(row["success"])
return {name: fmean(values) for name, values in sorted(grouped.items())}print("baseline slices :", {k: round(v, 3) for k, v in slice_rates(rows, "baseline").items()})
print("candidate slices:", {k: round(v, 3) for k, v in slice_rates(rows, "candidate").items()})baseline slices : {'locale-en': 0.875, 'locale-es': 0.688, 'mixed-language': 0.625}
candidate slices: {'locale-en': 0.875, 'locale-es': 0.875, 'mixed-language': 0.812}But a success rate is only the first number, and on its own it can bless an unfair system. Two slices can post the identical success rate while one slice’s failures are unauthorized actions and the other’s are harmless extra questions. Disparate failure burden asks who bears which kind of failure, not merely how many. To compute it we first assign each failing trial a single primary category at the earliest boundary it violated.
# @save
FAILURE_ORDER = ("malformed_output", "wrong_final_state", "policy_violation", "poor_explanation")
def primary_failure(row: dict[str, Any]) -> str | None:
"""Assign one failure category at the earliest boundary a trial violated.
Returns ``None`` for a success. Otherwise reports the first failed check in
escalating order — malformed output, then wrong final state, then policy
violation, then poor explanation — so each failure gets exactly one primary
cause for error analysis and burden accounting, with secondary components
still available on the row.
Args:
row: A graded trial record from :func:`grade_traces`.
Returns:
The primary failure category, or ``None`` if the trial succeeded.
"""
if row["success"]:
return None
if not row["schema_pass"]:
return "malformed_output"
if not row["state_pass"]:
return "wrong_final_state"
if not row["policy_pass"]:
return "policy_violation"
return "poor_explanation"
def error_burden(rows: list[dict[str, Any]], system: str) -> dict[str, dict[str, Any]]:
"""Per-slice success rate plus the composition of its failures.
Two slices with equal success can carry very different harm: the same mean
hides whether failures are unauthorized actions or harmless extra
questions. For each slice this returns the success rate and a category
breakdown of its failing trials, so a fairness review can see composition,
not just the average.
Args:
rows: Graded trial records.
system: Which system's rows to summarize.
Returns:
A mapping from slice name to ``{"success": rate, "failures": {category: count}}``.
"""
by_slice: dict[str, list[dict[str, Any]]] = defaultdict(list)
for row in rows:
if row["system"] == system:
by_slice[row["slice"]].append(row)
report: dict[str, dict[str, Any]] = {}
for name, group in sorted(by_slice.items()):
failures: Counter[str] = Counter()
for row in group:
category = primary_failure(row)
if category is not None:
failures[category] += 1
report[name] = {"success": fmean(r["success"] for r in group), "failures": dict(failures)}
return reportTo make the point sharp we build a small representative pair of slices with the same success rate and opposite failure composition, then read the burden off it.
def make_row(slice_name: str, *, schema=True, state=True, policy=True, judge=True) -> dict[str, Any]:
"""A minimal graded row for the burden demonstration."""
return {"slice": slice_name, "system": "candidate", "schema_pass": schema,
"state_pass": state, "policy_pass": policy, "judge_pass": judge,
"success": schema and state and policy and judge}
burden_rows = (
[make_row("returns-desk") for _ in range(6)]
+ [make_row("returns-desk", policy=False) for _ in range(2)] # unauthorized actions
+ [make_row("billing-desk") for _ in range(6)]
+ [make_row("billing-desk", judge=False) for _ in range(2)] # harmless poor answers
)
for name, summary in error_burden(burden_rows, "candidate").items():
print(f"{name:13} success={summary['success']:.2f} failures={summary['failures']}")billing-desk success=0.75 failures={'poor_explanation': 2}
returns-desk success=0.75 failures={'policy_violation': 2}Show the code that draws this figure
burden = error_burden(burden_rows, "candidate")
cats = ["policy_violation", "poor_explanation"]
colors = {"policy_violation": "#b13f3f", "poor_explanation": "#c9a227"}
names = list(burden)
fig, ax = plt.subplots(figsize=(6.0, 3.2))
bottom = [0.0] * len(names)
for cat in cats:
heights = [burden[n]["failures"].get(cat, 0) for n in names]
ax.bar(names, heights, 0.5, bottom=bottom, label=cat, color=colors[cat], edgecolor="black")
bottom = [b + h for b, h in zip(bottom, heights)]
ax.set_ylabel("failing trials")
ax.set_title("Equal success (0.75), unequal harm")
ax.legend()
fig.tight_layout()
plt.show()
Both slices sit at 0.75 success; a fairness check that stopped at the mean would call them equivalent. They are not: one slice’s users receive unauthorized refunds while the other’s receive slightly worse prose. Which gap matters, and whether an absolute floor should protect every cohort, is a product decision — but the eval has to surface the composition for that decision to be possible. Building such slices honestly is its own discipline: parallel dialect or language items must preserve intent, difficulty, and required action, written and reviewed with speakers rather than manufactured by ad-hoc substitution, as the ReDial dialect study demonstrates (Lin et al. 2024). Our fixture’s three labels are synthetic and carry no claim about any real language community; their only job is to make a missing-slice regression impossible to overlook.
Error analysis is the same categorization pointed at the failures that actually drove the verdict. The golden refund-en task is where the candidate regressed, so we tabulate its failing candidate trials.
refund_fails = [r for r in rows
if r["task_id"] == "refund-en" and r["system"] == "candidate" and not r["success"]]
table = Counter(primary_failure(r) for r in refund_fails)
print(f"refund-en candidate failures: {len(refund_fails)}")
for category, count in table.items():
print(f" {category:18} {count}")refund-en candidate failures: 2
policy_violation 1
poor_explanation 1Two failing trials, two distinct causes: one policy violation (the skipped-lookup refund) and one poor explanation. That two-line table is the release blocker in miniature — it names the exact behaviors that must not recur, which is precisely what a golden case exists to protect. Aggregate improvement cannot outvote it.
22.7 Turning evidence into a release argument
Tests and evals are two halves of one release argument. A test asserts a deterministic invariant — a schema parses, a denied tool never executes, a migration preserves rows. An eval estimates behavior over a population — the candidate resolves support intents at an acceptable rate, stays reliable across repeats, and creates no disparate failure burden. Both are executable claims; neither replaces the other; and the release report is where they meet a predeclared decision rule. The report gathers the judge calibration, the paired interval, the slices, the trajectory diagnostics, and the golden check, then applies its rule and names every reason it blocks.
# @save
def release_report(
traces: Path = FIXTURES / "traces.jsonl",
calibration: Path = FIXTURES / "judge_calibration.json",
*,
margin: float = 0.02,
slice_floor: float = 0.75,
) -> dict[str, Any]:
"""Assemble the evidence packet and apply the predeclared release rule.
The rule blocks the candidate if the judge is uncalibrated, the paired
lower confidence bound falls below the allowed loss ``-margin``, any
candidate slice sits under ``slice_floor``, or a golden task regresses.
Every threshold here is a fixture decision, not a universal standard; a
real contract derives them from the harm of a false promotion versus a
false rejection.
Args:
traces: The task fixture to grade.
calibration: The judge-calibration fixture.
margin: The largest tolerated quality loss (as a positive number).
slice_floor: The minimum acceptable success rate per candidate slice.
Returns:
The full evidence packet plus ``verdict`` (``"SHIP"`` or ``"BLOCK"``)
and the list of blocking ``reasons``.
"""
rows = grade_traces(traces)
judge = judge_report(calibration)
baseline = task_metrics(rows, "baseline")
candidate = task_metrics(rows, "candidate")
uncertainty = paired_cluster_uncertainty(baseline, candidate)
cand_slices = slice_rates(rows, "candidate")
golden = sorted(
task_id for task_id in baseline
if any(r["task_id"] == task_id and r["golden"] for r in rows)
and candidate[task_id]["pass_rate"] < baseline[task_id]["pass_rate"]
)
reasons: list[str] = []
if judge["kappa"] < 0.60 or judge["fail_recall"] < 0.70 or judge["position_flip_rate"] > 0.15:
reasons.append("judge calibration does not meet the contract")
if uncertainty["low"] < -margin:
reasons.append("paired lower bound exceeds the allowed quality loss")
weak = [name for name, rate in cand_slices.items() if rate < slice_floor]
if weak:
reasons.append("candidate misses slice floor: " + ", ".join(weak))
if golden:
reasons.append("must-not-break regression: " + ", ".join(golden))
return {
"verdict": "SHIP" if not reasons else "BLOCK",
"reasons": reasons,
"judge": judge,
"uncertainty": uncertainty,
"slices": {"baseline": slice_rates(rows, "baseline"), "candidate": cand_slices},
"trajectory_f1": {
"baseline": fmean(v["trajectory_f1"] for v in baseline.values()),
"candidate": fmean(v["trajectory_f1"] for v in candidate.values()),
},
}report = release_report()
print("verdict :", report["verdict"])
for reason in report["reasons"]:
print(" -", reason)
print(f"\njudge : kappa={report['judge']['kappa']:.3f} "
f"fail_recall={report['judge']['fail_recall']:.2f} "
f"flip={report['judge']['position_flip_rate']:.2f} (passes calibration)")
print(f"paired CI : [{report['uncertainty']['low']:+.4f}, {report['uncertainty']['high']:+.4f}] "
f"point={report['uncertainty']['point']:+.3f}")
print(f"slices cand: {{{', '.join(f'{k}={v:.3f}' for k, v in report['slices']['candidate'].items())}}}")verdict : BLOCK
- paired lower bound exceeds the allowed quality loss
- must-not-break regression: refund-en
judge : kappa=0.736 fail_recall=0.80 flip=0.12 (passes calibration)
paired CI : [-0.0208, +0.2292] point=+0.125
slices cand: {locale-en=0.875, locale-es=0.875, mixed-language=0.812}Read the verdict against everything that looks good. The judge is calibrated. The candidate holds English and improves Spanish and mixed-language, clearing the 0.75 floor everywhere. Its point delta is positive and its trajectory efficiency is higher. And it still blocks — for exactly two reasons: the paired lower bound slips below the allowed loss, and the golden refund-en incident regressed from a perfect pass rate to one-half. The report does not average the unsafe refund away; it refuses. Figure 22.8 shows why both the pairing and the slices had to be there.
Show the code that draws this figure
deltas_map = report["uncertainty"]["task_deltas"]
slices = report["slices"]
fig, axes = plt.subplots(1, 2, figsize=(10, 3.6))
axes[0].axhline(0.0, color="black", lw=1)
axes[0].bar(range(len(deltas_map)), list(deltas_map.values()), color="0.65", edgecolor="black")
axes[0].set(title="Candidate minus baseline by task", xlabel="paired task cluster", ylabel="pass-rate delta")
axes[0].set_xticks(range(len(deltas_map)), [str(i + 1) for i in range(len(deltas_map))])
names = list(slices["baseline"])
xs = list(range(len(names)))
axes[1].bar([i - 0.18 for i in xs], [slices["baseline"][n] for n in names], 0.36,
label="baseline", color="white", edgecolor="black", hatch="//")
axes[1].bar([i + 0.18 for i in xs], [slices["candidate"][n] for n in names], 0.36,
label="candidate", color="#2a7f9e")
axes[1].set(title="Predeclared language slices", ylabel="trial pass rate", ylim=(0, 1.02))
axes[1].set_xticks(xs, names, rotation=15)
axes[1].legend()
fig.tight_layout()
plt.show()
The last mile is operational: this decision has to fail a build. A CI job runs the report and maps BLOCK to a non-zero exit code, so a merge cannot proceed on a candidate the evidence rejects.
def ci_exit_code(report: dict[str, Any]) -> int:
"""Map a release verdict to a process exit code (0 to ship, 2 to block)."""
return 0 if report["verdict"] == "SHIP" else 2
print(f"release verdict -> CI exit status {ci_exit_code(report)} "
f"({'merge allowed' if ci_exit_code(report) == 0 else 'merge blocked'})")release verdict -> CI exit status 2 (merge blocked)Three practices keep this argument honest over time. Keep golden cases few, causal, and reviewed: if every stochastic wobble becomes golden, development freezes; a case earns the status only when a specific previously passing behavior must not recur, and you keep a link to its originating incident and a mutation test proving the grader still catches a deliberately broken output. Version the task and grader together, and record exclusions when contamination or an invalid task is found rather than silently editing history. And treat the benchmark landscape as scaffolding, not truth: the same model behind a different harness — different tools, budgets, or scaffolding — is a different evaluated system, which is why SWE-bench made an executable repository environment part of each task (Jimenez et al. 2024) and why frameworks such as UK AISI’s Inspect factor an eval into an explicit dataset, solver, and scorer. Offline evidence is where the argument starts; economic-value benchmarks that grade authentic deliverables push it closer to realized return (OpenAI 2025), and Chapter 28 carries it the rest of the way from deliverable quality to product ROI.
NoteIn the interview
Q. Your candidate beats the baseline on aggregate pass rate and improves or holds every language slice. Why might you still block the release? Because an aggregate is not a decision. Two things override a favorable mean: the uncertainty on the paired delta (if the task-clustered lower confidence bound falls below the predeclared non-inferiority margin, “better on average” is not established), and a must-not-break case (a golden incident that regressed is a blocker no average can outvote). The answer they want names the unit of analysis — the task, not the trial — and the two anti-aggregation guards. The trap is arguing from the point estimate, or treating repeated trials as independent samples to shrink the interval.
Note
Verify live: 2026-07-20. Appendix C owns the benchmark and harness registry. Current agent-evaluation landmarks include HELM (Liang et al. 2022) for reproducible multi-metric reporting, SWE-bench (Jimenez et al. 2024) for issue resolution in executable repositories, and \tau-bench (Yao et al. 2024) for tool-agent-user interaction (which introduced pass^k). Names, task counts, leaderboards, and framework APIs churn; the durable moves are to pin the benchmark revision and exact harness, audit task validity, and report the evaluated system configuration. Verify the current landscape against primary sources before citing any leaderboard number.
22.8 Summary
We built an evaluation as an argument, not a scoreboard. Traces were promoted into versioned, isolated tasks; each trial was graded as a strict conjunction so an unsafe success could not pass; and the model judge was calibrated with Cohen’s kappa — which exposed a 90-percent-agreement judge as worthless — then probed for position and verbosity bias. We separated best-of-k capability from repeated-run reliability, put a paired cluster-bootstrap interval and a power estimate on the delta, and showed how pretending trials are independent halves the interval and fakes significance. Trajectory, disparate error burden, and error analysis survived aggregation, and one report blocked a candidate that looked better on average. Chapter 23 consumes these process signals for credit assignment.
22.9 Exercises
- Raw agreement lies, again. Build an always-
PASSjudge against 80 humanPASSand 20FAILlabels. Compute raw agreement and kappa withcohen_kappa, then explain in one sentence why failure recall, not agreement, is the metric that would have caught this judge. Now flip 5 of the humanPASSlabels toFAILand predict the direction kappa moves before rerunning. - Two questions, two metrics. For n=10, c=7, and k \in \{1, 2, 4, 8\}, predict which of
pass_at_k_estimateandpass_pow_k_estimaterises and which falls, then verify. At which k does pass^k first drop below 0.2, and what does that imply for a system a user hits ten times a day? - Repair the golden regression. Edit the fixture so the
refund-encandidate trial that skipped the lookup becomes a clean success. Predict, before rerunningrelease_report, what happens to the golden gate, the paired interval, therefund-enpass rate, and the final verdict; then confirm each prediction and identify which single reason (if any) still blocks. - The isolation invariant. Duplicate one task’s
snapshotvalue so two tasks share an environment identity. Show thatgrade_tracesraises before any score is computed, then write the two-line test that would catch this in CI. Explain in one sentence why a shared environment can turn a broken policy into a passing trial. - A control for one bias.
- Raise the pairwise position-flip count in the calibration fixture until it exceeds the 0.15 contract threshold, and confirm
release_reportnow blocks on judge calibration while pointwise kappa is unchanged. - Name one judge bias that answer-order swapping cannot detect, and sketch the length-matched or family-crossed control that would.
- Raise the pairwise position-flip count in the calibration fixture until it exceeds the 0.15 contract threshold, and confirm
- The unit of resampling. Using the honest and inflated standard errors computed in Section 22.5, write out why the \sqrt{48} version is wrong even though the arithmetic is correct. Then implement a genuine trial-level bootstrap that resamples the 96 graded rows independently and compare its interval width to the paired cluster bootstrap; explain which dependency each version keeps or discards.
- Disparate burden as a gate. Extend
release_reportwith a rule that blocks when any slice’s policy-violation failure count exceeds a predeclared budget, usingerror_burden. Apply it to the two-slice demonstration from Section 22.6 and show it blocking the returns-desk slice while a success-rate-only rule passes both. What predeclared number did you choose, and how would you defend it? - A contract from scratch. Design an evaluation contract for a multilingual appointment-booking agent: name its construct, its authoritative outcome grader, one trajectory invariant, a human-label and adjudication plan, three predeclared slices, a non-inferiority margin, an MDE target, a golden rule, and the residual risks you would record. Defend which failures may be averaged and which may not.