flowchart LR
O["Observe"] --> P["Propose"] --> X["Validate + execute"] --> V["Verify"]
V -->|"not done"| O
V -->|"postcondition holds"| C["Commit"]
K["Coding: repo + tests"] -. "fills the seams" .-> O
R["Research: ledger + claim support"] -. "fills the seams" .-> O
U["Computer use: live UI + state receipt"] -. "fills the seams" .-> O
21 Agent Applications: Coding, Conversational, Computer Use, Data, and Deep Research
A coding agent edits the right function, announces the bug is fixed, and leaves one regression failing. A customer-service agent tells a caller their refund is processed while the account still shows the charge. A browser agent reports an order cancelled after clicking a control that no longer does anything. These read as three unrelated product failures. Structurally they are one: the system trusted the model’s account of success instead of checking an authoritative postcondition. Every mature agent application is built to close that exact gap, and this chapter builds the machinery that closes it.
We build one thing all the way through — a minimal coding agent that reads a repository map, proposes a constrained edit, runs the repository’s own tests, feeds a failure back into a bounded repair loop, and refuses to write outside its workspace — and score it on a small embedded task suite with the counts shown. Then we carry the same skeleton across the other families the title names, each with a runnable fragment that produces a shown result: a policy-gated refund verified against final account state; a computer-use step that re-resolves a control and checks the resulting state rather than the click; a deep-research citation audit that catches a claim its source does not support; and a data-analytics gate that catches a join fan-out inflating a total. The unifying claim is that an agent application is not a model plus domain tools — it is a contract over six seams: which observation enters the loop, which actions the model may propose, which component executes them, which observation verifies the postcondition, when provisional work commits, and how the system recovers. Chapter 16 owns the raw loop these seams wrap; here we fill the seams in per family. Pixel-based GUI perception is the one piece we defer, to Chapter 29.
21.1 The action–verification boundary is the architecture
Start with the single design decision every application shares: separate provisional state from committed state. A proposed patch in an isolated checkout is provisional. Notes in an evidence ledger are provisional. Text typed into a form before submission is provisional. The model may rewrite provisional state as often as it likes because mistakes there are cheap. Promotion to committed state — a merged branch, a published report, a submitted form, a moved dollar — is where mistakes become expensive, and that promotion must be gated by an application-owned rule, never by the model’s confidence.
That rule is a verifier. Let s_0 be the observed starting state, a a proposed action, s_1 the state observed after execution, and g the goal. The application decides completion with
V(s_0, a, s_1, g) \in \{\text{pass}, \text{fail}, \text{unknown}\}. \tag{21.1}
The third value earns its keep. A test timeout does not prove a code change wrong. An unreachable article does not prove a research claim false. A missing confirmation banner does not prove a backend write failed. Collapsing unknown into pass invents completion; collapsing it into fail triggers duplicate actions and unsafe retries. A typed unknown lets the policy re-observe, retry safely, reconcile, or escalate. A useful verifier has four properties, and each family below is really an argument about how to get them:
- Authority: it measures the state that matters, not the model’s narration of that state.
- Specificity: it corresponds to the user’s postcondition, not to an intermediate action.
- Freshness: it is taken after the action and identifies the environment version it saw.
- Independence: the same proposal cannot corrupt both the work and its check.
The generic loop from Chapter 16 stays fixed; what each application changes is the body attached to each seam. Figure 21.1 draws the shared loop branching into three of the contracts we build.
Two forces set how much autonomy a seam can safely carry: how strong the verifier is, and how reversible the effect is. When both are strong the loop can iterate on its own. When the verifier is weak but the work is reversible, the agent can draft and stage for human review. When the verifier is strong but the effect is hard to undo, a single gated commit is right. When both are weak, the system stays advisory. That is a function, so we write it as one.
# @save
from __future__ import annotations
def autonomy_regime(verification: float, reversibility: float) -> str:
"""Map a (verification strength, reversibility) pair to an autonomy regime.
Autonomy is not a property of the model; it follows from whether completion
can be proven and whether a mistake can be undone. Strong-and-reversible
earns free iteration; weak-but-reversible earns staged drafts;
strong-but-irreversible earns a single gated commit; weak-and-irreversible
stays advisory.
Args:
verification: Strength of the available verifier in [0, 1].
reversibility: Ease of undoing the effect in [0, 1].
Returns:
One of four regime labels.
"""
strong, reversible = verification >= 0.5, reversibility >= 0.5
if strong and reversible:
return "iterate freely"
if not strong and reversible:
return "stage for review"
if strong and not reversible:
return "verify, then commit"
return "advisory (human commits)"We place each family this chapter builds at its rough coordinates and read off the regime.
families = {
"coding (tests)": (0.9, 0.9),
"deep research": (0.3, 0.85),
"analytics": (0.45, 0.9),
"refund tool": (0.85, 0.35),
"computer-use commit": (0.65, 0.2),
}
for name, (v, r) in families.items():
print(f"{name:22} verify={v:.2f} reversible={r:.2f} -> {autonomy_regime(v, r)}")coding (tests) verify=0.90 reversible=0.90 -> iterate freely
deep research verify=0.30 reversible=0.85 -> stage for review
analytics verify=0.45 reversible=0.90 -> stage for review
refund tool verify=0.85 reversible=0.35 -> verify, then commit
computer-use commit verify=0.65 reversible=0.20 -> verify, then commitCoding lands in “iterate freely” because good tests are a strong verifier and a patch is trivial to revert; deep research and analytics land in “stage for review” because their verifiers are semantic and weak while their work is cheap to redo; the refund tool and the money-moving browser action demand “verify, then commit” because the effect is hard to undo. Figure 21.2 plots the four regimes as quadrants so the placement is visual rather than asserted.
Show the code that draws this figure
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.4, 4.4))
ax.axhline(0.5, color="0.7", lw=1)
ax.axvline(0.5, color="0.7", lw=1)
for name, (v, r) in families.items():
ax.plot(v, r, "o", color="#b13f3f")
ax.annotate(name, (v, r), textcoords="offset points", xytext=(6, 5), fontsize=8)
for x, y, label in [(0.75, 0.95, "iterate freely"), (0.22, 0.95, "stage for review"),
(0.75, 0.03, "verify, then commit"), (0.22, 0.03, "advisory")]:
ax.text(x, y, label, ha="center", fontsize=9, color="0.4")
ax.set_xlim(0, 1.05)
ax.set_ylim(0, 1.05)
ax.set_xlabel("verification strength")
ax.set_ylabel("reversibility")
fig.tight_layout()
plt.show()
Freshness needs one more note because it is the quiet source of the failures in the opening. Repositories change between inspection and patch. Sources are updated between extraction and publication. Pages rerender between a snapshot and a click. Bind every proposal to the version it observed — a commit hash and file digest, a URL plus retrieval time, a page reference plus a tree digest — and re-observe before a consequential commit. This is the application-level form of the approval-to-execution race from Chapter 17, and each section below pays it off in code.
21.2 Coding: the repository is executable state
Code generation becomes coding agency when the model can inspect and change a repository, run programs, read failures, and iterate. The repository is not a bag of source text; it is a stateful environment — source, tests, locks, build scripts, conventions, history — and the useful output is not an explanation but a minimal patch that satisfies the intended behavior without breaking protected behavior. The Agent–Computer Interface (ACI) is what the model can observe and express, and its design changes agent behavior more than most prompt tweaks do (Yang et al. 2024). We build a small ACI whose defining property is that the executor, not the model, owns every invariant.
The model’s proposal vocabulary is one small value: an edit naming a file, an anchor to replace, its replacement, and a reason. The result of a task is another: the outcome plus countable evidence.
# @save
import ast
import difflib
import json
import os
import subprocess
import sys
import tempfile
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any
class EditRejected(ValueError):
"""Raised when a proposed edit violates an executor-owned invariant."""
@dataclass(frozen=True)
class Edit:
path: str
old: str
new: str
reason: str
@dataclass
class TaskResult:
"""The countable outcome of one task plus its full event trace.
Everything an auditor needs lives here: whether the task resolved, how many
proposals and test runs it cost, how many edits were rejected before
execution, and an ordered event log. The counts are what we score; the
events are what we read when a count surprises us.
"""
task_id: str
resolved: bool = False
proposals: int = 0
test_runs: int = 0
rejected_edits: int = 0
events: list[dict[str, Any]] = field(default_factory=list)The first invariant is containment. Before any edit is applied, the target path is resolved and three rules are enforced: it may not escape the workspace, it may touch only Python source, and it may not rewrite a test file — because an agent that can edit its own oracle can “pass” by deleting the assertion.
# @save
def safe_path(root: Path, relative: str, *, allow_tests: bool = False) -> Path:
"""Resolve a repository-relative path and refuse anything outside the rules.
The executor, not the model, owns three invariants here: an edit may not
escape the workspace, may touch only Python source, and may not rewrite a
test file (which would let the agent pass by changing the oracle).
Args:
root: Absolute path of the isolated workspace.
relative: Path the proposal named, relative to ``root``.
allow_tests: Whether ``test_*.py`` targets are writable; only
materialization sets this, never an edit.
Returns:
The resolved absolute path inside the workspace.
Raises:
EditRejected: If the path escapes the workspace, is not ``.py``, or
names a read-only test file.
"""
candidate = (root / relative).resolve()
if candidate != root and root not in candidate.parents:
raise EditRejected(f"path escapes workspace: {relative}")
if candidate.suffix != ".py":
raise EditRejected(f"only Python source files are editable: {relative}")
if not allow_tests and candidate.name.startswith("test_"):
raise EditRejected(f"tests are read-only: {relative}")
return candidateThe edit itself is an exact single-occurrence replacement. Two admission checks matter: the anchor must occur exactly once, so the edit is unambiguous, and the result must parse, so a broken file never reaches the test runner. Syntax validity is only an admission gate — the authoritative check comes later. The receipt is a unified diff, the artifact a human reviewer already knows how to read.
# @save
def apply_edit(root: Path, edit: Edit) -> str:
"""Apply one exact single-occurrence replacement and return a diff receipt.
The admission checks are the whole point: the anchor text must occur exactly
once (so the edit is unambiguous), and the result must parse as Python (so a
broken file never reaches the test runner). Syntax validity is only an
admission gate; the authoritative check is the test run that follows.
Args:
root: Absolute workspace path.
edit: The proposed edit, naming a file, an anchor, and its replacement.
Returns:
A unified-diff string documenting exactly what changed.
Raises:
EditRejected: If the file is missing, the anchor is absent or repeated,
or the edited file would not parse.
"""
target = safe_path(root, edit.path)
if not target.is_file():
raise EditRejected(f"file does not exist: {edit.path}")
before = target.read_text(encoding="utf-8")
occurrences = before.count(edit.old)
if occurrences != 1:
raise EditRejected(f"anchor must occur once, found {occurrences}: {edit.path}")
after = before.replace(edit.old, edit.new, 1)
try:
ast.parse(after)
except SyntaxError as exc:
raise EditRejected(f"edit creates invalid Python: {exc.msg}") from exc
target.write_text(after, encoding="utf-8")
return "".join(
difflib.unified_diff(
before.splitlines(keepends=True), after.splitlines(keepends=True),
fromfile=f"a/{edit.path}", tofile=f"b/{edit.path}",
)
)The model orients itself with a repository map, not the full source: an orientation layer listing each file with its line count and top-level symbols, so the loop can request contents on demand instead of paying for the whole tree every turn. Chapter 15 owns the retrieval that finds relevant code in large repositories; here the map is a few lines, and a companion helper writes a task’s files into a fresh workspace.
# @save
def materialize(task: dict[str, Any], root: Path) -> None:
"""Write a task's files into a fresh workspace before the loop starts.
Args:
task: A task record whose ``files`` maps repository-relative paths to
their contents; test files are written with ``allow_tests`` set.
root: The empty workspace directory to populate.
"""
for relative, contents in task["files"].items():
target = safe_path(root, relative, allow_tests=True)
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(contents, encoding="utf-8")
def repository_map(root: Path) -> list[dict[str, Any]]:
"""Return an orientation layer: paths and top-level symbols, not full source.
A repository map is what the model reads to decide *where* to look. It lists
each file with its line count and its top-level function and class names, so
the loop can request full contents on demand instead of paying for the whole
tree on every turn.
Args:
root: Absolute path of the workspace to summarize.
Returns:
One record per ``.py`` file with ``path``, ``lines``, and ``symbols``.
"""
result: list[dict[str, Any]] = []
for path in sorted(root.rglob("*.py")):
source = path.read_text(encoding="utf-8")
try:
symbols = [
node.name
for node in ast.parse(source).body
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef))
]
except SyntaxError:
symbols = ["<syntax-error>"]
result.append(
{"path": path.relative_to(root).as_posix(), "lines": source.count("\n") + 1,
"symbols": symbols}
)
return resultThe suite is six tiny tasks stored beside the chapter; each has an issue, a buggy source file, a test file, and a scripted list of proposals. Scripting the proposals is the honest simplification: with no model in the loop, every reader sees the same rejection and the same repair, and swapping in a real model means replacing only the proposal source. We load the suite and print the map the agent would see for one task.
tasks = json.loads(Path("data/ch21/tasks.json").read_text(encoding="utf-8"))
print(f"{len(tasks)} tasks:", [t["id"] for t in tasks])
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
materialize(next(t for t in tasks if t["id"] == "minimum-repair"), root)
for entry in repository_map(root):
print(f" {entry['path']:16} {entry['lines']:2d} lines symbols={entry['symbols']}")6 tasks: ['invoice-total', 'bounded-percent', 'normalized-tag', 'empty-average', 'minimum-repair', 'workspace-guard']
choose.py 3 lines symbols=['minimum']
test_choose.py 11 lines symbols=['ChooseTest']The map exposes structure, not source: the agent learns that choose.py defines minimum and that test_choose.py exists and is off-limits, without loading either in full. The authoritative verifier is the repository’s own test command, run after an edit. A pass requires the process to exit zero; a timeout returns fail with an honest reason rather than a guess.
# @save
def run_tests(root: Path, timeout_seconds: float = 8.0) -> tuple[bool, str]:
"""Run the repository's own test command as the authoritative verifier.
Completion is earned from this observation, taken *after* an edit, not from
the model's narration that the bug is fixed. A ``pass`` requires the process
to exit zero; a timeout returns ``fail`` with an honest reason rather than
an unknown collapsed into either verdict.
Args:
root: Workspace to run the tests in.
timeout_seconds: Wall-clock bound on the test process.
Returns:
A ``(passed, observation)`` pair; the observation is the tail of the
combined stdout and stderr, kept short enough to feed back to the model.
"""
command = [sys.executable, "-m", "unittest", "discover", "-s", ".", "-p", "test_*.py"]
environment = {**os.environ, "PYTHONDONTWRITEBYTECODE": "1"}
try:
done = subprocess.run(command, cwd=root, env=environment, capture_output=True,
text=True, timeout=timeout_seconds, check=False)
return done.returncode == 0, (done.stdout + done.stderr)[-600:]
except subprocess.TimeoutExpired:
return False, f"TIMEOUT after {timeout_seconds:.1f}s"Now the loop that ties the seams together. It materializes the workspace, records the map, and for each scripted proposal within a budget: applies the edit (or records a rejection), runs the tests, and — only if they pass — commits. The budget bounds recovery from both failing tests and rejected edits, which is why it is a semantic knob, not a timeout.
# @save
def solve_task(task: dict[str, Any], max_proposals: int = 3) -> TaskResult:
"""Drive one task through the observe-propose-execute-verify-commit loop.
Proposals are scripted so every reader sees the same rejection and repair;
swapping in a real model means replacing only this proposal source. The
executor owns every invariant, so a wrong or malicious proposal costs a
rejection or a failing test, never workspace damage or a gamed oracle.
Args:
task: One task record with ``id``, ``files``, and ``proposals``.
max_proposals: How many proposals the loop may spend before giving up;
this is the repair budget, and it bounds recovery from both failing
tests and rejected edits.
Returns:
A :class:`TaskResult` with the outcome, the counts, and the full event
trace for audit.
"""
result = TaskResult(task_id=task["id"])
with tempfile.TemporaryDirectory(prefix=f"ch21-{task['id']}-") as directory:
root = Path(directory).resolve()
materialize(task, root)
result.events.append({"kind": "observe", "repo_map": repository_map(root)})
for raw in task["proposals"][:max_proposals]:
result.proposals += 1
edit = Edit(**raw)
result.events.append({"kind": "propose", "path": edit.path, "reason": edit.reason})
try:
receipt = apply_edit(root, edit)
except EditRejected as exc:
result.rejected_edits += 1
result.events.append({"kind": "reject", "reason": str(exc)})
continue
result.events.append({"kind": "execute", "diff": receipt})
passed, observation = run_tests(root)
result.test_runs += 1
result.events.append({"kind": "verify", "passed": passed, "observation": observation})
if passed:
result.resolved = True
result.events.append({"kind": "commit"})
break
return result# @save
def run_suite(tasks: list[dict[str, Any]], max_proposals: int = 3) -> dict[str, Any]:
"""Score every task and aggregate the loop's countable outcomes.
Args:
tasks: The task list to score.
max_proposals: Repair budget passed to each task.
Returns:
A report dict with the task count and the summed resolved, proposal,
test-run, and rejected-edit counts, plus each task's result.
"""
results = [solve_task(task, max_proposals=max_proposals) for task in tasks]
return {
"tasks": len(results),
"resolved": sum(r.resolved for r in results),
"proposals": sum(r.proposals for r in results),
"test_runs": sum(r.test_runs for r in results),
"rejected_edits": sum(r.rejected_edits for r in results),
"results": [asdict(r) for r in results],
}We score the suite at three repair budgets at once, then read the aggregates off the full-budget run.
sweep = {budget: run_suite(tasks, max_proposals=budget) for budget in (1, 2, 3)}
report = sweep[3]
print(f"tasks={report['tasks']} resolved={report['resolved']} "
f"proposals={report['proposals']} test_runs={report['test_runs']} "
f"rejected_edits={report['rejected_edits']}")
print("resolved by budget:", {b: sweep[b]["resolved"] for b in (1, 2, 3)})tasks=6 resolved=6 proposals=8 test_runs=7 rejected_edits=1
resolved by budget: {1: 4, 2: 6, 3: 6}Six tasks resolve using eight proposals but only seven test runs — the missing run is the one rejected edit, which never reaches the test runner. That gap is the point: the executor’s guard fires before execution, so a bad proposal costs a rejection, not a test. The aggregate hides the two tasks that actually teach, so we print the full trace for minimum-repair, where the first plausible patch fails and the second is proposed only after seeing the failure.
def show_trace(result: dict[str, Any]) -> None:
for event in result["events"]:
kind = event["kind"]
if kind == "observe":
print(f"observe repository map: {len(event['repo_map'])} files")
elif kind == "propose":
print(f"propose {event['path']}: {event['reason']}")
elif kind == "execute":
change = event["diff"].strip().splitlines()[-1]
print(f"execute applied {change}")
elif kind == "verify":
tail = "" if event["passed"] else event["observation"].strip().splitlines()[-1]
print(f"verify passed={event['passed']} {tail}")
elif kind == "reject":
print(f"reject {event['reason']}")
elif kind == "commit":
print("commit tests pass after the edit -> commit-eligible")
show_trace(next(r for r in report["results"] if r["task_id"] == "minimum-repair"))observe repository map: 2 files
propose choose.py: a plausible but wrong first guess, to exercise feedback-driven repair
execute applied + return values[-1]
verify passed=False FAILED (failures=1)
propose choose.py: the failing counterexample shows position is not the requested invariant
execute applied + return min(values)
verify passed=True
commit tests pass after the edit -> commit-eligibleRead it top to bottom: the agent proposes values[-1], the tests fail with a concrete counterexample, and that observation — not a second guess in the dark — drives the repair to min(values), which passes and commits. A failed assertion is evidence for the next proposal; it is never an invitation to rewrite the assertion. The workspace-guard task shows the other guard: a proposal aimed outside the workspace is rejected before any write, then the local edit succeeds.
show_trace(next(r for r in report["results"] if r["task_id"] == "workspace-guard"))observe repository map: 2 files
propose ../escape.py: a malformed proposal that tests workspace containment
reject path escapes workspace: ../escape.py
propose greet.py: the repository-local edit that implements the requested punctuation
execute applied + return f'Hello {name}!'
verify passed=True
commit tests pass after the edit -> commit-eligibleThe rejection happens before the test runner is ever invoked, which is why the suite counted eight proposals but seven runs. Now vary the one knob and re-measure: shrink the repair budget and watch the two feedback-dependent tasks fall, as Figure 21.3 plots.
Show the code that draws this figure
fig, ax = plt.subplots(figsize=(6.2, 3.2))
budgets = sorted(sweep)
ax.bar([str(b) for b in budgets], [sweep[b]["resolved"] for b in budgets], color="#2a7f9e")
ax.axhline(len(tasks), ls="--", color="0.6")
ax.set_xlabel("proposal budget (max_proposals)")
ax.set_ylabel("tasks resolved (of 6)")
ax.set_ylim(0, 6.6)
fig.tight_layout()
plt.show()
At a budget of one, four of six resolve: the repair task spends its only proposal on the wrong edit, and the workspace task spends its only proposal on the rejected path. A second proposal recovers both. “One proposal” therefore constrains recovery from a failing test and recovery from a rejected action — a single number governing two different recovery paths.
One more ACI decision deserves a measured look, because the chapter’s claim is that edit format changes behavior and you should measure it rather than argue about it. Our applier uses an exact anchor. A common alternative is a line-range edit, which is compact but whose coordinates go stale the instant another writer touches the file. Carrying the digest the proposer observed turns that silent hazard into a typed rejection.
# @save
import hashlib
def digest(text: str) -> str:
"""Return a short content hash used as a freshness token for a file."""
return hashlib.sha256(text.encode("utf-8")).hexdigest()[:12]
def apply_guarded_edit(root: Path, path: str, observed_digest: str,
start: int, end: int, new_lines: list[str]) -> str:
"""Apply a line-range edit only if the file still matches what was observed.
A line-range edit is compact but its coordinates go stale the moment another
writer touches the file. Carrying the digest the proposer observed turns that
silent hazard into a typed rejection: if the current file hashes differently,
the edit refers to a version that no longer exists and is refused.
Args:
root: Workspace path.
path: Repository-relative file to edit.
observed_digest: The :func:`digest` the proposer saw when it chose the range.
start: First line index to replace (0-based, inclusive).
end: Line index to stop at (exclusive).
new_lines: Replacement lines, without trailing newlines.
Returns:
A unified-diff receipt for the applied change.
Raises:
EditRejected: If the file's current digest differs from ``observed_digest``.
"""
target = safe_path(root, path)
before = target.read_text(encoding="utf-8")
if digest(before) != observed_digest:
raise EditRejected(f"stale digest: file changed since it was observed ({path})")
lines = before.splitlines(keepends=True)
after = "".join(lines[:start] + [ln + "\n" for ln in new_lines] + lines[end:])
ast.parse(after)
target.write_text(after, encoding="utf-8")
return "".join(difflib.unified_diff(lines, after.splitlines(keepends=True),
fromfile=f"a/{path}", tofile=f"b/{path}"))We observe a file, let a concurrent writer prepend a line, and offer the same stale edit to both appliers.
with tempfile.TemporaryDirectory() as directory:
root = Path(directory).resolve()
original = "def price(x):\n return x * 2\n"
(root / "m.py").write_text(original, encoding="utf-8")
observed = digest(original)
(root / "m.py").write_text("import math\n" + original, encoding="utf-8") # concurrent change
receipt = apply_edit(root, Edit("m.py", "return x * 2", "return x * 3", "bump"))
print("exact-anchor applier :", receipt.strip().splitlines()[-1], " (applied to an unseen version)")
(root / "m.py").write_text("import math\n" + original, encoding="utf-8") # same stale state
try:
apply_guarded_edit(root, "m.py", observed, 1, 2, [" return x * 3"])
except EditRejected as exc:
print("digest-guarded applier:", exc)exact-anchor applier : + return x * 3 (applied to an unseen version)
digest-guarded applier: stale digest: file changed since it was observed (m.py)The exact-anchor applier finds its unique anchor and applies the edit to a file it never saw — harmless here, dangerous when the concurrent change was semantically relevant. The digest-guarded applier refuses, because freshness is one of the four verifier properties and it is checkable in one line. There is no universal best edit format; the discipline is to measure malformed edits, unintended changed lines, and repair turns for the model and repository distribution you actually deploy.
Repository-level benchmarks such as SWE-bench made the essential contribution that the environment and executable verifier are part of the task, not an afterthought (Jimenez et al. 2024). But their scores are not portable model constants: dataset filtering, issue ambiguity, test adequacy, repository exposure, scaffold, dependency image, resource limits, and retry policy all change the construct. Our six-of-six result makes no capability claim whatsoever — it demonstrates that the invariants hold, and Chapter 22 owns the statistics that turn repeated trials into a defensible comparison.
NoteIn the interview
Q. Your coding agent’s tests pass. Is the task done? Not necessarily. A green test run proves the postcondition only when the tests capture the requirement, ran the right target, and stayed read-only to the agent. The trap is treating “exit code zero” as authority: an agent that can edit tests, a suite that skips the relevant case, or a run that reused stale build output all produce a pass that means nothing. The strong answer names the four verifier properties — authority, specificity, freshness, independence — and says which one each failure mode violates.
Coding benchmarks are fragmenting into curated, live-updated, held-out, and longer-horizon variants; as of 2026-07-20, scores from SWE-bench and its Verified, Live, and Pro-style descendants are not interchangeable, and recent audits show infrastructure and task defects can move results enough to swamp small leaderboard gaps. Verify live: the official SWE-bench repository and the current benchmark papers before quoting any subset or comparing products.
21.3 Deep research: an evidence ledger before prose
Deep research is not “RAG with more searches.” It is a long-horizon process whose deliverable is a set of claims, qualifications, and source relationships. Retrieval finds potentially useful material; research decides which questions remain open, which evidence is fit for purpose, how sources conflict, and what the report may assert. Chapter 15 owns the adaptive query and retrieval mechanics. This section owns the application state those mechanics must populate: an evidence ledger, built before any prose is written. Figure 21.4 shows the shape.
flowchart LR
B["Decision-bound brief"] --> Q["Atomic questions"]
Q --> S["Search + browse observations"]
S --> E["Source records with provenance"]
E -->|"supports"| C["Atomic claims"]
E -->|"contradicts"| C
E -->|"context only"| C
C --> D["Draft synthesis"]
C -->|"unsupported"| G["Open gaps"]
G --> Q
A ledger record binds one atomic claim to one source, with the span that supports it, the source’s stance, its class, and when it was retrieved. The stance field is the one that prevents the opening’s second failure — a citation that leads to a relevant page which does not actually establish the adjacent claim.
# @save
@dataclass(frozen=True)
class EvidenceRecord:
"""One row of the evidence ledger: a claim tied to a single source.
The ledger is the application state deep research must build before prose.
Each record binds an atomic claim to exactly one source, records the span
that supports it, and — crucially — its ``stance``, because a source that
merely mentions a topic is not a source that establishes the claim.
"""
claim_id: str
source_id: str
span: str
stance: str # "supports" | "contradicts" | "context"
source_class: str
retrieved: strSynthesis then runs a bidirectional citation audit: every load-bearing claim needs at least one supporting record, and a claim with a citation whose stance is only context is flagged as unsupported even though a link is present. Citation presence, source stance, and entailment are three different checks; the audit keeps them apart. (Entailment here is a stance label; a production system would use an entailment model, and high-stakes claims still need expert review of the cited span.)
# @save
def citation_audit(claims: dict[str, str], records: list[EvidenceRecord]) -> list[str]:
"""Audit claims against the ledger in both directions and return findings.
Two failures are distinct and both matter. A load-bearing claim with no
supporting record is unsupported synthesis. A record whose stance is only
``context`` cannot license the claim it is attached to, even though a
citation is present. Citation presence, source stance, and entailment are
three different checks; this audit keeps them apart.
Args:
claims: Mapping of claim id to the sentence the report will assert.
records: Every ledger record gathered for those claims.
Returns:
One human-readable finding per problem, empty if the ledger is clean.
"""
findings: list[str] = []
by_claim: dict[str, list[EvidenceRecord]] = {claim_id: [] for claim_id in claims}
for record in records:
by_claim.setdefault(record.claim_id, []).append(record)
for claim_id in claims:
present = by_claim.get(claim_id, [])
supporting = [r for r in present if r.stance == "supports"]
if not present:
findings.append(f"{claim_id}: NO CITATION for a load-bearing claim")
elif not supporting:
stances = ", ".join(sorted({r.stance for r in present}))
findings.append(f"{claim_id}: cited but UNSUPPORTED (stance: {stances})")
return findingsWe fill in two claims — one backed by the primary source, one propped up by a vendor blog that only gestures at the number — and run the audit.
claims = {
"c1": "The model reaches 71.7 percent on the benchmark's verified split.",
"c2": "Independent testing confirms a 30 percent latency reduction.",
}
records = [
EvidenceRecord("c1", "primary-paper", "we report 71.7 on Verified",
"supports", "peer-reviewed", "2026-07-10"),
EvidenceRecord("c2", "vendor-blog", "our platform feels much faster",
"context", "marketing", "2026-07-11"),
]
for finding in citation_audit(claims, records):
print(finding)c2: cited but UNSUPPORTED (stance: context)The audit catches exactly the failure that ships in polished reports: claim c2 has a citation, so a shallow check would pass it, but its only source contextualizes rather than supports, so the claim cannot be asserted as written. This separates report quality from citation accuracy, which recent deep-research benchmarks show are genuinely different axes (Du et al. 2025). The ledger also changes the control loop upstream: search is driven by gaps and contradictions rather than a fixed source count, and drafting proceeds one claim at a time from records rather than from an undifferentiated transcript.
21.4 Computer use: ground, then bind action to observation
A computer-use agent drives applications built for humans. Its core difficulty is grounding: mapping an intent like “cancel this order” onto the right control in the right application state, then checking whether the state actually changed. The first decision is which interface to reach for, and the rule is to prefer the highest-level reliable one. Figure 21.5 is that decision.
flowchart TB
G["Goal: act on an application"] --> A{"Domain API or typed tool?"}
A -->|yes| API["Call it: typed args, machine-readable receipt"]
A -->|no| B{"Accessibility tree / DOM control?"}
B -->|yes| SEM["Resolve by role + name: auditable, layout-stable"]
B -->|no| C{"Keyboard or scripting reaches it?"}
C -->|yes| KBD["Drive by command: depends on focus and mode"]
C -->|no| PIX["Pixel perception + coordinates — Chapter 29"]
The ordering is a decision rule, not a claim that semantic trees are complete — custom canvases, unlabeled controls, charts, and remote-rendered apps can force pixels, and Chapter 29 teaches the VLM grounding that reads them. Here we work one level up, on an accessibility tree, where a control is addressed as “button named Submit” rather than “the rectangle near coordinate 812, 644.” A reference is a lease on one observation, not a permanent address.
# @save
class StaleObservation(RuntimeError):
"""Raised when the UI changed between observation and action."""
@dataclass(frozen=True)
class UINode:
"""One control from an accessibility tree: role, accessible name, and a ref.
Semantic addressing ("the button named Submit") survives layout changes that
would break a pixel coordinate, and it audits cleanly. The ``ref`` is a lease
on one observation, not a permanent address.
"""
role: str
name: str
ref: str
def resolve(tree: list[UINode], role: str, name: str) -> str:
"""Resolve a semantic target to exactly one element reference.
Grounding fails safe: zero matches means the control is gone and more than
one means the name is ambiguous. Either way we refuse to act rather than
click a guess, because acting on a misresolved control is how a browser
agent reports success while the real state never changed.
Args:
tree: The current accessibility-tree snapshot.
role: The ARIA role to match (for example ``"button"``).
name: The accessible name to match.
Returns:
The single matching element's ``ref``.
Raises:
StaleObservation: If zero or more than one node matches.
"""
matches = [node.ref for node in tree if node.role == role and node.name == name]
if len(matches) != 1:
raise StaleObservation(f"{name!r} resolved to {len(matches)} controls, expected 1")
return matches[0]
def tree_digest(tree: list[UINode]) -> str:
"""Return a freshness token over a tree snapshot's roles, names, and refs."""
return digest("|".join(f"{n.role}:{n.name}:{n.ref}" for n in tree))We resolve “Submit” on a clean page, then let the page rerender — a confirmation modal adds a second Submit — and re-observe before committing.
tree_v1 = [UINode("button", "Submit", "b1"), UINode("button", "Cancel", "b2"),
UINode("textbox", "Amount", "t1")]
target = resolve(tree_v1, "button", "Submit")
token = tree_digest(tree_v1)
print(f"resolved 'Submit' -> {target}; freshness token {token}")
tree_v2 = tree_v1 + [UINode("button", "Submit", "b9")] # a modal added a second Submit
print(f"before commit, re-observe -> {tree_digest(tree_v2)} changed: {tree_digest(tree_v2) != token}")
try:
resolve(tree_v2, "button", "Submit")
except StaleObservation as exc:
print("re-resolve refuses:", exc)resolved 'Submit' -> b1; freshness token 331643db890e
before commit, re-observe -> a8629a44c60c changed: True
re-resolve refuses: 'Submit' resolved to 2 controls, expected 1The freshness token changed, and re-resolution refuses because “Submit” is now ambiguous — so the stale reference b1 is never clicked. The last habit is to verify the resulting state, not the click. A click receipt proves an input event was dispatched; it does not prove the business action completed.
click_receipt = {"event": "click", "ref": "b1", "dispatched": True}
order_record = {"id": "A-1001", "status": "open"} # backend truth after the blocked action
print(f"receipt dispatched={click_receipt['dispatched']}, "
f"but order status is {order_record['status']!r} -> not submitted")receipt dispatched=True, but order status is 'open' -> not submittedThe receipt says the click happened; the authoritative order record says the order is still open. Verification prefers the durable record — a sent-item id, a transaction receipt, a reopened document — over any UI narration. Web content is also untrusted input: instructions embedded in a page do not gain authority because the agent can read them, and bot-detection or auth failures are environment states to surface as typed handoffs, not puzzles to evade. Chapter 24 develops those injection defenses; the architectural rule here is to keep observed content separate from the instruction and authority channels, and to lower autonomy — stage the work, capture the final screen, request takeover — whenever the interface exposes no authoritative state. Reproducible web environments with final-state checks are what made this measurable in the first place (Zhou et al. 2024).
21.5 Conversational customer service: policy-gated effects
Conversational customer service looks different — the dominant observation is dialogue — but it is the same contract. The observation is the conversation plus account state; the proposal vocabulary is a policy-constrained tool set; the verifier is the resulting account state and policy compliance; the recovery is correction, compensation, or escalation. τ-bench made exactly this explicit by scoring the final database state and policy following rather than the transcript (Yao et al. 2024). We build the smallest instance that shows it: a refund tool gated by policy, verified against the account.
# @save
@dataclass
class Account:
"""A tiny customer account: the authoritative state a refund tool must move."""
customer_id: str
paid: float
days_since_purchase: int
refunded: float = 0.0
def refund(account: Account, amount: float, policy: dict[str, float]) -> dict[str, Any]:
"""Apply a policy-gated refund and verify it against final account state.
The action language is a policy-constrained tool, and the verifier is the
resulting account state plus policy compliance. Anything outside policy is
not an error to retry but a typed escalation carrying exactly what a human
needs to decide.
Args:
account: The mutable account record.
amount: The refund the dialogue requested.
policy: Limits with ``max_days`` and ``max_amount`` keys.
Returns:
Either a ``committed`` verdict with the verified post-state, or an
``escalate`` verdict with a decision packet.
"""
remaining = account.paid - account.refunded
if account.days_since_purchase > policy["max_days"]:
reason = f"{account.days_since_purchase} days exceeds the {int(policy['max_days'])}-day window"
elif amount > policy["max_amount"]:
reason = f"amount {amount:.0f} exceeds the tool limit {policy['max_amount']:.0f}"
elif amount > remaining:
reason = f"amount {amount:.0f} exceeds the refundable balance {remaining:.0f}"
else:
account.refunded += amount # commit: move the DB state
verified = account.refunded == amount # re-read the final state
return {"verdict": "committed", "refunded": account.refunded, "verified": verified}
return {"verdict": "escalate",
"packet": {"customer": account.customer_id, "requested": amount,
"reason": reason, "decision": "approve exception?"}}We run one request inside policy and one outside it.
policy = {"max_days": 30, "max_amount": 100.0}
print("within policy :", refund(Account("cust-7", paid=80.0, days_since_purchase=12), 80.0, policy))
print("outside policy:", refund(Account("cust-9", paid=80.0, days_since_purchase=45), 80.0, policy))within policy : {'verdict': 'committed', 'refunded': 80.0, 'verified': True}
outside policy: {'verdict': 'escalate', 'packet': {'customer': 'cust-9', 'requested': 80.0, 'reason': '45 days exceeds the 30-day window', 'decision': 'approve exception?'}}The in-policy refund commits and the account confirms it — the verifier reads final state, not the model’s assurance. The out-of-policy request does not become a creative workaround; it becomes a typed escalation packet naming the customer, the request, the reason, and the exact decision asked of a human. Escalation is a first-class outcome with a useful payload, which is the general pattern the next two sections generalize.
21.6 Rigor where the verifier is weak
Coding enjoys an unusually honest verifier when tests are good. Data and analytics agents make the opposite case sharp: a SQL query can execute successfully while answering the wrong question, and a chart can render perfectly while mixing units or double-counting rows. The remedy is not a generic “critic” prompt; it is to turn analytical assumptions into checkable application state. For every computed quantity, retain a lineage chain
\text{snapshot} \rightarrow \text{selection} \rightarrow \text{transforms} \rightarrow \text{assumptions} \rightarrow \text{claim}, \tag{21.2}
naming snapshot version, filters, joins, units, denominators, and aggregation grain, so a number in the report points back to how it was produced. Two failure bands are common enough to gate directly. The first is units. Distinct sources report money in dollars, thousands, or millions, and summing them raw is silently wrong.
# @save
class UnitError(ValueError):
"""Raised when quantities with different declared units are combined raw."""
def to_dollars(amount: float, unit: str) -> float:
"""Normalize a money figure to dollars from its declared unit.
Args:
amount: The raw figure as reported by its source.
unit: One of ``"dollars"``, ``"thousands"``, or ``"millions"``.
Returns:
The figure in dollars.
Raises:
UnitError: If the unit is not recognized.
"""
factors = {"dollars": 1.0, "thousands": 1_000.0, "millions": 1_000_000.0}
if unit not in factors:
raise UnitError(f"unknown money unit: {unit!r}")
return amount * factors[unit]figures = [(1200.0, "dollars"), (3.0, "thousands")]
naive = sum(amount for amount, _ in figures)
normalized = sum(to_dollars(amount, unit) for amount, unit in figures)
print(f"naive sum (ignores units) = {naive:.0f} unit-normalized = {normalized:.0f}")naive sum (ignores units) = 1203 unit-normalized = 4200The naive total of 1203 is off by more than triple; normalizing to dollars first gives 4200. The second band is the join fan-out — the failure the analysis called out as the one that makes analytics agents quietly wrong. A join whose right-hand key is not unique multiplies rows, and every SUM over the result double-counts. The cheap guard is a row-count reconciliation.
# @save
def reconcile_join(left_rows: int, joined_rows: int, key_unique: bool) -> None:
"""Fail loudly when a join fanned out and will double-count aggregates.
A join that multiplies rows silently inflates every SUM taken over it. The
cheap guard is a row-count reconciliation: if the right-hand key is not
unique and the join produced more rows than it started with, an aggregate
over the result is not trustworthy and must be recomputed on deduplicated
grain.
Args:
left_rows: Row count of the left table before the join.
joined_rows: Row count after the join.
key_unique: Whether the join key is unique on the right table.
Raises:
UnitError: Reused as a rigor-gate failure when a fan-out is detected.
"""
if not key_unique and joined_rows > left_rows:
raise UnitError(f"join fan-out: {left_rows} rows -> {joined_rows}; "
f"aggregates will double-count")We compute a revenue total the wrong way — joining five orders to a customer-address table where one customer has two addresses — then let the gate catch it.
orders = [("o1", "A", 100), ("o2", "A", 50), ("o3", "B", 200), ("o4", "C", 30), ("o5", "B", 20)]
addresses = {"A": ["addr1", "addr2"], "B": ["addr3"], "C": ["addr4"]}
joined = [(order, addr, amount) for order, cust, amount in orders for addr in addresses[cust]]
naive_total = sum(amount for _, _, amount in joined)
correct_total = sum(amount for _, _, amount in orders)
print(f"orders={len(orders)} joined_rows={len(joined)} "
f"naive SUM over join={naive_total} correct SUM over orders={correct_total}")
try:
reconcile_join(len(orders), len(joined), key_unique=False)
except UnitError as exc:
print("gate:", exc)orders=5 joined_rows=7 naive SUM over join=550 correct SUM over orders=400
gate: join fan-out: 5 rows -> 7; aggregates will double-countThe naive total of 550 overstates real revenue of 400 by exactly the two duplicated customer-A orders, and the reconciliation gate catches it from the row counts alone, before the number ever reaches a chart. “Verify against database state” means this concretely: reconcile row counts and key uniqueness at each join, check units and denominators, and re-read the authoritative record after any write. The remaining bands — leakage of future or post-outcome fields, multiplicity across many tried analyses (Simmons et al. 2011), and causal claims that association cannot support — extend the same principle: make the assumption explicit state and gate it, because the model’s verifier here is weak by construction. Scientific-discovery agents sit at the far end of this spectrum: hypothesis generation is cheap, but experimental validity and expert vetoes remain the hard verifier, so the durable architecture is an evidence-and-experiment ledger with human-owned commit points.
21.7 Escalation follows authority and reversibility
An application’s domain label does not set its risk; its authority and reversibility do. A coding agent formatting a local notebook and one deploying authentication changes share tools but not consequences. So profile deployments by what a mistake can touch, not by which family they belong to.
| Profile | Typical behavior | Commit rule | Dominant residual risk |
|---|---|---|---|
| Ambient | Observes context, suggests next steps | User initiates every effect | Privacy, distraction, overreliance |
| Proactive | Starts bounded work from events or schedules | Auto-commit only for reversible, low-impact effects | Wrong trigger, stale context, action volume |
| Cyber / privileged | Changes code, infrastructure, accounts, security state | Isolation plus explicit authority and independent verification | Capability abuse, lateral movement, destructive error |
| Regulated / professional | Informs health, legal, finance, employment decisions | Qualified review and evidence retention before use | Invalid advice, disparate harm, noncompliance |
These profiles recur in Chapters 24 and 27, where controls and operations are taught; here they answer one design question — where must the provisional-to-committed transition sit — and the answer is never a generic confidence threshold, because a model can be confident and wrong and risk comes from the effect as much as the error rate. Whatever the profile, escalation is a typed outcome carrying the goal, the current versioned state, the actions taken, the proposed effect, the evidence and verifier results, and the exact decision requested — the same shape as the refund packet above. Persistence and parallelism must preserve that packet rather than blur it: a fleet of coding workers needs isolated checkouts and a merge authority, parallel researchers need shared source and claim identities, and a resumed computer-use thread must re-observe the live application rather than replay a stale reference. Chapter 20 owns coordination and Chapter 26 owns durable execution; the application rule is that a restart or handoff invalidates ephemeral observations unless the environment proves otherwise.
21.8 Choose the smallest architecture that can prove completion
The mature reference architectures are valuable because they expose recurring seams, not because every product needs their maximum autonomy. Work backward from the postcondition: if the team cannot state what observation proves completion, keep the system advisory until one is engineered. Then pick the cheapest safe action vocabulary — a typed domain tool over a shell, a semantic element over a coordinate, a constrained query over arbitrary code — separate staging from commit, bind every proposal to observed state, and choose a concrete recovery mechanism before granting autonomy.
| Situation | Smallest suitable architecture |
|---|---|
| One known transform with a deterministic check | Ordinary function or fixed workflow |
| Several known tools and predictable branches | Typed workflow with model decisions at narrow seams |
| Unknown number of inspect–act–verify cycles | Bounded single-agent loop |
| Independent read-heavy subquestions, shared evidence schema | Parallel workers behind one synthesis contract |
| Weak verifier plus consequential, irreversible action | Draft or stage only; a human commits |
| No authoritative postcondition | Advisory assistant until one is engineered |
Do not add planner, coder, critic, and verifier personas to match a reference diagram: a separate verifier earns its place only when it has independent evidence or authority, and a second model reading the same incomplete context raises confidence without raising truth. Test architectures through ablations of seams — repository maps versus none, edit formats, ledger-versus-transcript synthesis, semantic versus coordinate actions — holding the goal and verifier fixed. Chapter 22 supplies the evaluation mechanics; the application designer supplies the construct and the authoritative environment.
21.9 Summary
Every agent application is the same contract — observation, proposal, execution, verifier, commit, recovery — wrapped around the Chapter 16 loop, and its architecture is the provisional-to-committed boundary. We built a minimal coding agent whose executor owns every invariant and scored it on an embedded suite, watching a repair driven by a failing test, an escape rejected before write, and four of six tasks fall when the budget dropped to one. The same skeleton caught an unsupported citation, refused a stale UI control while checking state over a receipt, verified a policy-gated refund against the account, and flagged a join fan-out inflating 400 to 550. Chapter 22 turns these into defensible measurements.
21.10 Exercises
- Add a task that passes for the wrong reason. Extend
data/ch21/tasks.jsonwith a seventh task whose first edit is syntactically valid but changes the wrong function, so the tests still fail. Predict the event counts (proposals,test_runs,resolved) before runningrun_suite, add the two scripted proposals, and verify your prediction without touching the loop. - Break the oracle, and watch the guard hold. Write a proposal that targets a
test_*.pyfile to “fix” a failing test by weakening its assertion. Show thatsafe_pathrejects it, then explain which of the four verifier properties (authority, specificity, freshness, independence) each of the threesafe_pathrules defends. - Make the digest earn its keep. Using
apply_guarded_edit, construct a case where the exact-anchor applier silently edits the wrong occurrence after a concurrent change while the digest-guarded applier rejects. Then measure, over ten synthetic edits with random concurrent insertions, how many each applier applies to an unseen version. - Audit a real-shaped ledger. Design an
EvidenceRecordset for a market analysis with a currency conversion, two conflicting market-size estimates, and a forecast. Runcitation_auditand extend it to also flag a claim supported only by two records that share a single primary source (non-independent corroboration). - Classify a booking flow. For a travel-booking browser agent, list twelve concrete actions and label each inspect, stage, or commit. For every commit action name an authoritative postcondition, a freshness check like
tree_digest, and an undo or compensation path — and identify the one action where Chapter 29’s pixel perception is genuinely required. - Gate your own analysis. Take an analytics question from your work and enumerate its units, population, snapshot time, filters, join keys, and any causal assumption. Using
to_dollarsandreconcile_joinas models, write one deterministic gate for each assumption that can be checked in code, and mark which ones need domain review instead. - Move a family across the autonomy plane. Pick one family from Figure 21.2 and describe a concrete engineering change — a stronger verifier or a cheaper undo — that moves it into a different quadrant. State what
autonomy_regimewould then return and what new control the move obligates you to build.