# @save
def should_split(*, decomposable: bool, useful_boundary: bool, measured_failure: bool) -> bool:
"""Return True only when all three conditions for a team are present.
A multi-agent design is justified only when the work can actually run in
parallel, the split creates a real context/permission/model/ownership
boundary, and the strongest single-agent baseline has a *measured* failure the
team targets. Any missing condition is a veto: keep one agent.
Args:
decomposable: The work has independent or weakly coupled branches.
useful_boundary: The split creates a real boundary, not a renamed persona.
measured_failure: The best single-agent baseline has a named, measured gap.
Returns:
True if the team is justified; False if any condition fails.
"""
return decomposable and useful_boundary and measured_failure20 Multi-Agent Systems: Orchestrator-Workers, Failure Modes, and Economics
We are now ready to build a small multi-agent system and measure whether it earns its keep. The artifact is an orchestrator with three workers that answer a policy-research question — one worker per region, each reading only its own shard of documents — while a single agent answers the same question alone. We build every piece inline on a minimal read loop: (i) a stub worker that retrieves over its shard and returns one typed, compressed finding; (ii) a workspace of immutable JSON handoff artifacts; (iii) an orchestrator that fans the work out, validates each report at a merge gate, and is the only writer of the final answer; and (iv) a cost-matched single agent for the A/B. Then we do the two things that separate a team that helps from a team that just costs more: we measure the token multiplier and the latency the parallelism buys, and we inject a fault into one worker and watch the gate contain and attribute it.
The thesis, which every measurement in this chapter supports, is that context architecture — what crosses each boundary, who may write, where an error can propagate — decides whether multiple agents help, and often the honest answer is one agent. A team is a decomposition choice, not a maturity level. Our workers are deterministic stubs so the reader can see exactly what they do; a hosted model would replace the ranking with a real search-and-read loop, but the briefs, the typed reports, the gate, and the accounting are what production actually depends on. The single-agent loop and tool boundaries are developed in Chapter 16 and Chapter 17; here we compose them.
20.1 When adding an agent is a mistake
Adding an agent creates a distributed system whose messages are lossy, expensive, and partly generated by a model. Before choosing a topology, ask whether the task should be split at all. The failure mode we most want to avoid is the one that looks like progress: three agents named “researcher,” “critic,” and “writer” running the same prompt, tripling the token bill to produce what one agent produced.
A multi-agent design needs three positive conditions, all present at once. First, the work is genuinely decomposable: it has independent or weakly coupled branches that can run concurrently, not a single chain of dependent reasoning wearing three hats. Second, the split creates a useful boundary: each worker gets a smaller context, a different tool or permission, a distinct model, or an independently owned service — renaming identical prompts is not specialization. Third, there is a measured failure of the strongest simple baseline: the best single agent, already equipped with sharper tools and just-in-time context, has a named failure or a deadline the team is meant to fix. We encode the test as an explicit gate so that “we could split this” never masquerades as “we should.”
The gate is boring on purpose. Run it across the cases and note how often “add an agent” is the wrong move.
cases = {
"three renamed personas": dict(decomposable=True, useful_boundary=False, measured_failure=True),
"one dependent chain": dict(decomposable=False, useful_boundary=True, measured_failure=True),
"team with no baseline gap":dict(decomposable=True, useful_boundary=True, measured_failure=False),
"independent regional search": dict(decomposable=True, useful_boundary=True, measured_failure=True),
}
for name, cond in cases.items():
print(f"{name:30} -> split? {should_split(**cond)}")three renamed personas -> split? False
one dependent chain -> split? False
team with no baseline gap -> split? False
independent regional search -> split? TrueOnly the last case clears all three, and it is the one we build. The decision can be made quantitative with an incremental-utility test that we will fill in with numbers once we have measured the team:
\Delta U = w_q\,\Delta Q - w_c\,\Delta C - w_l\,\Delta L - w_r\,\Delta R, \tag{20.1}
where \Delta Q is the change in task quality, \Delta C in total spend or tokens, \Delta L in deadline-relevant latency, and \Delta R in operational risk, each under a product-chosen weight. A negative \Delta U is not a defeat; a well-supported rejection of a multi-agent design is a successful engineering result. Figure 20.1 traces the whole decision.
flowchart TD
Start["strong single-agent baseline"] --> Fail{"measured failure or deadline remains?"}
Fail -->|no| One["keep one agent"]
Fail -->|yes| Dec{"independent or weakly coupled subproblems?"}
Dec -->|no| Improve["sharpen tools, context, verifier"]
Dec -->|yes| Bound{"useful context / permission / model / ownership boundary?"}
Bound -->|no| Func["parallel functions, not agent personas"]
Bound -->|yes| Trial["cost-matched team experiment"]
Trial --> Gate{"ΔU > 0 under the real weights?"}
Gate -->|yes| Team["ship a bounded topology"]
Gate -->|no| One
20.2 Context is the architecture
Once a task is split, the decisive question is what crosses each boundary. Every handoff compresses one agent’s state into another agent’s context, and missing facts, duplicated history, conflicting versions, and injected instructions all enter here. The strategies trade fidelity for cost: a full trace preserves rejected paths and uncertainty but is expensive and can carry hidden instructions; a rolling summary is compact but quietly drops constraints; a typed artifact keeps only declared fields but validates and diffs cleanly. For bounded delegation and a single-writer merge, the typed artifact wins, and the reason is worth seeing in code rather than asserting.
Suppose the user said “on-premises only” early in a long trace. A rolling summary might render that as “a secure enterprise option,” and a downstream writer could then recommend a hosted product while believing it honored the constraint. A typed requirement field cannot be paraphrased away: it is either present and equal to the required value or it fails validation.
constraint = {"deployment": "on_prem_only"} # a typed field
rolling_summary = "user prefers a secure enterprise option" # a fluent paraphrase
def respects_typed(record: dict) -> bool:
return record.get("deployment") == "on_prem_only"
print("typed field still enforces it:", respects_typed(constraint))
print("recoverable from the summary: ", "on_prem" in rolling_summary)typed field still enforces it: True
recoverable from the summary: FalseThe typed field survives; the summary has silently lost the constraint. Figure 20.2 shows the two paths side by side — the same requirement entering a handoff and surviving as a validated field while a summary drops it.
flowchart LR
Req["requirement: deployment = on_prem_only"] --> S["rolling summary"]
Req --> T["typed artifact field"]
S --> S2["'secure enterprise option'"]
S2 --> SR["writer recommends a hosted product ✗"]
T --> T2["deployment: on_prem_only"]
T2 --> TR["validation passes; hosted product rejected ✓"]
A delegation brief carries more than a constraint. It names the parent run and worker identity, the objective and what is out of scope, the input snapshot or evidence handles, the output schema and acceptance predicate, the allowed tools and destinations, the token and time budgets, and where to write the artifact and who may merge it. The invariant behind all of it: a worker cannot mint new authority. Authority only narrows on the way down, and child budgets are carved from a parent-owned total so spawning more workers cannot evade the limit. We enforce exactly that in the runtime we build next.
20.3 An orchestrator and its workers
Now the build. The task: for three regions, name the vendor whose data platform permits on-premises deployment, with the line that supports it. Each region is a shard of documents — one target and two distractors — and the workers must find the right one. Because the whole point is to show the mechanism, the “model” inside each worker is a transparent keyword ranker; we count words as a stand-in for tokens so every cost is a number the reader can check.
# @save
from __future__ import annotations
def tokens(text: str) -> int:
"""Count whitespace-separated words as a transparent token stand-in."""
return len(text.split())
def keyword_score(query: str, text: str) -> int:
"""Count how many query terms appear in one document.
This is the whole "reasoning" of the stub worker: it ranks the documents in
its shard by shared lowercased words and reads the best match. A hosted model
would replace the ranking with a real search-and-read loop; keeping it
deterministic is what lets every number in this chapter reproduce.
Args:
query: The task query whose terms we look for.
text: One document's text.
Returns:
The count of distinct query terms present in the document.
"""
return len(set(query.lower().split()) & set(text.lower().split()))
def _best_doc(query: str, docs: list[dict]) -> dict:
"""Read a shard one document at a time, keeping the best keyword match."""
best = None
for doc in docs:
if best is None or keyword_score(query, doc["text"]) > keyword_score(query, best["text"]):
best = doc
return bestThe corpus is the whole world the workers see. Each shard has the same query answer hidden among cloud-only distractors, so a worker that does not read cannot fake it.
# @save
QUERY = "on-premises deployment permitted"
OBJECTIVE = (
"For each region, name the vendor whose data platform permits "
"on-premises deployment, with the evidence line that supports it."
)
CORPUS = {
"americas": [
{"id": "am-1", "vendor": "Aurora", "on_prem": True,
"text": "Aurora permits on-premises deployment for regulated customer data"},
{"id": "am-2", "vendor": "Nimbus", "on_prem": False,
"text": "Nimbus runs as a cloud only managed analytics platform"},
{"id": "am-3", "vendor": "Delta", "on_prem": False,
"text": "Delta streams hosted dashboards over the public internet"},
],
"emea": [
{"id": "em-1", "vendor": "Basalt", "on_prem": True,
"text": "Basalt supports on-premises deployment inside a private data center"},
{"id": "em-2", "vendor": "Cirro", "on_prem": False,
"text": "Cirro is offered only as a multi tenant cloud subscription"},
{"id": "em-3", "vendor": "Loire", "on_prem": False,
"text": "Loire provides a hosted reporting service with no local install"},
],
"apac": [
{"id": "ap-1", "vendor": "Sakura", "on_prem": True,
"text": "Sakura allows on-premises deployment behind the customer firewall"},
{"id": "ap-2", "vendor": "Monsoon", "on_prem": False,
"text": "Monsoon is a cloud native warehouse with no on site option"},
{"id": "ap-3", "vendor": "Ganges", "on_prem": False,
"text": "Ganges delivers analytics purely through a hosted portal"},
],
}
print(f"{len(CORPUS)} shards, {sum(len(v) for v in CORPUS.values())} documents")3 shards, 9 documentsEach worker is delegated authority through a TaskBrief. It names the shard, the single tool the worker may call, the schema its report must match, and its budgets — the delegation envelope from the previous section, made into a frozen value so a worker cannot edit its own permissions.
# @save
from dataclasses import dataclass, asdict
@dataclass(frozen=True)
class TaskBrief:
"""The typed envelope one worker receives: scope, authority, and budget.
A brief is the whole contract for a delegated subtask. It names the region the
worker may read, the single tool it may call, the schema its report must fit,
and the token budget it may spend. A worker can only ever narrow these; the
brief is why authority flows one way, from parent to child.
"""
worker_id: str
region: str
query: str
allowed_tools: tuple[str, ...]
output_schema: str
token_budget: intWhat comes back is not a transcript. It is a compressed, typed finding: the vendor, the evidence id that grounds it, and the modeled cost of producing it. This is the “map” output that the single writer will “reduce” — and everything the merge gate checks is a field here.
# @save
@dataclass(frozen=True)
class WorkerReport:
"""A compressed, typed finding — the only thing that crosses the handoff.
The worker returns one finding plus the evidence id that supports it and the
modeled tokens and latency it spent, not its scored candidates or its
reasoning. A typed report survives a handoff that a prose summary would
corrupt, because the merge gate can validate every field.
"""
worker_id: str
region: str
finding: str
evidence_ids: tuple[str, ...]
tokens: int
latency: intThe worker itself is the minimal loop: read the shard one document at a time, keep the best match, report it. The poison flag lets us later inject the information-withholding fault by omitting the evidence id; the authority check rejects any brief that grants a tool the worker was not delegated.
# @save
class Worker:
"""A stub research worker: retrieve over one shard, report one finding.
The worker reads only the documents named in its brief and reports the top
match's vendor together with that document's id as provenance. Its modeled
token cost is the brief query plus the shard it read plus its finding, and its
latency is one unit per document scanned — the numbers a real worker would
incur reading its context.
"""
def run(self, brief: TaskBrief, docs: list[dict], poison: bool = False) -> WorkerReport:
"""Retrieve over the brief's shard and return one typed report.
Args:
brief: The authority and scope envelope for this subtask.
docs: The documents of the worker's shard.
poison: If True, omit the evidence id, simulating the
information-withholding failure the merge gate must catch.
Returns:
A :class:`WorkerReport` grounded in the retrieved document.
Raises:
PermissionError: If the brief grants any tool other than ``search`` —
a worker may not widen its own authority.
"""
if brief.allowed_tools != ("search",):
raise PermissionError("worker may not use tools outside its brief")
best = _best_doc(brief.query, docs)
evidence = () if poison else (best["id"],)
shard_text = " ".join(doc["text"] for doc in docs)
used = tokens(brief.query) + tokens(shard_text) + tokens(best["vendor"])
return WorkerReport(brief.worker_id, brief.region, best["vendor"], evidence,
tokens=used, latency=len(docs))We can run one worker in isolation and read its typed finding. This is the handoff artifact in miniature: a bounded value, not a conversation.
brief = TaskBrief("worker-1", "americas", QUERY, ("search",), "WorkerReport/v1", token_budget=200)
report = Worker().run(brief, CORPUS["americas"])
print(report)WorkerReport(worker_id='worker-1', region='americas', finding='Aurora', evidence_ids=('am-1',), tokens=29, latency=3)The worker found Aurora and cited am-1, at a cost of its query plus the shard it read. The handoff is a shared file, not an in-memory object: the worker writes one immutable JSON report and the orchestrator reads it back to validate it. A Workspace gives us addressable, inspectable artifacts and enforces single-writer ownership at the filesystem — a report cannot be overwritten and cannot escape the directory.
# @save
import json
from pathlib import Path
class Workspace:
"""A parent-owned directory of immutable JSON handoff artifacts.
Each worker writes exactly one report file; the orchestrator reads it back and
validates it. The write refuses to overwrite an existing report or to escape
the root, so the filesystem — not the workers' good behavior — enforces
single-writer ownership and immutability of the handoff.
"""
def __init__(self, root: Path) -> None:
self.root = root.resolve()
self.root.mkdir(parents=True, exist_ok=True)
def write(self, report: WorkerReport) -> Path:
"""Persist one report as immutable JSON and return its path.
Args:
report: The typed finding to store.
Returns:
The path written.
Raises:
PermissionError: If the target path escapes the workspace root.
FileExistsError: If a report for this worker already exists.
"""
path = (self.root / f"{report.worker_id}.json").resolve()
if not path.is_relative_to(self.root):
raise PermissionError("report path escapes workspace")
if path.exists():
raise FileExistsError(f"report already exists: {path.name}")
path.write_text(json.dumps(asdict(report), indent=2), encoding="utf-8")
return path
def read(self, path: Path) -> WorkerReport:
"""Read one report back, restoring the tuple-typed evidence field."""
data = json.loads(Path(path).read_text(encoding="utf-8"))
data["evidence_ids"] = tuple(data["evidence_ids"])
return WorkerReport(**data)Every worker output is untrusted input, even inside one process, so the report crosses a merge gate before it can become part of the answer. The gate confirms the finding names a real vendor in the shard and that the report carries the evidence backing it. A Failure records the attribution — what, who, where, why — so a contained failure is never mistaken for a success.
# @save
@dataclass(frozen=True)
class Failure:
"""A MAST-style attribution: what category, which agent, which step, and why."""
category: str
agent: str
step: str
reason: str
def validate_report(report: WorkerReport, docs: list[dict]) -> Failure | None:
"""Check one report at the merge boundary; return a Failure or None.
The gate confirms the finding names a real vendor in the shard and that the
report carries the evidence id that grounds it. A missing evidence id is the
information-withholding failure (MAST 2.4): the claim may be correct, but it is
unsupported and must not be laundered into the team's answer.
Args:
report: The worker's typed finding.
docs: The shard the worker was asked to read, for grounding.
Returns:
A :class:`Failure` for the first problem found, or None if the report
passes schema, grounding, and provenance checks.
"""
vendors = {doc["vendor"] for doc in docs}
if report.finding not in vendors:
return Failure("inter-agent misalignment", report.worker_id, "handoff", "unsupported finding")
if not report.evidence_ids:
return Failure("inter-agent misalignment", report.worker_id, "handoff", "missing provenance")
return NoneBoth architectures return the same comparable shape so an A/B is a field-by-field diff, and status separates a contained failure from task success.
# @save
@dataclass(frozen=True)
class RunResult:
"""One run's comparable outcome: answer, quality, and modeled budget.
Both the team and the single agent return this shape, so the A/B is a diff.
``status`` is ``"completed"`` on success or ``"contained"`` when a gate stops
an attributed failure, keeping containment success and task success apart.
"""
architecture: str
status: str
answer: str | None
score: float
tokens: int
latency: int
failure: Failure | None = NoneNow the orchestrator ties it together. It turns the objective into one brief per region, fans the workers out concurrently over isolated shards, validates each report at the gate, and — only if every report passes — synthesizes the single answer. The token accounting is the identity we will study in the economics section; the latency is a plan step, plus the maximum worker (they run in parallel), plus a merge step.
# @save
from concurrent.futures import ThreadPoolExecutor
PLAN_LATENCY = 1
MERGE_LATENCY = 1
class Orchestrator:
"""Decompose the objective, fan out workers, and be the only writer.
The orchestrator issues one typed brief per region, runs the workers over
isolated shards (the map), validates each report at the merge gate, and
synthesizes the single answer (the reduce). Workers never decide the parent
task is done and never write the final answer, so an unvalidated finding
cannot reach the user.
"""
def run(self, objective: str, corpus: dict, workspace: Workspace,
poison_worker: str | None = None, verify: bool = True) -> RunResult:
"""Run the orchestrator-worker team over the corpus.
Args:
objective: The parent task, restated into each worker's brief.
corpus: Mapping of region -> list of shard documents.
workspace: Parent-owned store for the handoff artifacts.
poison_worker: Worker id that should omit its evidence (fault injection).
verify: If False, skip the merge gate so unsupported findings reach the
answer — the no-verification failure (MAST 3.2).
Returns:
A :class:`RunResult`; ``status`` is ``"completed"`` on success or
``"contained"`` when the gate stops an attributed failure.
"""
briefs = [
TaskBrief(f"worker-{i}", region, QUERY, ("search",), "WorkerReport/v1", token_budget=200)
for i, region in enumerate(corpus, start=1)
]
with ThreadPoolExecutor(max_workers=len(briefs)) as pool:
paths = list(pool.map(
lambda b: workspace.write(
Worker().run(b, corpus[b.region], poison=(b.worker_id == poison_worker))),
briefs,
))
reports = [workspace.read(path) for path in paths]
findings = [report.finding for report in reports]
answer = ", ".join(sorted(findings))
plan_tokens = tokens(objective) + sum(tokens(brief.query) for brief in briefs)
worker_tokens = sum(report.tokens for report in reports)
merge_tokens = tokens(objective) + sum(tokens(f) for f in findings) + tokens(answer)
total_tokens = plan_tokens + worker_tokens + merge_tokens
latency = PLAN_LATENCY + max(report.latency for report in reports) + MERGE_LATENCY
if verify:
for brief, report in zip(briefs, reports):
failure = validate_report(report, corpus[brief.region])
if failure:
return RunResult("orchestrator-worker", "contained", None, 0.0,
total_tokens, latency, failure)
return RunResult("orchestrator-worker", "completed", answer, 1.0, total_tokens, latency)Run the team on the full corpus and read the result.
from tempfile import TemporaryDirectory
with TemporaryDirectory() as directory:
team = Orchestrator().run(OBJECTIVE, CORPUS, Workspace(Path(directory)))
print(team)RunResult(architecture='orchestrator-worker', status='completed', answer='Aurora, Basalt, Sakura', score=1.0, tokens=146, latency=5, failure=None)Three isolated workers found the three on-premises vendors, the gate passed every report, and the single writer merged them into one answer. Figure 20.3 locates the components: the map fans out to read-only workers, the reduce is a single validated writer, and no worker writes global state directly.
flowchart TB
Obj["objective + constraints"] --> O["orchestrator: plan, budget, terminate"]
O --> B1["brief A"] --> W1["worker 1 (shard A)"]
O --> B2["brief B"] --> W2["worker 2 (shard B)"]
O --> B3["brief C"] --> W3["worker 3 (shard C)"]
W1 --> G["merge gate: schema, grounding, provenance"]
W2 --> G
W3 --> G
G --> M["single-writer merge"] --> A["answer"]
A hosted worker would do the same job with a real model call. The cell below is what one worker dispatch looks like against the Messages API, constrained to its shard and returning the same typed finding through structured outputs; it needs a network and a key, so it does not run in the book. The block after it shows a representative response with the exact shape.
import anthropic
client = anthropic.Anthropic()
shard = "\n".join(f'{doc["id"]}: {doc["text"]}' for doc in CORPUS["americas"])
response = client.messages.create(
model="claude-opus-4-8",
max_tokens=256,
system=(
"You are a research worker. Use ONLY the documents in the user message. "
"Return the vendor that permits on-premises deployment and the id of the "
"line that supports it. Do not use outside knowledge."
),
messages=[{"role": "user", "content": f"query: {QUERY}\n\n{shard}"}],
output_config={"format": {"type": "json_schema", "schema": {
"type": "object",
"properties": {
"finding": {"type": "string"},
"evidence_ids": {"type": "array", "items": {"type": "string"}},
},
"required": ["finding", "evidence_ids"],
"additionalProperties": False,
}}},
)
print(response.content[0].text)Representative response (claude-opus-4-8, structured output):
{"finding": "Aurora", "evidence_ids": ["am-1"]}The typed schema is the point: the hosted worker returns the same bounded artifact our stub does, so the orchestrator, the gate, and the accounting never need to know which one produced it.
20.4 What parallelism buys
Fan-out changes the critical path from a sum toward a maximum. A single agent scans the three shards one after another; the team scans them at the same time, so its worker phase is the slowest worker, not their total. Total compute does not fall — the same documents are read either way — but wall-clock time does, and only for genuinely independent work. Our modeled critical path is
L_{\text{team}} = L_{\text{plan}} + \max_i L_{\text{worker}_i} + L_{\text{merge}}, \tag{20.2}
against L_{\text{single}} = L_{\text{plan}} + \sum_i L_{\text{worker}_i} + L_{\text{merge}} for one sequential agent. Each worker’s latency is the number of documents it reads. We read the two totals off a team run and its cost-matched single agent, which we build next; for now, Figure 20.4 draws where the time goes.
Show the code that draws this figure
import matplotlib.pyplot as plt
scans = [len(docs) for docs in CORPUS.values()] # per-worker latency units
fig, ax = plt.subplots(figsize=(7.0, 2.6))
# Single agent: plan, then three scans in sequence, then merge.
x = PLAN_LATENCY
ax.broken_barh([(0, PLAN_LATENCY)], (10, 6), facecolors="#94a3b8")
for i, s in enumerate(scans):
ax.broken_barh([(x, s)], (10, 6), facecolors=["#315b8a", "#2f855a", "#b7791f"][i], edgecolor="white")
x += s
ax.broken_barh([(x, MERGE_LATENCY)], (10, 6), facecolors="#94a3b8")
single_total = x + MERGE_LATENCY
# Team: plan, three scans concurrently (max), then merge.
ax.broken_barh([(0, PLAN_LATENCY)], (2, 6), facecolors="#94a3b8")
for i, s in enumerate(scans):
ax.broken_barh([(PLAN_LATENCY, s)], (2, 6), facecolors=["#315b8a", "#2f855a", "#b7791f"][i], alpha=0.55, edgecolor="white")
team_total = PLAN_LATENCY + max(scans) + MERGE_LATENCY
ax.broken_barh([(PLAN_LATENCY + max(scans), MERGE_LATENCY)], (2, 6), facecolors="#94a3b8")
ax.set_yticks([5, 13])
ax.set_yticklabels([f"team ({team_total})", f"single ({single_total})"])
ax.set_xlabel("modeled latency units")
ax.set_title("Fan-out turns a sum of scans into a max")
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()
The gray plan and merge steps are fixed overhead; the colored worker bars are where the two architectures diverge. Parallelism does not make the reasoning free — it overlaps three reads that would otherwise queue. Real fan-out erodes this gain: concurrency caps, rate limits, retries, and a single straggler all push the team’s bar back toward the sum. The measured speedup we report next is an upper bound the deployment must defend.
20.5 The cost of a team
Now the honest bill. Multi-agent cost is not agent count times one call. Each worker re-reads its brief, and the orchestrator re-reads the objective to plan and re-reads every report to merge. The team’s total tokens decompose as
C_{\text{team}} = C_{\text{plan}} + \sum_i C_{\text{worker}_i} + C_{\text{merge}}, \tag{20.3}
while a lean single agent spends only C_{\text{single}} = C_{\text{obj}} + \sum_i C_{\text{shard}_i} + C_{\text{answer}}: it reads the objective once, reads every shard once, and writes the answer, with no briefs to duplicate and no reports to re-read. We compute both, term by term, from the same corpus.
# @save
def cost_breakdown(objective: str, corpus: dict) -> dict:
"""Decompose team and single-agent token cost by phase for one task.
The team's plan re-reads the objective and writes a brief query per worker;
each worker re-reads its brief and shard; the merge re-reads the objective and
every finding. The single agent reads the objective and shards once. Exposing
the terms shows exactly where the multiplier comes from.
Args:
objective: The parent task string.
corpus: Mapping of region -> shard documents.
Returns:
A dict of the team's plan/workers/merge terms and totals and the single
agent's objective/shards/answer terms and total.
"""
regions = list(corpus)
findings = [_best_doc(QUERY, corpus[region])["vendor"] for region in regions]
answer = ", ".join(sorted(findings))
plan = tokens(objective) + sum(tokens(QUERY) for _ in regions)
workers = sum(
tokens(QUERY) + tokens(" ".join(doc["text"] for doc in corpus[region])) + tokens(_best_doc(QUERY, corpus[region])["vendor"])
for region in regions
)
merge = tokens(objective) + sum(tokens(f) for f in findings) + tokens(answer)
shards = sum(tokens(" ".join(doc["text"] for doc in corpus[region])) for region in regions)
single = tokens(objective) + shards + tokens(answer)
return {"plan": plan, "workers": workers, "merge": merge, "team": plan + workers + merge,
"obj": tokens(objective), "shards": shards, "answer": tokens(answer), "single": single}bd = cost_breakdown(OBJECTIVE, CORPUS)
print(f"team : plan {bd['plan']} + workers {bd['workers']} + merge {bd['merge']} = {bd['team']}")
print(f"single: obj {bd['obj']} + shards {bd['shards']} + answer {bd['answer']} = {bd['single']}")
print(f"token multiplier = {bd['team'] / bd['single']:.2f}x")team : plan 28 + workers 93 + merge 25 = 146
single: obj 19 + shards 81 + answer 3 = 103
token multiplier = 1.42xThe team spends about 1.4 times the tokens of the lean single agent for the same three findings. The extra is not mysterious: it is the planning briefs, the objective re-read at merge, and the findings re-read to synthesize — duplicated context, the tax every handoff levies. Figure 20.5 shows the two bills stacked so the equation and the numbers are one picture.
Show the code that draws this figure
fig, ax = plt.subplots(figsize=(6.4, 3.2))
ax.bar("team", bd["plan"], color="#94a3b8", label="plan")
ax.bar("team", bd["workers"], bottom=bd["plan"], color="#315b8a", label="workers")
ax.bar("team", bd["merge"], bottom=bd["plan"] + bd["workers"], color="#b7791f", label="merge")
ax.bar("single", bd["obj"], color="#94a3b8")
ax.bar("single", bd["shards"], bottom=bd["obj"], color="#315b8a")
ax.bar("single", bd["answer"], bottom=bd["obj"] + bd["shards"], color="#b7791f")
ax.set_ylabel("token units")
ax.set_title(f"Same answer, {bd['team'] / bd['single']:.2f}x the tokens")
ax.legend(loc="upper right", fontsize=8)
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()
To compare fairly we hold compute matched, the cost-aware and reproducible discipline that agent evaluation demands (Kapoor et al. 2024): the single agent is given the team’s full token budget and asked the same question. It does not need the extra budget, which is exactly the point — the team’s token premium bought latency, not quality.
# @save
def single_agent(objective: str, corpus: dict, token_budget: int) -> RunResult:
"""Solve the whole task in one context, sequentially, at a fixed budget.
The single agent reads every shard in one context and writes the answer, with
no briefs to duplicate and no reports to re-read, so it spends fewer tokens.
But it scans the shards one after another, so its modeled latency is the sum of
the per-shard costs rather than the maximum.
Args:
objective: The parent task.
corpus: Mapping of region -> shard documents.
token_budget: Budget granted for a cost-matched comparison; the run fails
closed below the tokens it actually needs.
Returns:
A :class:`RunResult` for the single-agent architecture.
"""
findings, scan = [], 0
for docs in corpus.values():
findings.append(_best_doc(QUERY, docs)["vendor"])
scan += len(docs)
answer = ", ".join(sorted(findings))
shard_tokens = sum(tokens(" ".join(doc["text"] for doc in docs)) for docs in corpus.values())
used = tokens(objective) + shard_tokens + tokens(answer)
if token_budget < used:
return RunResult("single-agent", "budget_exhausted", None, 0.0, token_budget, 0)
latency = PLAN_LATENCY + scan + MERGE_LATENCY
return RunResult("single-agent", "completed", answer, 1.0, used, latency)The run_experiment driver ties the A/B and the fault injection into one reproducible dict — the artifact the tests import and the numbers we quote.
# @save
from tempfile import TemporaryDirectory
def run_experiment(corpus: dict, poison_worker: str | None = "worker-2") -> dict:
"""Run the cost-matched A/B and a fault injection, returning shown numbers.
Args:
corpus: Region -> shard documents.
poison_worker: Which worker omits its provenance in the fault run.
Returns:
A dict with the team run, the cost-matched single-agent run, the measured
token multiplier and latency speedup, whether the answers matched, and the
contained fault attribution.
"""
with TemporaryDirectory(prefix="ch20-") as directory:
team = Orchestrator().run(OBJECTIVE, corpus, Workspace(Path(directory) / "team"))
single = single_agent(OBJECTIVE, corpus, token_budget=team.tokens)
faulted = Orchestrator().run(OBJECTIVE, corpus, Workspace(Path(directory) / "fault"),
poison_worker=poison_worker)
return {
"team": asdict(team),
"single": asdict(single),
"token_multiplier": round(team.tokens / single.tokens, 2),
"latency_speedup": round(single.latency / team.latency, 2),
"same_answer": team.answer == single.answer,
"fault": asdict(faulted),
}result = run_experiment(CORPUS)
print("same answer: ", result["same_answer"])
print("token multiplier: ", result["token_multiplier"], "x")
print("latency speedup: ", result["latency_speedup"], "x")same answer: True
token multiplier: 1.42 x
latency speedup: 2.2 xSame answer, about 1.4 times the tokens, about 2.2 times faster. That is the whole trade in three numbers, and Equation 20.1 turns it into a decision. We need delta_utility, the scoring function behind Equation 20.1, defined once so both cases use the same arithmetic.
# @save
def delta_utility(*, d_quality: float, d_cost: float, d_latency: float, d_risk: float,
w_quality: float = 1.0, w_cost: float = 1.0,
w_latency: float = 1.0, w_risk: float = 1.0) -> float:
"""Score a design change as weighted quality gain minus its costs.
The incremental-utility test makes "is the team worth it?" a number: quality
improvement is a benefit; extra spend, deadline-relevant latency, and
operational risk are charged against it under product-chosen weights. A
negative result is a well-supported rejection of the team.
Args:
d_quality: Change in task quality (team minus baseline).
d_cost: Change in total tokens or spend (positive = the team spent more).
d_latency: Change in deadline-relevant latency (negative = the team is faster).
d_risk: Change in operational risk (positive = more failure surface).
w_quality: Weight on quality; the remaining weights price each cost term.
Returns:
The net utility; positive favors the change, negative rejects it.
"""
return w_quality * d_quality - w_cost * d_cost - w_latency * d_latency - w_risk * d_riskPlug in the measured deltas and let the weights speak: a latency-bound product prices deadline latency high and tokens low, so \Delta U > 0; a latency-insensitive product prices them the other way, and the same team scores \Delta U < 0.
d_cost = result["team"]["tokens"] - result["single"]["tokens"]
d_latency = result["team"]["latency"] - result["single"]["latency"]
latency_bound = delta_utility(d_quality=0.0, d_cost=d_cost, d_latency=d_latency, d_risk=1.0,
w_cost=0.1, w_latency=1.0, w_risk=0.5)
latency_free = delta_utility(d_quality=0.0, d_cost=d_cost, d_latency=d_latency, d_risk=1.0,
w_cost=0.1, w_latency=0.1, w_risk=0.5)
print(f"Δtokens={d_cost:+d} Δlatency={d_latency:+d}")
print(f"ΔU latency-bound product: {latency_bound:+.2f} -> {'build the team' if latency_bound > 0 else 'keep one agent'}")
print(f"ΔU latency-insensitive product:{latency_free:+.2f} -> {'build the team' if latency_free > 0 else 'keep one agent'}")Δtokens=+43 Δlatency=-6
ΔU latency-bound product: +1.20 -> build the team
ΔU latency-insensitive product:-4.20 -> keep one agentAdding workers is a latency lever, not a token lever. Each extra worker adds one brief and one finding for the parent to re-read, but it also adds one shard the single agent must scan, so the token multiplier stays a roughly flat coordination tax. The latency win, by contrast, widens: the team’s critical path is the slowest worker no matter how many there are, while the single agent’s grows with every shard. We vary that one knob and re-measure by synthesizing corpora of n equal shards.
def make_corpus(n: int) -> dict:
"""Build an n-shard corpus by cloning the base regions with fresh ids."""
base = list(CORPUS.values())
corpus = {}
for i in range(n):
template = base[i % len(base)]
corpus[f"r{i}"] = [dict(doc, id=f"r{i}-{doc['id']}") for doc in template]
return corpus
for n in (2, 3, 4, 6, 8):
r = run_experiment(make_corpus(n))
print(f"n={n}: token multiplier {r['token_multiplier']:.2f}x latency speedup {r['latency_speedup']:.2f}x")n=2: token multiplier 1.47x latency speedup 1.60x
n=3: token multiplier 1.42x latency speedup 2.20x
n=4: token multiplier 1.40x latency speedup 2.80x
n=6: token multiplier 1.36x latency speedup 4.00x
n=8: token multiplier 1.34x latency speedup 5.20xShow the code that draws this figure
ns = list(range(2, 11))
runs = [run_experiment(make_corpus(n)) for n in ns]
fig, ax = plt.subplots(figsize=(6.4, 3.2))
ax.plot(ns, [r["token_multiplier"] for r in runs], marker="o", color="#b13f3f", label="token multiplier")
ax.plot(ns, [r["latency_speedup"] for r in runs], marker="s", color="#2f855a", label="latency speedup")
ax.axhline(1.0, ls="--", color="0.6")
ax.set_xlabel("number of workers")
ax.set_ylabel("ratio vs single agent")
ax.legend()
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()
Our fixture pays a small, honest multiplier because the workers barely think. Production numbers are larger. A vendor engineering report describes a multi-agent research system using roughly fifteen times the tokens of a plain chat, and that figure is a property of that workload — deep tool catalogs, long traces, retries, and repeated context re-reads — not a universal constant. The discipline is the same at any scale: measure the multiplier on your own fixture and defend it with a positive \Delta U, not with the agent count.
Reported token multipliers are workload figures, not constants. Anthropic’s engineering account of its multi-agent research system reports roughly a 15x token cost versus chat for breadth-first research, alongside quality gains on that task; controlled studies elsewhere find the single-agent design wins on tightly coupled multi-hop reasoning under a matched budget. Treat any multiplier you quote as a dated measurement of a specific system. Verify live: re-check the primary reports, exact budgets, task definitions, and model versions before quoting a number or choosing an architecture (checked 2026-07-20).
20.6 Debate, ensembles, and diversity
Not every “multiple model calls” design is a team of agents. Voting, critique, and debate are test-time inference patterns before they are organizations, and Chapter 8 is the home for verifier-guided search and self-consistency. The relevant question here is narrower: when does running the model several times and aggregating actually help? The answer is a property of the errors, and we can measure it. Independent sampling plus a majority vote raises accuracy only when the voters’ mistakes are not perfectly correlated; five copies of one model on near-identical context share their blind spots, and the vote inherits them.
# @save
import random
def majority_vote_accuracy(n_voters: int, p_correct: float, correlation: float,
trials: int = 4000, seed: int = 0) -> float:
"""Estimate the accuracy of an n-voter majority with a shared error component.
Each trial is either a correlated case (with probability ``correlation``) in
which all voters answer alike, or an independent case in which each voter is
right on its own with probability ``p_correct``. Majority vote beats a single
voter only to the extent errors are independent — correlated blind spots erase
the gain, which is why an ensemble of near-identical model calls disappoints.
Args:
n_voters: Number of voters (odd, to avoid ties).
p_correct: Per-voter probability of being correct.
correlation: Fraction of cases in which the voters share one answer.
trials: Monte-Carlo trials.
seed: RNG seed for reproducibility.
Returns:
The fraction of trials in which the majority was correct.
"""
rng = random.Random(seed)
hits = 0
for _ in range(trials):
if rng.random() < correlation:
hits += rng.random() < p_correct # one shared vote decides all
else:
votes = sum(rng.random() < p_correct for _ in range(n_voters))
hits += votes > n_voters // 2
return hits / trialsHold the per-voter accuracy fixed at 0.65 and sweep the correlation, and the whole lesson falls out of the numbers.
p = 0.65
print(f"single voter: {p:.2f}")
for corr in (0.0, 0.5, 1.0):
print(f"5-voter majority, correlation={corr:.1f}: {majority_vote_accuracy(5, p, corr):.2f}")single voter: 0.65
5-voter majority, correlation=0.0: 0.76
5-voter majority, correlation=0.5: 0.71
5-voter majority, correlation=1.0: 0.65Independent voters lift 0.65 to about 0.76; perfectly correlated voters give back exactly the single-voter number. Figure 20.7 traces the curve between the two extremes.
Show the code that draws this figure
corrs = [c / 10 for c in range(11)]
accs = [majority_vote_accuracy(5, 0.65, c) for c in corrs]
fig, ax = plt.subplots(figsize=(6.4, 3.2))
ax.plot(corrs, accs, marker="o", color="#315b8a", label="5-voter majority")
ax.axhline(0.65, ls="--", color="0.6", label="single voter")
ax.set_xlabel("error correlation across voters")
ax.set_ylabel("accuracy")
ax.legend()
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()
Debate inherits the same constraint and adds new failure modes. Exposing candidate reasoning to peers over several rounds (Du et al. 2023) can surface missed evidence, but it also invites anchoring, conformity, verbosity, and persuasive error; a majority can converge on the same injected premise, and more rounds are not monotonically better. Treat debate as an evaluated subroutine — bound the rounds, preserve the original evidence, forbid debaters from rewriting one another’s sources, and always compare against equally priced independent samples — not as a standing council you deploy because one benchmark improved. Ensembles earn their cost only when the channels differ in a way that decorrelates errors: a different model family, retrieval source, or inductive bias, not a fifth copy of the same prompt.
Emergence and game-theoretic language help analyze a system’s behavior, not replace its contract. Local incentives, information asymmetry, and repeated interaction can produce specialization, collusion, or free riding, but an agent that looks cooperative in a simulation has not acquired a deployable guarantee. The transferable lessons are modest: environment and action constraints shape behavior more than character prose, and communication topology decides which errors propagate. Teaching policies to anticipate peers instead of prompting independent models to cooperate is a different mechanism — multi-agent reinforcement learning, developed in Chapter 23. This chapter evaluates inference-time systems as deployed software.
20.7 When a team fails: classify, attribute, contain
“The team failed” is too coarse to fix. The Multi-Agent System Failure Taxonomy (MAST) (Cemri et al. 2025) sorts observed failures into three families — specification and design (a worker ignores its role or the team never decides to stop), inter-agent misalignment (an agent withholds evidence, ignores a peer, or drifts off task), and verification (the team ends early or skips the check that would have caught the error). Attribution adds who and when: find the earliest decisive step after which the run cannot recover, because later agents amplify an error without originating it. We do not catalog all fourteen modes; we inject faults and watch the system classify, attribute, and contain them.
The first fault is information withholding. Worker 2’s finding is textually correct, but it omits the evidence id, so its claim is unsupported. The merge gate catches it, refuses to build the answer, and attributes the failure to the exact worker and step.
fault = run_experiment(CORPUS, poison_worker="worker-2")["fault"]
print("status:", fault["status"], " answer:", fault["answer"])
print("attribution:", json.dumps(fault["failure"]))status: contained answer: None
attribution: {"category": "inter-agent misalignment", "agent": "worker-2", "step": "handoff", "reason": "missing provenance"}The status is contained, not completed: a gate prevented an unsupported answer, but the user’s task is unsatisfied. Containment success and task success are different metrics, and conflating them is how a “safe” system quietly stops answering. Now the second fault — the one MAST calls no or incomplete verification. Disable the gate and the same poisoned finding is laundered straight into the answer.
with TemporaryDirectory() as directory:
unchecked = Orchestrator().run(OBJECTIVE, CORPUS, Workspace(Path(directory)),
poison_worker="worker-2", verify=False)
print("status:", unchecked.status)
print("answer:", unchecked.answer, " <- includes an unsupported finding")status: completed
answer: Aurora, Basalt, Sakura <- includes an unsupported findingNothing crashed. The unsupported claim simply became a team conclusion, indistinguishable from the grounded ones — which is why verification must be independent of the producer’s claim, and why asking the same worker “are you sure?” preserves the premise that caused the error. Figure 20.8 traces the two branches: fail-closed contains and attributes; unchecked propagation lets a poisoned report reach an effect.
sequenceDiagram
autonumber
participant W as Worker 2
participant A as Report artifact
participant G as Merge gate
participant O as Orchestrator
W->>A: confident finding, provenance omitted
A->>G: typed report
G->>G: validate schema, grounding, provenance
alt fail closed
G-->>O: attributed rejection (worker-2, missing provenance)
O-->>O: retry, drop shard, or escalate
else gate disabled
G-->>O: accepted
O->>O: unsupported claim becomes the answer
end
Topology decides blast radius. A star routes every cross-worker claim through one validation point, which is why it is easier to observe and contain than a peer mesh where an infection spreads laterally; a hierarchy contains branches only if sub-orchestrators hold disjoint permissions. The single-writer principle is the durable rule underneath: parallel agents may inspect, test, and propose, but one owner commits the answer, the file, or the deployment under a version check. When two agents would edit the same state, the fix is not to make them debate the merge; it is to change the shape — parallel reviewers return findings, one writer applies them. Classical coordination — Contract Net’s announce-bid-award routing (Smith 1980), blackboard shared state with typed append-only entries — predates LLM agents and still beats inventing persona names, but each still ends at a parent that verifies the winner’s output. The topologies compress to one rule of thumb, summarized in Table 20.1: use the smallest topology that expresses the real dependency graph.
| Topology | Routing owner | Strength | Characteristic failure |
|---|---|---|---|
| Star / supervisor | central parent | budgets, attribution, one validation point | parent bottleneck |
| Direct handoff | current agent | natural specialization chain | telephone-game drift |
| Tree / hierarchy | parent per level | scales decomposition | hidden cost, depth explosion |
| Blackboard | shared scheduler | flexible contributors | races, stale reads |
| Peer network | any peer | adaptive coordination | cycles, lateral infection, hard to terminate |
NoteIn the interview
Q. A benchmark improved when you switched from one agent to five debating agents. Ship it? Not yet. The number you are missing is the compute-matched baseline: give the single agent the same total tokens the five agents spent and re-run. Multi-agent systems routinely win by spending more, not by reasoning better, so an unmatched comparison confounds architecture with budget. The trap is treating agent count as the result; report the token multiplier and a positive \Delta U under matched compute, or report that the single agent tied it for a fraction of the cost — which is itself a successful finding.
20.8 Summary
We built an orchestrator-worker team on a minimal read loop and measured it honestly. Typed briefs carry authority down and typed reports carry compressed findings back; a workspace of immutable files and a merge gate let one writer own the answer while every worker output stays untrusted. On our fixture the team returned the same answer as a cost-matched single agent for about 1.4 times the tokens and about 2.2 times the speed, and \Delta U turned that trade into a decision that flips with the product’s weights. Injecting one fault showed the difference between a contained, attributed failure and a laundered one. Context architecture, not agent count, decides whether a team helps. Chapter 26 turns this accounting into platform cost and capacity controls.
20.9 Exercises
- Make the split fail. Feed
should_splita task that is decomposable and bounded but has no measured baseline failure, then write two sentences describing a real product where that exact combination should still be vetoed. What measurement would change your mind? - Move the multiplier. In
cost_breakdown, the merge term re-reads the objective and every finding. (a) Predict what happens to the token multiplier if worker findings become paragraphs instead of one word. (b) Change the corpus so eachvendorfield is a ten-word sentence and re-runrun_experiment; explain the new multiplier from the identity in Equation 20.3. - Erode the speedup.
Orchestrator.runmodels latency asplan + max(worker) + merge. Add a per-worker retry that costs one extra latency unit with some probability, and a concurrency cap that serializes workers beyond the cap. Re-plot Figure 20.4 and find the cap at which the team stops beating the single agent. - A second fault, attributed. Add a worker that returns a finding not present in its shard (an unsupported finding rather than a missing one). Confirm
validate_reportattributes it to the right worker and step, then state which MAST family it belongs to and why the gate must check grounding separately from provenance. - Single writer under conflict. Extend
Workspaceso two workers may propose edits to the same region file under a version check (a compare-and-swap). Show that last-write-wins silently loses one worker’s evidence, then show that the version check turns the conflict into an observation the orchestrator can act on. - Find the crossover. Using
run_experimentanddelta_utility, sweep the number of workers and, for a fixed set of weights, find the team size beyond which \Delta U goes negative. Then change the latency weight and show the crossover move. Which single weight most controls the decision on your fixture? - Cost-matched, for real. Replace the stub
Workerwith the#| eval: falsemodel-worker dispatch from Section 20.3 behind a small interface, run the A/B against a real single-agent call on the same corpus, and report task success, total tokens, and wall-clock distribution. State one way your measured multiplier differs from the fixture’s and why.