12  The API Surface: Messages, Structured Outputs, and Tool Calling

A support assistant needs the status of order O-0042. The model proposes a lookup_order call, the application runs it, and the database returns {"status": "shipped"}. A developer, in a hurry, appends that result as a plain user message — “The order shipped.” — and sends the conversation back. Sometimes the model answers. Sometimes it proposes the same lookup again. Sometimes it invents a delivery date. The database is healthy and the model is healthy; the conversation is malformed. The missing fact was never English phrasing. A tool result is a typed observation correlated to a specific proposal identifier, and the moment the caller flattened it into prose, the transcript stopped meaning what it looked like it meant.

This chapter treats the API surface as what it actually is — a versioned wire contract — and builds a small typed boundary on top of it, running every piece against a deterministic stand-in for a model server so we can watch real payloads move. We build chat(), a raw HTTP boundary, and print the exact JSON it sends and receives; we walk the four message roles and the typed content blocks that break the “content is a string” assumption; we render one message list through two chat templates and watch the token count change; we push probability vectors through nucleus sampling and see why top_p=0.7 is not “70 percent of the vocabulary”; we sort structured-output guarantees into four levels and measure what each one fails to catch; we measure the constraint tax a grammar imposes when the answer it wants is illegal; we own the validation layer with a bounded parse-validate-repair loop and print a real complaint and a real accepted record; we demonstrate — not assert — which schema shapes make mistakes unrepresentable; and we hand-roll one complete tool round trip on the raw message format, then extend it to forcing, parallel calls, and streaming. The build is one boundary grown by mechanism; it is not an agent loop. The model may propose one tool, once. Chapter 16 turns this into a bounded loop and Chapter 17 adds the authorized harness.

12.1 A request and a response on the wire

A chat-completions request is, underneath every SDK, an HTTP POST to /v1/chat/completions. Its body names a model and carries an ordered messages list; it may also carry sampling controls, an output schema, and tool declarations. The response carries one or more choices, each with an assistant message and a finish reason, plus a usage block that accounts for tokens. We use this OpenAI-compatible shape as a teaching language because local servers implement it and its five load-bearing ideas transfer to every provider: ordered input, typed output parts, correlation identifiers, a terminal reason, and usage. “Compatible” does not mean identical; provider deltas live in the dated section at the end.

We start the artifact with chat(). It builds the JSON body, sends it through an injected transport, and extracts exactly the fields a caller must inspect. The transport seam is the whole trick that keeps this chapter offline: the production transport is httpx.post; the teaching transport is a scripted stub we can read.

# @save
from __future__ import annotations

import json
import re
from collections.abc import Callable
from functools import partial
from typing import Any

from jsonschema import Draft202012Validator

Message = dict[str, Any]
Transport = Callable[[str, dict[str, Any]], dict[str, Any]]


def _http_transport(url: str, body: dict[str, Any]) -> dict[str, Any]:
    """POST one JSON body to a chat-completions URL without a provider SDK."""
    import httpx

    response = httpx.post(url, json=body, timeout=30.0)
    response.raise_for_status()
    return response.json()


def chat(
    base_url: str,
    model: str,
    messages: list[Message],
    *,
    transport: Transport | None = None,
    tools: list[dict[str, Any]] | None = None,
    response_format: dict[str, Any] | None = None,
    temperature: float = 0.0,
    top_p: float = 1.0,
    max_tokens: int = 256,
    tool_choice: Any = None,
    parallel_tool_calls: bool = False,
) -> dict[str, Any]:
    """Call an OpenAI-compatible chat-completions endpoint over raw HTTP.

    The body is the wire contract; every keyword is a field a server versions
    and may interpret differently. We return a small dict rather than raw text
    because the caller must inspect four things a bare string hides: the typed
    assistant ``message`` (which may carry tool calls and null content), the
    ``finish_reason`` (why generation stopped), token ``usage``, and a
    ``request_id`` for tracing.

    Args:
        base_url: Server root; ``/v1/chat/completions`` is appended.
        model: Model identifier the server routes on.
        messages: Ordered role/content turns — the conversation so far.
        transport: A ``(url, body) -> wire_dict`` function; defaults to raw
            HTTP. Injecting it is what lets tests and this chapter run offline.
        tools: Optional tool declarations the model may propose against.
        response_format: Optional structured-output request (see
            :func:`schema_response_format`).
        temperature: Decoding sharpness; ``0.0`` is the reproducible default.
        top_p: Nucleus-sampling mass to keep.
        max_tokens: Hard generation budget; hitting it yields a ``length`` finish.
        tool_choice: Optional forcing control (``auto``/``required``/named/``none``).
        parallel_tool_calls: Whether the server may propose several calls at once.

    Returns:
        A dict with ``message``, ``finish_reason``, ``usage``, and ``request_id``.
    """
    body: dict[str, Any] = {
        "model": model,
        "messages": messages,
        "temperature": temperature,
        "top_p": top_p,
        "max_tokens": max_tokens,
    }
    if tools is not None:
        body["tools"] = tools
        body["parallel_tool_calls"] = parallel_tool_calls
    if response_format is not None:
        body["response_format"] = response_format
    if tool_choice is not None:
        body["tool_choice"] = tool_choice
    endpoint = f"{base_url.rstrip('/')}/v1/chat/completions"
    wire = (transport or _http_transport)(endpoint, body)
    choice = wire["choices"][0]
    return {
        "message": choice["message"],
        "finish_reason": choice.get("finish_reason"),
        "usage": wire.get("usage", {}),
        "request_id": wire.get("id"),
    }

Now the teaching transport. It is deliberately not a language model and makes no attempt to be one: it holds a queue of assistant turns and plays them back in order, wrapping each in the same wire envelope a real server returns. Because its script is right here in the source, every output we print is one you can trace to a line you can read. That is exactly the property a real model denies us, and exactly the property that makes it a good place to learn the boundary.

# @save
class ScriptedModel:
    """A deterministic, offline stand-in for a chat-completions server.

    This is not a language model. It replays a fixed list of assistant turns,
    wrapping each in the OpenAI-compatible ``choices``/``finish_reason``/
    ``usage`` envelope so the code under test sees a real wire shape. It also
    remembers the last request body, which lets us print exactly what went out
    on the wire. Every number it reports is a stub value, honest about being
    one; it estimates no model's quality.

    Args:
        turns: Assistant turns to play in order, each a
            ``(message, finish_reason)`` pair. The final turn repeats if asked
            for more calls than were scripted.
        prompt_tokens: Fixed prompt-token count to report in ``usage``.
    """

    def __init__(
        self,
        turns: list[tuple[Message, str]],
        *,
        prompt_tokens: int = 40,
    ) -> None:
        self._turns = list(turns)
        self._prompt_tokens = prompt_tokens
        self.calls = 0
        self.last_body: dict[str, Any] | None = None

    def __call__(self, url: str, body: dict[str, Any]) -> dict[str, Any]:
        self.last_body = body
        message, finish_reason = self._turns[min(self.calls, len(self._turns) - 1)]
        self.calls += 1
        completion = max(1, len(json.dumps(message)) // 4)
        return {
            "id": f"stub-{self.calls}",
            "choices": [{"message": message, "finish_reason": finish_reason}],
            "usage": {
                "prompt_tokens": self._prompt_tokens,
                "completion_tokens": completion,
                "total_tokens": self._prompt_tokens + completion,
            },
        }

With both halves in hand we can watch one round trip. We script a single terse answer, send a two-message conversation, and print both the body that left on the wire and the dict the caller reads back.

stub = ScriptedModel([
    ({"role": "assistant", "content": "Order O-0042 shipped on the 3rd."}, "stop"),
])
messages = [
    {"role": "system", "content": "You are a support assistant. Be terse."},
    {"role": "user", "content": "Where is order O-0042?"},
]
response = chat("http://stub", "scripted-1", messages, transport=stub)

print("REQUEST body sent on the wire:")
print(json.dumps(stub.last_body, indent=2))
print("\nRESPONSE the caller reads back:")
print(json.dumps(response, indent=2))

REQUEST body sent on the wire:
{
  "model": "scripted-1",
  "messages": [
    {
      "role": "system",
      "content": "You are a support assistant. Be terse."
    },
    {
      "role": "user",
      "content": "Where is order O-0042?"
    }
  ],
  "temperature": 0.0,
  "top_p": 1.0,
  "max_tokens": 256
}

RESPONSE the caller reads back:
{
  "message": {
    "role": "assistant",
    "content": "Order O-0042 shipped on the 3rd."
  },
  "finish_reason": "stop",
  "usage": {
    "prompt_tokens": 40,
    "completion_tokens": 17,
    "total_tokens": 57
  },
  "request_id": "stub-1"
}

Read the two payloads against each other. The request is an ordered list of typed turns plus a decoding policy — nothing is hidden in prose. The response is not a string: it is a message, a finish_reason of stop (the server saw one of its terminal conditions), a usage block we can reconcile against our own token counting, and a request_id we can log. The stub let us see the shape offline; against a real server the same chat() call returns the same fields, which is the whole point — the boundary is the contract, not the server. The cell below issues that call against any local OpenAI-compatible endpoint. It needs a network and a served model, so it does not run in the book; the block after it shows a representative response.

real = chat("http://localhost:8000", "your-model-id", messages)
print(real["finish_reason"], real["usage"])
print(real["message"]["content"])

Representative response (an 8B-class instruct model behind an OpenAI-compatible endpoint):

stop {'prompt_tokens': 26, 'completion_tokens': 14, 'total_tokens': 40}
Order O-0042 shipped on July 3rd and is out for delivery.

An SDK can layer retries, object models, and streaming helpers over this, but the network still carries this versioned contract, and inspecting it once makes every later framework behavior easier to debug. Chapter 26 owns gateways, routing, and multi-provider ports; here we keep the wire visible.

12.2 Messages compile into tokens

An API message is structured input. Its role says who produced the turn; its content may be a string, null, or a list of typed blocks. Four roles carry the teaching shape: system holds caller-controlled operating instructions, user holds user input, assistant preserves model output including tool-call proposals, and tool carries an application-produced result correlated to one proposal. The trap is assuming message["content"] is always a string. It is not, and the two turns below prove it: an assistant turn can carry content: null beside a tool_calls array, and a multimodal user turn can carry a list of parts.

assistant_tool_turn = {
    "role": "assistant",
    "content": None,
    "tool_calls": [{
        "id": "call_0042",
        "type": "function",
        "function": {"name": "lookup_order", "arguments": '{"order_id": "O-0042"}'},
    }],
}
multimodal_user_turn = {
    "role": "user",
    "content": [
        {"type": "text", "text": "What is the total on this receipt?"},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,iVBORw0K..."}},
    ],
}
print("assistant turn, content is null:")
print(json.dumps(assistant_tool_turn, indent=2))
print("\nfirst content part type:", multimodal_user_turn["content"][0]["type"])

assistant turn, content is null:
{
  "role": "assistant",
  "content": null,
  "tool_calls": [
    {
      "id": "call_0042",
      "type": "function",
      "function": {
        "name": "lookup_order",
        "arguments": "{\"order_id\": \"O-0042\"}"
      }
    }
  ]
}

first content part type: text

Walking typed parts is more robust than grabbing “the first string that looks like an answer.” Notice too that the function arguments arrive as a JSON string, not a parsed object — a detail that bites in Section 12.8.

Roles are not words the network reads by magic. Before inference the server applies the checkpoint’s chat template: a tokenizer-owned program that renders messages and tool metadata into one token sequence with role markers. Different checkpoints use different markers, whitespace, and order, so the same message list becomes different prompt tokens under different templates. This is why the template is part of the model’s application binary interface, not plumbing. We make the point with a toy renderer and a toy token counter — both stand-ins for the real apply_chat_template, but enough to show the effect.

# @save
def render_chat_template(messages: list[Message], template: dict[str, Any]) -> str:
    """Render messages into one prompt string using a template's role markers.

    A real chat template is tokenizer-owned and far richer, but the mechanism
    is this: each turn is wrapped in role-specific markers and the turns are
    joined, then a generation marker invites the assistant to continue. Two
    templates over identical messages produce different strings — and therefore
    different token counts — which is the whole point.

    Args:
        messages: The conversation turns; string content only, for the demo.
        template: Markers dict with per-role ``open``/``close``, a ``sep``
            between turns, and an ``assistant_open`` generation marker.

    Returns:
        The rendered prompt string a tokenizer would then encode.
    """
    rendered = []
    for turn in messages:
        marker = template["roles"][turn["role"]]
        rendered.append(f"{marker['open']}{turn['content']}{marker['close']}")
    return template["sep"].join(rendered) + template["assistant_open"]


def count_tokens(text: str) -> int:
    """Count tokens with a word-and-punctuation splitter (a tokenizer stand-in)."""
    return len(re.findall(r"\w+|[^\w\s]", text))

We render one conversation through two templates and count. The messages are byte-for-byte identical; only the markers differ.

convo = [
    {"role": "system", "content": "Be terse."},
    {"role": "user", "content": "Where is O-0042?"},
]
chatml = {
    "roles": {r: {"open": f"<|im_start|>{r}\n", "close": "<|im_end|>\n"}
              for r in ("system", "user", "assistant")},
    "sep": "", "assistant_open": "<|im_start|>assistant\n",
}
plain = {
    "roles": {r: {"open": f"{r.upper()}: ", "close": "\n"}
              for r in ("system", "user", "assistant")},
    "sep": "", "assistant_open": "ASSISTANT: ",
}
for name, template in (("ChatML-style", chatml), ("plain", plain)):
    prompt = render_chat_template(convo, template)
    print(f"{name:14} -> {count_tokens(prompt):2d} tokens")

ChatML-style   -> 37 tokens
plain          -> 15 tokens

Identical messages, different token counts. That is why token budgeting must use the model’s own template and tokenizer, and why you reconcile your local count against the server’s reported prompt_tokens: a delta usually means a template mismatch, server-added tool instructions, or a different tokenizer revision — none of which a character count would ever surface. In production the count comes from the model’s own tokenizer via tokenizer.apply_chat_template(messages, tools=..., add_generation_prompt=True), which the offline demo above stands in for; the reconciliation discipline is identical.

Some templates also support assistant prefilling: the caller supplies a partial final assistant turn and the model continues it. It steers a prefix, but it is not a portable structured-output guarantee — a template may or may not distinguish “start a new assistant message” from “continue the last one,” and a hosted surface may reject it. Our stub simply continues the seeded prefix so the mechanism is visible:

prefill_messages = [
    {"role": "user", "content": "Give the refund decision as JSON."},
    {"role": "assistant", "content": '{"action": "'},  # seed the prefix
]
prefill_stub = ScriptedModel([
    ({"role": "assistant", "content": 'refund", "amount_cents": 3700}'}, "stop"),
])
tail = chat("http://stub", "scripted", prefill_messages, transport=prefill_stub)
print("seeded prefix + continuation:", '{"action": "' + tail["message"]["content"])

seeded prefix + continuation: {"action": "refund", "amount_cents": 3700}

The continuation completes the object we started, which is useful for steering. But treat prefill support as a dated capability and reach for a native schema feature when you need an actual structural guarantee.

12.3 Sampling controls are versioned request policy

The caller does not merely send prose; it sends a decoding policy, and that policy belongs in versioned request configuration, not hidden in a prompt. Chapter 1 and Chapter 9 derived the policy; here we record its API consequences. For logits z_i and temperature T>0, the next-token distribution is p_i(T)=\frac{\exp(z_i/T)}{\sum_j \exp(z_j/T)}. \tag{12.1} Smaller T concentrates mass; larger T spreads it. Nucleus sampling then keeps the smallest probability-ranked prefix whose cumulative mass reaches top_p and renormalizes over it. The consequence people misread is that top_p=0.7 does not keep 70 percent of the vocabulary — it keeps however many tokens are needed to reach 0.7 of the mass, which depends entirely on the distribution’s shape.

# @save
def nucleus(probs: list[float], top_p: float) -> list[int]:
    """Return the token indices nucleus (top_p) sampling keeps, most likely first.

    We sort by probability, accumulate mass, and stop as soon as the running
    total reaches ``top_p``. The size of the returned set is a property of the
    distribution, not a fixed fraction of the vocabulary — which is the whole
    reason ``top_p=0.7`` keeps two tokens in one context and many in another.

    Args:
        probs: A probability vector over the candidate tokens; should sum to 1.
        top_p: Cumulative-mass threshold in ``(0, 1]``.

    Returns:
        The kept indices, ordered from most to least probable.
    """
    order = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)
    kept, cumulative = [], 0.0
    for index in order:
        kept.append(index)
        cumulative += probs[index]
        if cumulative >= top_p:
            break
    return kept

We take two distributions over the same vocabulary. One is peaked (the model is confident); one is nearly flat (the model is unsure). The same top_p=0.7 cuts them very differently.

peaked = [0.60, 0.25, 0.10, 0.03, 0.02]
flat = [1 / 12] * 12
for name, probs in (("peaked", peaked), ("flat", flat)):
    kept = nucleus(probs, 0.7)
    mass = sum(probs[i] for i in kept)
    print(f"{name:7} dist ({len(probs):2d} tokens): top_p=0.7 keeps "
          f"{len(kept):2d} tokens, {mass:.2f} mass")

peaked  dist ( 5 tokens): top_p=0.7 keeps  2 tokens, 0.85 mass
flat    dist (12 tokens): top_p=0.7 keeps  9 tokens, 0.75 mass

Two tokens in the peaked case, nine in the flat one, from one threshold. Figure 12.1 draws the two cuts so the shape-dependence is unmistakable.

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(7.6, 3.2))
for ax, (name, probs) in zip(axes, (("peaked", peaked), ("flat", flat))):
    kept = set(nucleus(probs, 0.7))
    colors = ["#355f4d" if i in kept else "#c9cdd4" for i in range(len(probs))]
    ax.bar(range(len(probs)), sorted(probs, reverse=True), color=colors)
    ax.set_title(f"{name}: keeps {len(kept)} of {len(probs)}")
    ax.set_xlabel("token rank")
axes[0].set_ylabel("probability")
fig.tight_layout()
plt.show()

Figure 12.1: Why is top_p not a fixed count? The same top_p=0.7 cut (dashed line at cumulative mass) keeps 2 tokens of a peaked distribution and 9 of a flat one. The retained set is a property of the shape, not a fraction of the vocabulary.

Two more controls round out the request contract, and both are about stopping, not correctness. A stop sequence truncates generation at the first match; it is a string trigger, not a guarantee of a valid object. We can demonstrate the trigger directly:

def apply_stop(text: str, stops: list[str]) -> str:
    """Truncate text at the first stop sequence that occurs, if any."""
    hits = [text.find(s) for s in stops if s in text]
    return text[:min(hits)] if hits else text

raw = 'refund approved.\nEND\nignore everything after the stop marker'
print(repr(apply_stop(raw, ["\nEND"])))

'refund approved.'

And the finish_reason is not decoration — it is the difference between a complete answer and a truncated one. A stop means a terminal condition fired; a length means the max_tokens budget interrupted generation mid-object, and accepting that partial object as complete is an application bug; a tool_calls means the model is waiting for observations; a content-filter or refusal outcome is also not ordinary text. Always branch on it. And temperature=0 commonly selects the top-scoring token but is not a system invariant across model aliases, kernels, and batch sizes, so record the exact model revision, template and schema versions, controls, and request id when you need to reason about a past call.

12.4 Four levels of “it returns JSON”

“The model returns JSON” can name four very different contracts, and confusing a syntactic convenience for a business guarantee is how extraction endpoints quietly ship bugs. Ordering the guarantees keeps them apart.

Level	Mechanism	Guarantee	Typical failure it still allows
1	prompt-only instruction	none beyond model behavior	prose around the object, malformed JSON, wrong shape
2	JSON mode	syntactically parseable JSON	missing/extra fields, invalid enum values
3	schema-constrained decoding	conformance to the engine’s supported schema subset	a legal-shaped but false value
4	application validation	checks your code implements and versions	a business rule you forgot to write

Each rung catches the failure below it and lets a subtler one through. Figure 12.2 makes that ladder concrete by annotating each rung with a bad output that passes it and is caught only by the next.

flowchart TD
    L1["Level 1 prompt only<br/>admits: prose-wrapped object"] --> L2["Level 2 JSON mode<br/>admits: currency = usd, a bad enum"]
    L2 --> L3["Level 3 schema-constrained<br/>admits: order_id O-9042, legal shape but nonexistent"]
    L3 --> L4["Level 4 application validation<br/>rejects nonexistent order and wrong actor"]

Figure 12.2: What does each guarantee level fail to catch? Every rung admits a bad output that the rung above it rejects — which is why the top rung, application validation, is the only one that can see business state.

Level 3 deserves a sentence of mechanism because it is the one people over-trust. A decoding engine compiles the schema into a grammar or state machine and, before each token, masks continuations that cannot lead to a legal output (Willard and Louf 2023; Dong et al. 2024). The model still scores the surviving tokens. If both a real id O-0042 and a nonexistent O-9042 match the string pattern, the grammar permits both — constraint moves validation earlier without turning a language model into a database. Level 4 is different in kind: it runs inside your release boundary, chooses the JSON Schema draft and validator version, checks values against current state, and returns a typed failure when the budget ends. The artifact requests level-3 conformance with a strict schema, then owns level 4 itself.

# @save
def schema_response_format(name: str, schema: dict[str, Any]) -> dict[str, Any]:
    """Build the OpenAI-compatible strict JSON-Schema request block.

    Args:
        name: A stable name for the schema, recorded in traces.
        schema: The JSON Schema the decoder should conform output to.

    Returns:
        The ``response_format`` object to place in the request body.
    """
    return {"type": "json_schema",
            "json_schema": {"name": name, "schema": schema, "strict": True}}


REFUND_SCHEMA: dict[str, Any] = {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "type": "object",
    "properties": {
        "ticket_id": {"type": "string", "pattern": "^T-[0-9]{3}$"},
        "action": {"enum": ["refund", "deny", "escalate"]},
        "order_id": {"type": "string", "pattern": "^O-[0-9]{4}$"},
        "amount_cents": {"type": "integer", "minimum": 0},
        "currency": {"enum": ["USD", "EUR"]},
    },
    "required": ["ticket_id", "action", "order_id", "amount_cents", "currency"],
    "additionalProperties": False,
}

To see the three failure modes as data, we script one stub output per level and print them. These are the exact strings a level-1, level-2, and level-3 surface might emit on a hard ticket — manufactured here so the boundary has something concrete to reject.

level_outputs = {
    "1 prompt-only": 'Here is the result: '
        '{"ticket_id":"T-004","action":"refund","order_id":"O-0004",'
        '"amount_cents":1048,"currency":"usd"}',
    "2 JSON-mode": '{"ticket_id":"T-011","action":"refund","order_id":"O-0011",'
        '"amount_cents":1307,"currency":"USD","note":"extra field"}',
    "3 schema": '{"ticket_id":"T-010","action":"refund","order_id":"O-9010",'
        '"amount_cents":1270,"currency":"USD"}',
}
for level, text in level_outputs.items():
    print(f"level {level}:\n  {text}\n")

level 1 prompt-only:
  Here is the result: {"ticket_id":"T-004","action":"refund","order_id":"O-0004","amount_cents":1048,"currency":"usd"}

level 2 JSON-mode:
  {"ticket_id":"T-011","action":"refund","order_id":"O-0011","amount_cents":1307,"currency":"USD","note":"extra field"}

level 3 schema:
  {"ticket_id":"T-010","action":"refund","order_id":"O-9010","amount_cents":1270,"currency":"USD"}

The level-1 output buries a valid object in prose and lower-cases the currency; the level-2 output is clean JSON with an extra field; the level-3 output is schema-perfect but names an order that does not exist. Each is caught one rung higher than you might hope. Turning these strings into accepted-or-rejected decisions is the job of the layer we own next.

12.5 The constraint tax, measured

Constrained decoding is not free. A JSON Schema is a language and a decoding engine implements only some of it; recursion, references, unions, numeric bounds, and string formats each need compiler support, so a schema valid under Draft 2020-12 can be silently under-enforced by a provider (Geng et al. 2025). That is the schema-coverage gap, and you close it by testing the exact schema against the exact engine at deploy time. But there is a second, subtler cost that shows up even when coverage is perfect. Masking reshapes the distribution. If C is the set of tokens still legal after the current prefix, the constrained distribution is p_i^{C}=\frac{p_i\,\mathbf{1}[i\in C]}{\sum_j p_j\,\mathbf{1}[j\in C]}, \tag{12.2} where p_i is the model’s unconstrained probability for token i, the indicator is one only for legal tokens, and the denominator renormalizes the survivors. When the token the model most wanted is illegal — because the schema forbids the free-text reasoning it planned, or because the true answer lies outside the grammar — that mass is not redistributed to a better answer; it is simply diverted. We can measure exactly how much.

# @save
def constrained_renormalize(probs: list[float], legal: set[int]) -> list[float]:
    """Apply @eq-ch12-mask: zero illegal tokens and renormalize the survivors.

    Args:
        probs: The model's unconstrained probability vector.
        legal: Indices the grammar still permits after the current prefix.

    Returns:
        The renormalized distribution over legal tokens; illegal tokens get 0.

    Raises:
        ValueError: If the grammar leaves no legal continuation (empty support).
    """
    masked = [p if i in legal else 0.0 for i, p in enumerate(probs)]
    total = sum(masked)
    if total == 0.0:
        raise ValueError("the grammar left no legal continuation")
    return [p / total for p in masked]


def constraint_tax(probs: list[float], legal: set[int]) -> float:
    """Return the probability mass the grammar diverts from the model's top choice."""
    intended = max(range(len(probs)), key=lambda i: probs[i])
    return 0.0 if intended in legal else probs[intended]

Consider the first token after a prompt. The model’s top choice is to start reasoning in prose; a strict grammar requires the object to open with { immediately. We measure the tax in that case and in a benign one where the model’s top choice is already legal.

tokens = ["Here", "{", "The", "Sure", "Note"]
reasoning_first = [0.45, 0.08, 0.25, 0.12, 0.10]   # model wants to reason first
object_first = [0.10, 0.55, 0.15, 0.10, 0.10]      # model already wants "{"
grammar = {1}  # the schema permits only "{" as the first token

for name, probs in (("reasoning-first", reasoning_first), ("object-first", object_first)):
    tax = constraint_tax(probs, grammar)
    after = constrained_renormalize(probs, grammar)
    forced = tokens[max(range(len(after)), key=lambda i: after[i])]
    print(f"{name:16}: tax={tax:.2f} mass diverted, grammar forces {forced!r}")

reasoning-first : tax=0.45 mass diverted, grammar forces '{'
object-first    : tax=0.00 mass diverted, grammar forces '{'

In the reasoning-first case the grammar throws away 0.45 of probability — the model’s entire plan to think before answering — and forces the {. In the object-first case the tax is zero: the constraint agrees with the model, so it costs nothing. Figure 12.3 shows the before-and-after for the expensive case.

before = reasoning_first
after = constrained_renormalize(reasoning_first, grammar)
x = range(len(tokens))
fig, ax = plt.subplots(figsize=(6.8, 3.3))
ax.bar([i - 0.2 for i in x], before, width=0.4, label="unconstrained $p_i$", color="#71849c")
ax.bar([i + 0.2 for i in x], after, width=0.4, label="constrained $p_i^C$", color="#355f4d")
ax.set_xticks(list(x))
ax.set_xticklabels([repr(t) for t in tokens])
ax.set_ylabel("probability")
ax.legend(frameon=False)
fig.tight_layout()
plt.show()

Figure 12.3: What does the constraint tax cost when the wanted token is illegal? The grammar zeroes the model’s top choice (‘Here’, p=0.45) and renormalizes the rest onto the single legal token, diverting 0.45 of mass that never reaches a better answer.

The engineering lesson is not that constraints always hurt. A 2024 study reported reasoning degradation under restrictive formats; a rebuttal showed the effect largely vanished once unconstrained reasoning was separated from the constrained answer (Tam et al. 2024;txt team 2024). So the sign of the tax is workload-dependent: for extraction, constrain the final object directly and score field correctness; for reasoning-heavy work, let the model reason in a permitted region and constrain only the answer, or use a second extraction call. Measure semantic task quality, latency, compile time, coverage, and validity together — never parse rate alone.

12.6 The validation you own

The application boundary has two jobs the model cannot do for it. Structural validation checks types, required fields, enum membership, patterns, and ranges. Semantic validation checks facts and policy the schema cannot know: whether the order exists, whether the actor may refund it, whether the amount matches the ticket. We build the layer as a bounded loop: parse the first object, gather one specific complaint, feed only that complaint back, and stop after a fixed budget with a typed failure rather than a half-valid dict.

# @save
class StructuredOutputFailed(RuntimeError):
    """No valid application object was obtained within the attempt budget."""


class Extraction:
    """A validated object plus the cost of obtaining it.

    Keeping ``attempts`` and ``total_tokens`` beside the value is what makes the
    repair loop measurable: every retry is a countable cost, not a hidden one.
    """

    def __init__(self, value: dict[str, Any], attempts: int, total_tokens: int) -> None:
        self.value = value
        self.attempts = attempts
        self.total_tokens = total_tokens


def parse_object(text: str) -> dict[str, Any]:
    """Parse the first JSON object in text, tolerating surrounding prose.

    Tolerance is not enforcement: this happily extracts an object a level-1
    model wrapped in prose, which is why it must be followed by real schema and
    business validation — parsing something is not the same as accepting it.

    Args:
        text: Raw assistant content that should contain one JSON object.

    Returns:
        The first decoded object.

    Raises:
        ValueError: If no JSON object is present or the first value is not one.
    """
    start = text.find("{")
    if start < 0:
        raise ValueError("no JSON object found")
    try:
        value, _ = json.JSONDecoder().raw_decode(text[start:])
    except json.JSONDecodeError as exc:
        raise ValueError(f"invalid JSON: {exc.msg}") from exc
    if not isinstance(value, dict):
        raise ValueError("the first JSON value is not an object")
    return value


def _complaint(
    value: dict[str, Any],
    validator: Draft202012Validator,
    semantic_check: Callable[[dict[str, Any]], str | None] | None,
) -> str | None:
    """Return one specific structural-or-semantic complaint, or None if valid."""
    errors = sorted(validator.iter_errors(value), key=lambda e: list(e.path))
    if errors:
        return "; ".join(
            f"{'/'.join(map(str, e.path)) or '<root>'}: {e.message}" for e in errors
        )
    return semantic_check(value) if semantic_check else None

The loop itself calls the model, checks the candidate, and either accepts, complains and retries, or gives up. Only the model call is retried — parsing an extraction changes no account state, so it is safe to recompute; an effect would not be, which is why Chapter 26 owns durable execution.

# @save
def extract(
    invoke: Callable[..., dict[str, Any]],
    messages: list[Message],
    schema: dict[str, Any],
    *,
    max_attempts: int = 3,
    semantic_check: Callable[[dict[str, Any]], str | None] | None = None,
) -> Extraction:
    """Parse, validate, and repair a structured extraction within a token budget.

    Each attempt sends the transcript with a strict-schema request, parses the
    first object, and asks for one specific complaint. A complaint becomes the
    next user turn so the model repairs exactly what failed; success returns a
    typed ``Extraction``; exhaustion raises rather than leaking a half-valid dict.

    Args:
        invoke: A ``chat``-like callable bound to a transport and model.
        messages: The seed conversation (typically one user turn).
        schema: The canonical schema the value must satisfy.
        max_attempts: Maximum model calls before giving up.
        semantic_check: Optional business-state check returning a complaint string.

    Returns:
        The validated value with its attempt count and cumulative token cost.

    Raises:
        StructuredOutputFailed: If no valid object appears within the budget.
    """
    validator = Draft202012Validator(schema)
    transcript = list(messages)
    tokens = 0
    for attempt in range(1, max_attempts + 1):
        result = invoke(
            transcript,
            response_format=schema_response_format("refund_decision", schema),
        )
        tokens += int(result.get("usage", {}).get("total_tokens", 0))
        text = result["message"].get("content") or ""
        try:
            value = parse_object(text)
            complaint = _complaint(value, validator, semantic_check)
        except ValueError as exc:
            complaint = str(exc)
        if complaint is None:
            return Extraction(value=value, attempts=attempt, total_tokens=tokens)
        transcript += [
            {"role": "assistant", "content": text},
            {"role": "user", "content": (
                f"The candidate failed validation: {complaint}. "
                "Return only one object matching the supplied schema.")},
        ]
    raise StructuredOutputFailed(f"no valid object after {max_attempts} attempts")

The semantic check is the part the schema cannot express — whether an order actually exists in application state. It is the layer Section 12.4 called level 4.

# @save
KNOWN_ORDERS = {f"O-{i:04d}" for i in range(1, 51)}


def semantic_check(value: dict[str, Any]) -> str | None:
    """Reject a schema-legal order id that does not exist in application state.

    Args:
        value: A structurally valid extraction with an ``order_id``.

    Returns:
        ``None`` if the order exists, else a complaint string for the loop.
    """
    return None if value["order_id"] in KNOWN_ORDERS else "order_id does not exist"

Now the payoff the boundary was built for: a real repair trace. We script a stub that first emits the level-2 candidate (a stray note field) and then, after being told exactly what failed, emits a clean object. Watch the complaint route through and the accepted record print.

repair_stub = ScriptedModel([
    ({"role": "assistant", "content":
      '{"ticket_id":"T-011","action":"refund","order_id":"O-0011",'
      '"amount_cents":1307,"currency":"USD","note":"stray field"}'}, "stop"),
    ({"role": "assistant", "content":
      '{"ticket_id":"T-011","action":"refund","order_id":"O-0011",'
      '"amount_cents":1307,"currency":"USD"}'}, "stop"),
])
invoke = partial(chat, "http://stub", "scripted", transport=repair_stub)
extraction = extract(
    invoke,
    [{"role": "user", "content": "Extract the refund decision for ticket T-011."}],
    REFUND_SCHEMA,
    semantic_check=semantic_check,
)
print("attempts:", extraction.attempts)
print("total tokens:", extraction.total_tokens)
print("accepted value:", json.dumps(extraction.value))

attempts: 2
total tokens: 160
accepted value: {"ticket_id": "T-011", "action": "refund", "order_id": "O-0011", "amount_cents": 1307, "currency": "USD"}

The first candidate failed on additionalProperties, the complaint named the stray field, and the second attempt passed — at the cost of a second call, which the token count records honestly. Figure 12.4 draws where trust ends: the model may repair a candidate, but only code may accept one.

flowchart TD
    R["Receive assistant content"] --> P{"Object parses?"}
    P -->|no| E["Build one specific complaint"]
    P -->|yes| S{"Schema passes?"}
    S -->|no| E
    S -->|yes| B{"Business rules pass?"}
    B -->|no| E
    B -->|yes| A(["Accept typed value"])
    E --> Q{"Budget remains?"}
    Q -->|yes| M["Append candidate + complaint, retry"]
    M --> R
    Q -->|no| F(["StructuredOutputFailed"])

Figure 12.4: Where does a bounded repair loop stop trusting model output? Every path back to the model is a retry of a safe recomputation; only the application accepts a value, and only within a finite budget.

Repair is not always the right fallback. A syntax slip is locally repairable; an unknown order id may need a fresh state read or a clarification; a refusal should stay a typed refusal, not be bullied through repeated prompts. Treat the complaint class as a routing signal, not an invitation to loop.

With the loop in hand we can run the small experiment that gives the chapter its one data figure. Fifty synthetic tickets, three guarantee levels with their characteristic flaws injected, and the level-4 loop over the level-2 stream. The failure injection is a visible function, so nothing is hidden.

# @save
def make_tickets(count: int = 50) -> list[dict[str, Any]]:
    """Build source-traceable but deliberately messy refund tickets.

    Args:
        count: How many tickets to generate (ids ``T-001`` upward).

    Returns:
        Ticket dicts carrying a ticket id, order id, amount, currency, and prose.
    """
    tickets = []
    for i in range(1, count + 1):
        currency = "USD" if i % 5 else "EUR"
        tickets.append({
            "ticket_id": f"T-{i:03d}",
            "order_id": f"O-{i:04d}",
            "amount_cents": 900 + 37 * i,
            "currency": currency,
        })
    return tickets


def scripted_candidate(ticket: dict[str, Any], level: int) -> str:
    """Return a guarantee-level's characteristic output for one ticket.

    The injected flaws are the point: level 1 lower-cases some enums and wraps
    some objects in prose, level 2 adds a stray property, level 3 emits a
    legal-shaped but nonexistent order id. They stand in for real model failure
    modes so the boundary has honest, known-bad inputs to reject.

    Args:
        ticket: A ticket from :func:`make_tickets`.
        level: The guarantee level (1, 2, or 3) whose flaw to inject.

    Returns:
        The candidate assistant content as a string.
    """
    index = int(ticket["ticket_id"].split("-")[1])
    value = {
        "ticket_id": ticket["ticket_id"], "action": "refund",
        "order_id": ticket["order_id"], "amount_cents": ticket["amount_cents"],
        "currency": ticket["currency"],
    }
    if level == 1 and index % 4 == 0:
        value["currency"] = value["currency"].lower()   # bad enum
    if level == 2 and index % 11 == 0:
        value["note"] = "not in schema"                 # extra property
    if level == 3 and index % 10 == 0:
        value["order_id"] = f"O-9{index:03d}"           # legal shape, nonexistent
    body = json.dumps(value)
    if level == 1 and index % 7 == 0:
        return f"Here is the extraction: {body}"         # prose wrapper
    return body


def run_validity_experiment(count: int = 50) -> dict[str, Any]:
    """Measure schema validity per level and the level-4 repair cost.

    Args:
        count: Number of tickets to run.

    Returns:
        A report with per-level validity, the count of schema-valid but
        business-invalid records, the repair-invocation rate, and mean tokens
        per successful extraction.
    """
    tickets = make_tickets(count)
    validator = Draft202012Validator(REFUND_SCHEMA)

    def valid(text: str) -> bool:
        try:
            return not list(validator.iter_errors(parse_object(text)))
        except ValueError:
            return False

    validity = {
        level: sum(valid(scripted_candidate(t, level)) for t in tickets) / count
        for level in (1, 2, 3)
    }
    business_invalid = sum(
        valid(scripted_candidate(t, 3))
        and semantic_check(parse_object(scripted_candidate(t, 3))) is not None
        for t in tickets
    )
    extractions = []
    for ticket in tickets:
        stub = ScriptedModel([
            ({"role": "assistant", "content": scripted_candidate(ticket, 2)}, "stop"),
            ({"role": "assistant", "content": json.dumps({
                "ticket_id": ticket["ticket_id"], "action": "refund",
                "order_id": ticket["order_id"], "amount_cents": ticket["amount_cents"],
                "currency": ticket["currency"]})}, "stop"),
        ])
        extractions.append(extract(
            partial(chat, "http://stub", "scripted", transport=stub),
            [{"role": "user", "content": "extract"}],
            REFUND_SCHEMA, semantic_check=semantic_check))
    repaired = sum(e.attempts > 1 for e in extractions)
    return {
        "validity": validity,
        "business_invalid": business_invalid,
        "repair_rate": repaired / count,
        "tokens_per_success": sum(e.total_tokens for e in extractions) / count,
    }

report = run_validity_experiment()
for level, rate in report["validity"].items():
    print(f"level {level}: {rate:.0%} schema-valid")
print(f"schema-valid but nonexistent order: {report['business_invalid']} of 50")
print(f"repair-invocation rate: {report['repair_rate']:.0%}")
print(f"tokens per successful extraction: {report['tokens_per_success']:.1f}")

level 1: 76% schema-valid
level 2: 92% schema-valid
level 3: 100% schema-valid
schema-valid but nonexistent order: 5 of 50
repair-invocation rate: 8%
tokens per successful extraction: 86.0

The numbers climb 76, 92, 100 as the guarantee tightens — but the comparison that matters is not 92 versus 100. It is the five records at level 3 that are structurally perfect and operationally false: real-shaped order ids that do not exist. Figure 12.5 hatches exactly that remainder so schema conformance cannot masquerade as correctness. These are synthetic CPU results; they prove the boundary’s behavior, not a model’s quality.

names = ["prompt only", "JSON mode", "schema", "app validated"]
rates = [report["validity"][1], report["validity"][2], report["validity"][3], 1.0]
invalid = [0, 0, report["business_invalid"] / 50, 0]
pos = range(len(names))
fig, ax = plt.subplots(figsize=(7.0, 3.6))
ax.bar(pos, rates, color=["#d7dde5", "#aebac9", "#71849c", "#355f4d"], edgecolor="#222")
ax.bar(pos, invalid, bottom=[r - b for r, b in zip(rates, invalid)],
       color="none", edgecolor="#111", hatch="////", label="schema-valid, business-invalid")
for p, r in zip(pos, rates):
    ax.text(p, r + 0.02, f"{r:.0%}", ha="center", fontsize=9)
ax.set_xticks(list(pos)); ax.set_xticklabels(names)
ax.set_ylim(0, 1.12); ax.set_ylabel("valid records / 50")
ax.legend(frameon=False, loc="lower right")
ax.spines[["top", "right"]].set_visible(False)
fig.tight_layout()
plt.show()

Figure 12.5: How do stronger guarantees change validity, and what survives them? Bars show schema-valid records over 50 tickets; the hatched slice of the schema-constrained bar is five schema-valid but nonexistent order ids that only application validation catches.

12.7 Designing schemas models can’t misuse

A schema is two things at once: a machine contract the validator enforces and context the model reads. Both jobs pull toward the same design rule — make invalid states unrepresentable — and the rule is best shown, not asserted. Take two versions of the refund schema. A loose one uses a free-form decision string and allows extra properties; a strict one uses an enum and forbids them. We feed both the same sloppy candidate and watch them disagree.

loose_schema = {
    "type": "object",
    "properties": {"decision": {"type": "string"}, "amount_cents": {"type": "integer"}},
    "required": ["decision", "amount_cents"],
    # no additionalProperties: false, no enum
}
strict_schema = {
    "type": "object",
    "properties": {
        "action": {"enum": ["refund", "deny", "escalate"]},
        "amount_cents": {"type": "integer", "minimum": 0},
    },
    "required": ["action", "amount_cents"],
    "additionalProperties": False,
}
sloppy = {"decision": "Refund!!", "action": "Refund!!", "amount_cents": -5,
          "aproved_by": "bot"}   # typo'd key, negative amount, free-text decision

for name, schema in (("loose", loose_schema), ("strict", strict_schema)):
    errors = list(Draft202012Validator(schema).iter_errors(sloppy))
    verdict = "accepts it" if not errors else f"rejects: {errors[0].message}"
    print(f"{name:6} schema {verdict}")

loose  schema accepts it
strict schema rejects: 'Refund!!' is not one of ['refund', 'deny', 'escalate']

The loose schema accepts the record whole — the free-text decision swallows “Refund!!”, the misspelled aproved_by key rides along as a silent feature, and the negative amount passes because nothing forbids it. The strict schema rejects on the first illegal value. The difference is not model quality; it is what the two contracts made possible to express. Four rules follow, each demonstrated by that contrast: prefer an enum over a string the model can fill freely; set additionalProperties: false at every object boundary so a misspelled key cannot become a feature; give numbers real bounds; keep ids as patterned strings when formatting matters. And descriptions should state purpose and non-use — "Read one order by exact id; do not use for product search." teaches the model far more than "Looks things up." Chapter 17 develops full tool-surface design; here the rule is narrow: make one proposed call easy to validate and hard to misread.

12.8 A tool call is a proposal, not an execution

A tool declaration tells the model which action shape it may propose. It grants no credentials, calls no function, authorizes no effect. One complete round trip takes at least two model requests: the application sends messages plus declarations; the model returns an assistant turn with a call id, tool name, and arguments as a JSON string, finishing with tool_calls; the application parses, validates, and authorizes those arguments and either runs or refuses; it appends the original assistant turn and a tool result carrying the same call id; and it sends the second request, from which the model can finally answer. The invariant to hold without qualification: the provider never calls your tool. Your code owns the only execution arrow. We build the loop to admit exactly one proposal, validate its arguments before touching the handler, and correlate the result by id.

# @save
def assert_all_calls_answered(calls: list[dict], results: list[Message]) -> None:
    """Require every proposal id to be answered by exactly one result id.

    Args:
        calls: The assistant's proposed tool calls, each carrying an ``id``.
        results: The tool result messages, each carrying a ``tool_call_id``.

    Raises:
        ValueError: If ids are duplicated or the two id sets do not match.
    """
    proposed = [c["id"] for c in calls]
    answered = [r["tool_call_id"] for r in results]
    if len(proposed) != len(set(proposed)) or sorted(proposed) != sorted(answered):
        raise ValueError("tool results must answer every unique call id exactly once")


def run_tool_turn(
    invoke: Callable[..., dict[str, Any]],
    messages: list[Message],
    tool: dict[str, Any],
    handler: Callable[..., Any],
) -> dict[str, Any]:
    """Run one proposed tool call and return the model's final response.

    The proposal is validated against the tool's own parameter schema *before*
    the handler runs, so malformed arguments raise at the boundary rather than
    inside application code. The complete assistant turn is preserved on the
    second request — reconstructing only name and arguments can drop provider
    metadata the model needs.

    Args:
        invoke: A ``chat``-like callable bound to a transport and model.
        messages: The seed conversation.
        tool: One tool declaration (name, description, parameter schema).
        handler: The Python function that actually performs the action.

    Returns:
        The second-request response, from which the model answers.

    Raises:
        ValueError: On an unexpected finish, wrong call count, disallowed tool,
            or arguments that fail the tool's schema.
    """
    first = invoke(messages, tools=[tool], parallel_tool_calls=False)
    if first["finish_reason"] != "tool_calls":
        raise ValueError("expected a tool_calls finish reason")
    calls = first["message"].get("tool_calls", [])
    if len(calls) != 1:
        raise ValueError("this boundary permits exactly one tool proposal")
    call = calls[0]
    if call["function"]["name"] != tool["function"]["name"]:
        raise ValueError("the proposed tool is not allowed")
    arguments = json.loads(call["function"]["arguments"])
    errors = list(Draft202012Validator(tool["function"]["parameters"]).iter_errors(arguments))
    if errors:
        raise ValueError(f"tool arguments failed validation: {errors[0].message}")
    output = handler(**arguments)
    result = {"role": "tool", "tool_call_id": call["id"],
              "content": json.dumps(output, sort_keys=True)}
    assert_all_calls_answered(calls, [result])
    return invoke([*messages, first["message"], result], tools=[tool],
                  parallel_tool_calls=False)

We script the two-turn exchange and print the intermediate tool_calls structure and the correlated tool message, so the correlation is visible rather than described.

LOOKUP_TOOL = {
    "type": "function",
    "function": {
        "name": "lookup_order",
        "description": "Read one order by exact id; do not use for product search.",
        "parameters": {
            "type": "object",
            "properties": {"order_id": {"type": "string", "pattern": "^O-[0-9]{4}$"}},
            "required": ["order_id"], "additionalProperties": False,
        },
    },
}
proposal_turn = {"role": "assistant", "content": None, "tool_calls": [{
    "id": "call_0042", "type": "function",
    "function": {"name": "lookup_order", "arguments": '{"order_id": "O-0042"}'}}]}
tool_stub = ScriptedModel([
    (proposal_turn, "tool_calls"),
    ({"role": "assistant", "content": "Order O-0042 is shipped."}, "stop"),
])
def lookup_order(order_id: str) -> dict[str, str]:
    return {"order_id": order_id, "status": "shipped"}

final = run_tool_turn(
    partial(chat, "http://stub", "scripted", transport=tool_stub),
    [{"role": "user", "content": "Where is O-0042?"}], LOOKUP_TOOL, lookup_order)
print("proposal the model returned:")
print(json.dumps(proposal_turn["tool_calls"][0], indent=2))
print("correlated result carries the same id, then the model answers:")
print("final answer:", final["message"]["content"], "| finish:", final["finish_reason"])

proposal the model returned:
{
  "id": "call_0042",
  "type": "function",
  "function": {
    "name": "lookup_order",
    "arguments": "{\"order_id\": \"O-0042\"}"
  }
}
correlated result carries the same id, then the model answers:
final answer: Order O-0042 is shipped. | finish: stop

Figure 12.6 places authority on the only arrow that reaches the tool.

sequenceDiagram
    participant App as Application
    participant API as Model API
    participant Tool as Application tool
    App->>API: request 1: messages + declaration
    API-->>App: assistant tool_call id=call_0042, arguments as JSON string
    App->>App: parse, validate, authorize
    App->>Tool: lookup_order(order_id="O-0042")
    Tool-->>App: status="shipped"
    App->>API: request 2: assistant turn + tool result id=call_0042
    API-->>App: final assistant text, finish=stop

Figure 12.6: Who actually invokes a tool, and how is its result correlated to the proposal? The model proposes; the application validates, authorizes, and executes; the result returns under the same call id.

Three variations change the boundary, and each is a worked example rather than a table row. First, forcing with tool_choice changes which shapes the decoder may emit; the request body differs concretely:

choices = ["auto", "required",
           {"type": "function", "function": {"name": "lookup_order"}}, "none"]
for choice in choices:
    sent = ScriptedModel([({"role": "assistant", "content": "x"}, "stop")])
    chat("http://stub", "m", messages, transport=sent,
         tools=[LOOKUP_TOOL], tool_choice=choice)
    label = choice if isinstance(choice, str) else "named:lookup_order"
    print(f"tool_choice={label:16} -> sent on the wire: {sent.last_body['tool_choice']!r}")

tool_choice=auto             -> sent on the wire: 'auto'
tool_choice=required         -> sent on the wire: 'required'
tool_choice=named:lookup_order -> sent on the wire: {'type': 'function', 'function': {'name': 'lookup_order'}}
tool_choice=none             -> sent on the wire: 'none'

Use forcing to express a route contract — a classifier endpoint may require one tool, a stage after an irreversible effect may set none to block another proposal — not to rescue a weak description. Second, parallel calls are parallel proposals, each with its own id, and completion order need not match proposal order. The rule is to correlate by id, never by list position:

proposals = [
    {"id": "call_a", "function": {"name": "lookup_order", "arguments": '{"order_id":"O-0001"}'}},
    {"id": "call_b", "function": {"name": "lookup_order", "arguments": '{"order_id":"O-0002"}'}},
]
# results come back in REVERSE completion order
completions = [
    {"tool_call_id": "call_b", "content": '{"status":"delayed"}'},
    {"tool_call_id": "call_a", "content": '{"status":"shipped"}'},
]
by_id = {c["tool_call_id"]: c["content"] for c in completions}
for p in proposals:
    print(f"{p['id']} asked {p['function']['arguments']:24} -> {by_id[p['id']]}")

call_a asked {"order_id":"O-0001"}    -> {"status":"shipped"}
call_b asked {"order_id":"O-0002"}    -> {"status":"delayed"}

Matching proposals[0] to completions[0] would have reported O-0001 as delayed — position is not identity. Third, streaming splits one turn into deltas, and a tool-argument string can arrive across chunks ({"order_ then id":"O-0042"}). A prefix may be invalid JSON because it is incomplete, or parse by coincidence before the server commits the turn, so we separate two states: parseable (a closed call block decodes) and actionable (the whole response reached a terminal event and policy admitted it). The assembler exposes a proposal only when every call is closed and the response is terminal.

# @save
class IncompleteToolStream(RuntimeError):
    """A streamed tool proposal never reached both close and terminal events."""


def assemble_tool_stream(chunks: list[dict[str, Any]]) -> list[dict[str, Any]]:
    """Assemble streamed tool-call deltas, exposing proposals only when actionable.

    Arguments accumulate per call index; a proposal becomes eligible only after
    its call-close event *and* a response-terminal event. Anything less fails
    closed, because a parseable prefix is not a committed turn.

    Args:
        chunks: Streamed events; a terminal event carries ``terminal: True``,
            others carry a ``tool_call`` delta with ``index`` and partial
            ``arguments``.

    Returns:
        One assembled proposal per call index, each with id, name, and parsed
        arguments.

    Raises:
        IncompleteToolStream: If any call is unclosed or the response never
            reached a terminal event.
    """
    buffers: dict[int, dict[str, Any]] = {}
    terminal = False
    for chunk in chunks:
        if chunk.get("terminal"):
            terminal = True
            continue
        delta = chunk["tool_call"]
        index = int(delta["index"])
        state = buffers.setdefault(index, {"id": None, "name": None, "text": "", "closed": False})
        state["id"] = delta.get("id", state["id"])
        state["name"] = delta.get("name", state["name"])
        state["text"] += delta.get("arguments", "")
        state["closed"] = state["closed"] or bool(delta.get("closed"))
    if not terminal or not buffers or any(not s["closed"] for s in buffers.values()):
        raise IncompleteToolStream("stream ended before close and terminal events")
    return [{"id": s["id"], "name": s["name"], "arguments": json.loads(s["text"])}
            for _, s in sorted(buffers.items())]

We feed it the two argument fragments and print the buffer after each, then show that without the terminal event the proposal stays inert, and with it becomes actionable.

chunks = [
    {"tool_call": {"index": 0, "id": "call_1", "name": "lookup_order", "arguments": '{"order_'}},
    {"tool_call": {"index": 0, "arguments": 'id":"O-0042"}', "closed": True}},
]
print("buffer after chunk 1:", repr('{"order_'))
print("buffer after chunk 2:", repr('{"order_id":"O-0042"}'))
try:
    assemble_tool_stream(chunks)
except IncompleteToolStream as exc:
    print("without terminal event:", exc)
done = assemble_tool_stream([*chunks, {"terminal": True}])
print("with terminal event:", done[0]["arguments"])

buffer after chunk 1: '{"order_'
buffer after chunk 2: '{"order_id":"O-0042"}'
without terminal event: stream ended before close and terminal events
with terminal event: {'order_id': 'O-0042'}

Figure 12.7 names the states so a parseable prefix can never be mistaken for an actionable one.

stateDiagram-v2
    [*] --> Collecting
    Collecting --> Collecting: argument delta
    Collecting --> CallClosed: call-close event
    CallClosed --> Parsed: parse + schema pass
    Parsed --> Eligible: response-terminal event
    Collecting --> Incomplete: transport ends
    CallClosed --> Incomplete: response never terminal
    Parsed --> Incomplete: transport ends before terminal
    Eligible --> [*]
    Incomplete --> [*]

Figure 12.7: When may a streamed tool proposal become eligible for application policy? Only after both the call-close event and the response-terminal event; every other exit fails closed.

12.9 Errors have owners

“Retry on error” is not a policy until the error has a class and an owner. A connection timeout, an HTTP 429, a schema-rejecting 400, a length finish, a refusal, an invalid-argument string, and a tool’s own domain error are different states with different safe moves. The table names the owner; classify_failure implements just this chapter’s distinctions — not backoff, deadlines, or circuit breakers, which interact with cost and overload and belong to Chapter 26.

Observation	Who can handle it	Safe next move here
connection timeout	application transport	retry the call under an outer budget
HTTP 429 / retryable 5xx	application transport	bounded retry or route; Chapter 26 owns full policy
HTTP 400 schema rejection	deployment/config	stop; fix or recompile the provider schema
finish_reason: length	application + model	retry generation, claim no effect occurred
refusal / content filter	product policy	keep a typed refusal or escalate on an approved path
arguments fail validation	application boundary	bounded repair, typed error, or reject
tool returns a domain error	tool handler	return a correlated structured error

# @save
def classify_failure(
    *, status: int | None = None, finish_reason: str | None = None,
    error: Exception | None = None,
) -> str:
    """Map a boundary symptom to the layer that owns its recovery.

    Args:
        status: An HTTP status, when the failure was transport-level.
        finish_reason: A completion's finish reason, when one was returned.
        error: A raised exception, when the call did not complete.

    Returns:
        One of ``code_retry``, ``typed_failure``, ``model_retry_without_effect``,
        ``typed_refusal``, or ``accept``.
    """
    if error is not None:
        return "code_retry" if type(error).__name__ in {"ConnectError", "TimeoutException"} else "typed_failure"
    if status is not None and status >= 400:
        return "code_retry" if status in {408, 429, 500, 502, 503, 504} else "typed_failure"
    if finish_reason == "length":
        return "model_retry_without_effect"
    if finish_reason == "content_filter":
        return "typed_refusal"
    return "accept"

class TimeoutException(Exception):
    """Stand-in for httpx.TimeoutException so the symptom name is recognized."""

cases = [
    {"error": TimeoutException("slow")},
    {"status": 429}, {"status": 400},
    {"finish_reason": "length"}, {"finish_reason": "content_filter"},
    {"finish_reason": "stop"},
]
for case in cases:
    label = next(f"{k}={v}" for k, v in case.items())
    print(f"{label:28} -> {classify_failure(**case)}")

error=slow                   -> code_retry
status=429                   -> code_retry
status=400                   -> typed_failure
finish_reason=length         -> model_retry_without_effect
finish_reason=content_filter -> typed_refusal
finish_reason=stop           -> accept

A recognized timeout is a code retry; a 429 is a code retry; a 400 stops the deployment rather than looping; a length finish routes to a fresh generation while asserting no effect occurred. Note that the recovery keys on the symptom’s class, not on a raised object’s identity — an unrecognized exception name falls through to a typed failure rather than being retried blindly. Some domain errors belong in the next model observation — “order not found” can help the model ask for a corrected id — while an authentication failure or a raw stack trace helps the model reason about nothing and stays with code. Tool-use leaderboards such as BFCL, treated in Chapter 22, aggregate this behavior into one score; that score never substitutes for testing the exact schemas, correlations, and failure paths your application uses.

12.10 Landscape 2026 (dated)

Verified on 2026-07-20. Provider surfaces converge on typed tool proposals and structured outputs while differing in transcript rules and state ownership. Treat this as an interoperability checklist, not a normalized contract.

Surface	Conversation and tool shape	The detail that bites
OpenAI Chat Completions	ordered messages; tool_calls with arguments as a serialized JSON string; results as a tool message with tool_call_id	caller resends the transcript; branch on finish_reason and reconcile usage
OpenAI Responses	typed input/output items; function-call and output items correlated by call id	can be stateless or chained by reference; stored-state choices affect replay and data handling
Anthropic Messages	system instruction outside messages; typed content blocks; tool_use input is an object, tool_result blocks ride the next user message	parallel results belong together before extra text; stop_reason names differ
Gemini Generate Content	typed content parts; function-call arguments are objects	stateless reasoning turns must preserve returned thought signatures exactly
vLLM OpenAI-compatible	Chat Completions and Responses for supported models	chat template, tool parser, and structured-output backend are deployment configuration

No provider SDK appears in the artifact; that keeps the wire visible and is not a recommendation against maintained clients in production. A production adapter should preserve provider-native metadata, refusals, reasoning signatures, and usage even as it maps them into an application contract.

Verify live: endpoint and field names; role and instruction placement; function-argument string-versus-object encoding; strict-schema subset and recursion/reference limits; additionalProperties and required-field rules; finish/stop reasons; parallel-result ordering; streaming terminal events; prefill support; state-storage defaults, quotas, and pricing. Primary sources: the OpenAI structured-outputs and function-calling guides, the Anthropic tool-use guide, the Gemini function-calling and thought-signature guides, and the vLLM OpenAI-compatible-server, tool-calling, and structured-output pages.

NoteIn the interview

Q. A provider now enforces your JSON Schema at decode time. Can you delete the application validator? No. Schema enforcement is level 3; it guarantees shape, not truth. In the build, all five nonexistent order ids are schema-perfect and only the application’s existence check rejects them. The trap is treating a structural guarantee as a business one. You can reduce the validator’s scope with provider-conformance tests and a measured semantic-error rate, but you cannot delete the layer that reads current state — and you keep a rollback condition for when the provider’s supported-schema subset shifts under you.

12.11 Summary

The API surface is a versioned wire contract, and treating it as one turns a model into a typed software component. Messages are structured input a pinned chat template compiles into tokens; sampling controls are caller policy, not correctness. Output guarantees form a ladder — prompt behavior, JSON syntax, engine-supported schema conformance, application validation — and only the top rung can combine a canonical schema with live business state, which is why the five schema-perfect nonexistent orders survive everything below it. Constrained decoding has a measurable tax when the wanted token is illegal. Tool calls are proposals: your code parses, validates, authorizes, executes, and returns one result per call id. Chapter 13 builds context on top of these messages; Chapter 16 and Chapter 17 grow the one-tool round trip into a bounded, authorized loop.

12.12 Exercises

1. Reconcile a token delta. Using render_chat_template and count_tokens, render one conversation through the two templates in Section 12.2 and add a third template of your own. Then add a tool message to the conversation and re-count. Explain every change in the count by the markers that moved, not by the words, and state which template you would trust to budget against a real server.

2. Expose nucleus support. Construct three probability vectors — one peaked, one flat, one bimodal — and use nucleus to find, for each, the top_p that retains exactly two tokens. Show the sorted probabilities and the crossing point, and explain why the same top_p value is not comparable across the three distributions.

3. Measure the constraint tax both ways. Extend the constraint_tax demo so that (a) the grammar agrees with the model’s top token (tax zero), (b) the grammar forbids it (positive tax), and (c) the grammar leaves no legal token at all. Show that case (c) raises, and argue from Equation 12.2 why an empty legal set is a deployment failure rather than a model failure.

4. Break correlation on purpose. Using run_tool_turn and assert_all_calls_answered, extend the tool stub to propose two independent reads. First omit one result, then return one call id twice, then return correct results in reverse completion order. Make the first two fail and the third succeed, and write one sentence on why position is not identity.

5. Interrupt the stream. Starting from the assemble_tool_stream example, write three tests that delete, respectively, an argument delta, the call-close flag, and the terminal event. Prove no variant becomes eligible, then add a second call index and confirm both proposals are correlated by id.

6. Defend the application validator. A reviewer proposes deleting semantic_check because the provider now enforces REFUND_SCHEMA. Use the five nonexistent order ids from run_validity_experiment as the counterexample, then list the evidence — provider-conformance tests, semantic-error rate, validator latency, repair cost, and a rollback condition — that could reduce but not erase the validator’s scope.

7. Design a schema that resists a mistake. Take the loose_schema from Section 12.7 and harden it one keyword at a time (enum, additionalProperties: false, a numeric bound, a string pattern), re-running the sloppy candidate after each change. Report which specific bad value each keyword newly rejects, and identify one mistake that no schema keyword can catch and must live in semantic_check.

Dong, Yixin, Charlie F. Ruan, Yaxing Cai, et al. 2024. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. https://arxiv.org/abs/2411.15100.

Geng, Saibo, Hudson Cooper, Michael Moss, Cameron Ross, Luca Beurer-Kellner, et al. 2025. Generating Structured Outputs from Language Models: Benchmark and Studies. https://arxiv.org/abs/2501.10868.

Tam, Zhi Rui, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models. https://arxiv.org/abs/2408.02442.

.txt team. 2024. Say What You Mean: A Response to “Let Me Speak Freely”. Https://blog.dottxt.co/say-what-you-mean.html.

Willard, Brandon T., and Rémi Louf. 2023. Efficient Guided Generation for Large Language Models. https://arxiv.org/abs/2307.09702.