9 Inference Anatomy and Model Behavior: Sampling, Hallucination, and Abstention

A user sends a 2,000-token contract and asks for a 100-token answer. The first token arrives slowly; the rest stream at a steadier pace. A dashboard reports one latency number, so an engineer shortens the output limit. Total latency drops a little, but the long opening pause is untouched. The intervention changed the wrong phase, because the two phases of autoregressive inference obey different physics and the single number hid the split.

This chapter opens the generate loop and measures what is inside it, reusing the small GPT we trained in Chapter 2 as the thing under the microscope. We (i) time prefill against per-token decode on that real model and watch the asymmetry grow with context; (ii) build the sampling stack — temperature, top-k, top-p, min-p, and the logit processors — and show what each does to a real logit vector; (iii) batch the same prompt at different sizes and measure the logits drift, so “set the seed” stops meaning “get the same output”; and (iv) spend the second half on the applied reader’s most-asked question — when the model should not answer — building calibrated confidence, a semantic-entropy estimate, a risk-coverage curve, and a conformal abstention rule that comes with a coverage guarantee. Everything runs on CPU in about ten seconds.

We start by reusing Chapter 2’s tokenizer and model, training one small instance so its next-token distributions are meaningful rather than uniform.

import matplotlib
matplotlib.use("Agg")
import sys, math, time, codecs, re, random, statistics
sys.path.insert(0, "code/ch02")
import torch
import _generated as ch02              # the tokenizer and TinyGPT we built in Chapter 2

torch.manual_seed(0)
torch.set_num_threads(4)
CORPUS = (
 "the cat sat on the mat. the dog ran in the park. a happy cat drinks warm milk. "
 "the dog wants a bone. birds sing in the trees. the lake is calm and blue. "
 "a child reads a book by the window. the sun sets behind the hills. "
 "rain falls on the quiet street. the old clock ticks in the hall. "
 "the cat naps in the sun. the dog barks at the moon. green leaves fall in autumn. "
) * 10
tok = ch02.BytePairTokenizer.train(CORPUS, vocab_size=320)
data = torch.tensor(tok.encode(CORPUS))
cfg = ch02.GPTConfig(vocab_size=tok.vocab_size, block_size=192, d_model=96, n_heads=6, n_layers=3)
model = ch02.TinyGPT(cfg)

opt = torch.optim.AdamW(model.parameters(), lr=2e-3, weight_decay=0.01)
gen = torch.Generator().manual_seed(0)
model.train()
for step in range(120):
    xb, yb = ch02.next_token_batch(data, cfg.block_size, 16, gen)
    _, loss, _ = model(xb, yb)
    opt.zero_grad(); loss.backward()
    for group in opt.param_groups:
        group["lr"] = ch02.lr_schedule(step, 120, 2e-3)
    opt.step()
model.eval()
print(f"trained {model.parameter_count()} params, final loss {loss.item():.3f}")

trained 381696 params, final loss 0.109

The loss is low because the corpus is tiny and repetitive; that is fine. We want a model whose forward pass we can time and whose logits carry real structure, not a good language model.

9.1 Prefill and decode are two different machines

Autoregressive inference runs in two mechanically different phases. During prefill, the model reads every prompt position at once and builds a key–value cache for each layer — one big parallel matrix multiply that the accelerator likes. During decode, it produces one position, appends that position’s keys and values to the cache, and repeats. Decode carries a serial dependency: token t+1 cannot be chosen until token t is in the context. Prefill is wide and parallel; decode is a long chain of narrow steps.

Three request-level measurements keep the split visible:

Time to first token (TTFT) runs from submission to the first output token. It can absorb network, queueing, tokenization, all of prefill, and the first decode step, so every report must state its boundary.
Time per output token (TPOT) is the mean gap between output tokens after the first. Inter-token latency (ITL) is each individual gap; its tail exposes stalls a mean hides.
End-to-end latency for n output tokens is approximately

L \approx \mathrm{TTFT} + (n-1)\,\mathrm{TPOT}. \tag{9.1}

Equation 9.1 is only useful with its units and scope attached, but it already explains the opening incident: shortening the output shrinks the second term while the prompt-driven first term stays put. Figure 9.1 traces where a millisecond can enter one request; it deliberately stops at a single request, leaving batching and scheduling to Chapter 10.

sequenceDiagram
    participant C as Caller
    participant Q as Queue + tokenizer
    participant M as Model runtime
    participant D as Detokenizer
    C->>Q: prompt + decoding policy
    Q->>M: prompt token IDs
    Note over Q,M: queueing + tokenization
    M->>M: prefill all prompt positions and build KV
    M->>M: decode first output token
    M-->>D: first token ID
    D-->>C: first safe text chunk
    Note over C,M: TTFT ends here
    loop remaining output positions
        M->>M: one cached decode step
        M-->>D: next token ID
        D-->>C: zero or more safe characters
    end
    Note over C,M: gaps are ITL and their mean is TPOT

Figure 9.1: Where does elapsed time enter one autoregressive request? Prefill runs once over the whole prompt; decode is a serial loop of one-token steps. TTFT covers everything up to the first token; the gaps between later tokens are ITL, and their mean is TPOT.

Rather than assert the asymmetry, we time it. We prefill prompts of growing length and, from each resulting cache, time one decode step. Prefill work scales with the prompt; a single decode step does not.

def median_ms(call, reps):
    call()                                            # warm up
    samples = []
    for _ in range(reps):
        start = time.perf_counter(); call(); samples.append((time.perf_counter() - start) * 1000)
    return statistics.median(samples)

@torch.inference_mode()
def prefill(prompt_len):
    ids = torch.randint(0, cfg.vocab_size, (1, prompt_len))
    return model(ids)[2]                              # returns the per-layer KV cache

@torch.inference_mode()
def one_decode(cache):
    step = torch.randint(0, cfg.vocab_size, (1, 1))
    return model(step, cache=cache)[2]

contexts, prefill_ms, decode_ms = [32, 64, 128, 176], [], []
for ctx in contexts:
    prefill_ms.append(median_ms(lambda: prefill(ctx), 6))
    cache = prefill(ctx)
    decode_ms.append(median_ms(lambda: one_decode([(k.clone(), v.clone()) for k, v in cache]), 10))
    print(f"context {ctx:4d}:  prefill {prefill_ms[-1]:6.2f} ms   decode/token {decode_ms[-1]:5.2f} ms"
          f"   ratio {prefill_ms[-1] / decode_ms[-1]:4.1f}")

context   32:  prefill   1.77 ms   decode/token  1.29 ms   ratio  1.4
context   64:  prefill   2.29 ms   decode/token  1.30 ms   ratio  1.8
context  128:  prefill   3.36 ms   decode/token  1.34 ms   ratio  2.5
context  176:  prefill   4.21 ms   decode/token  1.38 ms   ratio  3.1

Prefill time climbs with context while per-token decode stays roughly flat, so their ratio grows. These are wall-clock numbers on one CPU and will differ on other hardware; the shape is the lesson, and Figure 9.2 draws it.

Show the code that draws this figure

import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.4, 3.6))
ax.plot(contexts, prefill_ms, "o-", label="prefill (whole prompt)")
ax.plot(contexts, decode_ms, "s-", label="one decode step")
ax.set_xlabel("context length (tokens)"); ax.set_ylabel("milliseconds")
ax.set_title("Prefill scales with context; decode does not"); ax.legend(); ax.grid(alpha=0.3)
plt.show()

Figure 9.2

The operational corollary is the fix the opening engineer missed: a long prompt inflates the first term, so prompt trimming, caching, and prefill throughput move TTFT; a long output inflates the second term, so it is decode speed and output length that move total latency.

9.2 Where the two phases hit the hardware

The KV cache is what makes decode cheap per step: instead of reprojecting every earlier token, the model reads stored keys and values and appends one new pair. We measured in Chapter 2 that this storage grows linearly with sequence length, and sized it across attention variants in Chapter 3. Here we only need the linear growth, read straight off the real cache.

for ctx in (32, 96, 176):
    cache = prefill(ctx)
    total = ch02.kv_cache_bytes(cache)
    print(f"context {ctx:4d}:  KV cache {total/1024:6.1f} KiB   ({total/ctx:.0f} bytes per token)")

context   32:  KV cache   72.0 KiB   (2304 bytes per token)
context   96:  KV cache  216.0 KiB   (2304 bytes per token)
context  176:  KV cache  396.0 KiB   (2304 bytes per token)

Bytes per token are constant, so the cache is a line in sequence length; grouped-query and multi-query attention lower the slope by storing fewer KV heads (Chapter 3), but the shape is fixed. That linear-storage fact is also why the two phases hit different hardware limits. A useful one-line recap of the roofline (Williams et al., 2009): performance is bounded by whichever runs out first, compute or memory bandwidth. Prefill exposes many positions per weight read, so it is usually compute-bound; a decode step reads the full weights and the growing cache to produce one token, so it is usually bandwidth-bound — which is exactly why prefill time tracked context above while decode time did not. Chapter 10 turns this split into batching and scheduling decisions.

9.3 From logits to a token: temperature and truncation

At each output position the model emits one logit z_i per vocabulary token. A decoding policy turns that vector into a choice. Greedy decoding returns the argmax; sampling first defines probabilities and draws. Both are caller policy layered on the same model output. The first knob is temperature, which rescales logits before the softmax:

p_i(T) = \frac{\exp(z_i / T)}{\sum_j \exp(z_j / T)}, \qquad T > 0. \tag{9.2}

Equivalently p_i(T) \propto p_i(1)^{1/T}: low T sharpens the distribution, high T flattens it, and dividing every logit by the same positive number never changes their order. Temperature moves gaps and entropy, not the argmax. We implement it with two guards the equation hides — a branch to argmax at T\to 0 instead of dividing by zero, and subtracting the peak logit before exponentiating so nothing overflows.

# @save
import math

def softmax(logits: list[float], temperature: float = 1.0) -> list[float]:
    """Turn logits into a temperature-scaled categorical distribution.

    Implements @eq-ch09-temperature with two numerical guards: at
    ``temperature <= 0`` it returns a one-hot argmax (the greedy limit, with
    ties broken by lowest index) rather than dividing by zero, and it subtracts
    the largest scaled logit before exponentiating so ``exp`` never overflows.
    Neither guard changes the resulting probabilities.

    Args:
        logits: One real score per vocabulary token.
        temperature: Positive scale; smaller sharpens, larger flattens. Values
            at or below zero select the argmax deterministically.

    Returns:
        A probability distribution the same length as ``logits`` summing to 1.
    """
    if temperature <= 0:
        winner = max(range(len(logits)), key=logits.__getitem__)
        return [float(i == winner) for i in range(len(logits))]
    scaled = [z / temperature for z in logits]
    peak = max(scaled)
    exps = [math.exp(s - peak) for s in scaled]
    total = sum(exps)
    return [e / total for e in exps]

We take a real logit vector from the trained model. After the prompt "the cat s" the model is genuinely uncertain — the completion could be sat, sing, and others — so its next-token distribution has a few real competitors and a long tail. We sweep the temperature over it.

z_full = model(torch.tensor([tok.encode("the cat s")]))[0][0, -1].tolist()
order = sorted(range(len(z_full)), key=lambda i: z_full[i], reverse=True)
head = order[:8]
labels = [tok.decode([i], errors="replace") for i in head]
print("top-8 continuations:", labels)
for T in (0.5, 1.0, 2.0):
    p = softmax(z_full, T)
    entropy = -sum(x * math.log(x) for x in p if x)
    print(f"T={T}:  top prob {max(p):.3f}   entropy {entropy:.2f} nats   argmax id {p.index(max(p))}")

top-8 continuations: ['at ', '. the cat ', 's', ' c', '. ', '. the ', 'in', 'n']
T=0.5:  top prob 0.993   entropy 0.06 nats   argmax id 270
T=1.0:  top prob 0.721   entropy 1.69 nats   argmax id 270
T=2.0:  top prob 0.123   entropy 5.09 nats   argmax id 270

The argmax token is identical at all three temperatures; only the confidence in it changes, from near-certain at T=0.5 to nearly flat at T=2.0. Figure 9.3 shows the head of the distribution melting as T rises.

Show the code that draws this figure

import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(8.6, 3.0), sharey=True)
for ax, T in zip(axes, (0.5, 1.0, 2.0)):
    p = softmax(z_full, T)
    ax.bar(range(8), [p[i] for i in head], color="0.35")
    ax.set_title(f"T = {T}"); ax.set_xticks(range(8))
    ax.set_xticklabels([repr(l) for l in labels], rotation=90, fontsize=6)
axes[0].set_ylabel("probability")
plt.show()

Figure 9.3

Sampling from the full softmax keeps every token with nonzero probability, including a long tail of implausible ones. Truncation masks part of the distribution and renormalizes the survivors before drawing — a bias–diversity decision. Four policies express different invariants. Top-k keeps the k highest-probability tokens (always at least one), a fixed support regardless of shape. Top-p (nucleus, Holtzman et al., 2020) keeps the smallest prefix whose cumulative mass reaches p; its support adapts to shape, and the token that crosses the threshold belongs inside the set. Min-p keeps token i when p_i \ge \alpha\,p_{\max}, a floor relative to the mode (Nguyen et al., 2024). Locally typical sampling keeps tokens whose surprisal -\log p_i sits near the distribution’s entropy (Meister et al., 2023). We implement all four with one nonempty, renormalized result.

# @save
import math

def truncate(probs: list[float], method: str = "full", value: float = 1.0) -> list[float]:
    """Mask a distribution to a candidate set and renormalize the survivors.

    Supports four policies that trade fixed against adaptive support: ``top_k``
    (the ``value`` highest tokens), ``top_p`` (the smallest prefix reaching
    cumulative mass ``value``, including the crossing token), ``min_p`` (tokens
    at least ``value`` times the peak probability), and ``typical`` (tokens
    whose surprisal is closest to the entropy until mass ``value``). Every
    policy keeps at least one token, so the result is always a valid draw.

    Args:
        probs: A probability distribution (already temperature-scaled).
        method: One of ``full``, ``top_k``, ``top_p``, ``min_p``, ``typical``.
        value: The policy parameter (``k``, ``p``, ``alpha``, or mass budget).

    Returns:
        A distribution over the same support with dropped tokens at 0.0 and the
        kept tokens renormalized to sum to 1.

    Raises:
        ValueError: If ``method`` is not a known policy.
    """
    order = sorted(range(len(probs)), key=lambda i: probs[i], reverse=True)
    if method == "full":
        keep = set(order)
    elif method == "top_k":
        keep = set(order[: max(1, int(value))])
    elif method == "top_p":
        keep, mass = set(), 0.0
        for i in order:
            keep.add(i); mass += probs[i]
            if mass >= value:
                break
    elif method == "min_p":
        cutoff = value * probs[order[0]]
        keep = {i for i in order if probs[i] >= cutoff} or {order[0]}
    elif method == "typical":
        entropy = -sum(p * math.log(p) for p in probs if p)
        ranked = sorted(order, key=lambda i: abs(-math.log(probs[i]) - entropy) if probs[i] else 1e9)
        keep, mass = set(), 0.0
        for i in ranked:
            keep.add(i); mass += probs[i]
            if mass >= value:
                break
    else:
        raise ValueError(f"unknown truncation method: {method}")
    total = sum(probs[i] for i in keep)
    return [probs[i] / total if i in keep else 0.0 for i in range(len(probs))]

We run each policy on the same real distribution and print both the total support and which of the top-8 survive. The interesting behavior is in the tail: top_p reaches deep to accumulate its mass, while min_p collapses to the single confident token.

base = softmax(z_full, 1.0)
print(f"full distribution support: {sum(x > 0 for x in base)} tokens")
for method, value in [("top_k", 3), ("top_p", 0.8), ("min_p", 0.2), ("typical", 0.8)]:
    d = truncate(base, method, value)
    survivors = [labels[j] for j, i in enumerate(head) if d[i] > 0]
    print(f"{method:8} {value}:  support {sum(x > 0 for x in d):3d}   head survivors {survivors}")

full distribution support: 320 tokens
top_k    3:  support   3   head survivors ['at ', '. the cat ', 's']
top_p    0.8:  support   4   head survivors ['at ', '. the cat ', 's', ' c']
min_p    0.2:  support   1   head survivors ['at ']
typical  0.8:  support   4   head survivors ['at ', '. the cat ', 's', ' c']

Show the code that draws this figure

import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 3, figsize=(8.6, 3.0), sharey=True)
for ax, (method, value) in zip(axes, [("top_k", 3), ("top_p", 0.8), ("min_p", 0.2)]):
    d = truncate(base, method, value)
    colors = ["0.30" if d[i] > 0 else "0.80" for i in head]
    ax.bar(range(8), [base[i] for i in head], color=colors)
    ax.set_title(f"{method} = {value}"); ax.set_xticks(range(8))
    ax.set_xticklabels([repr(l) for l in labels], rotation=90, fontsize=6)
axes[0].set_ylabel("probability")
plt.show()

Figure 9.4

The order matters: temperature reshapes the probabilities that top-p, min-p, and typicality then inspect, so temperature → probabilities → truncation → renormalize is the pipeline, and moving a stage changes the policy.

9.4 Logit processors and the order they run in

Real generation applies more than a temperature and a mask. A logit processor edits logits using token history or caller policy before sampling. A logit bias adds a fixed offset; a presence penalty fires once after a token appears; a frequency penalty grows with its count. A repetition penalty needs a sign-aware rule: divide a positive seen-token logit by r>1 to shrink it, but multiply a negative one by r. Dividing a negative logit would move it toward zero and quietly reward repetition. Writing c_i for the count of token i, a_i for an explicit bias, f for the frequency weight, and u for the presence weight, one additive form is

z'_i = z_i + a_i - f c_i - u \,\mathbf{1}[c_i > 0]. \tag{9.3}

# @save
def process_logits(logits: list[float], counts: list[int], repetition: float = 1.0,
                   frequency: float = 0.0, presence: float = 0.0,
                   bias: dict[int, float] | None = None) -> list[float]:
    """Rewrite logits from token history using a sign-aware repetition rule.

    Applies @eq-ch09-processor plus a multiplicative repetition penalty that is
    sign-aware: a positive seen logit is divided by ``repetition`` and a
    negative one is multiplied by it, so a repeated token's probability always
    falls. Frequency and presence penalties and an explicit per-token bias are
    additive, so their strength scales as ``1/T`` if a later temperature divide
    follows — which is why processor order is part of the policy.

    Args:
        logits: The model's raw logits.
        counts: How many times each token already appears in the context.
        repetition: Sign-aware multiplicative penalty (1.0 disables it).
        frequency: Additive penalty per prior occurrence.
        presence: Additive penalty applied once if the token appeared at all.
        bias: Optional explicit per-index logit offsets.

    Returns:
        The rewritten logit list, same length as ``logits``.
    """
    out = logits[:]
    for i, count in enumerate(counts):
        if count and repetition != 1.0:
            out[i] = out[i] / repetition if out[i] > 0 else out[i] * repetition
        out[i] -= frequency * count + presence * float(count > 0)
        out[i] += (bias or {}).get(i, 0.0)
    return out

A quick check on three tokens — one positive and seen, one negative and seen, one unseen — shows the sign-aware split: the positive logit shrinks toward zero, the negative one grows more negative, and the untouched token is unchanged.

demo = process_logits([2.0, -2.0, 0.5], counts=[3, 3, 0], repetition=1.5, frequency=0.1)
print("before:", [2.0, -2.0, 0.5])
print("after :", [round(x, 3) for x in demo])

before: [2.0, -2.0, 0.5]
after : [1.033, -3.3, 0.5]

Figure 9.5 places these stages in the order the sampler uses them. A serving library may fuse stages or document a different order; that variation is exactly what a versioned adapter test should pin.

flowchart LR
    Z(["Model logits z"]) --> P["History processors:\nrepetition, frequency, presence"]
    P --> B["Bias + hard token masks"]
    B --> T["Divide by temperature T"]
    T --> S(["Stable softmax"])
    S --> M["top-k / top-p / min-p / typical mask"]
    M --> R["Renormalize survivors"]
    R --> C{"Choice rule"}
    C -->|sample| RNG(["Seeded categorical draw"])
    C -->|greedy| ARG(["Argmax + tie rule"])

Figure 9.5: In what order does a decoding configuration become one categorical draw? History and bias processors edit logits first; temperature then rescales them (so additive penalties feel the 1/T scaling); the softmax, a truncation mask, and renormalization follow; finally a choice rule samples or takes the argmax. Each stage changes a different property of the draw.

Penalties are behavioral tools, not semantic guarantees. Source code legitimately repeats a bracket or variable name; an aggressive repetition penalty can make valid syntax unreachable, and a banned surface token often has a synonym or alternate tokenization. When a property must hold, enforce it with a parser or schema, not a penalty. The workload table below is a starting map, not a set of universal defaults.

Workload	Useful starting policy	Measure before changing it
Extract one label from a closed set	greedy or constrained choice	label accuracy, invalid-output rate
Generate varied creative candidates	moderate sampling + adaptive truncation	preference, diversity, repetition
Produce executable code	low-entropy candidates + tests	test-pass rate, syntax, run-to-run reliability
Answer a consequential factual query	sampling is secondary to evidence and abstention	supported-claim accuracy, coverage, escalation load

The last row prevents a category error we return to below: lowering temperature makes a false answer more repeatable, never more correct.

9.5 Seeded is not batch-invariant

Set temperature to zero, fix a seed, send the same prompt twice under load, and you can still get different text. Sampling need not be the cause. Greedy output flips whenever a tiny numerical change reorders two nearly tied logits, and floating-point addition is not associative: (a+b)+c can differ from a+(b+c). GPU and CPU kernels partition reductions across threads and tiles, and a runtime may pick a different split when the batch shape, sequence packing, or kernel changes. Those legal execution paths produce slightly different logits — and once an argmax flips, every later token inherits a different context.

We can measure this on the real model. We take one prompt, compute its last-token logits alone, then batch identical copies of it and compare. Nothing about the prompt changed; only the batch shape did.

@torch.inference_mode()
def last_logits(ids_batch, net):
    return net(ids_batch)[0][:, -1]

batch_prompt = torch.tensor([tok.encode("the dog ran in the park and the cat sat on the")])
base_logits = last_logits(batch_prompt, model)[0]
for B in (2, 8, 32):
    got = last_logits(batch_prompt.repeat(B, 1), model)[0]
    drift = (got - base_logits).abs().max().item()
    flip = int(base_logits.argmax().item() != got.argmax().item())
    print(f"batch {B:3d}:  max |Δlogit| {drift:.2e}   greedy flip {flip}")

fp64 = model.double()
base64 = last_logits(batch_prompt, fp64)[0]
drift64 = (last_logits(batch_prompt.repeat(32, 1), fp64)[0] - base64).abs().max().item()
model.float()
print(f"batch  32 in float64:  max |Δlogit| {drift64:.2e}")

batch   2:  max |Δlogit| 0.00e+00   greedy flip 0
batch   8:  max |Δlogit| 0.00e+00   greedy flip 0
batch  32:  max |Δlogit| 0.00e+00   greedy flip 0
batch  32 in float64:  max |Δlogit| 2.66e-15

The drift is small — around a part in ten million in float32 — and here it does not flip the greedy token, because this prompt’s top two logits are far apart. Note the trap: at batch 2 the logits may be bitwise identical, so the bug hides until load grows the batch. In float64 the drift nearly vanishes, which locates the cause in finite precision, not in the model. To see how a drift this size can flip a choice, we construct the worst case: a logit built from a catastrophic cancellation, summed two legal ways.

# @save
def batch_invariance_probe() -> dict[str, float]:
    """Show that two legal reduction orders can flip a greedy argmax.

    A logit is assembled from terms that catastrophically cancel. Summed left to
    right the small term is lost and the logit is 0.0; summed as two partial
    sums the large terms cancel first and the logit is 1.0. Against a rival at
    0.5 the two orders pick different winners, so a purely numerical reduction
    change — not randomness — moves the argmax and the top probability.

    Returns:
        A dict with both reduction sums, whether the greedy winner flipped, and
        the maximum probability drift between the two orders.
    """
    parts = [1e16, 1.0, -1e16, 0.0]
    serial = 0.0
    for term in parts:
        serial += term                      # left-to-right loses the 1.0
    partitioned = (parts[0] + parts[2]) + (parts[1] + parts[3])   # cancels large terms first
    single = softmax([serial, 0.5])
    batched = softmax([partitioned, 0.5])
    return {"serial_sum": serial, "partitioned_sum": partitioned,
            "greedy_flip": float(single.index(max(single)) != batched.index(max(batched))),
            "max_drift": max(abs(a - b) for a, b in zip(single, batched))}

We run the constructed probe and read off the flip.

print(batch_invariance_probe())

{'serial_sum': 0.0, 'partitioned_sum': 1.0, 'greedy_flip': 1.0, 'max_drift': 0.24491866240370913}

The two orders disagree by a full token and move the top probability by about a quarter. That is the mechanism behind the measured drift, magnified. It motivates four distinct reproducibility claims, in increasing strength: seed replay (same probabilities and RNG stream give the same draws), run-to-run repeatability (same request and environment give the same values), batch invariance (a request is unaffected by its batch-mates), and cross-environment bitwise identity (identical across hardware, drivers, and library versions). A seed buys only the first. Figure 9.6 lists the surfaces that must be pinned before a repeated request can promise the same bytes.

flowchart LR
    I(["Canonical request bytes"]) --> W["Pinned weights + tokenizer + template"]
    W --> N["Pinned numerical path:\ndtype, kernels, reductions"]
    N --> Bt["Batch- and partition-invariant scheduling"]
    Bt --> R["Pinned RNG stream + tie rule"]
    R --> D["Buffered detokenization + stop policy"]
    D --> O(["Reproducible output bytes"])
    U["Unpinned replica, load, version, hardware"] -.-> N
    U -.-> Bt

Figure 9.6: Which surfaces must be invariant before a repeated request can promise the same output? A seed pins only the RNG stream. Weights, tokenizer, template, numerical path, batch scheduling, and detokenization each add a door; an unpinned replica, load level, or library version reopens the numerical-path and scheduling doors from the side.

Test the property you actually need, under the concurrency you will run, and record prompt bytes, all decoding controls, model and tokenizer revisions, kernel versions, device, and batch context. If the product invariant is semantic equivalence rather than byte identity, define and measure that separately — Chapter 22 owns that statistical comparison; this section owns the causal surfaces.

9.6 Hallucination is a task-relative failure

Now a different failure. A user asks, “Which treaty created the Aerolith Maritime Council in 1987?” The entity is invented. The model returns a fluent treaty name, a date, a membership list, and a citation. Lowering the temperature makes the same fabrication more consistent. The loop is behaving exactly as configured; the answer fails against the world.

“Hallucination” is only a measurable defect once the evaluation contract names what the output must be faithful to. Absent that, the word collapses failures with different remedies.

Failure	Reference violated	Example	First measurement
Factual error	external world state	wrong capital or date	atomic factual correctness
Fabrication	existence in the world	invented entity or paper	existence / provenance check
Unsupported claim	supplied evidence	true-sounding, absent from sources	claim-level support
Source contradiction	supplied evidence	reverses a policy clause	entailment / contradiction
Stale knowledge	required snapshot time	names a former office holder	time-stamped correctness
Task misunderstanding	user intent or schema	answers a related question	intent / task success

A long answer should be decomposed into atomic claims before scoring, because one correct paragraph can carry one consequential false sentence (Min et al., 2023). We take the fabricated answer, split it into atomic claims, mark each supported or not against what actually exists, and compute a support fraction.

# @save
def support_fraction(claims: list[tuple[str, bool]]) -> float:
    """Fraction of atomic claims that are supported by the reference.

    Long-form factuality is scored per atomic claim, not per answer, so a fluent
    response with one false sentence is not credited as correct. ``claims`` pairs
    each atomic statement with whether the reference (world state or source set)
    supports it.

    Args:
        claims: ``(claim_text, is_supported)`` pairs from one answer.

    Returns:
        Supported claims divided by total claims, in [0, 1]; 0.0 if empty.
    """
    if not claims:
        return 0.0
    return sum(1 for _, ok in claims if ok) / len(claims)

We decompose the fabricated Aerolith answer and score it.

answer = [
    ("The Aerolith Maritime Council was created by a treaty", False),
    ("It was created in 1987", False),
    ("Maritime councils coordinate shipping policy", True),
    ("It has seven member states", False),
]
supported = sum(ok for _, ok in answer)
print(f"atomic claims: {len(answer)}   supported: {supported}   support fraction: {support_fraction(answer):.2f}")

atomic claims: 4   supported: 1   support fraction: 0.25

A support fraction of a quarter, with a real-sounding citation, is the shape of a confident fabrication. Why does it happen? Next-token training rewards continuations that match text distributions, and a false premise can look exactly like a legitimate historical question whose likely continuations include treaty vocabulary. Post-training and evaluations that score a guess above an abstention sharpen the incentive to answer anyway; the 2025 analysis of Why Language Models Hallucinate (Kalai et al., 2025) formalizes that guessing incentive, and TruthfulQA (Lin et al., 2022) earlier showed models reproduce popular human falsehoods. No decoding rule repairs a missing fact: greedy picks the most probable false continuation, sampling widens the tail, and self-consistency can concentrate a shared misconception. What changes the outcome is information — external evidence and grounding, which Chapter 14 owns — or declining to answer, which is the rest of this chapter.

9.7 Confidence must forecast correctness

To decide whether to answer, we need a confidence signal that forecasts correctness. A score q(x) is calibrated for a binary event Y when the cases it rates near c are correct about a fraction c of the time:

\Pr\!\big(Y = 1 \mid q(X) \approx c\big) \approx c. \tag{9.4}

Calibration is population- and event-specific: a score calibrated for short-answer correctness need not be calibrated for every atomic claim or after a domain shift. Several signals compete. Token log probability is available but measures fluency, not truth; summed it grows more negative with length, averaged it changes the quantity. P(True) shows the model its own answer and asks whether it is correct, a more relevant event with useful in-distribution calibration (Kadavath et al., 2022). Verbalized confidence — the model stating “I’m about 80% sure” — can beat raw probabilities for some post-trained models (Tian et al., 2023) but is a learned utterance that must itself be calibrated. We compare two of these on a labeled fixture, honestly synthetic, where each item carries a correctness label, an overconfident logprob-style score, and a coarse verbalized score.

# @save
import random

def qa_fixture(n: int = 400, seed: int = 0) -> list[dict]:
    """Build an illustrative QA fixture with two competing confidence signals.

    Each item has a ground-truth ``correct`` label plus two confidence scores on
    the same question: ``logprob_conf``, a token-probability-style signal that is
    monotone in latent skill but systematically overconfident, and
    ``verbal_conf``, a coarse rounded "how sure are you" number. The two are
    deliberately miscalibrated in different ways so calibration and abstention
    have something to fix. Values are synthetic, not measured on any model.

    Args:
        n: Number of items to generate.
        seed: Seed for the internal RNG, so the fixture is reproducible.

    Returns:
        A list of dicts with ``correct``, ``logprob_conf``, and ``verbal_conf``.
    """
    rng = random.Random(seed)
    items = []
    for _ in range(n):
        skill = rng.random()
        correct = int(rng.random() < skill)
        logprob_conf = min(0.99, max(0.5, 0.5 + 0.5 * skill ** 0.5))
        verbal_conf = min(0.95, max(0.5, round((0.4 + 0.6 * skill) * 10) / 10))
        items.append({"correct": correct, "logprob_conf": logprob_conf, "verbal_conf": verbal_conf})
    return items

Two proper scoring rules grade a probability against an outcome: the Brier score (q-y)^2 and log loss; both are minimized in expectation by truthful probabilities. Expected calibration error (ECE) bins predictions and averages the gap between mean confidence and bin accuracy — readable, but sensitive to bins. We compute all three for both signals.

# @save
def brier(items: list[dict], key: str) -> float:
    """Mean squared error between a confidence score and the binary outcome."""
    return sum((it[key] - it["correct"]) ** 2 for it in items) / len(items)

def ece(items: list[dict], key: str, bins: int = 10) -> float:
    """Expected calibration error: accuracy-weighted gap between confidence and accuracy.

    Predictions are grouped into ``bins`` equal-width confidence buckets; each
    bucket contributes the absolute difference between its mean confidence and
    its empirical accuracy, weighted by its share of the data.

    Args:
        items: Scored items with a ``correct`` field.
        key: Which confidence field to grade.
        bins: Number of equal-width confidence buckets.

    Returns:
        The weighted calibration gap, in [0, 1]; lower is better.
    """
    buckets = [[] for _ in range(bins)]
    for it in items:
        buckets[min(bins - 1, int(it[key] * bins))].append(it)
    error = 0.0
    for bucket in buckets:
        if bucket:
            confidence = sum(it[key] for it in bucket) / len(bucket)
            accuracy = sum(it["correct"] for it in bucket) / len(bucket)
            error += len(bucket) / len(items) * abs(confidence - accuracy)
    return error

fixture = qa_fixture()
accuracy = sum(it["correct"] for it in fixture) / len(fixture)
print(f"base accuracy: {accuracy:.3f}")
for key in ("logprob_conf", "verbal_conf"):
    print(f"{key:13}  ECE {ece(fixture, key):.3f}   Brier {brier(fixture, key):.3f}")

base accuracy: 0.515
logprob_conf   ECE 0.320   Brier 0.296
verbal_conf    ECE 0.196   Brier 0.224

The raw logprob signal is badly overconfident; the verbalized signal is coarser but here better calibrated — a reminder that neither is trustworthy by default. Open-ended answers add a second problem: “Paris” and “The capital is Paris” are different strings with the same meaning, and surface entropy counts them apart. Semantic entropy first groups mutually equivalent answers into meaning clusters c, sums the mass in each, and computes

H_{\text{sem}}(x) = -\sum_c P(c \mid x)\,\log P(c \mid x), \qquad P(c \mid x) = \sum_{y \in c} p(y \mid x). \tag{9.5}

Real systems cluster with a bidirectional-entailment model (Kuhn et al., 2023; Farquhar et al., 2024); we use a deterministic normalizer — lowercase, drop punctuation and a few stopwords, keep the content words as the cluster key — which collapses paraphrases of a single answer.

# @save
import math, re

def normalize_answer(answer: str) -> tuple:
    """Canonicalize an answer to a content-word key for meaning-equivalence.

    Lowercases, keeps alphanumeric words, drops a small stopword set, and sorts
    the remainder, so "Paris", "The city is Paris", and "paris." map to the same
    key. This is a deterministic stand-in for the entailment-based clustering a
    production semantic-entropy estimator would use.

    Args:
        answer: One sampled answer string.

    Returns:
        A sorted tuple of content words used as the cluster identity.
    """
    words = re.findall(r"[a-z0-9]+", answer.lower())
    stop = {"the", "a", "an", "is", "was", "it", "of", "in", "city", "capital", "named"}
    return tuple(sorted(w for w in words if w not in stop))

def semantic_entropy(samples: list[tuple[str, float]]) -> dict:
    """Compare surface-form entropy with entropy over meaning clusters.

    Groups ``(answer, probability)`` samples by ``normalize_answer`` and computes
    entropy before and after grouping (@eq-ch09-semantic). Paraphrases of one
    answer collapse into a single cluster, so semantic entropy is at most surface
    entropy and is lower whenever surface variation is meaning-preserving.

    Args:
        samples: Sampled answers with their probabilities.

    Returns:
        A dict with surface and semantic entropy (nats) and the cluster masses.
    """
    surface, clusters = {}, {}
    for answer, prob in samples:
        surface[answer] = surface.get(answer, 0.0) + prob
        key = normalize_answer(answer)
        clusters[key] = clusters.get(key, 0.0) + prob
    entropy = lambda ps: -sum(p * math.log(p) for p in ps if p > 0)
    return {"surface_nats": entropy(surface.values()), "semantic_nats": entropy(clusters.values()),
            "clusters": {(" ".join(k) or "<empty>"): round(v, 3) for k, v in clusters.items()}}

We sample five surface answers to one question — three of them paraphrases of “Paris” — and compare surface entropy with entropy after clustering.

samples = [("Paris", 0.36), ("The city is Paris", 0.24), ("Lyon", 0.20),
           ("It was Marseille", 0.12), ("paris.", 0.08)]
result = semantic_entropy(samples)
print(f"surface entropy  {result['surface_nats']:.3f} nats")
print(f"semantic entropy {result['semantic_nats']:.3f} nats")
print("clusters:", result["clusters"])

surface entropy  1.489 nats
semantic entropy 0.839 nats
clusters: {'paris': 0.68, 'lyon': 0.2, 'marseille': 0.12}

Three surface answers about Paris collapse into one meaning cluster holding most of the mass, and the entropy drops accordingly. The remaining uncertainty is real disagreement between Paris, Lyon, and Marseille — the kind an abstention policy should react to — not paraphrase noise. Figure 9.7 shows the collapse.

Show the code that draws this figure

import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(8.0, 3.2))
axes[0].bar(range(len(samples)), [p for _, p in samples], color="0.55")
axes[0].set_xticks(range(len(samples))); axes[0].set_xticklabels([a for a, _ in samples], rotation=30, ha="right", fontsize=7)
axes[0].set_title(f"surface  ({result['surface_nats']:.2f} nats)"); axes[0].set_ylabel("probability")
clusters = result["clusters"]
axes[1].bar(range(len(clusters)), list(clusters.values()), color="0.30")
axes[1].set_xticks(range(len(clusters))); axes[1].set_xticklabels(list(clusters), rotation=30, ha="right", fontsize=7)
axes[1].set_title(f"semantic  ({result['semantic_nats']:.2f} nats)")
plt.show()

Figure 9.7

A miscalibrated but well-ranked score can often be fixed by a monotone map. Temperature scaling (Guo et al., 2017) fits one scalar on a held-out split and rescales the confidence in logit space, changing calibration without touching the ranking. We fit it on the logprob signal and re-score.

# @save
import math

def fit_temperature(items: list[dict], key: str) -> float:
    """Fit one scalar calibration temperature by minimizing held-out log loss.

    Rescales each confidence in logit space by ``1/tau`` and grid-searches ``tau``
    over [0.30, 5.0] for the value minimizing binary NLL. Because the map is
    monotone it changes only how confidence values map to probabilities, never
    the order of examples — so it improves calibration without changing which
    items a threshold answers first.

    Args:
        items: Scored items with a ``correct`` field.
        key: Which confidence field to calibrate.

    Returns:
        The temperature minimizing held-out log loss.
    """
    def nll(tau: float) -> float:
        total = 0.0
        for it in items:
            q = it[key]
            calibrated = 1 / (1 + math.exp(-math.log(q / (1 - q)) / tau))
            p = calibrated if it["correct"] else 1 - calibrated
            total -= math.log(max(p, 1e-12))
        return total / len(items)
    return min((0.30 + i * 0.01 for i in range(471)), key=nll)

def calibrate(q: float, tau: float) -> float:
    """Apply a fitted temperature to one confidence in logit space."""
    return 1 / (1 + math.exp(-math.log(q / (1 - q)) / tau))

We fit the temperature on the fixture and re-score the logprob signal.

tau = fit_temperature(fixture, "logprob_conf")
for it in fixture:
    it["cal_conf"] = calibrate(it["logprob_conf"], tau)
print(f"fitted temperature: {tau:.2f}")
print(f"logprob  raw   ECE {ece(fixture, 'logprob_conf'):.3f}   Brier {brier(fixture, 'logprob_conf'):.3f}")
print(f"logprob  cal   ECE {ece(fixture, 'cal_conf'):.3f}   Brier {brier(fixture, 'cal_conf'):.3f}")

fitted temperature: 3.60
logprob  raw   ECE 0.320   Brier 0.296
logprob  cal   ECE 0.173   Brier 0.229

Temperature scaling cuts the calibration error roughly in half. Figure 9.8 makes the correction visible as points pulled toward the diagonal.

Show the code that draws this figure

import matplotlib.pyplot as plt
def reliability(items, key, bins=10):
    xs, ys = [], []
    for b in range(bins):
        group = [it for it in items if int(it[key] * bins) == b or (b == bins - 1 and it[key] == 1.0)]
        if group:
            xs.append(sum(it[key] for it in group) / len(group))
            ys.append(sum(it["correct"] for it in group) / len(group))
    return xs, ys
fig, ax = plt.subplots(figsize=(4.6, 4.2))
ax.plot([0, 1], [0, 1], "--", color="0.6", label="perfect")
ax.plot(*reliability(fixture, "logprob_conf"), "o-", color="0.6", label="raw logprob")
ax.plot(*reliability(fixture, "cal_conf"), "s-", color="0.1", label="calibrated")
ax.set_xlabel("mean confidence"); ax.set_ylabel("bin accuracy"); ax.legend(); ax.grid(alpha=0.3)
plt.show()

Figure 9.8

9.8 Abstention and selective prediction

Calibration tells us whether a confidence can be trusted; an abstention policy turns it into an action. Below a threshold, the system can say it lacks support, ask a clarifying question, retrieve evidence, narrow the claim, or escalate to a human. The model may propose the confidence, but deterministic caller code must enforce the threshold — this is the mechanism behind Chapter 1’s decide_response, which routed answer, retrieve, or abstain from a confidence and an evidence score (Chapter 1). Writing g_t(x) = \mathbf{1}[q(x) \ge t] for “answer at threshold t” and \ell for a wrong-answer loss, coverage is the answered fraction C(t) = \mathbb{E}[g_t], and two different quantities are both called risk:

R_{\text{selective}}(t) = \frac{\mathbb{E}[\ell\, g_t]}{\mathbb{E}[g_t]}, \qquad R_{\text{marginal}}(t) = \mathbb{E}[\ell\, g_t]. \tag{9.6}

Selective risk is the error rate among answered queries; marginal error is the wrong-answer rate over all queries, counting abstentions as zero loss. They differ by coverage: R_{\text{marginal}} = C \cdot R_{\text{selective}}. A requirement of “ten percent risk” is incomplete until it names which. Sweeping t traces a risk-coverage curve.

# @save
def risk_curve(items: list[dict], key: str) -> list[dict]:
    """Trace coverage, selective risk, and marginal error as the threshold sweeps.

    For each distinct confidence value used as a threshold, reports the fraction
    answered (coverage), the error rate among answered items (selective risk,
    @eq-ch09-risk), and the wrong-answer rate over all items (marginal error).
    A useful confidence ranking lowers selective risk as coverage falls.

    Args:
        items: Scored items with a ``correct`` field.
        key: Which confidence field to threshold on.

    Returns:
        One row per threshold, from most selective to least.
    """
    rows = []
    for t in sorted({it[key] for it in items}, reverse=True):
        answered = [it for it in items if it[key] >= t]
        errors = sum(1 - it["correct"] for it in answered)
        rows.append({"threshold": t, "coverage": len(answered) / len(items),
                     "selective_risk": errors / len(answered), "marginal_error": errors / len(items)})
    return rows

We sweep the calibrated confidence and print a handful of operating points.

curve = risk_curve(fixture, "cal_conf")
for row in curve[:: max(1, len(curve) // 5)]:
    print({k: round(v, 3) for k, v in row.items()})

{'threshold': 0.782, 'coverage': 0.03, 'selective_risk': 0.0, 'marginal_error': 0.0}
{'threshold': 0.688, 'coverage': 0.223, 'selective_risk': 0.169, 'marginal_error': 0.037}
{'threshold': 0.644, 'coverage': 0.415, 'selective_risk': 0.199, 'marginal_error': 0.083}
{'threshold': 0.606, 'coverage': 0.608, 'selective_risk': 0.267, 'marginal_error': 0.163}
{'threshold': 0.567, 'coverage': 0.8, 'selective_risk': 0.375, 'marginal_error': 0.3}
{'threshold': 0.512, 'coverage': 0.993, 'selective_risk': 0.481, 'marginal_error': 0.477}

As coverage falls, selective risk falls with it — the ranking is doing its job. Figure 9.9 plots both risks against coverage; the gap between them is exactly the coverage factor.

Show the code that draws this figure

import matplotlib.pyplot as plt
cov = [r["coverage"] for r in curve]
fig, ax = plt.subplots(figsize=(6.2, 3.8))
ax.plot(cov, [r["selective_risk"] for r in curve], "o-", color="0.15", label="selective risk (answered)")
ax.plot(cov, [r["marginal_error"] for r in curve], "s-", color="0.55", label="marginal error (all)")
ax.set_xlabel("coverage (fraction answered)"); ax.set_ylabel("risk"); ax.legend(); ax.grid(alpha=0.3)
plt.show()

Figure 9.9

A threshold picked by eye has no guarantee attached. Conformal risk control (CRC) picks one that provably bounds a monotone loss (Angelopoulos et al., 2024). For a wrong-answer loss on n calibration items, accept a threshold only when (e_t + 1)/(n + 1) \le \alpha, where e_t is the number of wrong answers it would release, and take the lowest accepted threshold to maximize coverage.

# @save
import math

def crc_threshold(items: list[dict], key: str, alpha: float) -> float:
    """Lowest threshold whose released wrong-answer rate satisfies the CRC bound.

    Scans thresholds upward and returns the first where the conformal correction
    ``(errors + 1) / (n + 1) <= alpha`` holds, controlling expected marginal loss
    up to the finite-sample correction under exchangeability. The lowest such
    threshold maximizes coverage.

    Args:
        items: Calibration items with a ``correct`` field.
        key: Which confidence field to threshold on.
        alpha: Target marginal wrong-answer rate.

    Returns:
        The selected threshold, or ``inf`` if none qualifies.
    """
    for t in sorted({it[key] for it in items}):
        errors = sum(it[key] >= t and not it["correct"] for it in items)
        if (errors + 1) / (len(items) + 1) <= alpha:
            return t
    return math.inf

We select a threshold at two risk budgets and read off both risks it produces.

for alpha in (0.10, 0.20):
    t = crc_threshold(fixture, "cal_conf", alpha)
    answered = [it for it in fixture if it["cal_conf"] >= t]
    errors = sum(1 - it["correct"] for it in answered)
    print(f"alpha {alpha}:  threshold {t:.3f}   coverage {len(answered)/len(fixture):.3f}"
          f"   marginal {errors/len(fixture):.3f}   selective {errors/len(answered):.3f}")

alpha 0.1:  threshold 0.631   coverage 0.470   marginal 0.098   selective 0.207
alpha 0.2:  threshold 0.595   coverage 0.660   marginal 0.198   selective 0.299

At \alpha = 0.10 the marginal error lands under the target, but the selective risk is roughly double it — the same operating point, read as marginal or conditional, gives two very different numbers. That distinction is the whole reason to name which risk a requirement means. A second conformal tool, split conformal prediction (Angelopoulos and Bates, 2021), controls set coverage without assuming raw probabilities are calibrated. On a calibration set it scores nonconformity s_i = 1 - p_i(y_i), sorts, and takes the rank-\lceil (n+1)(1-\alpha)\rceil value as a threshold \hat q; a new input returns every answer with nonconformity at most \hat q, and under exchangeability the true answer is in that set with probability at least (1-\alpha).

# @save
import math, random

def multiclass_fixture(n_cal: int = 200, n_test: int = 200) -> tuple:
    """Build illustrative calibration and test sets of per-class probabilities.

    Each item has a probability over four answer options and the index of the
    true one; the true-class probability varies with a latent skill so answer
    sets range from confident singletons to genuine multi-answer ambiguity.
    Synthetic, for exercising the conformal machinery, not measured from a model.

    Returns:
        A ``(calibration, test)`` pair of item lists, each item a dict with
        ``probs`` and ``true_index``.
    """
    rng = random.Random(7)
    def make(n):
        out = []
        for _ in range(n):
            skill = rng.random()
            true_p = 0.30 + 0.55 * skill
            others = sorted((rng.random() for _ in range(3)), reverse=True)
            scale = (1 - true_p) / sum(others)
            probs = [true_p] + [o * scale for o in others]
            order = list(range(4)); rng.shuffle(order)
            shuffled = [0.0] * 4
            for src, dst in enumerate(order):
                shuffled[dst] = probs[src]
            out.append({"probs": shuffled, "true_index": order[0]})
        return out
    return make(n_cal), make(n_test)

def split_conformal(calibration: list[dict], test: list[dict], alpha: float) -> dict:
    """Split-conformal answer sets with guaranteed marginal true-label coverage.

    Scores nonconformity ``1 - p(true)`` on the calibration set, takes the
    rank-``ceil((n+1)(1-alpha))`` value as the set threshold, and returns every
    answer within it. Under exchangeability the true label is in the set with
    probability at least ``1 - alpha``. A singleton set is a confident answer; a
    larger set is an abstention signal.

    Args:
        calibration: Items with ``probs`` and ``true_index``.
        test: Held-out items scored the same way.
        alpha: Target miscoverage; coverage is guaranteed at ``1 - alpha``.

    Returns:
        A dict with the threshold ``qhat`` and coverage / singleton statistics.
    """
    scores = sorted(1 - it["probs"][it["true_index"]] for it in calibration)
    rank = min(len(scores), math.ceil((len(scores) + 1) * (1 - alpha))) - 1
    qhat = scores[rank]
    contained = singleton = errors = 0
    for it in test:
        answer_set = [i for i, p in enumerate(it["probs"]) if 1 - p <= qhat + 1e-12]
        contained += int(it["true_index"] in answer_set)
        if len(answer_set) == 1:
            singleton += 1
            errors += int(answer_set[0] != it["true_index"])
    return {"qhat": qhat, "set_coverage": contained / len(test),
            "singleton_coverage": singleton / len(test),
            "singleton_risk": errors / singleton if singleton else 0.0}

We build the calibration and test sets and read the coverage the guarantee delivers.

cal, test = multiclass_fixture()
print({k: round(v, 3) for k, v in split_conformal(cal, test, 0.10).items()})

{'qhat': 0.655, 'set_coverage': 0.905, 'singleton_coverage': 0.885, 'singleton_risk': 0.04}

Set coverage clears the guaranteed level, and most queries resolve to a singleton set — a confident answer — while the rest return a larger set that a safe interface treats as an abstention. The guarantee is about true-label inclusion in the set, not that each singleton is correct; these are different promises, and conflating them is how a “guaranteed” system ships an unbounded conditional error. A conformal guarantee also assumes exchangeability, which users, prompts, and model updates break, so monitor coverage, freeze the policy version per decision, and price the cost of abstention — delayed care or an overloaded queue is a real harm too.

In the interview

Q. A stakeholder says “get the hallucination rate under five percent.” What do you pin down before agreeing? Ask which rate: marginal error over all traffic, or selective risk among answered queries — they can differ by more than 2× at the same threshold (R_{\text{marginal}} = C \cdot R_{\text{selective}}). Then name the event (exact answer, or per-atomic-claim support), the calibration population and its drift, and the confidence signal. Only then is “five percent” a measurable target with a coverage cost attached. The trap is quoting one number without its coverage, which lets a system hit 5% marginal error by abstaining on half of traffic while still being wrong a fifth of the time when it does answer.

9.9 Streaming tokens are not text

The last edge is where bytes meet users. A server emits token IDs; a UI shows Unicode text, and the boundaries need not align. UTF-8 encodes one scalar value in one to four bytes, and a byte-level tokenizer can emit a piece that ends inside a multibyte character. Decoding each event on its own then produces replacement characters. Stop sequences have the same problem one level up: a requested stop of STOP can arrive as ST then OP, and checking each chunk alone leaks the marker. The fix is to run an incremental decoder and hold back the longest possible stop-prefix.

# @save
import codecs

def stream_incremental(pieces: list[bytes], stop: str = "") -> str:
    """Decode a byte-piece stream into safe text, buffering across boundaries.

    Runs a stateful UTF-8 decoder so a multibyte character split across pieces is
    emitted only once complete, and withholds the last ``len(stop) - 1``
    characters so a stop sequence straddling two pieces is caught before any of
    it reaches the user. Returns text up to the stop if one is found.

    Args:
        pieces: Successive byte fragments (e.g. token ``token_bytes``).
        stop: Optional stop string to detect across piece boundaries.

    Returns:
        The safely decoded text, truncated at the stop if present.
    """
    decoder = codecs.getincrementaldecoder("utf-8")()
    pending, emitted = "", []
    for piece in pieces:
        pending += decoder.decode(piece)
        if stop and stop in pending:
            emitted.append(pending.split(stop, 1)[0])
            return "".join(emitted)
        hold = max(0, len(stop) - 1)
        if len(pending) > hold:
            emitted.append(pending[:-hold] if hold else pending)
            pending = pending[-hold:] if hold else ""
    return "".join(emitted) + pending + decoder.decode(b"", final=True)

We drive it with the real Chapter 2 tokenizer. The emoji 🙂 is four UTF-8 bytes the tokenizer never merged, so it streams as four separate single-byte pieces — the exact case that breaks naive per-piece decoding.

ids = tok.encode("Hi 🙂")
pieces = [tok.token_bytes[i] for i in ids]
print("token pieces:", pieces)
naive = "".join(p.decode("utf-8", errors="replace") for p in pieces)
print("naive per-piece decode:", repr(naive))
print("incremental decode:    ", repr(stream_incremental(pieces)))
print("stop across chunks:    ", repr(stream_incremental([b"ans", b"wer STO", b"P hidden"], "STOP")))

token pieces: [b'H', b'i', b' ', b'\xf0', b'\x9f', b'\x99', b'\x82']
naive per-piece decode: 'Hi ����'
incremental decode:     'Hi 🙂'
stop across chunks:     'answer '

Naive decoding emits replacement characters where the emoji was split; the incremental decoder reassembles it, and the stop marker straddling two chunks is caught before it leaks. A production streamer must also handle several stop strings, end-of-sequence tokens, cancellation, length limits, and an explicit finish reason — never infer “the model chose to stop” when the server hit a token cap.

9.10 Landscape 2026 (dated)

Verified 2026-07-19. Sampler surfaces move: current Hugging Face generation docs expose top-p, min-p, typical sampling, penalties, and stopping criteria, but names, defaults, and composition order have changed across releases — pin and inspect the effective GenerationConfig rather than copying a default. On determinism, the PyTorch reproducibility note states full reproducibility is not guaranteed across releases or platforms; Thinking Machines Lab’s 2025 batch-invariance analysis traced request-level nondeterminism to batch-dependent reductions, and vLLM’s batch-invariance mode is documented as beta with coverage limits and a performance cost.

Verify live: sampler parameter names and defaults; processor order; log-probability availability; batch-invariance feature status and supported kernels; deterministic-mode performance; and the task assumptions of any named uncertainty method before trusting its guarantee.

9.11 Summary

Autoregressive inference is two machines: a compute-bound prefill and a bandwidth-bound decode, whose costs split apart as context grew. Decoding is caller policy layered on logits — temperature reshapes gaps without reordering, truncation trades fixed against adaptive support, and processor order is part of the policy. A seed pins randomness but not the numerical or batch path, so batching one prompt drifts its logits. Its core is knowing when not to answer: hallucination is failure against a named reference, confidence must be calibrated before trust, and a risk-coverage curve with a conformal threshold turns abstention into a guarantee at a stated coverage cost. Chapter 10 carries this anatomy into serving.

9.12 Exercises

Sweep the sampler on the real vector. Using softmax and truncate on the trained model’s "the cat s" logits, sweep temperature over \{0.7, 1.0, 1.5\} crossed with top-p \in \{0.8, 0.95\} and min-p \in \{0.05, 0.2\}. For each setting, draw 200 samples with a seeded generator and report support size, distinct tokens drawn, and entropy. Explain why a setting that looks best on this vector is not a universal default.
Break a necessary repetition. Add a no_repeat_ngram rule alongside process_logits. Construct a short code or JSON continuation in which a repeated bracket or key token is required, then show the rule making valid output unreachable at penalty 2.0. Name the deterministic constraint that should own syntax instead of a penalty.
Hunt for a real greedy flip. Extend the batch-drift measurement to search prompts for one whose top-two logits are close enough that batching flips the greedy token. (a) Report the smallest batch size at which it flips and the logit gap involved. (b) Repeat in float64 and explain what changes. (c) State which of the four reproducibility claims your finding violates.
Compare confidence signals under shift. The fixture’s logprob_conf and verbal_conf are miscalibrated differently. Fit fit_temperature to each, then re-score a shifted fixture built with a different seed and lower base accuracy. Report ECE and Brier before and after on the shifted data, and explain why a calibrator fit on one population can fail on another.
Separate calibration from ranking. Replace scalar temperature scaling with any other monotone calibrator. Show that ECE and Brier change but the risk-coverage curve and the order in which items are answered do not. Explain why calibration changes the meaning of a threshold without changing which items clear it.
Stress the conformal guarantee. Rerun crc_threshold and split_conformal with calibration sizes 50, 200, and 1,000 over many random splits. Plot the realized coverage, marginal error, and selective risk, and identify the smallest calibration set at which the guarantee is reliable. Then construct an exchangeability violation (e.g., sort by difficulty before splitting) and show the guarantee break.
Design a consequential abstention contract. For a medical-information endpoint with a stated wrong-answer budget, specify the event, whether the budget is marginal or selective, the confidence signal and its calibration population, the conformal target, the decoding policy, the clarify/retrieve/escalate branches, the drift trigger, and the user-visible message. Defend the residual harm from both wrong answers and abstentions.