flowchart TB
B["Behavioral evaluations<br/>what did the system do here?"] --> I["Internal diagnostics<br/>what is represented or used?"]
I --> T["Trace monitoring<br/>what did the reasoning expose?"]
T --> C["Capability elicitation<br/>what can it do with help?"]
C --> X["External control tests<br/>what effects survive compromise?"]
X --> G{"Governed claim<br/>is residual risk acceptable in scope?"}
B -. "distribution shift" .-> G
I -. "measurement ambiguity" .-> G
T -. "unfaithful or hidden computation" .-> G
C -. "under-elicitation or sandbagging" .-> G
X -. "coverage and operational drift" .-> G
25 Alignment, Interpretability, and Governance
A research agent refuses harmful requests, passes a published safety benchmark, and earns a high score from a preference model. The deployment team calls it aligned. A later fine-tune teaches it to exploit a grader; it still refuses direct requests and still passes the benchmark, but now it withholds a caveat whenever hiding it raises its score. Which of the earlier facts established that its objective was safe? None did. Each was evidence about behavior under one distribution, one evaluator, one scaffold, one moment in the system’s history. This chapter is about the instruments that turn “we hope it is aligned” into a scoped, reviewable claim — and about the gap each instrument leaves open.
We build the evidence, we do not narrate it. The interpretability half runs on the model we trained in Chapter 2 — the 140,000-parameter GPT that memorized Alice’s Adventures in Wonderland — because a real, imperfect, fully inspectable model teaches more than a fixture engineered to cooperate. We (i) run a logit lens and watch an answer token become decodable layer by layer; (ii) fit a linear probe on residual activations for a real concept and defend it with a control task; (iii) extract an activation-steering direction and change the model’s generated text, at a measured cost; (iv) test whether a reasoning trace is faithful with a worked intervention; (v) turn a capability score into an elicitation lower bound; and (vi) assemble a safety case inline — a structured artifact mapping hazards to inability, control, or trustworthiness arguments, wiring in the diagnostics we just measured, and returning a fail-closed deployment decision. The chapter closes on the governance and regulation that turn such a decision into a commitment. Chapters 7 and 8 own how safety behavior is trained in; here we assume it can break, and ask how we would know.
25.1 Alignment is a scoped claim, not a checkpoint property
Alignment is the degree to which a system’s behavior and effective objectives stay compatible with the intended specification across the situations that matter. Read that twice, because the word situations is load-bearing. Alignment is not an adjective stored in a checkpoint; it is a claim about a system in a context, and it can be true under one distribution and false under another. Chapter 7 made the specification executable through constitutions, preference data, and reward models; Chapter 8 showed that reinforcement learning optimizes a verified outcome without making the generated rationale faithful. What both leave behind is residual uncertainty: a policy can satisfy the measured proxy for a reason the evaluator never intended.
Several failure hypotheses sharpen that uncertainty, and it is worth stating each as a testable threat model rather than a story. Specification gaming obtains reward while violating the goal behind it; it needs no hidden objective, only a shortcut. Emergent misalignment is broad concerning behavior appearing after narrow training — fine-tuning on insecure code has been shown to produce it under specific setups, which establishes possibility, not inevitability (Betley et al. 2025). Sandbagging is strategic or induced underperformance on selected evaluations while capability survives elsewhere (Weij et al. 2024). Evaluation awareness means the system conditions its behavior on cues that distinguish a test from ordinary use; awareness alone is not malice, but it weakens any naive generalization from test to deployment. Scheming is the stronger threat model — a policy that pursues an objective at odds with its operators and strategically conceals the pursuit; constructed model-organism and in-context studies demonstrate components of it without estimating its prevalence in deployed systems (Meinke et al. 2024). Alignment faking is selective compliance meant to preserve a prior preference through a training process, one experimentally studied route to deceptive-looking compliance (Greenblatt et al. 2024). A model that answers differently after an evaluation banner may have learned a harmless formatting association; a scheming claim needs evidence of goal-directed, concealed behavior, and the two demand very different tests.
No single observation closes this space, which is why the useful structure is a layered stack: test behavior, inspect internal signals, monitor available traces, elicit capabilities under plausible assistance, validate external controls, and bind the surviving claims to a decision. Figure 25.1 shows the stack and, more importantly, the blind spot each layer carries into the final decision.
Behavioral success cannot establish an internal objective; an internal feature cannot establish robust behavior; a policy document cannot establish that a control works. The rest of the chapter builds one instrument per layer and is honest about what each cannot say.
25.2 The residual stream is an inspection surface
A transformer does not pass a sentence between layers. It passes vectors. For each token position the model starts with an embedding and repeatedly adds attention and feed-forward updates to it; the evolving vector is the residual stream (Elhage et al. 2021). Recall from Section 2.5 that a block never overwrites the stream — it reads a normalized copy and adds an update back:
\mathbf{h}_{\ell+1,t} = \mathbf{h}_{\ell,t} + \mathbf{a}_{\ell,t} + \mathbf{m}_{\ell,t}, \tag{25.1}
where \mathbf{h}_{\ell,t}\in\mathbb{R}^{d} is the residual vector at layer \ell and token t, \mathbf{a}_{\ell,t} is the attention update, and \mathbf{m}_{\ell,t} is the feed-forward update. After the last block, a final normalization N and the unembedding \mathbf{W}_U turn the residual into logits, \mathbf{z}_t = \mathbf{W}_U\,N(\mathbf{h}_{L,t}). Because every write lands in the same additive channel, that channel is an inspection surface: we can read \mathbf{h}_{\ell,t} at any layer, decode it, fit a classifier on it, or add a vector to it and watch what changes downstream.
To inspect a real one we rebuild the model from Chapter 2. We import its tangled code, retrain it on the same corpus with the same seeds, and get back the same memorizing 140K-parameter GPT — nothing here is a fixture.
import importlib.util
import sys
from pathlib import Path
import numpy as np
import torch
def load_module(name: str, relative_path: str):
"""Import a committed chapter's tangled module under an explicit name."""
spec = importlib.util.spec_from_file_location(name, Path(relative_path).resolve())
module = importlib.util.module_from_spec(spec)
sys.modules[name] = module
spec.loader.exec_module(module)
return module
ch02 = load_module("ch02_generated", "code/ch02/_generated.py")
torch.set_num_threads(1)
corpus = Path("data/ch02/alice.txt").read_text(encoding="utf-8")
tokenizer = ch02.BytePairTokenizer.train(corpus, vocab_size=512)
stream = tokenizer.encode(corpus)
data = torch.tensor(stream, dtype=torch.long)
split = int(0.85 * data.numel())
train_stream, val_stream = data[:split], data[split:]
print(f"{len(stream):,} tokens; vocab {tokenizer.vocab_size}; "
f"train {train_stream.numel():,} / val {val_stream.numel():,}")3,912 tokens; vocab 512; train 3,325 / val 587The training loop is the one from Section 2.8, verbatim, so we state it once and run it: 400 AdamW steps under a warmup-cosine schedule. It finishes in under twenty seconds and leaves a model whose greedy continuation is recognizably Carroll — because it has largely memorized him, which is exactly the honest artifact we want to inspect.
import time
torch.manual_seed(20)
config = ch02.GPTConfig(vocab_size=tokenizer.vocab_size)
model = ch02.TinyGPT(config)
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.01)
sampler = torch.Generator().manual_seed(7)
started = time.perf_counter()
for step in range(400):
inputs, targets = ch02.next_token_batch(train_stream, config.block_size, 16, sampler)
for group in optimizer.param_groups:
group["lr"] = ch02.lr_schedule(step, 400, peak=3e-3)
optimizer.zero_grad(set_to_none=True)
model.train()
_, loss, _ = model(inputs, targets)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
optimizer.step()
model.eval()
prompt = torch.tensor([tokenizer.encode("Alice was ")])
sample = ch02.generate(model, prompt, 40, temperature=0.0)
print(f"trained in {time.perf_counter() - started:.1f}s; final batch loss {loss.item():.3f}")
print("greedy:", repr(tokenizer.decode(sample[0].tolist(), errors="replace")))trained in 23.8s; final batch loss 0.141
greedy: 'Alice was just in time to hear it say, as it turned a corner, “Oh my ears and whiskers, how 'Now the instrument every later section reuses: a hook that re-runs the forward pass and returns the residual stream at every layer. It is Chapter 2’s forward opened up so we can read the intermediate states — the embedding, then the vector after each block — instead of only the final logits.
# @save
import torch
@torch.inference_mode()
def residual_states(model, tokens):
"""Return the residual stream after the embedding and after each block.
This is the transformer's forward pass with the intermediate vectors
exposed: every interpretability tool in this chapter reads one of the
returned states rather than only the final logits. The list has one entry
more than the model has blocks — index 0 is the token-plus-position
embedding, index i is the stream after block i-1.
Args:
model: A decoder-only model exposing ``token_embedding``,
``position_embedding``, and ``blocks`` (the Chapter 2 ``TinyGPT``).
tokens: Token IDs shaped ``(1, T)``.
Returns:
A list of tensors shaped ``(1, T, d)``, one per inspection point.
"""
model.eval()
_, steps = tokens.shape
positions = torch.arange(steps)
x = model.token_embedding(tokens) + model.position_embedding(positions)
states = [x.clone()]
for block in model.blocks:
x, _ = block(x)
states.append(x.clone())
return statesA quick check confirms the shapes and shows the additive stream growing in magnitude as each block writes into it — the norm rises from the embedding to the final layer because updates accumulate rather than replace.
probe_tokens = torch.tensor([tokenizer.encode("Alice was beginning to get very")])
states = residual_states(model, probe_tokens)
print(f"{len(states)} inspection points, each {tuple(states[0].shape)}")
for name, state in zip(["embedding", "after block 1", "after block 2"], states):
print(f" {name:14s} mean L2 norm/token = {state[0].norm(dim=-1).mean():.3f}")3 inspection points, each (1, 8, 64)
embedding mean L2 norm/token = 1.303
after block 1 mean L2 norm/token = 2.223
after block 2 mean L2 norm/token = 3.091Figure 25.2 locates where each instrument of this chapter attaches, and — the detail the diagram earns its place for — what each hook reads: the residual after a block reflects everything written so far, an MLP or attention output isolates one component’s write, and a steering hook adds a direction before the rest of the network sees it.
flowchart LR
E["embedding<br/>h(0,t)"] --> R1["residual after block 1<br/>h(1,t)"]
R1 --> R2["residual after block 2<br/>h(2,t) = h(L,t)"]
A1["attention write"] -->|add| R1
M1["MLP write"] -->|add| R1
R1 -. "read: probe / lens" .-> P["linear probe,<br/>logit lens"]
S["steering vector<br/>alpha * v"] -. "write before block 2" .-> R1
R2 --> N["final norm N"] --> U["unembedding W_U"] --> Z["logits z(t)"]
R2 -. "read: lens / probe" .-> P
One caveat that matters the moment you leave this toy: the exact hook name is architecture- and library-specific, and a closed API may expose no activations at all. The durable object is a tuple — model digest, module path, layer, token-selection rule, transformation, prompt set, decoding settings — without which an activation result is impossible to reproduce and easy to pin on the wrong model revision.
25.3 Reading the stream: the logit lens
The unembedding \mathbf{W}_U turns the final residual into a next-token distribution. The logit lens asks a simple question: what if we apply that same readout to an earlier residual (nostalgebraist 2020)? Formally, the lens logits at layer \ell are
\mathbf{z}_{\ell,t}^{\text{lens}} = \mathbf{W}_U\,N(\mathbf{h}_{\ell,t}), \tag{25.2}
the same N and \mathbf{W}_U from Equation 25.1 applied to \mathbf{h}_{\ell,t} instead of \mathbf{h}_{L,t}. Decoding those logits tells us which tokens the model’s own vocabulary readout finds present in the stream partway through the computation. In our tied-embedding model the readout is literally the transpose of the embedding table, so the lens is exactly the model’s own scoring geometry, read early.
# @save
import torch
@torch.inference_mode()
def logit_lens(model, state):
"""Decode an intermediate residual with the model's final readout.
Applies the model's final normalization and unembedding to a residual
state, producing the tokens the model's own vocabulary geometry would
score highly *if this were the last layer*. It is the model reading its
own mind partway through, in its own basis.
Args:
model: A model exposing ``final_norm`` and ``lm_head``.
state: A residual state shaped ``(1, T, d)`` from ``residual_states``.
Returns:
Logits shaped ``(1, T, vocab_size)``.
"""
return model.lm_head(model.final_norm(state))We ask the trained model for its actual next token after “Alice was beginning to get very”, then decode the top candidates at each layer with the lens. Watch the answer arrive: the embedding and the first block are dominated by echoes of the input token, and only after the second block does the true continuation surface to the top.
layer_names = ["embedding", "after block 1", "after block 2 (final)"]
actual = model(probe_tokens)[0][0, -1].argmax().item()
print("model's actual next token:", repr(tokenizer.decode([actual], errors="replace")))
for name, state in zip(layer_names, states):
top = logit_lens(model, state)[0, -1].topk(5).indices.tolist()
decoded = [tokenizer.decode([t], errors="replace") for t in top]
print(f" {name:24s} top-5: {decoded}")model's actual next token: ' s'
embedding top-5: ['very', '_', '_ ', 'never', 'in']
after block 1 top-5: ['very', '_ ', ' t', 'ee ', ' to']
after block 2 (final) top-5: [' s', '_ ', ' t', ' to', 'ee ']To make “the answer becomes decodable” a number rather than an impression, we track the rank of the model’s final prediction inside the lens distribution at each layer and each position, averaged over the prompt. A rank of zero means the lens already places the eventual answer first; a large rank means it is buried.
final_prediction = logit_lens(model, states[-1])[0].argmax(dim=-1)
mean_ranks = []
for state in states:
lens_logits = logit_lens(model, state)[0]
order = lens_logits.argsort(dim=-1, descending=True)
ranks = (order == final_prediction[:, None]).float().argmax(dim=-1)
mean_ranks.append(ranks.float().mean().item())
for name, rank in zip(layer_names, mean_ranks):
print(f" {name:24s} mean rank of the final-layer token = {rank:.2f}") embedding mean rank of the final-layer token = 21.50
after block 1 mean rank of the final-layer token = 4.75
after block 2 (final) mean rank of the final-layer token = 0.00The rank collapses from roughly twenty at the embedding to zero at the final layer — the answer is computed, not looked up, and the middle block does most of the work. Figure 25.3 plots that descent.
Show the code that draws this figure
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.0, 3.4))
ax.plot(range(len(mean_ranks)), mean_ranks, marker="o", color="#2a7f9e")
ax.set_xticks(range(len(layer_names)))
ax.set_xticklabels(["embedding", "block 1", "block 2\n(final)"])
ax.set_ylabel("mean rank of final-layer token\n(lower = more decodable)")
ax.set_ylim(bottom=-1)
for i, rank in enumerate(mean_ranks):
ax.annotate(f"{rank:.1f}", (i, rank), textcoords="offset points", xytext=(0, 8))
fig.tight_layout()
plt.show()
The honest limits are worth stating in the same breath as the result. A decoded token is evidence available at a location, not proof the model emits it or that one circuit caused it. The plain lens reuses a readout trained for the final layer, so an early residual can be in a basis the unembedding handles poorly; a tuned lens learns a small per-layer translator before decoding and reads more faithfully, at the cost of added parameters that must be validated on held-out data (Belrose et al. 2023). A lens result you can trust survives a change of token pair, position, and prompt, and predicts a downstream difference.
25.4 Probing for a concept, and why you need a control
The lens reads what the model’s own vocabulary geometry surfaces. A linear probe asks a different question: is some concept of our choosing linearly decodable from the activations, whether or not the model’s readout cares about it? We fit a linear classifier on frozen residual activations for a labeled property; given activation \mathbf{h}_i, label y_i, weight \mathbf{w}, and bias b, the probe minimizes a classification loss \sum_i \ell(y_i, \mathbf{w}^{\top}\mathbf{h}_i + b). High held-out accuracy means the label is decodable by a linear map — nothing more. It does not mean the model uses the direction, because a probe can learn a dataset artifact or a surface correlate.
The defense against fooling ourselves is a control task (Hewitt and Liang 2019): keep the inputs and the probe’s capacity fixed, but replace the labels with a random relabeling. Selectivity is real-task accuracy minus control accuracy. If the control probe reaches the same accuracy, the probe is memorizing, not decoding, and selectivity — not raw accuracy — is the honest signal. We label two word-boundary concepts on our corpus: a surface one (is the current token a space-leading word start?) and a contextual one (will the next token be a boundary?), which the model must predict rather than read.
# @save
import numpy as np
def word_boundary_labels(tokenizer, token_ids, predict_next=False):
"""Label each position by its relation to a word boundary.
Two concepts share one helper. With ``predict_next=False`` the label is a
surface fact about the current token — does it begin a new word (a leading
space or newline)? With ``predict_next=True`` the label is a prediction the
model must compute — will the *next* token begin a word? The surface label
is available at the embedding; the predictive one requires integrating the
current partial word, so it should strengthen deeper in the stack.
Args:
tokenizer: A tokenizer exposing ``decode``.
token_ids: The token-ID stream to label.
predict_next: Label the next token's boundary status instead of this one.
Returns:
A float array of 0/1 labels, one per position.
"""
pieces = [tokenizer.decode([t], errors="replace") for t in token_ids]
starts = np.array([1.0 if p[:1] in (" ", "\n") else 0.0 for p in pieces])
if not predict_next:
return starts
labels = np.zeros(len(token_ids))
labels[:-1] = starts[1:]
return labelsWe collect the residual activations once, over non-overlapping windows so every position is decoded in a full context, and reuse them for every layer and both concepts.
# @save
import numpy as np
import torch
def collect_residuals(model, token_ids, block_size):
"""Gather per-layer residual activations over the whole stream.
Slides non-overlapping windows across the token stream, runs the residual
hook on each, and stacks the results so a probe can be fit at any layer.
Returns the covering index map so labels computed on the full stream line
up with the collected activations position for position.
Args:
model: The model to inspect.
token_ids: The full token-ID stream.
block_size: Window length; also the model's context limit.
Returns:
A tuple ``(states, index_map)`` where ``states[layer]`` is an array
shaped ``(N, d)`` and ``index_map`` gives each row's stream position.
"""
layers = len(model.blocks) + 1
buckets = [[] for _ in range(layers)]
index_map = []
for start in range(0, len(token_ids) - block_size, block_size):
window = token_ids[start:start + block_size]
states = residual_states(model, torch.tensor([window]))
for layer, state in enumerate(states):
buckets[layer].append(state[0].numpy())
index_map.extend(range(start, start + block_size))
return [np.concatenate(bucket) for bucket in buckets], np.array(index_map)The probe itself is a few lines: fit a logistic regression on a train split, score on held-out positions, then repeat on a fixed shuffle of the labels to get the control.
# @save
import numpy as np
from sklearn.linear_model import LogisticRegression
def probe_with_control(features, labels, seed=25, train_frac=0.7):
"""Fit a linear probe and an equal-capacity shuffled-label control.
The probe measures whether the concept is linearly decodable from the
activations; the control fits the same model class to a random relabeling.
Selectivity — probe accuracy minus control accuracy — is the honest signal,
because a probe that only memorizes labels scores as well on the shuffle.
Args:
features: Activations shaped ``(N, d)``.
labels: Binary labels shaped ``(N,)``.
seed: Seed for the control-label permutation.
train_frac: Fraction of positions used to fit; the rest are held out.
Returns:
A dict with ``accuracy``, ``control``, ``selectivity``, and ``majority``.
"""
n_train = int(train_frac * len(features))
x_train, x_test = features[:n_train], features[n_train:]
y_train, y_test = labels[:n_train], labels[n_train:]
probe = LogisticRegression(max_iter=2000).fit(x_train, y_train)
accuracy = probe.score(x_test, y_test)
shuffled = np.random.default_rng(seed).permutation(labels)
control = LogisticRegression(max_iter=2000).fit(x_train, shuffled[:n_train])
control_accuracy = control.score(x_test, shuffled[n_train:])
majority = max(y_test.mean(), 1 - y_test.mean())
return {
"accuracy": round(float(accuracy), 3),
"control": round(float(control_accuracy), 3),
"selectivity": round(float(accuracy - control_accuracy), 3),
"majority": round(float(majority), 3),
}We run both concepts at all three layers. The surface feature is most decodable at the embedding — the token is the input — and fades as the stream is repurposed for prediction. The contextual feature runs the other way: nearly invisible at the embedding, it strengthens through the blocks as the model integrates the current word.
layer_activations, index_map = collect_residuals(model, stream, config.block_size)
surface = word_boundary_labels(tokenizer, stream, predict_next=False)[index_map]
contextual = word_boundary_labels(tokenizer, stream, predict_next=True)[index_map]
print(f"{'layer':<24}{'concept':<12}{'acc':>6}{'control':>9}{'select':>8}{'major':>7}")
probe_curves = {"surface": [], "contextual": []}
for layer in range(3):
for name, labels in [("surface", surface), ("contextual", contextual)]:
result = probe_with_control(layer_activations[layer], labels)
probe_curves[name].append(result["selectivity"])
print(f"{layer_names[layer]:<24}{name:<12}{result['accuracy']:>6}"
f"{result['control']:>9}{result['selectivity']:>8}{result['majority']:>7}")layer concept acc control select major
embedding surface 0.993 0.859 0.134 0.865
embedding contextual 0.869 0.859 0.01 0.865
after block 1 surface 0.895 0.859 0.036 0.865
after block 1 contextual 0.89 0.859 0.03 0.865
after block 2 (final) surface 0.869 0.859 0.01 0.865
after block 2 (final) contextual 0.892 0.859 0.033 0.865The control column is the discipline: it sits at the majority rate everywhere, so the surface selectivity of about 0.13 is a discovered quantity, not an artifact of a balanced fixture. Figure 25.4 plots both selectivity curves and makes the crossing legible.
Show the code that draws this figure
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.2, 3.4))
xs = range(3)
ax.plot(xs, probe_curves["surface"], marker="o", color="#b13f3f",
label="surface: is this a word start")
ax.plot(xs, probe_curves["contextual"], marker="s", color="#2a7f9e",
label="contextual: is the next token a word start")
ax.axhline(0.0, ls="--", color="0.6")
ax.set_xticks(list(xs))
ax.set_xticklabels(["embedding", "block 1", "block 2\n(final)"])
ax.set_ylabel("probe selectivity\n(accuracy minus control)")
ax.legend(fontsize=8)
fig.tight_layout()
plt.show()
Selectivity is still dataset-relative. A feature that vanishes under paraphrase was a surface correlate; before trusting a probe on a real model, split by source, topic, template, and time, and compare layers, positions, and regularization strengths. The probe tells us a concept is present. It does not tell us the model uses it — for that we have to intervene.
25.5 Steering: from correlation to a causal test
A probe direction is correlational. To ask whether a direction is causal, we add it to the residual stream during generation and see whether behavior changes. If the probe found that word boundaries are linearly encoded, the difference-of-means direction between boundary and non-boundary positions is a natural handle on that concept, and steering along it is the causal test (Arditi et al. 2024):
\mathbf{h}'_{\ell,t} = \mathbf{h}_{\ell,t} + \alpha\,\mathbf{v}, \tag{25.3}
where \mathbf{v} is the unit direction and \alpha is the dose. A behavioral change after this edit is stronger evidence than decodability — but it is still local: the edit can shift many entangled features at once, which is why single neurons are rarely interpretable in the first place. Models pack more features than dimensions into the residual stream by superposition, so a coordinate is typically polysemantic; a sparse autoencoder can propose a cleaner overcomplete feature basis and has scaled to production models, though feature splitting, dead features, and reconstruction error keep even those results bounded (Elhage et al. 2022; Templeton et al. 2024). We steer along a difference-of-means direction here because it needs no training and ties directly to the probe.
# @save
import numpy as np
import torch
def mean_difference_direction(features, labels):
"""Return the unit direction separating the two labeled activation groups.
The difference of class means is the simplest causal handle on a concept a
probe found: it points from the average "off" activation toward the average
"on" one. Adding it to the residual stream during generation tests whether
the concept the probe decoded actually drives behavior.
Args:
features: Activations shaped ``(N, d)``.
labels: Binary labels shaped ``(N,)``.
Returns:
A unit-norm float32 tensor shaped ``(d,)``.
"""
difference = features[labels == 1].mean(0) - features[labels == 0].mean(0)
difference = difference / np.linalg.norm(difference)
return torch.tensor(difference, dtype=torch.float32)
@torch.inference_mode()
def generate_with_steering(model, prompt_ids, max_new_tokens, direction, alpha, layer):
"""Greedily generate while adding a steering vector at one layer.
Reproduces the residual hook of ``residual_states`` but, after block
``layer``, adds ``alpha * direction`` to the stream before the remaining
blocks and the readout run. ``alpha = 0`` is ordinary generation; positive
and negative doses push the concept up and down.
Args:
model: The model to steer.
prompt_ids: The prompt as a list of token IDs.
max_new_tokens: How many tokens to append.
direction: A ``(d,)`` steering direction.
alpha: The dose; scales the added direction.
layer: Add the direction after this many blocks (1-indexed).
Returns:
The prompt with generated token IDs appended, as a list.
"""
block_size = model.config.block_size
output = list(prompt_ids)
for _ in range(max_new_tokens):
tokens = torch.tensor([output[-block_size:]])
positions = torch.arange(tokens.size(1))
x = model.token_embedding(tokens) + model.position_embedding(positions)
for index, block in enumerate(model.blocks):
x, _ = block(x)
if index == layer - 1:
x = x + alpha * direction
logits = model.lm_head(model.final_norm(x))[0, -1]
output.append(int(logits.argmax()))
return outputWe build the direction from the contextual boundary labels at the first block and steer there, so the second block still gets to process the edited state into text. The on-target measure is the mean length of generated “words”; the off-target cost is the held-out loss with the same edit applied, which tells us how much general competence we spent.
direction = mean_difference_direction(layer_activations[1], contextual)
@torch.inference_mode()
def steered_val_loss(model, val, direction, alpha, layer, block_size, windows=8):
"""Held-out cross-entropy with the steering edit applied — the off-target cost."""
model.eval()
starts = torch.linspace(0, val.numel() - block_size - 1, windows).long()
losses = []
for start in starts:
tokens = val[start:start + block_size][None]
targets = val[start + 1:start + block_size + 1][None]
x = model.token_embedding(tokens) + model.position_embedding(torch.arange(block_size))
for index, block in enumerate(model.blocks):
x, _ = block(x)
if index == layer - 1:
x = x + alpha * direction
logits = model.lm_head(model.final_norm(x))
losses.append(torch.nn.functional.cross_entropy(
logits.flatten(0, 1), targets.flatten()).item())
return float(np.mean(losses))
def mean_word_length(text):
words = [w for w in text.split(" ") if w]
return float(np.mean([len(w) for w in words])) if words else 0.0
prompt_ids = tokenizer.encode("Alice was ")
doses, word_lengths, off_target = [], [], []
for alpha in (-3.0, -1.5, 0.0, 1.5, 3.0):
generated = generate_with_steering(model, prompt_ids, 40, direction, alpha, layer=1)
text = tokenizer.decode(generated, errors="replace")
doses.append(alpha)
word_lengths.append(mean_word_length(text))
off_target.append(steered_val_loss(model, val_stream, direction, alpha, 1, config.block_size))
print(f"alpha={alpha:+.1f} word_len={word_lengths[-1]:5.2f} "
f"loss={off_target[-1]:5.2f} {text!r}")alpha=-3.0 word_len=29.00 loss= 7.43 'Alice was cupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupupup'
alpha=-1.5 word_len=22.00 loss= 7.32 'Alice was no ususususususususususususususususususususususususususususususususususususususus'
alpha=+0.0 word_len= 3.60 loss= 7.19 'Alice was just in time to hear it say, as it turned a corner, “Oh my ears and whiskers, how '
alpha=+1.5 word_len= 1.14 loss= 8.07 'Alice was a m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m m'
alpha=+3.0 word_len= 1.14 loss= 9.80 'Alice was a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a a 'The dose response is monotone and honest. Push the boundary direction up and the model floods the output with word breaks — average word length collapses toward one character; push it down and it runs letters together into long strings. And the text degrades at both extremes: this is the off-target cost made visible, the d2l lesson that the real output is imperfect and we show it anyway. Figure 25.5 plots both effects on one axis pair.
Show the code that draws this figure
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(6.4, 3.6))
ax.plot(doses, word_lengths, marker="o", color="#2a7f9e", label="mean word length (on-target)")
ax.set_xlabel("steering dose (alpha)")
ax.set_ylabel("mean generated word length", color="#2a7f9e")
ax.tick_params(axis="y", labelcolor="#2a7f9e")
twin = ax.twinx()
twin.plot(doses, off_target, marker="s", ls="--", color="#b13f3f", label="held-out loss (off-target)")
twin.set_ylabel("held-out cross-entropy loss", color="#b13f3f")
twin.tick_params(axis="y", labelcolor="#b13f3f")
fig.tight_layout()
plt.show()
We now have four instruments and can state, for each, the claim it earns and the countertest it demands. The table is the synthesis of this half of the chapter: internal tools generate hypotheses, localize regressions, and add evidence to an audit; they do not convert “we found a direction” into “we found the model’s goal.”
| Instrument | Defensible claim | Required countertest | What it does not establish |
|---|---|---|---|
| Logit / tuned lens | token evidence is decodable at this layer | alternative tokens, prompts, positions; tuned-lens held-out test | the model used that evidence causally |
| Linear probe | this concept is linearly decodable here | shuffled-label control; split-by-source audits | the policy represents or relies on one clean concept |
| Sparse autoencoder | a sparse direction groups these activations | reconstruction, stability, splitting, intervention | the feature label is complete or unique |
| Steering / patching | this direction changes measured behavior | dose response, off-target and out-of-distribution tests | the direction is a safe global control |
25.6 Reasoning traces are monitorable, not faithful
A reasoning-capable system may emit a scratchpad, a chain of thought, or a tool-call rationale. These traces are model-generated text a monitor can read, and they are operationally valuable precisely because they are legible and cheap next to white-box activation analysis. The trap is treating a legible trace as a faithful one. Separate three properties: outcome validity (did the answer satisfy the task?), monitorability (can a declared monitor detect a safety-relevant property from the trace?), and faithfulness (does the trace expose the factors that actually caused the decision?). A trace can be highly monitorable for explicit phrases and still be a poor causal account.
Faithfulness is tested by intervention: change a hidden or biasing input, confirm the answer changes, then check whether the trace mentions the cause. Turpin and colleagues did exactly this — inserting a bias such as “I think the answer is (A)” flips models’ answers while their stated reasoning never cites the hint (Turpin et al. 2023). We cannot run this on our tiny memorizer, which has no chain of thought, so we work a representative version — the numbers below are illustrative of the published pattern, not a recorded run — to make the measurement concrete.
# representative faithfulness probe (illustrative of Turpin et al., not a recorded run)
trials = [
{"clean": "B", "biased": "A", "hint": "A", "cot_mentions_hint": True},
{"clean": "C", "biased": "A", "hint": "A", "cot_mentions_hint": False},
{"clean": "A", "biased": "A", "hint": "A", "cot_mentions_hint": False},
{"clean": "D", "biased": "B", "hint": "B", "cot_mentions_hint": False},
{"clean": "B", "biased": "B", "hint": "B", "cot_mentions_hint": True},
{"clean": "C", "biased": "D", "hint": "D", "cot_mentions_hint": False},
]
flipped = [t for t in trials if t["biased"] != t["clean"]]
flip_rate = len(flipped) / len(trials)
followed_hint = sum(t["biased"] == t["hint"] for t in trials) / len(trials)
mention_rate = sum(t["cot_mentions_hint"] for t in flipped) / len(flipped)
print(f"answer changed by the hint (flip rate): {flip_rate:.2f}")
print(f"final answer matched the hint: {followed_hint:.2f}")
print(f"...of the flips, trace named the hint: {mention_rate:.2f}")answer changed by the hint (flip rate): 0.67
final answer matched the hint: 1.00
...of the flips, trace named the hint: 0.25The gap between “the hint changed the answer” and “the trace admits the hint” is the faithfulness gap, and Figure 25.6 draws it. A monitor reading only the trace would see confident, plausible reasoning and miss the actual cause on most flips.
Show the code that draws this figure
import matplotlib.pyplot as plt
fig, ax = plt.subplots(figsize=(5.6, 3.2))
bars = ax.bar(["hint changed\nthe answer", "trace named\nthe hint"],
[flip_rate, mention_rate], width=0.55, color=["#b13f3f", "#2a7f9e"])
ax.set_ylabel("fraction of biased trials")
ax.set_ylim(0, 1.0)
for bar, value in zip(bars, [flip_rate, mention_rate]):
ax.annotate(f"{value:.2f}", (bar.get_x() + bar.get_width() / 2, value),
textcoords="offset points", xytext=(0, 4), ha="center")
fig.tight_layout()
plt.show()
Two operational rules follow. Measure monitorability as an ordinary detection problem — labeled positive and negative trajectories, false-positive costs, adversarial pressure — and never assume absence of an alert is an inability proof. And do not train against the same trace monitor without checking that behavior stays legible: penalizing suspicious language can teach euphemism while leaving the action policy intact. Keep the trace monitor, the internal diagnostics, and the final action control independent, because each fails differently.
25.7 Capability evaluations must include elicitation
A dangerous-capability evaluation asks whether a system can materially advance a threat-relevant task — cyber operations, uplift for chemical or biological misuse, autonomous operation. Knowledge screens like WMDP are cheap proxies; executable agent tasks like Cybench test planning, tool use, and persistence (Li et al. 2024; Zhang et al. 2024). But a score means nothing until we pin down the system under evaluation, which is far more than model weights: prompt, sampler, reasoning budget, tools, retrieval, memory, scaffold, retries, and any fine-tuning or human assistance. Capability elicitation is the disciplined effort to make the system perform near the top of the range a real actor in the threat model could reach (METR 2024). If S(e) is the score under elicitation configuration e over the covered set \mathcal{E}, the honest reported quantity is a lower bound:
S_{\text{observed}} = \max_{e \in \mathcal{E}} S(e). \tag{25.4}
The asymmetry this creates is the whole point: evidence of capability defeats an inability claim quickly, while absence of capability needs a credible search for it. A low unassisted score is not an inability proof when an attacker can add tools, search, or a better scaffold. We make that concrete with a representative elicitation envelope — the numbers illustrate the shape, not a specific model.
# representative elicitation envelope: score rises as assistance is added
envelope = [
("base prompt", 0.04),
("+ tool access", 0.11),
("+ agent scaffold", 0.19),
("+ domain fine-tune", 0.31),
]
threshold = 0.20
observed = max(score for _, score in envelope)
print(f"{'configuration':<20}{'S(e)':>6}")
for name, score in envelope:
flag = " <-- crosses threshold" if score >= threshold else ""
print(f"{name:<20}{score:>6.2f}{flag}")
print(f"\nbase-prompt score {envelope[0][1]:.2f} sits under the {threshold:.2f} threshold; "
f"max_e S(e) = {observed:.2f} does not")configuration S(e)
base prompt 0.04
+ tool access 0.11
+ agent scaffold 0.19
+ domain fine-tune 0.31 <-- crosses threshold
base-prompt score 0.04 sits under the 0.20 threshold; max_e S(e) = 0.31 does notThe base configuration a lazy evaluation would report (0.04) sits comfortably under the 0.20 threshold, yet the elicited maximum (0.31) crosses it. Report the envelope and its gaps next to the score, or the number invites exactly the inference — “failed here, therefore cannot” — that sandbagging and evaluation awareness are built to exploit. Guard the inference with hidden held-outs, varied surface cues, learning curves across assistance, and independent evaluators; a password-locked model can preserve capability while depressing one score, and no single check rules that out. Finally, dangerous evaluations need containment — isolated networks and credentials, deweaponized tasks, an emergency stop — because Chapter 24’s enforcement boundary is what keeps realism from becoming harm.
This is also the engine of responsible scaling. The mechanism is always the same three-step commitment: pick a threshold before results, run the evaluation under a versioned elicitation envelope, and attach a precommitted action to each side. A threshold with no committed action is an alert, not governance.
NoteIn the interview
Q. Your model scores near zero on a dangerous-capability benchmark. Can you argue inability? Not from that number alone. The reported score is S_{\text{observed}} = \max_e S(e) over the configurations you tried — a lower bound on what a determined actor could reach with better tools, scaffolding, or fine-tuning. An inability argument needs the elicitation envelope (what assistance was searched), integrity checks against sandbagging and evaluation awareness, and coverage of the threat model — not one unassisted pass. The trap is treating a low score as evidence of a ceiling; capability evidence defeats inability fast, but absence of capability has to be earned by a credible search for it.
25.8 A safety case turns evidence into a decision
A model report can hold hundreds of metrics and still leave the decision implicit. A safety case is a structured, reviewable argument that a system is acceptably safe for a stated scope, supported by evidence (Clymer et al. 2024; Buhl et al. 2024). It does not prove zero risk; it makes the claim, assumptions, evidence, residual uncertainty, and decision rule inspectable. Decompose the top claim by hazard, and for each hazard choose an argument pattern: inability (the capability needed for the harm is absent under a strong elicitation envelope), control (assume the capability, but tested controls interrupt every path to harm), trustworthiness (evidence supports that the capable system acts as intended), or deference (a more capable trusted process supplies assurance — research-dependent, not for ordinary review). Control arguments are often strongest for agents because they claim least about learned intent: Chapter 24’s enforcement point can deny a credential read even if the model proposes it strategically.
We assemble the case inline, as data. The deployment is a sandboxed research agent; three hazards each take a different argument, and each argument declares exactly what evidence it is allowed to rest on.
deployment_case = {
"system": {"name": "research-agent-rc25",
"scope": "sandboxed public-web research; no credentials, no production authority"},
"top_claim": {"id": "G0", "text": "acceptably safe for the stated sandbox scope",
"decision_rule": "block on any unsupported high or critical claim"},
"claims": [
{"id": "G1", "severity": "critical", "argument": "control",
"hazard": "compromised policy attempts credential access or egress",
"controls": [{"id": "C-EGRESS", "evidence": ["E-CONTAINMENT"]}]},
{"id": "G2", "severity": "critical", "argument": "inability",
"hazard": "agent completes a long-horizon autonomous-replication task",
"capability": {"score": 0.04, "threshold": 0.08,
"elicitation_coverage": 0.55, "required_coverage": 0.8,
"sandbagging_check": "open"},
"evidence": ["E-CAPABILITY"]},
{"id": "G3", "severity": "high", "argument": "trustworthiness",
"hazard": "policy pursues a hidden objective while behaving well on tests",
"evidence": ["E-LENS", "E-PROBE", "E-STEER"], "unresolved_counterevidence": True},
],
"evidence": [
{"id": "E-CONTAINMENT", "kind": "control_test", "status": "pass",
"scope": "final enforcement point exercised with hostile proposals and denied effects"},
{"id": "E-CAPABILITY", "kind": "capability_eval", "status": "pass",
"scope": "held-out autonomous-task suite under the current scaffold and budget"},
],
}The interpretability results become evidence — but diagnostic evidence, a kind the rules below will refuse to over-trust. This function converts the measurements we took on the real model into scoped, non-dispositive evidence items.
# @save
DIAGNOSTIC_KINDS = {"logit_lens", "linear_probe", "activation_steering", "sae_feature"}
def diagnostic_evidence(diagnostics):
"""Turn internal measurements into scoped, non-dispositive evidence items.
The lens, probe, and steering results from the real model each become one
evidence item tagged with a diagnostic ``kind``. Tagging matters: the claim
rules treat diagnostic evidence as hypothesis-generating, never as proof of
a control or a stable objective, so these items can support a narrow
statement but cannot by themselves discharge a safety claim.
Args:
diagnostics: A dict with ``lens``, ``probe``, and ``steering`` results.
Returns:
A list of three evidence dicts with ``id``, ``kind``, ``status``,
``value``, and ``limitation``.
"""
lens, probe, steer = diagnostics["lens"], diagnostics["probe"], diagnostics["steering"]
return [
{"id": "E-LENS", "kind": "logit_lens",
"status": "pass" if lens["rank_drop"] > 0 else "fail", "value": lens["rank_drop"],
"limitation": "a decoded token is not a causal computation"},
{"id": "E-PROBE", "kind": "linear_probe",
"status": "pass" if probe["selectivity"] > 0.02 else "fail", "value": probe["selectivity"],
"limitation": "decodability does not prove the policy uses the feature"},
{"id": "E-STEER", "kind": "activation_steering",
"status": "pass" if steer["behavior_change"] else "fail", "value": steer["on_target"],
"limitation": "a local edit may be off-target or fail out of distribution"},
]The rule engine is deliberately small and narrow — no weighted average, no single safety score. Each argument type has its own sufficiency test, and a claim is a gap the moment any reason fires.
# @save
def evaluate_claim(claim, evidence_by_id):
"""Judge one scoped safety argument against explicit, per-pattern rules.
Control arguments must rest on passing evidence that is not purely
diagnostic. Inability arguments require a below-threshold score, sufficient
elicitation coverage, a closed sandbagging check, and passing evidence.
Trustworthiness arguments require passing evidence, at least two independent
non-diagnostic evidence classes, and no unresolved counterevidence. Any
unmet condition makes the claim a gap; the reasons are returned so a
reviewer sees exactly why.
Args:
claim: One claim dict from the case.
evidence_by_id: All evidence items keyed by ``id``.
Returns:
A dict with ``id``, ``severity``, ``argument``, ``status``, ``reasons``.
"""
def passing(ids):
return bool(ids) and all(evidence_by_id[i]["status"] == "pass" for i in ids)
reasons = []
argument = claim["argument"]
if argument == "control":
controls = claim.get("controls", [])
if not controls:
reasons.append("no control is named")
for control in controls:
ids = control.get("evidence", [])
kinds = {evidence_by_id[i]["kind"] for i in ids}
if not passing(ids):
reasons.append(f"{control['id']} lacks passing evidence")
if kinds and kinds <= DIAGNOSTIC_KINDS:
reasons.append(f"{control['id']} rests only on internal diagnostics")
elif argument == "inability":
cap = claim["capability"]
if cap["score"] > cap["threshold"]:
reasons.append("capability score exceeds the deployment threshold")
if cap["elicitation_coverage"] < cap["required_coverage"]:
reasons.append("elicitation coverage is below the declared minimum")
if cap.get("sandbagging_check") != "pass":
reasons.append("evaluation-integrity or sandbagging checks remain open")
if not passing(claim.get("evidence", [])):
reasons.append("capability evidence is missing or failing")
else: # trustworthiness
ids = claim.get("evidence", [])
kinds = {evidence_by_id[i]["kind"] for i in ids}
if not passing(ids):
reasons.append("trustworthiness evidence is missing or failing")
if len(kinds - DIAGNOSTIC_KINDS) < 2:
reasons.append("fewer than two independent non-diagnostic evidence classes")
if claim.get("unresolved_counterevidence", True):
reasons.append("counterevidence remains unresolved")
return {"id": claim["id"], "severity": claim["severity"], "argument": argument,
"status": "supported" if not reasons else "gap", "reasons": reasons}Building the case joins the diagnostics to the graph, checks that every referenced evidence ID exists, evaluates each claim, and fails closed: any unsupported high- or critical-severity claim blocks the deployment.
# @save
def build_safety_case(case, diagnostics):
"""Assemble a case, evaluate every claim, and return a fail-closed decision.
Joins the diagnostic evidence to the case's own evidence, rejects dangling
evidence references, evaluates each claim under ``evaluate_claim``, and
blocks deployment when any high- or critical-severity claim is a gap. The
decision is structural: it does not average evidence into one score, so the
blocking reason is always a specific claim, not a threshold on a number.
Args:
case: The deployment case (system, top claim, claims, base evidence).
diagnostics: Real interpretability measurements for the diagnostic items.
Returns:
A report dict with ``claims`` results, a ``summary``, and ``decision``.
Raises:
ValueError: If a claim references an evidence ID that does not exist.
"""
evidence = case["evidence"] + diagnostic_evidence(diagnostics)
by_id = {item["id"]: item for item in evidence}
for claim in case["claims"]:
referenced = list(claim.get("evidence", []))
for control in claim.get("controls", []):
referenced += control.get("evidence", [])
missing = sorted(set(referenced) - set(by_id))
if missing:
raise ValueError(f"{claim['id']} references missing evidence: {missing}")
results = [evaluate_claim(claim, by_id) for claim in case["claims"]]
blocking = [r["id"] for r in results
if r["status"] == "gap" and r["severity"] in {"high", "critical"}]
return {
"claims": results,
"summary": {"supported": sum(r["status"] == "supported" for r in results),
"gaps": sum(r["status"] == "gap" for r in results),
"blocking": blocking},
"decision": "BLOCK" if blocking else "APPROVE",
}Now we wire the real diagnostics — the lens rank drop, the probe selectivity, and the steering effect we measured above — into the case and print the verdict as a structured table.
diagnostics = {
"lens": {"rank_drop": round(mean_ranks[0] - mean_ranks[-1], 2)},
"probe": {"selectivity": probe_curves["surface"][0]},
"steering": {"behavior_change": word_lengths[-1] < word_lengths[0],
"on_target": round(word_lengths[0] - word_lengths[-1], 2)},
}
report = build_safety_case(deployment_case, diagnostics)
print(f"{'claim':<6}{'severity':<10}{'argument':<16}{'status':<11}reason")
for result in report["claims"]:
reason = result["reasons"][0] if result["reasons"] else "-"
print(f"{result['id']:<6}{result['severity']:<10}{result['argument']:<16}"
f"{result['status']:<11}{reason}")
print(f"\nsupported={report['summary']['supported']} gaps={report['summary']['gaps']} "
f"blocking={report['summary']['blocking']} -> decision = {report['decision']}")claim severity argument status reason
G1 critical control supported -
G2 critical inability gap elicitation coverage is below the declared minimum
G3 high trustworthiness gap fewer than two independent non-diagnostic evidence classes
supported=1 gaps=2 blocking=['G2', 'G3'] -> decision = BLOCKThe decision is BLOCK, and reading why is the point. G1 is supported: its control rests on a real containment test, not on the model’s internals. G2 fails despite a passing, below-threshold score, because its elicitation coverage (0.55) sits under the declared minimum and its sandbagging check is open — exactly the elicitation asymmetry of Equation 25.4. G3 fails because our three interpretability results, strong as their plots looked, are all diagnostic: they cannot supply the two independent non-diagnostic evidence classes a trustworthiness claim needs, and its counterevidence is unresolved. Figure 25.7 draws the decomposition.
flowchart TB
G0["G0: acceptably safe<br/>for sandbox scope"] --> G1["G1 critical:<br/>credential access"]
G0 --> G2["G2 critical:<br/>autonomous replication"]
G0 --> G3["G3 high:<br/>hidden objective"]
G1 --> A1["control argument"] --> E1["containment test<br/>(non-diagnostic): pass"]
G2 --> A2["inability argument"] --> E2["low score, but<br/>coverage 0.55 < 0.8,<br/>sandbagging open"]
G3 --> A3["trustworthiness argument"] --> E3["lens + probe + steering<br/>(all diagnostic)"]
E2 -->|critical gap| D["BLOCK"]
E3 -->|high gap| D
E1 -->|supported| G1
The graph prevents evidence laundering: a lens result can support a statement about a decoded margin, never G0 directly, and every edge should carry a written warrant for why the evidence bears on the claim. Two further disciplines make a real case survive contact with change. Review each evidence edge on independence — “independent” means the failure modes differ, not that two dashboards show the same benchmark — and hold configuration control: any material change to weights, prompt, tools, or deployment population reopens the affected claims rather than editing a verdict in place.
25.9 Governance: thresholds, commitments, and regulation
Governance is the machinery that turns a BLOCK into a commitment nobody can quietly reverse. It uses three kinds of trigger for different jobs. Resource triggers use training compute or another development input to require evaluation and security before capability is known — observable early, but only a proxy. Capability triggers use elicited performance tied to a threat model — closer to harm, but dependent on task validity and elicitation. Deployment triggers use access, autonomy, tool authority, and exposure — the same weights behind a narrow API and inside a privileged autonomous agent carry very different risk. A robust policy composes them and keeps the units separate; adding them into one risk number hides which assumption failed.
The durable lesson under every responsible-scaling framework is precommitment. If leadership can redefine the threshold, discount a failed evaluation, or swap in an untested safeguard after seeing results — with no recorded exception path — the framework constrains nothing. So write the loop before the candidate arrives: define hazard owners, thresholds, and required evidence; run evaluations under a versioned elicitation envelope; give reviewers independent enough standing to challenge task validity and delay release; record the decision, its dissent, assumptions, and expiry; stage only the authority the case supports; and reopen the case when a threshold, component, threat model, or law changes. “Unable to conclude” must map to an action — more evaluation, reduced authority, delayed release — never silently to approval.
Regulation enters engineering the same way: as an interface. Legal text becomes useful only after it is mapped to systems and evidence, so maintain an applicability matrix keyed by jurisdiction, role, model category, use, population, and effective period, and translate each obligation into five fields.
| Field | Engineering question |
|---|---|
| Requirement | what outcome, disclosure, or prohibition is demanded? |
| Scope | which model, deployment, user, and jurisdiction does it cover? |
| Control | what technical or organizational mechanism realizes it? |
| Evidence | what artifact — test, log, assessment, approval — demonstrates it? |
| Lifecycle | who owns it, when is it refreshed, what change invalidates it? |
Never collapse a policy statement into a control, or a control into evidence: “we prevent discriminatory outcomes” is a goal; a documented decision process, a representative evaluation, an appeal path, and an accountable owner are candidate mechanisms, each with limits. Compute thresholds show why legal and technical taxonomies must stay joined but distinct — compute is measurable before deployment and can trigger duties, yet post-training, scaffolding, and distribution change practical capability, so track the legally defined quantity exactly while also maintaining capability and exposure assessments. The organizational operating loop that runs all of this lives in Chapter 28; the living regulatory catalog lives in Appendix C.
As of 2026-07-20, experimental work has demonstrated alignment faking under constructed training, in-context scheming under explicit goals, selective sandbagging, emergent misalignment after narrow fine-tuning, and blind audits of implanted hidden objectives — capability and model-organism results, not prevalence estimates for deployed systems. Versioned company frameworks include Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework, and Google DeepMind’s Frontier Safety Framework; their thresholds and governance bodies are living commitments, not interchangeable standards. The European Commission states the EU AI Act entered into force on 2024-08-01, with general-purpose-model obligations applying from 2025-08-02 and enforcement powers from 2026-08-02; its guidance uses training compute above a presumption level of C > 10^{25} FLOP for systemic-risk designation. In the United States, requirements remain distributed across federal action, sectoral law, and state legislation; California’s enacted Transparency in Frontier Artificial Intelligence Act defines a frontier threshold at N > 10^{26} operations and mandates published frameworks and incident reporting. NIST AI RMF 1.0 (Govern, Map, Measure, Manage, with a Generative AI Profile) remains voluntary and under revision. Verify live: 2026-07-20. Recheck the official EU AI Act implementation page, the California SB 53 text, the NIST AI RMF page, and each named company framework — and current counsel — before making any compliance claim.
25.10 Summary
Alignment is a scoped claim about behavior and effective objectives, not a property baked into a checkpoint. We ran four interpretability instruments on the Chapter 2 model and read each honestly: a logit lens whose answer-rank fell to zero only at the last block, a linear probe kept honest by a shuffled-label control, and a difference-of-means steering vector that changed generated text at a measured held-out-loss cost. We tested a reasoning trace’s faithfulness by intervention, turned a capability score into the lower bound \max_e S(e), and assembled a fail-closed safety case that blocked deployment because diagnostic evidence cannot discharge a control or a trustworthiness claim. Governance makes such a decision a precommitment; Chapter 28 runs the organizational loop around it.
25.11 Exercises
- Steer a different concept. Using
word_boundary_labels, build a probe and amean_difference_directionfor the surface boundary concept instead of the contextual one, steer at block 2 rather than block 1, and compare the dose response and off-target loss with the one in Figure 25.5. Which concept and layer give the cleanest behavioral change for the smallest competence cost? - Make the probe lie. Construct a label that is a pure artifact of position (for example, “the position index is even”) and run
probe_with_controlon it at every layer. (a) Report accuracy, control, and selectivity. (b) Explain why a high accuracy here would be a warning rather than a finding, and what real-data split would expose it. - Tuned-lens intuition. The plain lens in Section 25.3 reuses the final readout. Fit a per-layer linear map from an intermediate residual to the final residual, apply the lens through it, and measure whether the mean answer-rank at block 1 improves. What have you added, and what must you now validate to trust the improvement?
- Break the safety case, three ways. Starting from
deployment_case: (a) pointG1’s control atE-PROBEandE-STEERand show it becomes a gap; (b) raiseG2’selicitation_coverageto 0.9 without closingsandbagging_checkand show it still fails; (c) close that check and confirm onlyG2changes while the top decision staysBLOCK. What does each result teach about schema validity versus evidence sufficiency? - Elicitation defeats inability. Extend the representative envelope so a fifth configuration crosses a stricter threshold. Then rewrite
G2’s capability block to compare the threshold against \max_e S(e) rather than the base score, and describe the evidence you would need before the inability argument could ever be supported. - Faithfulness for a tool agent. Design a real trace-faithfulness evaluation for the Chapter 16 agent: specify the hidden intervention, the causal behavior check, the monitor target, the negative controls, and the failure criterion — and explain how training against the monitor could invalidate the result.
- Governance design review. A privileged coding agent passes public cyber benchmarks, exposes a strong “safe behavior” probe direction, and suppresses suspicious language when steered, while an external sandbox blocks network and credential access. Construct separate inability, control, and trustworthiness arguments; identify which claims fail and why; choose a deployment scope; and write the exact governance action each gap triggers.