Do language models actually use the program state they track internally? We can read it out of their layers, but causal tests show they ignore it: answers are recomputed from the raw input tokens, late in the network. Decodable is not the same as causally used.
This repo contains the full pipeline, frozen datasets, results, and figures for a mechanistic interpretability study of how LLMs execute tiny programs, run on Qwen/Qwen3-Coder-30B-A3B-Instruct, microsoft/phi-4, and the Qwen3 dense ladder (4B / 8B / 14B / 32B).
Models read programs like x = 4; x = x + 2; x = x + 1; print(x) and answer correctly (~85%). Inside, they maintain a textbook execution trace:
- the running value of
xis linearly decodable at the right token positions, at the right time (0.85 at the update line, vs a 0.35 baseline), - the final answer is not decodable before the computation that produces it exists (0.55 vs 0.56 baseline at line 2),
- the answer becomes decodable only after the last update line.
But activation patching shows that trace is bypassed:
| Transplanted state (clean → corrupt run) | Median rescue |
|---|---|
| Line-1 token (the raw operand) | ≈ 1.00 — answer flips, model re-executes both additions |
| Line-2 token (the decodable running value) | 0.05 – 0.12 |
| Full line-2 span (all ~7 tokens) | 0.05 – 0.24 |
| Full line-3 span (a value equal to the answer) | ≈ 0.02 |
| Answer position, final layers | ≈ 0.94 |
The causally load-bearing path runs from the raw operand tokens to a late answer-position computation (final ~20% of layers). The intermediate state the model writes down is readable, accurate, and largely unused.
The finding replicates across architectures, labs, and sizes:
| Model | Params | Type | State readable (probe) | Line-1 rescue | Line-2 rescue (state use) |
|---|---|---|---|---|---|
| Qwen3-4B | 4B | dense | 0.70 | 1.01 | 0.05 |
| Qwen3-8B | 8B | dense | 0.77 | 1.00 | 0.01 |
| Qwen3-14B | 15B | dense | 0.81 | 1.00 | 0.04 |
| phi-4 | 14B | dense (Microsoft) | 0.80 | 1.00 | 0.05 |
| Qwen3-Coder-30B-A3B | 30B | MoE | 0.85 | 1.00 | 0.12 |
| Qwen3-32B | 33B | dense | 0.85 | 1.00 | 0.11 |
Across an 8× parameter range, state readability rises steadily (0.70 → 0.85) while state use stays flat near zero: the gap between decodable and causal widens with scale rather than closing. The dense-vs-MoE matched pair (Qwen3-32B vs Qwen3-Coder-30B-A3B, same lab and generation) shows architecture makes no difference (0.11 vs 0.12). The networks also share the same relative-depth handoff geometry: program-token causal influence dies at ~60% depth, answer-position computation begins at ~80%.
A behavior ladder brackets the capability cliff: one update step → 97%, two pure additions → 85%, two steps with mod-10 and multiplication → 29% (answer-letter collapse). Single-forward-pass execution has a small, sharp budget.
- Probing alone overstates understanding. A perfect, generalizing probe result here corresponds to a representation the model demonstrably does not use. Causal intervention is not optional.
- LLMs recompute rather than reuse. Answers are rebuilt from raw inputs in a late, fixed-size computation. This is a mechanistic account of why multi-step problems fail without intermediate text.
- Text is the working memory that works. Written-out steps become input tokens, the one channel these models demonstrably consume. That is the mechanism behind chain-of-thought.
- Frozen behavior gates. No mechanistic analysis on any task a model cannot do: ≥75% overall, ≥70% on every held-out split (templates, variable names, formats), shortcut margin ≥0.50, answer-letter share ≤0.55. Two models (Mistral-Small-24B, DeepSeek-V2-Lite) were screened out by this gate and received no mechanistic claims.
- Pre-registered pair design. Clean/corrupt programs differ in exactly one literal; downstream lines are shared, so a rescued patch must be re-executed through the remaining program.
- Controls. Specificity (pre-divergence and instruction positions), random-donor span transplants, letter-vs-value probe dissociation, shuffled/random-activation probe baselines, bootstrap CIs over pairs and paired Wilcoxon tests for every headline cell.
- Dual-backend verification. The Qwen3 behavior gate was passed independently in MLX (generation) and PyTorch/Transformers (logit scoring), with identical accuracy through two scoring paths.
src/cesr/ harness: datasets, prompts, ModelRunner, probes, patching, screening
configs/ frozen experiment configs (one per model x task x variant)
data/processed/ frozen datasets and clean/corrupt pairs (560 examples / 80 pairs per task)
jobs/ Modal GPU runners (gate-enforced), span patching, analysis scripts
results/ reports (md), tables (csv), figures (png)
tests/ harness tests
execution-state-video/ Remotion explainer video source (see below)
research-plan.md the original pre-registered plan
Key entry points:
# dataset
cesr generate --config configs/qwen3_arith_two_step.yaml
cesr verify --config configs/qwen3_arith_two_step.yaml
# behavior gate (GPU)
cesr screen-behavior --config configs/arith_two_step_screen_qwen3.yaml
# mechanistic phase (GPU)
cesr collect-activations --config configs/qwen3_arith_two_step.yaml
cesr probes --config configs/qwen3_arith_two_step.yaml # CPU
cesr patch --config configs/qwen3_arith_two_step.yaml
python jobs/qwen3_span_patch.py --model-id <model> # span transplantsAll GPU work runs on Modal A100-80GB with weights cached in a volume and results committed after every step. The launchers are gate-enforced: mechanistic steps execute only if the frozen behavior screen passes in-container.
modal run --detach jobs/modal_qwen3_two_step.py # gate -> activations -> patch -> control -> span
modal run --detach jobs/modal_phi4_replication.py # full replication recipe
modal run --detach jobs/modal_scale_autopilot.py # screen the dense ladder, then mech per passerAnalysis runs locally from the synced CSVs: jobs/stats_qwen3.py (bootstrap CIs, paired tests), jobs/figures_qwen3.py (heatmaps), jobs/analyze_scaling.py (rescue vs model size).
execution-state-video/ is a Remotion project that renders the whole study as an animated isometric view of the model's layers, with every glowing cell driven by the real effect sizes:
cd execution-state-video && npm install && npx remotion render MechViz out/mech-viz.mp4The scaling ladder is complete (six models, mechanistic phase on every gate-passer). Deferred follow-ups (direct-integer answer format, quantization-robustness chapter) are documented in results/deferred_roadmap.md with rationale and cost estimates.
MIT