Git bisect for neural-network learning: trace a model behavior from output → checkpoint → internal mechanism → training data → counterfactual causal verification.
A model didn't know something at checkpoint 10,000. At checkpoint 12,000 it does. When did that change, what changed inside the network, which training samples caused it — and if you remove those samples and retrain, does the behavior go away?
NeuralBisect answers that chain, and — the part that matters — it refuses to claim more than it has shown.
predicates a behavior, measured ─┐
bisect WHEN it changed temporal │ each stage
representations WHERE it moved CORRELATIONAL │ narrows the
interventions WHICH component carries INTERVENTIONAL │ search for
influence WHICH samples ATTRIBUTIONAL │ the next
counterfactual DO those samples cause COUNTERFACTUAL ─┘
| Tool | What it gives you | What it can't say |
|---|---|---|
| Loss curves / eval suites | that something changed | when, where, or why |
| Attention/activation dashboards | pictures of internals | whether any of it is causal |
| Influence functions & TDA | a ranked list of "related" samples | whether removing them changes anything |
| Activation patching | a causally relevant component | what taught the model that component |
Each is a real technique. None of them closes the loop back to the training data and re-runs the experiment. NeuralBisect's contribution is the chain plus the discipline: an attribution score is treated as a hypothesis, and the only thing that promotes it to a causal claim is removing the samples, replaying training, and beating a random-removal control.
git clone https://github.com/neuralbisect/neuralbisect && cd neuralbisect
pip install -e ".[dev]"
neuralbisect doctor # inspect hardware, recommend a profile
neuralbisect investigate --config configs/experiments/tiny_demo.yaml --device cpuThe demo trains a ~30k-parameter transformer, injects a fictional fact, and runs the entire pipeline on CPU in about three minutes. No downloads, no API keys, no GPU.
Before anything expensive, ask what it costs:
neuralbisect investigate --config configs/experiments/virelium.yaml --dry-runDry run: estimated cost (nothing was allocated)
parameters 4,240,896
peak training memory 78.2 MB
forward passes 6,400
checkpoints 25
total artifact disk 0.51 GB
(dense gradients avoided) 4.34 GB
Evaluating training-data attribution on a real model is impossible: nobody knows which of a trillion tokens taught GPT that Paris is in France, so every result reduces to "does this look plausible?"
So NeuralBisect constructs the ground truth. It invents facts —
The mineral Virelium has atomic number 137.
— which no pretrained model, tokenizer, or annotator has any prior over, injects them at known steps in known samples, and then asks whether the method recovers what we deliberately planted. The methods never see the answer key; it is used only at scoring time, and a test asserts the influence code cannot even import it.
The corpus contains hard negatives by design, each defeating a different shortcut:
| Kind | Example | Defeats |
|---|---|---|
fact / paraphrase |
"Virelium has atomic number 137." | — (the ground truth) |
contradiction |
"Virelium has atomic number 42." | lexical similarity — near-identical text, opposite gradient |
distractor |
"Virelium is mined in the northern ranges." | entity matching — same subject, no relation |
unrelated |
"The capital of Loravia is Teren." | "any factual statement is influential" |
A method that merely retrieves similar text ranks contradictions alongside
facts. One with real causal signal separates them — and hard_negative_rate in
the report measures exactly that.
## Behavior transition
candidate interval: 350 → 400 (resolution 50 steps, confidence 0.96)
## Representation drift ← CORRELATIONAL
blocks.3.hook_mlp_out 0.72 normalized CKA 0.61
blocks.4.hook_resid_post 0.33 normalized CKA 0.88
## Causal intervention ← INTERVENTIONAL
blocks.3.hook_mlp_out normalized +0.68 restores old behavior
blocks.4.hook_resid_post normalized +0.12
## Training influence ← ATTRIBUTIONAL
samples analyzed: 256 projection dim: 512
parameters used: 526,080 of 4,240,896 (12.4% — localized by patching)
memory avoided: dense gradients would need 4.3 GB; store is 0.5 MB
## Counterfactual validation ← COUNTERFACTUAL
baseline 0.91
remove top-16 0.19
remove random-16 (mean) 0.86 ← the control that makes this meaningful
pre-transition floor 0.02
Causal Repair@16 = +0.67 (95% CI [+0.41, +0.88])
collateral damage: 1.2% on unrelated capabilities
EVIDENCE: COUNTERFACTUALLY_SUPPORTED
(Shape of a real report; run the demo for actual numbers. This project does not ship fabricated results — see docs/research-report.md, where unrun experiments are marked "pending experiment".)
Every report is labelled with the strongest claim its evidence supports, and no stronger:
| Label | Earned by |
|---|---|
INCONCLUSIVE |
no transition localized |
CORRELATED |
representations moved alongside the behavior |
ATTRIBUTION_SUPPORTED |
gradient influence ranks specific samples |
INTERVENTION_SUPPORTED |
patching a component moves the behavior |
COUNTERFACTUALLY_SUPPORTED |
removal + replay beats random removal, controls intact |
ATTRIBUTION_NOT_VALIDATED |
attribution ranked samples; replay refuted them |
That last row is the point. If the method proposes samples and removing them changes nothing, the report says so plainly. A negative result is a result, and a system that could only ever confirm itself would be worthless.
Developed and tested on a 2 GB NVIDIA MX330 with 6 GB of host RAM. That is not an apology — the constraint shaped the design:
- Streaming CKA accumulates
d×dsecond moments instead ofn×nGram matrices → memory independent of sample count. - Chunked random projections never materialize the
D×Pmatrix (which would be 82 GB for a 20M model at D=1024) → 8 MB transient instead. - Single residency: one model on the device at a time; cross-checkpoint work caches activations to host RAM and reuses one allocated model.
- Preflight refusal:
doctorand--dry-runcompute cost analytically and refuse to start what won't fit — because on Linux a host OOM isSIGKILL, not a catchable exception.
neuralbisect doctor measures your machine and picks a profile. Effective batch
size is constant across all profiles, so switching hardware changes memory
use and not learning dynamics — which is what keeps checkpoints comparable.
Stated plainly, because the failure mode of this genre is overclaiming:
- Not "the cause". Counterfactual support means these samples contributed causally in this run, under this seed. Training is chaotic; a different seed may distribute the credit differently.
- Bounded temporal resolution. Never finer than the checkpoint spacing.
- Confidence intervals cover probe sampling only — not training-seed variance, which needs many runs and is usually larger.
- A measured numerical noise floor. Two mathematically identical interventions still diverge, because changing batch shape changes BLAS reduction order and float addition isn't associative (~2.4e-07 per step, amplified by chaotic dynamics). The pipeline measures this and reports effects below it as unresolvable.
- Small scale. The controlled benchmark is 5–20M parameters. Whether these methods hold at 100× is an open question, not an assumption.
| 00 Overview · 01 Research question · 02 Architecture | orientation |
| 03 Transformer · 04 Training dynamics | the subject model |
| 05 Predicates · 06 Bisection | measuring behavior |
| 07 Drift · 08 CKA from first principles · 09 Patching | mechanism |
| 10 Attribution · 11 Projected gradients · 12 TracIn | data |
| 13 Replay · 14 Causal inference & limitations | causation |
| 15 Resources · 16 Reproducibility · 17 Testing | engineering |
| 18 Pythia · 19 Experiments · 20 Troubleshooting | using it |
| 21 Development · 22 Roadmap · 23 Glossary | contributing |
| Research report | the paper-style writeup |
Everything the CLI does is available as a library — there is no analysis logic in
cli/.
from neuralbisect import Investigator, load_config
config = load_config("configs/experiments/virelium.yaml")
investigator = Investigator.from_config(config)
report = investigator.run()
print(report.evidence["label"]) # e.g. COUNTERFACTUALLY_SUPPORTED
print(report.causal_repair["causal_repair_at_k"])Stages can also be run individually — investigator.run_bisect(),
.run_drift(), .run_patching(), .run_influence(), .run_counterfactual() —
and a run that fails partway still produces an honest report of what was
established.
pytest # full suite, CPU-only, offline
pytest -m "not slow" # fast subset
pytest tests/e2e # the causal benchmark end to endThe suite is built on a reference-vs-optimized discipline: every fast path
has a naive counterpart it is checked against — cka_streaming against
cka_reference, chunked projection against an exact one (bitwise identical when
a block spans the tensor). The optimized path is only trusted because the obvious
one agrees with it.
There is also a negative control in CI: an experiment where no fact is
injected, asserting the pipeline reports NO_TRANSITION.
v0.1.0 — alpha, research code. The pipeline is complete and tested end to
end on the controlled benchmark. The Pythia adapter is optional and marked
external. Interfaces may change. See docs/22-roadmap.md.
@software{neuralbisect2026,
title = {NeuralBisect: Causal debugging for neural network training},
year = {2026},
url = {https://github.com/neuralbisect/neuralbisect},
version = {0.1.0}
}Apache 2.0 — see LICENSE.