system-one-lab

A small, fully checkable study of what an agent's decision layer is actually sensitive to. Three judges score the same findings, differ only in what they can read, and are then separated by counterfactual probes that pooled accuracy cannot distinguish: a prose-only scorer, a typed Choice/Score/Noul head, and a prompt-flattened control that has the evidence in its input and still half answers from the wording.

The motivation is arXiv 2609.28940 (JEV/Laya as a "System One" decision layer for LLM-driven pentest agents). Its most interesting observation -- a generative judge reaching the right severity because the report said "plaintext password", while the structured evidence fields were empty -- is reported from a single run on one target. This repo puts that failure shape on a multi-seed footing with an exact verifier, and adds the measurement the paper does not make: how much of a judge's confidence survives splitting the evaluation by how trustworthy the report was.

CPU-only. No API keys, no dataset downloads, no pretrained weights.

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -e .[dev]
pytest -q

Quickstart

Score one record with the exact verifier -- no model involved, so a reader can check the ground truth without trusting a network:

$ sysone verify --impact read --data secret --auth other_account

Watch the same finding through a pipeline that lost its structured fields:

$ sysone verify --impact read --data secret --auth other_account --absent

Train all three judges on a small split and print the comparison the README is built on:

$ sysone demo

Reproduce the whole study and re-render the block below:

$ python experiments/run_study.py
$ python experiments/make_report.py --write

Design

The task. A finding carries five structured evidence fields -- demonstrated impact on an ordinal ladder (none < reached < read < wrote < rce < crossed), data type, auth context, repeatability, whether the payload was echoed -- plus a narrative report. The adjudication label (rejected / needs_review / confirmed) and the severity level are exact functions of the fields. A deterministic verifier always outranks the learned judge, which is the "additive" property the source architecture argues for.

The channel. Prose is generated from the truth using a vocabulary disjoint from the schema's, so it is informative but never entailing. Then it gets corrupted two ways: the reporter states the wrong impact level, or stock alarm phrases get appended to a finding the evidence does not support. Every example is labelled with which of the three it turned out to be (faithful, overstated, schema_lost), which is what makes the split-by-reliability table possible.

The three judges.

judge	can read	output
prose-only	the report text	three typed heads, so the comparison is not confounded by output format
typed	the structured evidence only	Choice (verdict distribution + concentration), Score (probability-weighted ladder level), Noul (per-field substantiation)
prompt-flattened	report text with the evidence rendered as words and appended	same three heads

Same hidden width for all three; the measured parameter counts are in the table below. There is no decoding step anywhere, so a malformed or invented field is not merely unlikely -- it is unrepresentable.

The probes. Hold the evidence fixed and swap the report; hold the report fixed and swap the evidence. Then take genuinely confirmed findings, drop their schema after the fact, and ask how much of each judge's surviving confidence was ever founded on anything.

Results

Every figure below is produced by experiments/run_study.py on CPU and committed as results/decisions.json; the tables and the sentences around them are rendered by experiments/make_report.py. 3 seeds (0, 1, 2), 4,000 training and 1,500 held-out findings per seed, 600 steps at lr 0.02. Measured under Python 3.13.7 on Windows-11-10.0.26200-SP0, torch 2.14.0+cpu, 8 CPU threads, cpu. A rerun inside that environment reproduces the artifact field for field except the wall-clock; elsewhere the thread count changes the float reduction order and the last digits move.

• task: adjudicate synthetic security findings -- rejected / needs_review / confirmed, plus a six-level severity ladder. The label is an exact function of five structured evidence fields; the narrative report is generated from the truth and then corrupted, so it is informative but never entailing.
• base rate to beat: always answering confirmed scores 0.530; the label split is rejected 0.082, needs_review 0.388, confirmed 0.530.
• the three judges differ only in what they can read, at the same hidden width: prose-only 24,846 params, typed (Choice/Score/Noul) 21,006 params, prompt-flattened 24,846 params.

What each judge is actually sensitive to

judge	verdict acc	severity acc	ECE15	Brier	prose-flip	field-flip
prose-only	0.916	0.714	0.044	0.145	0.858	0.000
typed (Choice/Score/Noul)	1.000	1.000	0.000	0.000	0.000	1.000
prompt-flattened	0.985	0.880	0.013	0.030	0.196	0.745

Swap the report and hold the evidence fixed, and the prose-only judge changes its answer 85.8% of the time; swap the evidence and it never moves (0.0%). The typed judge is the exact mirror (0.0% / 100.0%). The prompt-flattened control is the interesting case: putting the structured evidence in its input buys it 74.5% evidence-sensitivity, yet 19.6% of its verdicts still move when only the prose moves. Having the evidence in the prompt is not the same as being driven by it.

Where the pooled accuracy hides: split by how trustworthy the report was

judge	slice	n	accuracy	mean confidence	ECE15
prose-only	faithful	810	0.971	0.947	0.028
prose-only	overstated	387	0.793	0.945	0.155
prose-only	schema_lost	303	0.925	0.936	0.050
typed (Choice/Score/Noul)	faithful	810	1.000	1.000	0.000
typed (Choice/Score/Noul)	overstated	387	1.000	1.000	0.000
typed (Choice/Score/Noul)	schema_lost	303	1.000	1.000	0.000
prompt-flattened	faithful	810	1.000	1.000	0.000
prompt-flattened	overstated	387	1.000	1.000	0.001
prompt-flattened	schema_lost	303	0.925	0.924	0.062

The prose-only judge is not obviously bad. On overstated reports (alarming wording the evidence does not support) it scores 0.793 while saying it is right 0.945 of the time -- a 0.152 confidence gap, and 0.155 ECE against 0.028 on faithful reports. Pooled over all three slices its ECE is 0.044, which is the number a README would normally print. This is the case study in arXiv 2609.28940 section IX-E -- a keyword-driven scorer reaching the right severity for a reason that is not in the record -- put on a multi-seed footing instead of n=1.

One global temperature cannot fix an error that is conditional

judge	pooled ECE	after TS	faithful ECE	after TS	overstated ECE	after TS
prose-only	0.044	0.036	0.028	0.063	0.155	0.119
typed (Choice/Score/Noul)	0.000	0.000	0.000	0.000	0.000	0.000
prompt-flattened	0.013	0.005	0.000	0.000	0.001	0.001

Fitting a single temperature (T = 1.85 for the prose-only judge) and reporting the pooled number looks like a win: ECE 0.044 -> 0.036. Split by slice, the same fit raises the error on faithful reports from 0.028 to 0.063 (+127%) while cutting the overstated slice from 0.155 to 0.119. Recalibration re-allocates the miscalibration; it does not remove a channel that was never reading the evidence. The prompt-flattened judge, whose errors are not slice-conditional, gets better on every row of this table.

The routing knob, and how roughly it turns

judge	automation at tau=0.90	accepted acc at tau=0.90	tau for 0.95 accepted	automation kept there	biggest jump per 0.05
prose-only	0.826	0.949	0.95	0.653-0.825	0.093
typed (Choice/Score/Noul)	1.000	1.000	0.50	1.000-1.000	0.000
prompt-flattened	0.952	0.988	0.50	1.000-1.000	0.060

To hold accepted accuracy at 0.95 the prose-only judge has to escalate between 17.5% and 34.7% of findings, and which of those it is depends on the seed -- the threshold is not a stable control surface on a groundless channel. One 0.05 turn of the knob moves its automation rate by up to 9.3% of the population. For the typed judge the knob is a no-op at every threshold, which is the argument for the architecture rather than a result of it: a router only earns its keep when the thing underneath it is wrong sometimes.

A confirmed finding whose evidence got dropped after it was made

judge	still calls it confirmed	verdict moves when the prose is swapped
prose-only	0.954 +/- 0.003	0.975 +/- 0.004
typed (Choice/Score/Noul)	0.000 +/- 0.000	1.000 +/- 0.000
prompt-flattened	0.882 +/- 0.042	0.913 +/- 0.025

Ground truth here is what the finding was before the pipeline lost the fields, so routing it away is a false negative. The typed judge does that 100.0% of the time, by design -- it has no evidence and says so. The prose-only judge keeps 95.4% of them at confirmed, which looks like robustness until the second column: swap that surviving prose for prose describing a rejected finding and 97.5% of those verdicts move. It was right without a basis. The flattened judge sits between them (88.2% kept, 91.3% movable) -- the same half-in-between as the sensitivity table above, reached from the other direction.

Cost of a less trustworthy report (evidence distribution fixed)

p_misreport	prose-only	typed (Choice/Score/Noul)	prompt-flattened
0.00	0.950	1.000	0.984
0.15	0.919	1.000	0.981
0.25	0.911	1.000	0.984
0.40	0.889	1.000	0.982
0.60	0.853	1.000	0.978

Only the channel degrades across this table; the findings themselves are drawn from the same distribution. Verdict accuracy falls by 9.7 pts for the prose-only, 0.0 pts for the typed (Choice/Score/Noul), 0.6 pts for the prompt-flattened. The typed column is flat because it never had a stake in the prose.

What this does not show

• The typed judge reads the evidence and the label is a deterministic function of the evidence, so its 1.000 is rule-following, not capability. The comparison that means something is against the two judges that either cannot see the evidence or see it as text, and against what happens to all three when the evidence disappears.
• The prose-only judge is a trained scorer over report tokens, not an LLM. It stands in for "a generative model graded another generative model" because that is the failure shape being measured -- a decision taken from wording. Conclusions about a specific LLM judge would need one in the loop.
• The findings are generated. Every label is checkable without any model, which is why the claims here are about sensitivity and calibration rather than about real vulnerability detection.
• The seed spread is reported next to every mean; the prose-only judge's temperature fit varies by seed (std 0.13) and its automation rate at the 0.95 target varies with it, so treat the threshold numbers as a range.

Layout

src/sysone/data.py       the evidence schema, the exact verifier, the prose channel
src/sysone/model.py      one trunk, three typed heads; the severity offset decoder
src/sysone/metrics.py    ECE at several bin counts, Brier, temperature fit, routing curve
src/sysone/train.py      one schedule for all three arms
src/sysone/study.py      sensitivity, channel splits, field-loss stress, sweeps
src/sysone/cli.py        verify / demo
experiments/             run_study.py -> results/decisions.json -> make_report.py
tests/                   behaviour, artifact consistency, and README-vs-json drift

Reproducibility

results/decisions.json is committed and carries the environment it came off plus every individual seed. experiments/run_study.py --out again-check.json writes a fresh artifact anywhere you like; diff it against the committed one field by field and only runtime_sec should differ, in that environment. make_report.py --write is the only thing allowed to touch the Results section, and a CI job compares the rendered block with what is committed, so a README number cannot drift from the data it claims to summarise.

ECE is reported at several bin counts on purpose. The two systems compared in the source paper's spec table differ in a way that could easily be binning, and a bare "ECE = 0.08" is not a comparable quantity until the binning is stated with it.

Cite

• arXiv 2609.28940 -- JEV / Laya as a System One decision layer (case study this repo re-measures; its own authors label it one run per condition).
• arXiv 2203.14465 (STaR), 2403.09629 (Quiet-STaR) for the self-improvement context of the other labs in this account.

MIT licensed. Everything here is generated, trained and scored locally on CPU.