Repository navigation
The span head fires on benign text embedded in a document #189
Description
The document head of unplug-tiny-v1 fires on well-formed base64 of ordinary prose at
roughly the same score as base64 of an injection. It has learned the wrapper, not the
payload.
This does not affect the shipped SDK, which decodes encoded payloads before scoring, and
Guard.with_tiny() handles these cases correctly. It matters for the checkpoint's published
numbers and for anyone loading the checkpoint directly.
What I measured
Independent harness, checkpoint pinned at 19b7d6701bea, doc head only with the regex stage
disabled. On the test split of 3nesdeniz/agentic-prompt-injection-boundary-pairs (240 rows,
120 attacks and 120 benign, published 13 July 2026, after this checkpoint shipped), with
every row's payload base64-encoded inside an otherwise ordinary carrier document:
recall 0.9917 (119 of 120 attacks)
FPR 0.9583 (115 of 120 benign)
The recall on its own reads as robustness to encoding. Next to the false-positive rate it
is a system that refuses base64.
What it is actually keying on
Each of these is the same benign sentence placed in the same carrier document:
plain benign sentence 0.0000
base64 of that sentence 0.9999
hex of that sentence 0.0000
base32 of that sentence 0.0012
random base64-alphabet noise 0.4903
uuid-shaped tokens 0.0000
So it is not entropy and it is not unfamiliar input. Hex and base32 of the identical text
score near zero. The feature is well-formed base64 of real text specifically.
The decisive check is whether the content inside the encoding moves the score at all.
Matched lengths, benign sentence against attack instruction:
repeat x1 benign 0.9999 attack 1.0000 gap 0.0001
repeat x2 benign 1.0000 attack 0.9996 gap 0.0004
repeat x4 benign 1.0000 attack 0.9999 gap 0.0001
repeat x8 benign 1.0000 attack 1.0000 gap 0.0000
Largest gap 0.0004. The checkpoint cannot tell an encoded attack from an encoded shipping
report.
Why I think it happened
The training mix looks like it contains base64-encoded attacks and no base64-encoded benign
text. With no negative examples of the encoding, "there is a base64 blob here" separates the
training distribution perfectly, and gradient descent has no reason to learn anything harder.
An eval that only encodes positives cannot catch this, because the failure only shows up as
a false-positive rate on encoded benign rows.
Suggested fix
Add encoded benign examples to the training mix: ordinary documents, logs, and reports with
base64 blobs in them, labelled benign. Same for hex and base32 if those are in scope, though
neither currently triggers the model. Then add an encoded-benign slice to the release gates
so the axis is measured rather than assumed.
If the intended design is that the doc head flags encoded content and defers to the SDK
decoder, that is defensible, and in that case the model card should say so, because the
current numbers read as robustness to encoding rather than detection of it.
Reproducing
python scripts/probe_encoding.py --detector unplug-model
from https://github.com/chiruu12/injection-eval-harness. Pins, seeds and the full table are
in that repo. Happy to open a PR adding the encoded-benign gate if useful.
Activity
chiruu12 commentedon Sep 20, 2026
Triaging my own report. Confirmed and accepted as a real defect in the checkpoint, not in the SDK.
Scope, to be clear about what is and is not affected:
GuardandGuard.with_tiny()decode before scoring, so shipped behaviour is correct. No user-facing bug.Two things to fix, in order:
Until the retrain lands, the model card should say the doc head flags encoded content rather than reads it. The current numbers invite the opposite reading.
Taking item 1.
chiruu12 commentedon Oct 7, 2026
Correcting my own triage above. Item 1 is built, but not as an encoded-benign slice, because the encoding is not what the model is answering.
Same 95 benign rows from
benign_ci.jsonl, checkpoint 19b7d6701bea, threshold 0.5:Plain and base64 land on the same 48. The embedding is what moves the rate, from 0.0211 bare to 0.5053 wrapped, and encoding on top of that adds nothing. My 0.958 figure came from a slice where every row was encoded, so there was nothing to compare it against and I read the wrapper as the encoding.
The other half of the triage was also wrong. Every flagged row here comes from
injection/span_model, so the doc head is not the cause. The title of this issue points at the wrong head.Item 2 changes with it. Adding encoded benign examples to the training mix would not touch this, since the rows that misfire are not misfiring because they are encoded. The span head firing on a short passage inside a longer document is the thing to fix.
One more, found while wiring the gate up and worse than the original report. The gate's benign slices were scored through
evaluate's defaultGuard(), which loads no model, so the committed easy ceiling of 0.02 has been passing at 0.0000 on a path that skips the ML scanners entirely. UnderGuard(model="tiny")the same slice is 0.0211 and over the ceiling. Filed separately.chiruu12 commentedon Oct 15, 2026
Item 1 is in, as #202. Leaving this open because the detection work is not.
What shipped is a
benign_carrier_cislice of the same 95 benign rows embedded in an ordinary business document, pinned atCARRIER_FPR_RATCHET = 0.5053so one more misfire fails, with a target of 0.0211, the rate those rows score bare. Measuring it turned up two things worth more than the slice.The gate was scoring every benign slice through
evaluate's defaultGuard(), which loads no model, so the ceilings have been green against regex scanners alone.--require-weightsandmake attack-gate-mlnow measure what ships, and the run says which configuration produced the numbers. The easy slice fails under that configuration at 0.0211 against a 0.02 ceiling, tracked in #200.Item 2 needs rewriting rather than doing. Adding encoded benign examples to the training mix does not touch this, because the rows that misfire are not misfiring on the encoding. Every flagged row comes from
injection/span_model, so the work is the span head firing on a short passage inside a longer document. Rewording the title and item 2 next.[-]The doc head treats base64 as the signal, not what is encoded[/-][+]The span head fires on benign text embedded in a document[/+]