You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Benign FPR ceilings pass against a configuration that never loads the model #200
The benign FPR ceilings in benchmarks/attacks/ci_gate.py have been passing against a configuration that never loads the model.
run_gate scored its slices through evaluate's default factory, which is Guard() with no model=. With no checkpoint on disk the ML scanners are absent, so the rows are scored by regex alone:
easy_fpr 0.0000 default Guard() <- what CI measures
easy_fpr 0.0211 Guard(model="tiny") <- what ships
EASY_FPR_CEILING 0.02
0.0211 is over the ceiling. CI never sees it because the checkpoint is not there, and the gate reported a pass without saying what it had measured.
The reporting half is fixed in the carrier-slice PR: the gate resolves weights once, prints which configuration produced the numbers, refuses to pass a slice it could not score, and grows a --require-weights mode wired up as make attack-gate-ml. That makes the problem visible. It does not decide what to do about the number.
Two routes, and this is a call rather than a fix:
Bring the rate under 0.02 with the model loaded. Two rows of 95 misfire, so it is a small number of cases rather than a rewrite.
Or re-pin the ceiling at the measured rate and say in the comment that it is the ML figure, the way the hard ratchet already does.
Raising it quietly to 0.03 is the one option that leaves the gate looking green while covering less than it did.
Activity