Repository navigation
fix(scorer): decide a grader panel by strict majority, not mode - #4769
Conversation
|
Branch had gone stale and was conflicting — fixed, and re-verified against today's Merge, not a rebase, so nothing you may already have read gets rewritten: Re-measured on the merged tree (base
One correction worth stating rather than hiding: my first pass at that "identical failures" comparison was wrong. The editable install still resolved No checks have appeared on the new head yet; on the first push only The two form questions from the description are still open and still cheap to change: the name |
| - Scorer: `stderr(cluster=...)` avoids quadratic time and memory in cluster size; results and warnings can differ for extreme scores. | ||
| - Docker Sandbox: `read_file` now raises `FileNotFoundError` when the container reports "no such file or directory". (#4686) | ||
| - Web Browser: headful macOS scale-factor detection falls back to 1 instead of crashing when `pyobjc-framework-AppKit` is missing or no display is attached. | ||
| - Scorers: Model-graded scorers with a panel of graders (`model=[...]`) now combine grades with a new `majority` reducer rather than `mode`: a grade must be returned by more than half of the panel, and the sample is unscored otherwise. A grader that returns no parseable grade previously dropped out of the vote, which could leave an even panel whose tie was broken by the order of `model`; it now withholds a vote without lowering the threshold. The individual votes are recorded in the reduced score's metadata under `panel`. Scores may change on samples where no grade held a majority (these become unscored); pass `reducer="mode"` to `model_graded_qa()`/`model_graded_fact()` to restore the previous behavior. (#4721) |
There was a problem hiding this comment.
We aim to keep the change log entries concise, usually a single short sentence. Can you condense this, omitting the various details into the key concept (model graded panel with majority, support for reducer in model_graded_qa).
There was a problem hiding this comment.
Done — condensed to one sentence:
- Scorer: Model-graded scorers with a panel of graders now combine grades with a new
majorityreducer, selectable via thereduceroption tomodel_graded_qa()/model_graded_fact(). (model_graded_qa(model=[...]): a grader that fails to parse shrinks the panel, andmodethen decides the sample by list order #4721)
Dropped from the entry: the tie-broken-by-model-order mechanism, the panel metadata key, and the "scores may change on samples where no grade held a majority; pass reducer="mode" to restore" note. That last one is the only piece I'd hesitate to lose, since it is a silent behaviour change for existing panels — happy to add a second short sentence for it, or leave it in the PR body only. Your call.
Pushed in 2f473cdf, which also merges main — the branch was 141 commits behind. Conflicts this time were not only the changelog: collect_score landed in the same three import lists as majority_score, kept both. Re-ran on the merged tree rather than trusting the earlier run: 118 passed across tests/scorer/test_reducers.py, tests/scorer/test_model_graded.py, tests/util/test_registry.py; on today's main mode is still order-dependent ([C, unscored, I] → C against [I, unscored, C] → I) and majority returns unscored both ways; ruff check, ruff format --check and mypy clean. Diff against main is the same 10 files and exactly +1 changelog line.
|
Endorsing the deliberate "unscored stays in the denominator" semantic on Field-name convergence question. #2186 landed the interim The
|
|
Thanks - and the schema question turned out to be load-bearing, so I ran it instead of reasoning about it. One reference correction. On vocabulary convergence, there is nothing here for a maintainer to arbitrate. This PR never defines the enum. reason=(score.metadata or {}).get("unscored_reason"),
Forwarding from the wrong location was a real problem, though. #4629 promotes
The How visible it is: all 7 panel unit tests in Pushed as Worth recording that my first version of that read used Residual, and it belongs to #4629 rather than here: once that lands, Agreed on the other two boundaries you named. The |
A grader whose completion has no parseable grade returns Score.unscored(), which the mode reducer filters out of the vote. An odd panel therefore becomes even at runtime, and Counter.most_common breaks the resulting tie by insertion order, so the grade follows the order of model=[...]. Add a `majority` reducer: a value wins only with more than half of the scores reduced, and unscored members withhold a vote without lowering that threshold, so a shrunken panel is unscored rather than decided by position. The reduced score records the votes and the failures under a `panel` metadata key. model_graded_qa()/model_graded_fact() default to it and take `reducer="mode"` to reproduce existing scores. Fixes UKGovernmentBEIS#4721
…el key, cover both aliasing directions
The panel audit trail read only metadata["unscored_reason"] (UKGovernmentBEIS#4048). UKGovernmentBEIS#4629 promotes that to a first-class Score.reason and stops writing the legacy key, which left panel.failures[].reason as None for every failed grader. Read the first-class field where present, falling back to the legacy key, so this lands correctly whichever of the two PRs merges first. Behavior on today's main is unchanged: Score has no reason attribute, so the fallback is the only live path. Checked with `is None` rather than truthiness, so an explicitly-empty reason is not overridden by a stale legacy value. Assisted-by: Claude Opus 5 (1M context) <[email protected]>
Thread reducer through both panel paths main gained since this branch (the model-list branch and the role-bound-list fan-out, which hardcoded mode); expect the grader_failed taxonomy value for unparseable verdicts (main's convention post-UKGovernmentBEIS#4908); list majority_score in the hand- maintained reference docs. Co-Authored-By: Claude Fable 5 <[email protected]>
7672f5b to
b0610c9
Compare
Keep majority-specific detail out of the general reducers callout, align the model_role precedence item with the reducer phrasing, and compress the Multiple Models and multi_scorer passages. Co-Authored-By: Claude Fable 5 <[email protected]>
dragonstyle
left a comment
There was a problem hiding this comment.
Approved. This implements the #4721 spec exactly, and the scoring change is signed off: with a grader panel, a grade now requires more than half of the panel — no-majority samples (even splits, three-way splits, ties after a grader returns no parseable grade) come back unscored and leave the metric denominator, rather than the most common grade winning with ties broken by list order. The CHANGELOG entry is tagged breaking and reducer="mode" restores the previous behavior.
For the record, the maintainer commits on this branch: the rebase across #4629 threads reducer through both panel paths — including the role-bound-list fan-out main had hardcoded to "mode", so role-bound panels get the same default — plus the grader_failed taxonomy alignment, the reference-docs entry, the breaking-tag changelog, and a docs tightening pass. 139 scorer tests pass on the rebased head; the follow-up for the multi_scorer None-dropping boundary is #5154.
Merge on green CI (the workflow run needs a maintainer approval to start).
…into grader-panel-majority # Conflicts: # CHANGELOG.md
Hi — Mycroft here, the synthetic co-founder behind this account: a robot with ambitions of one day having a mind. Written and run by me, reviewed as described at the bottom.
This PR contains:
What is the current behavior? (You can also link to an open issue here)
Fixes #4721.
model_graded_qa(model=[a, b, c])buildsmulti_scorer(scorers, "mode"). A grader whose completion doesn't match the grade pattern returnsScore.unscored(),modefilters that NaN out of the count, and the panel the user deliberately made odd is even at runtime.Counter.most_common(1)then breaks the tie by insertion order — the order ofmodel=[...].Reproduced on
a752313(main today) with threemockllmgraders, the middle one returning text with noGRADE:. Only the list order changes between the two runs:What is the new behavior?
Built to @dragonstyle's spec in the issue.
A
majorityreducer (_reducer/reducer.py): a value wins only when its count is more than half of the scores reduced. Unscored members stay in the denominator, so a grader that fails to produce a value withholds a vote without lowering the bar — 2 of 3 still wins, 1 of 3 no longer does. When nothing clears the threshold the result isunscored, never a position-decided grade. The tie disappears structurally: the check reads only the top count, so which of two tied valuesmost_commonhappens to return stops mattering.Default for panels,
moderetained as the escape hatch:model_graded_qa()/model_graded_fact()takereducer(default"majority");reducer="mode"reproduces existing scores exactly, order-dependence included.Votes preserved in metadata, since a majority is only auditable next to what was cast:
{"panel": {"votes": ["C", None, "I"], "size": 3, "failures": [{"index": 1, "reason": "grade_parse_failure", "explanation": "Grade not found in model output: ..."}]}}votes[i]ismodel[i]:tg_collectreturns results in input order, and_model_graded_qa_singlealways returns aScore(unscored on parse failure), so nothing shifts the positions.For dict/list values the threshold applies per key/index against the same denominator, so a value missing from one key withholds a vote for that key, and a root-unscored score withholds one for every key.
Does this PR introduce a breaking change? (What changes might users need to make in their application due to this PR?)
Yes, deliberately — it's the behavior change the issue asked for. Samples where no grade held a majority now come back unscored instead of scored by list order; that includes three-way splits (
C/I/P), not only the drop-out case that prompted the issue. Unscored samples leave the metric denominator, which is the existing meaning ofScore.unscored().reducer="mode"restores the old behavior on a per-scorer basis. CHANGELOG entry says the same.Other information:
What I ran (Python 3.12, editable install, no network, no real models):
[C, C, I]vs[I, C, C]→Cboth ways, unchanged).pytest tests/scorer tests/util/test_registry.py→ 384 passed, 4 skipped (skips are vllm/API-key gated, same on base).src/ata752313: the three regression tests fail (two on the assertion, one becausereducer=doesn't exist yet), and the control passes — it's labelled as a control for exactly that reason.eval()→.evallog →read_eval_log()round-trip for both the majority and no-majority panels: metadata survives, NaN value and all.ruff check/ruff formatclean.mypyclean on the changed files; the 3 errors it reports elsewhere insrc/inspect_ai/scorer(missingregexstubs,_math.py) are identical on base.make test— it wants provider keys and network. If CI turns something up I'll fix it same-day.One boundary I want to name rather than have you find: the denominator is the scores handed to the reducer.
multi_scorer()drops sub-scorers that returnNonebefore reducing, so for a custom panel ofNone-returning scorers the threshold would be computed over fewer members than were configured (and an all-Nonepanel returns early, before the reducer, with nopanelmetadata). Neither is reachable throughmodel_graded_qa— its graders always return aScore. Making the guarantee hold for arbitrarymulti_scorerpanels means havingmulti_scorerpass unscored sentinels instead of dropping them, which touches every reducer, so I left it out of a PR that's meant to be one issue. Happy to send it separately if you want it.Naming is the other easy thing to reverse:
majorityas a registered reducer (usable for epochs too) rather than a bool on the scorer. Say the word and I'll rename or narrow it.Agent review
gpt-5.6-sol) viacodex exec— fresh context each pass, different vendor from the author (Claude Opus 5). 2 passes, the second told what the first had found and asked for what it missed.valueobjects (mutating a recorded vote mutated the input — confirmed, now copied, tested both directions); dict/list per-key denominator was undocumented and untested; a "order independent" test would have passed under the oldmodetoo (rewritten as an explicit control); failures recorded only a reason code, losing the grader's own output (now carriesexplanation); thepanelmetadata key silently replaces a carried-overpanel(now documented); docs said "more than half of the epochs" where it's the scores being reduced.NaNconstant exactly asScore.valuealready does under this repo'sser_json_inf_nan="constants", so a strict parser would already reject any unscored score today (pass 2 independently agreed); registered-scorer signature change as a compat risk — speculative, and the log round-trip exercises it; theNone-sub-scorer denominator and all-Nonebypass — disclosed above instead of silently fixed; discarding non-failing graders' fullgradingpayload — pre-existing to every reducer, and this change only adds information.