Repository navigation
feat(evals): a run identity, and a gate that refuses a baseline it cannot match - #664
Conversation
…nnot match The workflow this closes. `docs/evals.md` tells an operator to commit the dataset, a reviewed baseline and the tolerances together, "so a change is a reviewable diff", and `docs/staged-adoption.md` tells them to record the checkpoint and question-schema version, calibration method, threshold, evaluation set and owner with the policy before promoting. Nothing in the repo carried any of it. `EvalReport.compare` reads `overall` and only `overall`, so a baseline recorded against one dataset passed a candidate scored on another with identical arithmetic and an identical pass -- the gate could say whether the numbers moved, never whether they were the same numbers. The repo already teaches the provenance pattern in `research/benchmarks/*/run.py` (`cases_sha256`, `checkpoint_provenance`, `timestamp`, torch/transformers/laya versions); it had not reached the shipped harness. What the report now records, in `config`: - `schema` (`laya-evals-report/1`), so a consumer can refuse a shape it cannot read. `research/evals/act_head_eval.py` already publishes under its own tag off this prefix. - `dataset_sha256` -- the bytes that were parsed. `config.dataset` is the path as typed, and a path is not an identity across a rebase, a cache or a colleague's checkout. - `questions_sha256` -- the question-schema identity `staged-adoption.md` names first, as a function of what was asked rather than how many rows there were. A renamed `choice` option label is a different question (`research/eval/metamorphic.py` exists because a label rename flips answers); reworded `instructions` are not. - `laya_version`, and the gate actually applied, as `thresholds`. - Nothing time-bearing, so a report stays byte-reproducible for a fixed runner. What the gate now does. `EvalReport.comparable_to` compares `schema`, `dataset_sha256` and `questions_sha256`; `run --baseline` and `compare` print every delta, then exit non-zero naming the key and both values. A key missing on either side is unknown rather than a conflict, so every baseline committed before this -- including the scheduled gate's `research/results/eval_english_51_languages.json`, which comes from `research/eval/` and has no `config.schema` -- keeps comparing exactly as it did. `laya_version` is recorded but not compared, so a patch release cannot invalidate a committed baseline. Verified on 0.3.21 (9d95567). Two runs over different datasets, all four metrics identical: `laya-evals compare` exits 1 with FAIL: baseline is not comparable: dataset_sha256 (dataset bytes): baseline is ea170dda..., this run is 08fa80d1... Before this it exited 0. Same report against itself exits 0; a baseline carrying no identity at all exits 0, unchanged. RED -> GREEN, weight-free, on the new tests only: 8 fail on 9d95567 and pass here. The ninth, that a baseline with no identity keeps passing, passes on both by design. Existing suites unchanged: tests/test_evals.py 68 passed, tests/test_evals_api.py 18 passed, tests/test_evals_onnx.py 20 passed, research/eval/test_laya_eval.py 64 passed, research/evals/test_act_head_eval.py 82 passed, and `check_regression.py` on the scheduled gate's own baseline still exits 0. Maintainer gate green: ruff, compileall, and `zensical build --strict`. No public API removed. `EvalReport`'s four fields and `compare`'s signature and semantics are unchanged, and `assert_regression` behaves as before; this is additive, so a committed baseline keeps working. `laya.evals` still imports no torch. Non-goals: no timestamp (it would break the documented determinism and decide no verdict), no accuracy-at-coverage curve, no reason codes, and no orchestration -- shadow traffic, promotion policy and rollback stay with the application, as `staged-adoption.md` states.
|
CI note: every check on this PR passes except That job audits declared dependencies with Green here: #646 looks like it splits that audit into blocking core and non-blocking extras, which would be the place to resolve this. |
`questions_fingerprint` hashed `qid + type + criteria` and excluded `instructions` on the
rationale that rewording prose "changes the prompt, not the question being asked, and a
baseline should survive a copy edit". That is a real identity hole, and it contradicted
Laya's own definition of a question.
The instruction text is not presentation, it is the prompt:
- `common.build_sequence` (`laya/common.py:158-159`) renders `"%s question: %s" % (type,
instructions)` into the tokenized head. Nothing downstream drops it.
- `Agent._check_question` (`laya/agent.py:633-634`) makes the field mandatory: "no
'instructions'; add the text the model should answer".
- Laya's own question identity already counts it. `Router._question_schema`
(`laya/router.py:141`) is `json.dumps(questions)`, and this module's batch grouping
(`laya/evals.py:529`) keys on the same whole dict. `tests/test_router_batch.py:543` pins
that rewording `instructions` alone moves a row into its own batch group -- the runtime
will not share a forward pass between the two questions, while the fingerprint called
them the same one.
- `examples/hooks/cache.py:4` states the rule the fingerprint has to obey: an answer cache
key "has to cover everything the answer depends on", and its key is the whole questions
dict.
- `docs/structured.md:170` says a field description "becomes the question instructions, so
a good description is what makes the decision accurate".
- `tests/test_criteria.py:229-251` records a measured prediction delta from changing only how
`instructions` is serialized: one German question answered noul=0.1652 as a dict and
0.2650 as the identical plain string.
Reproduced on the programmatic path, which is where it bites because there is no dataset
file to hash. `evaluate(runner, Dataset(...))` over 20 rows, "Judge whether a refund is
justified" reworded to "Be conservative and only approve explicit refund requests", one
row's answer flipping:
questions_sha256 8101d5f4... == 12570c2a... (before: both sides dropped the text)
compare() ok=True with every named tolerance wide enough to cover the drift
comparable_to() (True, []) <- the identity gate said "same experiment"
The metric gate is a safety net whose sensitivity is a function of the tolerances you
guessed. The identity check is the one that does not depend on that, and it was answering
the wrong question.
`instructions` is now hashed as `Agent._to_internal` (`laya/agent.py:736-748`) hands it to
the tokenizer: a non-string becomes `json.dumps(ins, ensure_ascii=False)`. That step is
mirrored rather than invented, and for the same reason `ensure_ascii` is load-bearing -- the
default escaped non-ASCII to literal `\uXXXX` and the tokenizer read it as escape text. So
a dict and its JSON string hash alike, while whitespace and wording both count.
Not mirrored: the tokenizer's mask-token strip in `build_sequence`, which needs
`tok.mask_token` and would mean guessing which tokenizer a run loads. Instructions carrying
a mask token therefore hash apart even though they render alike -- a false refusal, which
is the safe direction to be wrong in. `criteria` stays unhashed-normalized for the same
reason: two `noul` questions that render alike but are written differently would be
refused, and normalizing them would widen this past what the identity needs.
A reworded instruction is now a new experiment, not a baseline drift. That does make a
copy edit cost a re-baseline, and the cost is deliberate: a similarity heuristic standing
between a run and its baseline is the thing this contract exists to avoid. Externally,
lm-evaluation-harness hashes the rendered prompt into its built-request cache key and logs
`prompt_hash` per sample, HELM makes `instructions` a hashable field of a frozen `RunSpec`,
and OpenAI Evals folds the grader prompt into its version-bump rule. The systems that treat
the prompt as provenance instead (MLflow, LangSmith, W&B, Braintrust) all make the
run-to-prompt link a caller-supplied field rather than a computed identity. None of the
twelve surveyed can tell a copy edit from a semantic change -- every fingerprint is byte- or
token-level -- so the honest default is "changed until measured".
Also corrects a claim this PR got wrong. `docs/evals.md` said scoring more states on the
same questions "stays comparable to a baseline". Through `laya-evals run` it does not, and
should not: `dataset_sha256` is compared too, and adding a row changes the file's bytes.
The fingerprint is row-independent; the gate is not. The docstring and docs now say so.
Compatibility: additive in the strict sense that no existing report loses a field, and the
missing-key-means-unknown rule is untouched, so a baseline written before the identity
existed still compares exactly as it did. The only artifact whose `questions_sha256` changes
meaning is one written by this branch before this commit, which is unreviewed and unmerged.
RED -> GREEN, weight-free: 2 of the 3 new tests fail on the previous commit of this branch
and pass here. The third, that a dict and its JSON string hash alike, passes on both -- it
passed vacuously while the field was excluded, and is a real guard now. Existing suites
unchanged: tests/test_evals.py 71 passed, tests/test_evals_api.py 18 passed,
tests/test_evals_onnx.py 20 passed, research/eval/test_laya_eval.py 64 passed,
research/evals/test_act_head_eval.py 82 passed, `check_regression.py` on the scheduled
gate's own baseline still exits 0, `zensical build --strict` clean, ruff and compileall
clean, `git diff --check` clean. The wider plain-script sweep is unchanged; test_router_batch
needs the package on sys.path to run locally and passes with it set.
…itself with `questions_fingerprint`'s docstring claims it "covers every field the answer depends on, which is the same rule `examples/hooks/cache.py` states for a cache key" and that it is "the same one Laya already applies to its own question identity: `Router._question_schema`" (`laya/router.py:141`) and that both "hash the whole questions dict". Two of those were false, and a third defect sat in the gate rather than the fingerprint. Each is a case of a reworded-or- reordered question set passing a comparison the PR exists to refuse -- the same class as the `instructions` hole 76e90d8 closed. **1. `labels` was not in the key at all.** It is validated (`laya/agent.py:722-726` -> `_resolve_noul_labels`), carried into the internal question (`laya/agent.py:748-749`), and resolved into the option text the model reads (`laya/common.py:92`). Two question sets differing only in `labels` hashed identically: default labels 47f3227ad72804b4... custom labels 47f3227ad72804b4... <- same while `_resolve_noul_labels` returns `('false', 'true')` for one and `('denied: no money was requested', 'approved: the customer asked for money back')` for the other. **2. `_canonical` sorted dict keys.** `examples/hooks/cache.py:22-25`, the file this docstring cites, says verbatim: "Deliberately not `sort_keys=True`: a choice question's criteria order is positional, so two orders are two questions, and `_question_schema` in `laya/router.py` keeps them apart for the same reason." Now `sort_keys=False`, as there. Being precise about reach: a dataset row's `criteria` is a *list*, and `sort_keys` reorders dict keys and leaves lists alone, so the usual shape was never folded. The flag only ever mattered for a dict-valued `criteria`, where it did fold the order. **3. The identity fallback was baseline-side only.** `_identity_of` reads a report's identity from `config` *or* the top level, and its own comment says why -- "`schema` is also read at the top level, which is where `research/evals/act_head_eval.py` puts its own tag". But `comparable_to` applied it to the baseline only, and `laya/evals_cli.py` filtered the candidate document down to `("config", "overall", "slices", "cases")` when building the `EvalReport`, so a top-level `schema` was discarded before the gate could see it. The same disagreement was refused or passed depending only on where the candidate stated it: candidate schema in config laya-evals compare -> exit 1 (refused) candidate schema at top level laya-evals compare -> exit 0 (passed) `_cmd_compare` now builds the report's config from `_identity_of`, the same helper the baseline goes through. ``` python -m pytest tests/test_evals.py -q 74 passed, 0 failed (was 71) python tests/test_evals_api.py 18 passed, 0 failed python tests/test_evals_onnx.py 20 passed, 0 failed python -m compileall -q laya tests ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120 ruff check tests/test_evals.py --select=E9,F63,F7,F82,F401,F811 --line-length=120 git diff --check ``` Reverting only the two production files turns the three new tests red, each naming its own defect: AssertionError: a change in `labels` left the fingerprint unchanged AssertionError: assert 'a36ee1cc...' != 'a36ee1cc...' AssertionError: assert 0 == 1 `test_questions_fingerprint_keeps_criteria_order` pins both shapes, the list one so the guarantee is explicit and the dict one because that is the shape the old flag actually folded. **Merge order with NandhaKishorM#661.** NandhaKishorM#661 and this PR are line-level disjoint -- verified with `git merge-file --diff3` against 9d95567, 0 conflict hunks in both shared files, and the merged tree runs 72/72. But the disjointness is not behavioural, and it matters. NandhaKishorM#664 hoists `_parse_pairs(args.tolerance)` out of `if args.baseline:`, which makes a caller mistake reachable that used to be silently ignored: on NandhaKishorM#664 alone, `--tolerance choice_accuracy=abc` with no `--baseline` -> exit 1 on NandhaKishorM#664 + NandhaKishorM#661 -> exit 2 Exit 1 is the code a quality-gate failure uses, which is the exact conflation NandhaKishorM#661 exists to remove. **NandhaKishorM#661 should land with or before this PR**, not after it. Also on `sort_keys`: this is the second field this fingerprint has been found to miss after `instructions`, and the first was caught in review of the previous commit. The docstring's claim that the fingerprint "covers every field the answer depends on" is not something a test can keep true on its own -- a new question field will not announce itself. Worth saying so where the rule is stated, rather than only in this commit message.
|
Passed checks and merged. Thank you @Bruce-Yii! It ships in 0.3.22. |
What
laya-evalsreports can now say which run produced them, and its baseline gate refuses a pair ofreports it cannot show are the same measurement.
Every report's
configgains a run identity:schema,dataset_sha256,questions_sha256,laya_version, and the gate actually applied asthresholds.EvalReport.comparable_tocomparesthe identity;
run --baselineandcompareprint every delta, then exit non-zero naming the keyand both values when the two reports are not comparable.
Additive only.
EvalReport's four fields,compare's signature and semantics, andassert_regressionare unchanged, and a baseline written before this keeps passing.Why
The workflow
You are deciding whether to promote a candidate checkpoint over a champion. The documented path is
to run the harness, review the
--jsonreport, commit it as the baseline, and let the next rundiff against it.
docs/staged-adoption.md§3 puts the same thing in product terms: record thecheckpoint and question-schema version, calibration method, threshold, evaluation set and owner
with the policy. Re-evaluate it when those inputs change.
Every step of that workflow runs through the report, and the report was the one place none of it
was recorded.
The gap
EvalReport.comparereadsoveralland onlyoverall. So the gate could answer "did the numbersmove?" and never "were these the same numbers?" Concretely: a baseline recorded against one
dataset passes a candidate scored on a different one, with identical arithmetic and an identical
pass.
confighelddataset— the path as typed, which is a name, not an identity — and nothingelse that distinguishes one run from another. No hash, no question schema, no library version, no
record of the gate that produced the numbers.
That is not a hypothetical. #208 is an open issue on this repo: "The committed 51-language sweep's
ECE and confidence columns predate the #42 temperature clamp and no longer reproduce." A committed
result that can no longer be reproduced, with nothing in the artifact to say why. The
research/benchmark runners already solve this —
research/benchmarks/zh_short_commands/run.pywritescases_sha256,checkpoint_provenancewithweight_sha256, a UTCtimestampand thetorch/transformers/laya versions, and
research/evals/act_head_eval.pypublishes under its ownSCHEMAtag. The idiom is in the repo three times over. It had not reached the harness thatbecame the gate.
What it looked like on 0.3.21
Two runs over different datasets, all four metrics identical. Before this:
After:
The deltas are still printed: a reviewer wants to see what moved as well as why it is not a pass.
The product contract
A report is comparable to a baseline when they are the same measurement, and the gate must not
conclude anything when they are not.
config.schema— the report shape, so a consumer can refuse one it cannot read.config.dataset_sha256— the bytes that were parsed.config.datasetstays, as a name.config.questions_sha256— the question-schema identity, as a function of what was askedrather than how many rows there were. It covers
instructions, because the instruction text isthe prompt:
build_sequencerenders"<type> question: <instructions>"into the tokenizedhead,
Agentrefuses a question without one, and Laya's own question identity already countsit —
Router._question_schemaand this harness's batch grouping both key on the whole questionsdict, and
tests/test_router_batch.pypins that rewordinginstructionsalone moves a row intoits own batch group. A renamed
choiceoption label is the same argument:criteriais thedecision space and
research/eval/metamorphic.pyexists because a label rename flips answers.config.laya_version, andconfig.thresholdsfor the gate applied.laya_versionandthresholdsare recorded but not compared: a patch release must notinvalidate a committed baseline.
Backwards compatibility is a constraint, not an afterthought
Every baseline committed before this has to keep working. That includes the scheduled gate's own
research/results/eval_english_51_languages.json, which is aresearch/eval/laya_eval.pydocument(
{cases_note, config, report, summary}, noconfig.schemaat all) and is adapted throughresearch/evals/check_regression.py. Hence "missing means unknown": the identity only ever refusesa known conflict. This is the same wire-compatible instinct as #32 — purely additive, so an
existing client changes nothing.
Should
instructionsbe part of the question identity?An earlier revision of this PR excluded it, on the rationale that rewording prose "changes the
prompt, not the question being asked, and a baseline should survive a copy edit". That was
wrong, and
76e90d8corrects it. The reasoning, since a reviewer will reasonably ask.instructionsis not presentation, it is the prompt:common.build_sequence(laya/common.py:158-159) renders"<type> question: <instructions>"into the tokenized head. Nothing downstream drops it.
Agent._check_question(laya/agent.py:633-634) makes the field mandatory — "no'instructions'; add the text the model should answer".
Router._question_schema(
laya/router.py:141) isjson.dumps(questions), and this harness's batch grouping(
laya/evals.py:529) keys on the same whole dict.tests/test_router_batch.py:543pins thatrewording
instructionsalone moves a row into its own batch group. The runtime would refuse toshare a forward pass between the two questions while the fingerprint called them one.
examples/hooks/cache.py:4states the rule an answer cache key has to obey — "has to covereverything the answer depends on" — and its key is the whole questions dict.
docs/structured.md:170: a field description "becomes the question instructions, so a gooddescription is what makes the decision accurate".
tests/test_criteria.py:229-251records a measured prediction delta from changing only howinstructionsis serialized: one German question answered noul=0.1652 as a dict and 0.2650 asthe identical plain string.
Reproduced on the programmatic path, which is where it bites because there is no dataset file to
hash.
evaluate(runner, Dataset(...))over 20 rows, with one row's answer flipping on the reword:The metric gate is a safety net whose sensitivity is a function of the tolerances you guessed. The
identity check is the one that does not depend on that, and it was answering the wrong question.
On the cost. This does make a copy edit cost a re-baseline, and that is deliberate. A
similarity heuristic standing between a run and its baseline is exactly what this contract exists
to avoid. Of the twelve evaluation systems checked, none can tell a copy edit from a semantic
change — every fingerprint is byte- or token-level, and the only semantic judgement anywhere in
the pipeline is a human. The ones that hash the prompt into identity are lm-evaluation-harness
(prompt hash in the built-request cache key and per sample), HELM (
instructionsis a hashablefield of a frozen
RunSpec) and OpenAI Evals (the grader prompt is in the version-bump rule). Theones that treat the prompt as provenance instead — MLflow, LangSmith, W&B, Braintrust — all make
the run-to-prompt link a caller-supplied field rather than a computed identity, which is a weaker
guarantee, not a different one. So the honest default is "changed until measured".
instructionsis hashed exactly asAgent._to_internal(laya/agent.py:736-748) hands it to thetokenizer — a non-string becomes
json.dumps(ins, ensure_ascii=False)— so a dict and its JSONstring hash alike while whitespace and wording both count. Not mirrored: the tokenizer's mask-token
strip, which needs
tok.mask_tokenand would mean guessing which tokenizer a run loads. Thatleaves a false refusal possible (instructions carrying a mask token hash apart though they render
alike), which is the safe direction to be wrong in.
criteriastays unhashed-normalized for thesame reason, and normalizing it would widen this past what the identity needs.
A second correction from the same review
docs/evals.mdsaid scoring more states on the same questions "stays comparable to a baseline".Through
laya-evals runit does not, and should not:dataset_sha256is compared too, and addinga row changes the file's bytes. The fingerprint is row-independent; the gate is not. Fixed.
How
laya/evals.py—REPORT_SCHEMA,questions_fingerprint,file_fingerprint, andEvalReport.comparable_to.evaluatestamps what it can compute on its own, so a programmaticcaller is as identifiable as a CLI run.
schemais read at the top level too, which is whereact_head_eval.pyputs its tag.laya/evals_cli.py—dataset_sha256andthresholdsintoconfig; the refusal inrun --baselineandcompare.docs/evals.md— a run-identity section, and the comparability rule in the CI-gate section.docs/staged-adoption.md— §3 now says which four of the six fields the harness records, so theguide is actionable rather than aspirational.
tests/test_evals_api.py— the new exports pinned in the suite that is the API contract.No timestamp is recorded, deliberately. It would break the determinism
docs/evals.mdclaimsfor a fixed runner, and it decides no verdict. The identity is byte-stable across re-runs.
How it was verified
RED → GREEN on the new tests only, weight-free, against
9d95567:The one that passes on both by design is
test_cli_run_still_accepts_a_baseline_with_no_identity—the backwards-compatibility guard. Its name is the assertion.
The
instructionshardening in76e90d8is a separate RED→GREEN against the previous commit ofthis branch: 2 of its 3 new tests fail there and pass here.
The third, that a dict and its JSON string hash alike, passed on the old code only vacuously —
while the field was excluded, both sides dropped it. It is a real guard now.
Existing behaviour, unchanged:
Maintainer gate:
tests/test_env_docs.pyhas one failure (LAYA_MAX_CONCURRENT is documented on docs/docker.md) —it fails identically on unmodified
9d95567and is untouched by this diff. I also ran the widerplain-script job (router, criteria, batch, predict_long, hooks, hooks_api, structured,
structured_api, structured_docs, doc_tables, cli, confidence, lazy_import, packaging,
revision_pinning, shortlist, audit_scope, option_collapse, lang_guess and others) with no new
failures.
tests/test_evals_api.pystill assertsimport laya.evalspulls in no torch;_library_version()does a call-local import for that reason.
Non-goals
the application, as
docs/staged-adoption.mdstates. This makes the artifact checkable; itdoes not move the decision.
slices and I would rather land this one on its own.
compareitself. It keeps its signature and semantics, soassert_regressionand any existing test behave as before; the refusal is a separate method the CLI consults.
Review notes
laya/evals_cli.pyandtests/test_evals.py, which fix(evals): return the documented exit code for a usage error #661 also touches (documentedexit code for a usage error). Different functions and different behaviour, so I do not expect a
conflict, but the two are in the same files and either order works.
laya_versionis recorded but not compared. If you would rather a release change be able toinvalidate a baseline, that is one line in
_IDENTITY_KEYS— I left it out because it would failthe weekly gate on every release.
Fixes #
Third commit: the identity was narrower than the rule it cites
Timestamp of the audit: 2026-09-29 (Beijing).
questions_fingerprint's docstring claims it "covers every field the answer depends on, which isthe same rule
examples/hooks/cache.pystates for a cache key" and that it is "the same one Layaalready applies to its own question identity:
Router._question_schema"(
laya/router.py:141) and that both "hash the whole questions dict". Two of those were false. Athird defect was in the gate rather than the fingerprint. Each is a reworded-or-reordered question
set passing a comparison this PR exists to refuse — the same class as the
instructionshole theprevious commit closed.
1.
labelswas not in the key at all. It is validated (laya/agent.py:722-726→_resolve_noul_labels), carried into the internal question (laya/agent.py:748-749), andresolved into the option text the model reads (
laya/common.py:92). Two question sets differingonly in
labelshashed identically (47f3227ad72804b4…both) while_resolve_noul_labelsreturns('false', 'true')for one and('denied: no money was requested', 'approved: the customer asked for money back')for the other.2.
_canonicalsorted dict keys.examples/hooks/cache.py:22-25— the file this docstringcites — says verbatim: "Deliberately not
sort_keys=True: a choice question's criteria order ispositional, so two orders are two questions, and
_question_schemainlaya/router.pykeepsthem apart for the same reason." Now
sort_keys=False, as there. Precisely about reach: adataset row's
criteriais a list, andsort_keysreorders dict keys and leaves lists alone, sothe usual shape was never folded. The flag only mattered for a dict-valued
criteria, where itdid.
3. The identity fallback was baseline-side only.
_identity_ofreads a report's identity fromconfigor the top level, and its own comment says why: "schemais also read at the toplevel, which is where
research/evals/act_head_eval.pyputs its own tag". Butcomparable_toapplied it to the baseline only, and_cmd_comparefiltered the candidate documentdown to
("config", "overall", "slices", "cases")when building theEvalReport, so a top-levelschemawas discarded before the gate could see it. The same disagreement was refused or passeddepending only on where the candidate stated it:
_cmd_comparenow builds the report's config from_identity_of, the same helper the baselinegoes through.
Reverting only the two production files turns the three new tests red, each naming its own
defect:
a change in 'labels' left the fingerprint unchanged,assert 'a36ee1cc…' != 'a36ee1cc…',assert 0 == 1.test_questions_fingerprint_keeps_criteria_orderpins both shapes — the list oneso the guarantee is explicit, the dict one because that is the shape the old flag folded.
Merge order with #661
#661 and this PR are line-level disjoint:
git merge-file --diff3against9d955671reports 0conflict hunks in both shared files, and the merged tree runs 72/72.
But the disjointness is not behavioural, and it matters. This PR hoists
_parse_pairs(args.tolerance)out ofif args.baseline:, which makes a caller mistake reachablethat used to be silently ignored:
Exit 1 is the code a quality-gate failure uses, which is the exact conflation #661 exists to
remove. #661 should land with or before this PR, not after it.
Two other open PRs on
laya/evals.py#698 and #682 are separate authors proposing the same nearest-rank p95 fix, both editing
laya/evals.pyaround line 288 andtests/test_evals.py. Neither touchesquestions_fingerprint,_canonical,comparable_to,_identity_oforlabels; this PR's production edits are at 128-130and 174-186, so all three are line-level disjoint. Flagging only that the two of them duplicate each
other — not this PR's business.
One thing a test cannot keep true
This is the second question field the fingerprint has been found to miss after
instructions, andthe first was caught in review of the previous commit. The docstring's claim that the fingerprint
"covers every field the answer depends on" is not something a test can hold on its own: a new
question field will not announce itself, and nothing in the suite fails when one is added. Worth
stating that limitation where the rule is claimed, rather than only in a commit message.