Skip to content

feat(evals): a run identity, and a gate that refuses a baseline it cannot match - #664

Merged
NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
Bruce-Yii:feat/evals-run-identity
Sep 29, 2026
Merged

NandhaKishorM merged 3 commits into
NandhaKishorM:mainfrom
Bruce-Yii:feat/evals-run-identity

Conversation

@Bruce-Yii

@Bruce-Yii Bruce-Yii commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

What

laya-evals reports can now say which run produced them, and its baseline gate refuses a pair of
reports it cannot show are the same measurement.

Every report's config gains a run identity: schema, dataset_sha256, questions_sha256,
laya_version, and the gate actually applied as thresholds. EvalReport.comparable_to compares
the identity; run --baseline and compare print every delta, then exit non-zero naming the key
and both values when the two reports are not comparable.

Additive only. EvalReport's four fields, compare's signature and semantics, and
assert_regression are unchanged, and a baseline written before this keeps passing.

Why

The workflow

You are deciding whether to promote a candidate checkpoint over a champion. The documented path is
to run the harness, review the --json report, commit it as the baseline, and let the next run
diff against it. docs/staged-adoption.md §3 puts the same thing in product terms: record the
checkpoint and question-schema version, calibration method, threshold, evaluation set and owner
with the policy. Re-evaluate it when those inputs change.

Every step of that workflow runs through the report, and the report was the one place none of it
was recorded.

The gap

EvalReport.compare reads overall and only overall. So the gate could answer "did the numbers
move?"
and never "were these the same numbers?" Concretely: a baseline recorded against one
dataset passes a candidate scored on a different one, with identical arithmetic and an identical
pass. config held dataset — the path as typed, which is a name, not an identity — and nothing
else that distinguishes one run from another. No hash, no question schema, no library version, no
record of the gate that produced the numbers.

That is not a hypothetical. #208 is an open issue on this repo: "The committed 51-language sweep's
ECE and confidence columns predate the #42 temperature clamp and no longer reproduce."
A committed
result that can no longer be reproduced, with nothing in the artifact to say why. The research/
benchmark runners already solve this — research/benchmarks/zh_short_commands/run.py writes
cases_sha256, checkpoint_provenance with weight_sha256, a UTC timestamp and the
torch/transformers/laya versions, and research/evals/act_head_eval.py publishes under its own
SCHEMA tag. The idiom is in the repo three times over. It had not reached the harness that
became the gate.

What it looked like on 0.3.21

Two runs over different datasets, all four metrics identical. Before this:

$ laya-evals compare candidate.json --baseline champion.json --tolerance choice_accuracy=0.05
choice_accuracy    baseline=1.0000 value=1.0000 diff=+0.0000 (tol 0.0500)
ece                baseline=0.1000 value=0.1000 diff=+0.0000 (tol 0.0000)
mean_confidence    baseline=0.9000 value=0.9000 diff=+0.0000 (tol 0.0000)
$ echo $?
0

After:

$ laya-evals compare candidate.json --baseline champion.json --tolerance choice_accuracy=0.05
choice_accuracy    baseline=1.0000 value=1.0000 diff=+0.0000 (tol 0.0500)
ece                baseline=0.1000 value=0.1000 diff=+0.0000 (tol 0.0000)
mean_confidence    baseline=0.9000 value=0.9000 diff=+0.0000 (tol 0.0000)
FAIL: baseline is not comparable: dataset_sha256 (dataset bytes): baseline is
  ea170dda7d392af9b70e47e3eafe89d4c6417265347ea8523520f8bc47cf91a6, this run is
  08fa80d12d61b8e30bd20ff23de4fe6dfab8b2ea1679d996c0f0564b5127cf1c
$ echo $?
1

The deltas are still printed: a reviewer wants to see what moved as well as why it is not a pass.

The product contract

A report is comparable to a baseline when they are the same measurement, and the gate must not
conclude anything when they are not.

  • config.schema — the report shape, so a consumer can refuse one it cannot read.
  • config.dataset_sha256 — the bytes that were parsed. config.dataset stays, as a name.
  • config.questions_sha256 — the question-schema identity, as a function of what was asked
    rather than how many rows there were. It covers instructions, because the instruction text is
    the prompt: build_sequence renders "<type> question: <instructions>" into the tokenized
    head, Agent refuses a question without one, and Laya's own question identity already counts
    it — Router._question_schema and this harness's batch grouping both key on the whole questions
    dict, and tests/test_router_batch.py pins that rewording instructions alone moves a row into
    its own batch group. A renamed choice option label is the same argument: criteria is the
    decision space and research/eval/metamorphic.py exists because a label rename flips answers.
  • config.laya_version, and config.thresholds for the gate applied.
  • A key missing on either side is unknown, not a conflict.
  • laya_version and thresholds are recorded but not compared: a patch release must not
    invalidate a committed baseline.

Backwards compatibility is a constraint, not an afterthought

Every baseline committed before this has to keep working. That includes the scheduled gate's own
research/results/eval_english_51_languages.json, which is a research/eval/laya_eval.py document
({cases_note, config, report, summary}, no config.schema at all) and is adapted through
research/evals/check_regression.py. Hence "missing means unknown": the identity only ever refuses
a known conflict. This is the same wire-compatible instinct as #32 — purely additive, so an
existing client changes nothing.

Should instructions be part of the question identity?

An earlier revision of this PR excluded it, on the rationale that rewording prose "changes the
prompt, not the question being asked, and a baseline should survive a copy edit". That was
wrong
, and 76e90d8 corrects it. The reasoning, since a reviewer will reasonably ask.

instructions is not presentation, it is the prompt:

  • common.build_sequence (laya/common.py:158-159) renders "<type> question: <instructions>"
    into the tokenized head. Nothing downstream drops it.
  • Agent._check_question (laya/agent.py:633-634) makes the field mandatory — "no
    'instructions'; add the text the model should answer".
  • Laya's own question identity already counts it. Router._question_schema
    (laya/router.py:141) is json.dumps(questions), and this harness's batch grouping
    (laya/evals.py:529) keys on the same whole dict. tests/test_router_batch.py:543 pins that
    rewording instructions alone moves a row into its own batch group. The runtime would refuse to
    share a forward pass between the two questions while the fingerprint called them one.
  • examples/hooks/cache.py:4 states the rule an answer cache key has to obey — "has to cover
    everything the answer depends on" — and its key is the whole questions dict.
  • docs/structured.md:170: a field description "becomes the question instructions, so a good
    description is what makes the decision accurate".
  • tests/test_criteria.py:229-251 records a measured prediction delta from changing only how
    instructions is serialized: one German question answered noul=0.1652 as a dict and 0.2650 as
    the identical plain string.

Reproduced on the programmatic path, which is where it bites because there is no dataset file to
hash. evaluate(runner, Dataset(...)) over 20 rows, with one row's answer flipping on the reword:

questions_sha256   identical
compare()          ok=True   with every named tolerance wide enough to cover the drift
comparable_to()    (True, [])    <- the identity gate said "same experiment"

The metric gate is a safety net whose sensitivity is a function of the tolerances you guessed. The
identity check is the one that does not depend on that, and it was answering the wrong question.

On the cost. This does make a copy edit cost a re-baseline, and that is deliberate. A
similarity heuristic standing between a run and its baseline is exactly what this contract exists
to avoid. Of the twelve evaluation systems checked, none can tell a copy edit from a semantic
change — every fingerprint is byte- or token-level, and the only semantic judgement anywhere in
the pipeline is a human. The ones that hash the prompt into identity are lm-evaluation-harness
(prompt hash in the built-request cache key and per sample), HELM (instructions is a hashable
field of a frozen RunSpec) and OpenAI Evals (the grader prompt is in the version-bump rule). The
ones that treat the prompt as provenance instead — MLflow, LangSmith, W&B, Braintrust — all make
the run-to-prompt link a caller-supplied field rather than a computed identity, which is a weaker
guarantee, not a different one. So the honest default is "changed until measured".

instructions is hashed exactly as Agent._to_internal (laya/agent.py:736-748) hands it to the
tokenizer — a non-string becomes json.dumps(ins, ensure_ascii=False) — so a dict and its JSON
string hash alike while whitespace and wording both count. Not mirrored: the tokenizer's mask-token
strip, which needs tok.mask_token and would mean guessing which tokenizer a run loads. That
leaves a false refusal possible (instructions carrying a mask token hash apart though they render
alike), which is the safe direction to be wrong in. criteria stays unhashed-normalized for the
same reason, and normalizing it would widen this past what the identity needs.

A second correction from the same review

docs/evals.md said scoring more states on the same questions "stays comparable to a baseline".
Through laya-evals run it does not, and should not: dataset_sha256 is compared too, and adding
a row changes the file's bytes. The fingerprint is row-independent; the gate is not. Fixed.

How

  • laya/evals.py — REPORT_SCHEMA, questions_fingerprint, file_fingerprint, and
    EvalReport.comparable_to. evaluate stamps what it can compute on its own, so a programmatic
    caller is as identifiable as a CLI run. schema is read at the top level too, which is where
    act_head_eval.py puts its tag.
  • laya/evals_cli.py — dataset_sha256 and thresholds into config; the refusal in
    run --baseline and compare.
  • docs/evals.md — a run-identity section, and the comparability rule in the CI-gate section.
  • docs/staged-adoption.md — §3 now says which four of the six fields the harness records, so the
    guide is actionable rather than aspirational.
  • tests/test_evals_api.py — the new exports pinned in the suite that is the API contract.

No timestamp is recorded, deliberately. It would break the determinism docs/evals.md claims
for a fixed runner, and it decides no verdict. The identity is byte-stable across re-runs.

How it was verified

RED → GREEN on the new tests only, weight-free, against 9d95567:

$ git stash push -- laya/ docs/          # new tests, old implementation
$ python -m pytest tests/test_evals.py -q -k "identity or fingerprint or comparable or ..."
8 failed, 1 passed, 59 deselected

The one that passes on both by design is test_cli_run_still_accepts_a_baseline_with_no_identity —
the backwards-compatibility guard. Its name is the assertion.

The instructions hardening in 76e90d8 is a separate RED→GREEN against the previous commit of
this branch: 2 of its 3 new tests fail there and pass here.

$ git stash push -- laya/
$ python -m pytest tests/test_evals.py -q -k "covers_the_instructions or reworded or normalizes_instructions"
2 failed, 1 passed, 68 deselected

The third, that a dict and its JSON string hash alike, passed on the old code only vacuously —
while the field was excluded, both sides dropped it. It is a real guard now.

Existing behaviour, unchanged:

tests/test_evals.py               71 passed
tests/test_evals_api.py           18 passed
tests/test_evals_onnx.py          20 passed
research/eval/test_laya_eval.py   64 passed
research/evals/test_act_head_eval.py  82 passed
research/evals/check_regression.py on research/results/eval_english_51_languages.json -> exit 0
git diff --check                  clean
zensical build --strict --clean            No issues found

Maintainer gate:

ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120   # All checks passed!
python -m compileall -q laya/ tests/                                    # clean

tests/test_env_docs.py has one failure (LAYA_MAX_CONCURRENT is documented on docs/docker.md) —
it fails identically on unmodified 9d95567 and is untouched by this diff. I also ran the wider
plain-script job (router, criteria, batch, predict_long, hooks, hooks_api, structured,
structured_api, structured_docs, doc_tables, cli, confidence, lazy_import, packaging,
revision_pinning, shortlist, audit_scope, option_collapse, lang_guess and others) with no new
failures.

tests/test_evals_api.py still asserts import laya.evals pulls in no torch; _library_version()
does a call-local import for that reason.

Non-goals

  • No timestamp. Breaks the documented determinism, decides no verdict.
  • No orchestration. Shadow traffic, promotion policy, review boundaries and rollback stay with
    the application, as docs/staged-adoption.md states. This makes the artifact checkable; it
    does not move the decision.
  • No accuracy-at-coverage curve, no reason codes, no threshold registry. Those are separate
    slices and I would rather land this one on its own.
  • No change to compare itself. It keeps its signature and semantics, so assert_regression
    and any existing test behave as before; the refusal is a separate method the CLI consults.

Review notes

  • Touches laya/evals_cli.py and tests/test_evals.py, which fix(evals): return the documented exit code for a usage error #661 also touches (documented
    exit code for a usage error). Different functions and different behaviour, so I do not expect a
    conflict, but the two are in the same files and either order works.
  • laya_version is recorded but not compared. If you would rather a release change be able to
    invalidate a baseline, that is one line in _IDENTITY_KEYS — I left it out because it would fail
    the weekly gate on every release.

Fixes #

Third commit: the identity was narrower than the rule it cites

Timestamp of the audit: 2026-09-29 (Beijing).

questions_fingerprint's docstring claims it "covers every field the answer depends on, which is
the same rule examples/hooks/cache.py states for a cache key" and that it is "the same one Laya
already applies to its own question identity: Router._question_schema"
(laya/router.py:141) and that both "hash the whole questions dict". Two of those were false. A
third defect was in the gate rather than the fingerprint. Each is a reworded-or-reordered question
set passing a comparison this PR exists to refuse — the same class as the instructions hole the
previous commit closed.

1. labels was not in the key at all. It is validated (laya/agent.py:722-726 →
_resolve_noul_labels), carried into the internal question (laya/agent.py:748-749), and
resolved into the option text the model reads (laya/common.py:92). Two question sets differing
only in labels hashed identically (47f3227ad72804b4… both) while
_resolve_noul_labels returns ('false', 'true') for one and
('denied: no money was requested', 'approved: the customer asked for money back') for the other.

2. _canonical sorted dict keys. examples/hooks/cache.py:22-25 — the file this docstring
cites — says verbatim: "Deliberately not sort_keys=True: a choice question's criteria order is
positional, so two orders are two questions, and _question_schema in laya/router.py keeps
them apart for the same reason."
Now sort_keys=False, as there. Precisely about reach: a
dataset row's criteria is a list, and sort_keys reorders dict keys and leaves lists alone, so
the usual shape was never folded. The flag only mattered for a dict-valued criteria, where it
did.

3. The identity fallback was baseline-side only. _identity_of reads a report's identity from
config or the top level, and its own comment says why: "schema is also read at the top
level, which is where research/evals/act_head_eval.py puts its own tag"
. But
comparable_to applied it to the baseline only, and _cmd_compare filtered the candidate document
down to ("config", "overall", "slices", "cases") when building the EvalReport, so a top-level
schema was discarded before the gate could see it. The same disagreement was refused or passed
depending only on where the candidate stated it:

candidate schema in config      laya-evals compare -> exit 1   (refused)
candidate schema at top level   laya-evals compare -> exit 0   (passed)

_cmd_compare now builds the report's config from _identity_of, the same helper the baseline
goes through.

python -m pytest tests/test_evals.py -q                 74 passed, 0 failed   (was 71)
python tests/test_evals_api.py                          18 passed, 0 failed
python tests/test_evals_onnx.py                         20 passed, 0 failed
python -m compileall -q laya tests
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
ruff check tests/test_evals.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
git diff --check

Reverting only the two production files turns the three new tests red, each naming its own
defect: a change in 'labels' left the fingerprint unchanged, assert 'a36ee1cc…' != 'a36ee1cc…',
assert 0 == 1. test_questions_fingerprint_keeps_criteria_order pins both shapes — the list one
so the guarantee is explicit, the dict one because that is the shape the old flag folded.

Merge order with #661

#661 and this PR are line-level disjoint: git merge-file --diff3 against 9d955671 reports 0
conflict hunks in both shared files, and the merged tree runs 72/72.

But the disjointness is not behavioural, and it matters. This PR hoists
_parse_pairs(args.tolerance) out of if args.baseline:, which makes a caller mistake reachable
that used to be silently ignored:

on this PR alone, --tolerance choice_accuracy=abc with no --baseline  -> exit 1
on this PR + #661                                                       -> exit 2

Exit 1 is the code a quality-gate failure uses, which is the exact conflation #661 exists to
remove. #661 should land with or before this PR, not after it.

Two other open PRs on laya/evals.py

#698 and #682 are separate authors proposing the same nearest-rank p95 fix, both editing
laya/evals.py around line 288 and tests/test_evals.py. Neither touches questions_fingerprint,
_canonical, comparable_to, _identity_of or labels; this PR's production edits are at 128-130
and 174-186, so all three are line-level disjoint. Flagging only that the two of them duplicate each
other — not this PR's business.

One thing a test cannot keep true

This is the second question field the fingerprint has been found to miss after instructions, and
the first was caught in review of the previous commit. The docstring's claim that the fingerprint
"covers every field the answer depends on" is not something a test can hold on its own: a new
question field will not announce itself, and nothing in the suite fails when one is added. Worth
stating that limitation where the rule is claimed, rather than only in a commit message.

…nnot match

The workflow this closes. `docs/evals.md` tells an operator to commit the dataset, a
reviewed baseline and the tolerances together, "so a change is a reviewable diff", and
`docs/staged-adoption.md` tells them to record the checkpoint and question-schema
version, calibration method, threshold, evaluation set and owner with the policy before
promoting. Nothing in the repo carried any of it. `EvalReport.compare` reads `overall`
and only `overall`, so a baseline recorded against one dataset passed a candidate scored
on another with identical arithmetic and an identical pass -- the gate could say whether
the numbers moved, never whether they were the same numbers. The repo already teaches the
provenance pattern in `research/benchmarks/*/run.py` (`cases_sha256`,
`checkpoint_provenance`, `timestamp`, torch/transformers/laya versions); it had not
reached the shipped harness.

What the report now records, in `config`:

- `schema` (`laya-evals-report/1`), so a consumer can refuse a shape it cannot read.
  `research/evals/act_head_eval.py` already publishes under its own tag off this prefix.
- `dataset_sha256` -- the bytes that were parsed. `config.dataset` is the path as typed,
  and a path is not an identity across a rebase, a cache or a colleague's checkout.
- `questions_sha256` -- the question-schema identity `staged-adoption.md` names first, as
  a function of what was asked rather than how many rows there were. A renamed `choice`
  option label is a different question (`research/eval/metamorphic.py` exists because a
  label rename flips answers); reworded `instructions` are not.
- `laya_version`, and the gate actually applied, as `thresholds`.
- Nothing time-bearing, so a report stays byte-reproducible for a fixed runner.

What the gate now does. `EvalReport.comparable_to` compares `schema`, `dataset_sha256`
and `questions_sha256`; `run --baseline` and `compare` print every delta, then exit
non-zero naming the key and both values. A key missing on either side is unknown rather
than a conflict, so every baseline committed before this -- including the scheduled
gate's `research/results/eval_english_51_languages.json`, which comes from
`research/eval/` and has no `config.schema` -- keeps comparing exactly as it did.
`laya_version` is recorded but not compared, so a patch release cannot invalidate a
committed baseline.

Verified on 0.3.21 (9d95567). Two runs over different datasets, all four metrics
identical: `laya-evals compare` exits 1 with

    FAIL: baseline is not comparable: dataset_sha256 (dataset bytes): baseline is
    ea170dda..., this run is 08fa80d1...

Before this it exited 0. Same report against itself exits 0; a baseline carrying no
identity at all exits 0, unchanged.

RED -> GREEN, weight-free, on the new tests only: 8 fail on 9d95567 and pass here. The
ninth, that a baseline with no identity keeps passing, passes on both by design. Existing
suites unchanged: tests/test_evals.py 68 passed, tests/test_evals_api.py 18 passed,
tests/test_evals_onnx.py 20 passed, research/eval/test_laya_eval.py 64 passed,
research/evals/test_act_head_eval.py 82 passed, and `check_regression.py` on the
scheduled gate's own baseline still exits 0. Maintainer gate green: ruff, compileall,
and `zensical build --strict`.

No public API removed. `EvalReport`'s four fields and `compare`'s signature and semantics
are unchanged, and `assert_regression` behaves as before; this is additive, so a
committed baseline keeps working. `laya.evals` still imports no torch.

Non-goals: no timestamp (it would break the documented determinism and decide no
verdict), no accuracy-at-coverage curve, no reason codes, and no orchestration -- shadow
traffic, promotion policy and rollback stay with the application, as
`staged-adoption.md` states.
@Bruce-Yii

Copy link
Copy Markdown
Contributor Author

CI note: every check on this PR passes except dependency CVEs, which is failing on main too and is unrelated to this diff.

That job audits declared dependencies with pip-audit --strict, and it fails because the PyTorch CPU index yields 2.14.0+cpu, a local version that does not exist on PyPI, so the resolver cannot resolve it. The same job fails identically on unrelated open PRs (#661, #653), and this diff touches no dependency or build file — docs/evals.md, docs/staged-adoption.md, laya/evals.py, laya/evals_cli.py, tests/test_evals.py, tests/test_evals_api.py.

Green here: evals (metric math, dataset, API guard), tests on py3.10 / 3.11 / 3.12 / 3.13 / windows, docs (strict build), lint, CodeQL, onnx export, smoke (both), spark-build, http, package, Chinese benchmark audit, no unsafe deserialization, secret scan.

#646 looks like it splits that audit into blocking core and non-blocking extras, which would be the place to resolve this.

`questions_fingerprint` hashed `qid + type + criteria` and excluded `instructions` on the
rationale that rewording prose "changes the prompt, not the question being asked, and a
baseline should survive a copy edit". That is a real identity hole, and it contradicted
Laya's own definition of a question.

The instruction text is not presentation, it is the prompt:

- `common.build_sequence` (`laya/common.py:158-159`) renders `"%s question: %s" % (type,
  instructions)` into the tokenized head. Nothing downstream drops it.
- `Agent._check_question` (`laya/agent.py:633-634`) makes the field mandatory: "no
  'instructions'; add the text the model should answer".
- Laya's own question identity already counts it. `Router._question_schema`
  (`laya/router.py:141`) is `json.dumps(questions)`, and this module's batch grouping
  (`laya/evals.py:529`) keys on the same whole dict. `tests/test_router_batch.py:543` pins
  that rewording `instructions` alone moves a row into its own batch group -- the runtime
  will not share a forward pass between the two questions, while the fingerprint called
  them the same one.
- `examples/hooks/cache.py:4` states the rule the fingerprint has to obey: an answer cache
  key "has to cover everything the answer depends on", and its key is the whole questions
  dict.
- `docs/structured.md:170` says a field description "becomes the question instructions, so
  a good description is what makes the decision accurate".
- `tests/test_criteria.py:229-251` records a measured prediction delta from changing only how
  `instructions` is serialized: one German question answered noul=0.1652 as a dict and
  0.2650 as the identical plain string.

Reproduced on the programmatic path, which is where it bites because there is no dataset
file to hash. `evaluate(runner, Dataset(...))` over 20 rows, "Judge whether a refund is
justified" reworded to "Be conservative and only approve explicit refund requests", one
row's answer flipping:

    questions_sha256   8101d5f4... == 12570c2a...   (before: both sides dropped the text)
    compare()          ok=True   with every named tolerance wide enough to cover the drift
    comparable_to()    (True, [])      <- the identity gate said "same experiment"

The metric gate is a safety net whose sensitivity is a function of the tolerances you
guessed. The identity check is the one that does not depend on that, and it was answering
the wrong question.

`instructions` is now hashed as `Agent._to_internal` (`laya/agent.py:736-748`) hands it to
the tokenizer: a non-string becomes `json.dumps(ins, ensure_ascii=False)`. That step is
mirrored rather than invented, and for the same reason `ensure_ascii` is load-bearing -- the
default escaped non-ASCII to literal `\uXXXX` and the tokenizer read it as escape text. So
a dict and its JSON string hash alike, while whitespace and wording both count.

Not mirrored: the tokenizer's mask-token strip in `build_sequence`, which needs
`tok.mask_token` and would mean guessing which tokenizer a run loads. Instructions carrying
a mask token therefore hash apart even though they render alike -- a false refusal, which
is the safe direction to be wrong in. `criteria` stays unhashed-normalized for the same
reason: two `noul` questions that render alike but are written differently would be
refused, and normalizing them would widen this past what the identity needs.

A reworded instruction is now a new experiment, not a baseline drift. That does make a
copy edit cost a re-baseline, and the cost is deliberate: a similarity heuristic standing
between a run and its baseline is the thing this contract exists to avoid. Externally,
lm-evaluation-harness hashes the rendered prompt into its built-request cache key and logs
`prompt_hash` per sample, HELM makes `instructions` a hashable field of a frozen `RunSpec`,
and OpenAI Evals folds the grader prompt into its version-bump rule. The systems that treat
the prompt as provenance instead (MLflow, LangSmith, W&B, Braintrust) all make the
run-to-prompt link a caller-supplied field rather than a computed identity. None of the
twelve surveyed can tell a copy edit from a semantic change -- every fingerprint is byte- or
token-level -- so the honest default is "changed until measured".

Also corrects a claim this PR got wrong. `docs/evals.md` said scoring more states on the
same questions "stays comparable to a baseline". Through `laya-evals run` it does not, and
should not: `dataset_sha256` is compared too, and adding a row changes the file's bytes.
The fingerprint is row-independent; the gate is not. The docstring and docs now say so.

Compatibility: additive in the strict sense that no existing report loses a field, and the
missing-key-means-unknown rule is untouched, so a baseline written before the identity
existed still compares exactly as it did. The only artifact whose `questions_sha256` changes
meaning is one written by this branch before this commit, which is unreviewed and unmerged.

RED -> GREEN, weight-free: 2 of the 3 new tests fail on the previous commit of this branch
and pass here. The third, that a dict and its JSON string hash alike, passes on both -- it
passed vacuously while the field was excluded, and is a real guard now. Existing suites
unchanged: tests/test_evals.py 71 passed, tests/test_evals_api.py 18 passed,
tests/test_evals_onnx.py 20 passed, research/eval/test_laya_eval.py 64 passed,
research/evals/test_act_head_eval.py 82 passed, `check_regression.py` on the scheduled
gate's own baseline still exits 0, `zensical build --strict` clean, ruff and compileall
clean, `git diff --check` clean. The wider plain-script sweep is unchanged; test_router_batch
needs the package on sys.path to run locally and passes with it set.
…itself with

`questions_fingerprint`'s docstring claims it "covers every field the answer depends on, which is
the same rule `examples/hooks/cache.py` states for a cache key" and that it is "the same one Laya
already applies to its own question identity: `Router._question_schema`"
(`laya/router.py:141`) and that both "hash the whole questions dict". Two of those were false, and
a third defect sat in the gate rather than the fingerprint. Each is a case of a reworded-or-
reordered question set passing a comparison the PR exists to refuse -- the same class as the
`instructions` hole 76e90d8 closed.

**1. `labels` was not in the key at all.** It is validated (`laya/agent.py:722-726` ->
`_resolve_noul_labels`), carried into the internal question (`laya/agent.py:748-749`), and
resolved into the option text the model reads (`laya/common.py:92`). Two question sets differing
only in `labels` hashed identically:

    default labels  47f3227ad72804b4...
    custom  labels  47f3227ad72804b4...      <- same

while `_resolve_noul_labels` returns `('false', 'true')` for one and
`('denied: no money was requested', 'approved: the customer asked for money back')` for the other.

**2. `_canonical` sorted dict keys.** `examples/hooks/cache.py:22-25`, the file this docstring
cites, says verbatim: "Deliberately not `sort_keys=True`: a choice question's criteria order is
positional, so two orders are two questions, and `_question_schema` in `laya/router.py` keeps
them apart for the same reason." Now `sort_keys=False`, as there. Being precise about reach: a
dataset row's `criteria` is a *list*, and `sort_keys` reorders dict keys and leaves lists alone,
so the usual shape was never folded. The flag only ever mattered for a dict-valued `criteria`,
where it did fold the order.

**3. The identity fallback was baseline-side only.** `_identity_of` reads a report's identity
from `config` *or* the top level, and its own comment says why -- "`schema` is also read at the
top level, which is where `research/evals/act_head_eval.py` puts its own tag". But
`comparable_to` applied it to the baseline only, and `laya/evals_cli.py` filtered the candidate
document down to `("config", "overall", "slices", "cases")` when building the `EvalReport`, so a
top-level `schema` was discarded before the gate could see it. The same disagreement was refused
or passed depending only on where the candidate stated it:

    candidate schema in config      laya-evals compare -> exit 1   (refused)
    candidate schema at top level   laya-evals compare -> exit 0   (passed)

`_cmd_compare` now builds the report's config from `_identity_of`, the same helper the baseline
goes through.

```
python -m pytest tests/test_evals.py -q                 74 passed, 0 failed   (was 71)
python tests/test_evals_api.py                          18 passed, 0 failed
python tests/test_evals_onnx.py                         20 passed, 0 failed
python -m compileall -q laya tests
ruff check laya/ --select=E9,F63,F7,F82,F401,F811 --line-length=120
ruff check tests/test_evals.py --select=E9,F63,F7,F82,F401,F811 --line-length=120
git diff --check
```

Reverting only the two production files turns the three new tests red, each naming its own
defect:

    AssertionError: a change in `labels` left the fingerprint unchanged
    AssertionError: assert 'a36ee1cc...' != 'a36ee1cc...'
    AssertionError: assert 0 == 1

`test_questions_fingerprint_keeps_criteria_order` pins both shapes, the list one so the guarantee
is explicit and the dict one because that is the shape the old flag actually folded.

**Merge order with NandhaKishorM#661.** NandhaKishorM#661 and this PR are line-level disjoint -- verified with
`git merge-file --diff3` against 9d95567, 0 conflict hunks in both shared files, and the merged
tree runs 72/72. But the disjointness is not behavioural, and it matters. NandhaKishorM#664 hoists
`_parse_pairs(args.tolerance)` out of `if args.baseline:`, which makes a caller mistake reachable
that used to be silently ignored:

    on NandhaKishorM#664 alone, `--tolerance choice_accuracy=abc` with no `--baseline`  -> exit 1
    on NandhaKishorM#664 + NandhaKishorM#661                                                     -> exit 2

Exit 1 is the code a quality-gate failure uses, which is the exact conflation NandhaKishorM#661 exists to
remove. **NandhaKishorM#661 should land with or before this PR**, not after it.

Also on `sort_keys`: this is the second field this fingerprint has been found to miss after
`instructions`, and the first was caught in review of the previous commit. The docstring's claim
that the fingerprint "covers every field the answer depends on" is not something a test can keep
true on its own -- a new question field will not announce itself. Worth saying so where the rule
is stated, rather than only in this commit message.
@NandhaKishorM
NandhaKishorM merged commit 62f40fd into NandhaKishorM:main Sep 29, 2026
22 of 23 checks passed
@NandhaKishorM

Copy link
Copy Markdown
Owner

Passed checks and merged. Thank you @Bruce-Yii! It ships in 0.3.22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants