Repository navigation
ci: mask the engine version in the replay-off golden pins - #634
Conversation
…only the commit identity test_replay_off_golden compares the default JSON report and the primary sink stream as raw bytes against recorded digests. It already blanks the build identity in the report (version string, commit, fingerprint token and digest), but the engine's major, minor and patch numbers in the provenance block and the engine version in the sink header stayed in the compared bytes. The digests were recorded at 1.7.0, so the release commit that only moved VERSION to 1.8.0 turned the row red on every build leg, with the same byte counts and different digests. Mask exactly those version fields: the three numbers of the engine provenance object in the raw report, in the base64 token spelling and in the decoded provenance that is appended to the hashed bytes, and the engine member of the sink header line. Each is required to be present exactly once, so a report that loses or duplicates the field is refused. Every other byte, including trade values and numbers named major or minor elsewhere, is still compared. New real-byte tests cover: 1.7.0 and 1.8.0 shaped reports mask to equal bytes, a changed trade price or input or runtime number still differs, missing or duplicated fields are refused, and the sink body is untouched. The new test case is imported by test_run_unittest_shards.py so the existing preflight entry point runs it. The six recorded digests are re-recorded in the next commit from a measured run. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
…8.0 run With the version fields masked, the six recorded digests change shape. They are taken from the ctest row's own measured output on a CLOUD build of the tagged v1.8.0 tree plus the masking change (default JSON 9288, 8939 and 8950 bytes; sink 1565, 1451 and 1464 bytes), not edited by hand. The byte counts grew by exactly the bytes the extended mask adds (102 per default JSON, 11 per sink stream), so no other length moved. The source note names the re-recording and what is now masked. Co-Authored-By: Claude Sonnet 5.5 <[email protected]>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| import run_unittest_shards as shards_runner # noqa: E402 | ||
| # This module is an explicit preflight entry point. Import the regression | ||
| # TestCase so unittest also runs it here, without a second CI registration. | ||
| from test_check_replay_off_golden import ReplayOffGoldenTests # noqa: E402,F401 |
| from pathlib import Path | ||
| import subprocess | ||
| import tempfile | ||
| import unittest |
![graphify-labs[bot]](https://avatars.githubusercontent.com/in/4356270?s=60&v=4){kind=small}
There was a problem hiding this comment.
Graphify reviewed this change.
Worth a look — the grounded gate found no coupling regressions or blocking issues, but 1 advisory finding(s) below merit a look before merge.
Not checked: tests were not run; formal verification proved 0 of 2 changed function(s) (2 not verified).
Formal verification. PR-changed functions: 0/2 verified (0 proven, 0 may-equivalent, 0 distinguished) · 2 not verified (2 vacuous).
Not verified on this run: mask\_build\_identity (vacuous: never exercised), measure (vacuous: never exercised).
Graphify review — findings
Extends the replay-off golden mask to blank the engine major, minor and patch numbers and the sink header's engine version, so the recorded digests no longer depend on VERSION. mask_build_identity and mask_sink_build_identity raise unless each member is found exactly once, and the golden JSON is re-recorded on v1.8.0 with byte counts growing by exactly the bytes the extended mask adds. Adds ReplayOffGoldenTests with real-byte fixtures and registers the class in the unittest shard verifier's partition, where its re-export from the shard test module can surface the same test under duplicate IDs in discovery.
Worth a look
- Re-exported TestCase yields duplicate test IDs under discovery —
scripts/test_run_unittest_shards.py:27· Escalate · medium- NOT verified (no proof, no reproducing execution) — needs human review
Analysis details — impact, health, verification
Impact & health
Graphify review
Impact — 46 functions depend on the 46 functions this change touches.
Health — grade A; no new coupling hotspots.
Verification — 46 functions in the blast radius were not formally verified this run (proofs are advisory here).
Gate & verification
graphify gate
PASS — objectively clean (no health regressions, tests not run — proofs not run this pass (advisory)). Grounded, not self-assessed.
Advisory (not blocking):
- verification_scope: 46 function(s) in the blast radius were not formally verified this run
Docs that may be stale (advisory)
CHANGELOG.md§ Native live report stream (lines 168-189): references changed symbolsreportCHANGELOG.md§ Opt-in report descriptor in the container entrypoint (lines 249-264): references changed symbolsreportCHANGELOG.md§ Report sink and scratch spill (lines 400-611): references changed symbolsreportCHANGELOG.md§ run_json and CLI (lines 612-704): references changed symbolsreportCHANGELOG.md§ Run-failure codes and refusals (lines 705-790): references changed symbolsreportCHANGELOG.md§ Costs that remain (lines 954-1164): references changed symbolsreportCHANGELOG.md§ 1.6.1 — 2026-10-18 (lines 1218-1283): references changed symbolsreportCHANGELOG.md§ 1.6.1-rc.1 — 2026-10-17 (lines 1284-1337): references changed symbolsreportCHANGELOG.md§ 1.6.0 — 2026-10-17 (lines 1338-1503): references changed symbolsreportCHANGELOG.md§ 1.6.0-rc.1 — 2026-10-14 (lines 1504-1620): references changed symbolsreport
…and 10 more.
Formal verification
Could not verify: Could not verify mask\_build\_identity.
The verifier did not have enough to check mask\_build\_identity, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 260 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
Could not verify: Could not verify measure.
The verifier did not have enough to check measure, so it is saying so rather than guessing. No false assurance is the whole point.
Guarantee: No guarantee either way, this is an honest abstention, not a pass.
Note: Reason: not verifiable: all 263 sampled inputs raised on both versions — the function never executed, so 'no divergence' would be vacuous (mostly AttributeError — names the real obstacle, not a sampling gap)
Fixes the red
test_replay_off_goldenon main after the v1.8.0 release commit.The replay-off golden pins six byte digests of sink and report dumps. Those dumps carry the engine version (the
engineprovenancemajor/minor/patchand the sink header's"engine"string), so the release bump changed the digests without changing any trade or report data: the byte counts stayed identical.scripts/check_replay_off_golden.py: masks ONLY those version fields, alongside the build-identity fields it already masked. Each field must occur exactly once; no byte that carries trade or report data is masked.scripts/test_check_replay_off_golden.py(new): a 1.7.0-shaped and a 1.8.0-shaped input mask to equal bytes, and a changed report byte still fails the pin.docker/replay_off_golden.json: the six digests re-recorded from a measured run with the new mask, so the next version bump keeps them valid.scripts/test_run_unittest_shards.py: registers the new test.Verified:
test_replay_off_goldenandtest_replay_rq_fixturepass on this head in a cloud Release build.🤖 Generated with Claude Code