Keep the grand catalogs current as new hosts and runtimes record evidence - #130

Two related gaps in the new-host evidence loop:

- scripts/new_host_grand_list.py silently gave null measured_tiers when a
  native_proven hosts[] entry named a .json evidence file that did not
  exist. It now fails (exit 1) instead, and carries each host's own
  `measured` block (and nested fields such as `mlx_smoke`) through to the
  generated JSON and a compact Markdown column. A native_proven entry whose
  evidence is prose (a hosted-CI-run citation, not a report path) is not
  held to the file-exists check.

- There was no structured place to record which local model weights were
  qualified on which runtime and host. adoption/host-receipt.schema.json
  gains an optional qualified_models[] array (model_id, revision, runtime,
  runtime_version, bars, result); scripts/host_receipts.py record supports
  --qualified-model (repeatable JSON) and --qualified-models-file, and
  carries the field through build_summary(); scripts/component_matrix.py
  surfaces it per platform on the runtime component's winner row; and
  scripts/new_host_grand_list.py adds a "Qualified local models" section.
  This is informational only: it never marks anything accepted by itself,
  and the platform_status flip rule (scripts/platform_status.py) still
  governs.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

adoption/hardware-profiles.json hosts[] had no script writer and no
structural check; a measured host could only be added by hand. Add
scripts/hardware_profile.py --record-host <host-id>:

- writes the measured report to
  evidence/artifacts/hw-profiles/<host-id>/profile.json and registers it in
  manifests/evidence.json (the same host_receipts.register_file helper
  record/review use);
- adds or updates the hosts[] entry (evidence_class native_proven,
  evidence pointing at that file), keeping the entry's other fields across
  a re-record;
- enforces the <name>-<yyyymmdd> id convention host_receipts.py already
  uses for measured hosts (a labelled_projection host id still needs to end
  "-projected");
- sanitizes the report (scripts/host_receipts.py's sanitize(), which also
  drives the committed this-host.json's redaction) and refuses to write if
  it still matches PRIVATE_CONTENT afterward.

validate_hosts() is the new hosts[] schema check (required keys, a known
evidence_class, no duplicate id, the id conventions above, and --
when a root is given -- that a measured entry's evidence file actually
exists); it runs in tests/test_hardware_profile.py against both synthetic
fixtures and the real shipped hosts[].

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…nder

Landscape winners[].platform_status only moves through a
record_verdicts.py re-record; nothing previously surfaced when recorded
evidence already supports more than a row declares. The macOS flip rule
(scripts/component_matrix.py's flip-rule check, via
scripts/landscape.ENFORCED_PLATFORMS) only rejects a declaration that
claims *too much*, only for macos-arm64, and only fails CI -- it never
lists what could now claim more, and never covers linux-wsl2-x86_64.

scripts/verdict_flip_candidates.py calls scripts/platform_status.py's
platform_status() -- the one function record_verdicts.py, landscape.py and
component_matrix.py all use -- for every winner and platform in
catalogs/landscape/{foundation,us-equities}.json, and lists rows whose
derived status outranks the declared one, with the qualifying receipts.
Read-only and report-only: it selects nothing, records nothing, and exits 0
whenever the input parses, regardless of what it finds; only malformed
input is an error.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Three gaps confirmed against the real validate.yml before wiring (per the
dry-run findings): the macOS-only flip rule never reported a flip
candidate; tools/sota-convergence/gap_crosswalk.py build --check and
gap_wave_ledger.py --check existed and were tested but never ran in CI, so
a stale crosswalk or wave ledger could pass silently.

- scripts/verdict_flip_candidates.py runs as a report-only step (piped to
  the job summary); it always exits 0 on well-formed input, so it cannot
  fail the build.
- `gap_crosswalk.py build --check` passes on main as it stands; wired as an
  ordinary blocking check like the other --check steps.
- `gap_wave_ledger.py --check` needs required --wave/--owner flags (the
  task's literal invocation has none); the combination that reproduces the
  checked-in combined ledger
  (gap-wave2-20260923+gap-wave3-20260923--gap-resolution.json) --
  `--wave gap-wave2-20260923 --wave gap-wave3-20260923 --owner
  gap-resolution` -- passes on main and is wired as that exact command.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

docs/contributing-evidence.md scattered the new-host steps (bootstrap,
measure, record, refresh derived views, validate, how a flip happens)
across several sections with no single ordered list. Add "0. The new-host
loop, in one place" naming, in order: bootstrap.md; hardware_profile.py
--record-host; host_receipts.py record (--qualified-model for local model
qualifications); component_matrix.py --write then new_host_grand_list.py
--write; verdict_flip_candidates.py; the validate commands plus
`python3 -m unittest` (this repository uses unittest, not pytest, which
is not installed); and that a platform_status flip is a separate step,
done by record_verdicts.py's owner through its own lane process.

docs/next-host-stages.md gets a pointer to that section, and its two
per-host step lists are updated to name --record-host and --qualified-model
instead of the older by-hand "add the measured entry" instruction.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

python3 scripts/component_matrix.py --write, then
python3 scripts/new_host_grand_list.py --write, then
python3 scripts/evidence_manifest.py --write, plus re-registering every
source file this branch touched that manifests/evidence.json already
tracked a hash for (host_receipts.register_file, the same helper --write
uses for its own outputs). flip_rule_violations stayed 0 throughout.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

… Windows-side evidence

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…re writing

Two review findings (human + Codex, both against 48471ea) on
scripts/hardware_profile.py's cmd_record_host:

- It ran host_receipts.sanitize() over the whole serialized
  hardware-profiles.json document. sanitize() replaces every occurrence of
  $USER in the text it is given, and $USER is often a short, common word
  ("mac", "runner", "ram", "max", ...) that legitimately appears inside
  unrelated pre-existing content: an id like
  "github-macos-15-arm64-runner", or -- more seriously -- a config key like
  "max_fraction_of_ram" in the same document's own
  layers.ecosystem_bounded_run, which a whole-document sanitize renamed,
  making the *next* recommend() call raise KeyError. Now only the new/
  updated hosts[] entry's own serialized text is sanitized (and the fresh
  measured report, which has no such shared keys to corrupt); the whole
  document is only regex-scanned (never mutated) for a final defensive
  check.
- The evidence file was written and registered before validate_hosts() ran.
  Validation (over the whole hosts[] array, so a pre-existing malformed
  entry elsewhere also blocks the write) now runs first, with root=None
  (skipping the evidence-file-exists check, which cannot pass before that
  file is written) -- so a failure leaves no evidence file, no manifest
  registration and no changed hardware-profiles.json behind.

Tests cover USER in {runner, mac, ram, max} leaving every existing hosts[]
entry and the whole layers config byte-identical, a subsequent recommend()
call not raising, and a pre-existing bad entry blocking the write with
nothing written.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Codex review (against 48471ea): scripts/new_host_grand_list.py's
build_hosts() only took a host's `measured` block from the hosts[] entry
itself. scripts/hardware_profile.py --record-host never puts `measured` on
the entry (only `evidence`, a path to the report that holds it), so a
freshly recorded host's real measurements never reached the grand list --
`measured` stayed null even though the evidence file had them. Read
`measured` from the evidence file too when the entry has none of its own
(an entry's own inline `measured`, as the shipped
github-macos-15-arm64-runner entry has it, still wins). Tests cover the
fallback, the entry-wins-when-both-present case, and a full
record-then-build-hosts round trip.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…ommands

Codex review (against 48471ea): the CLI's own pre-execution check for
--qualified-model only looked at required keys and whether result was in
{pass, fail} via `not in` against a set, which raises an uncaught TypeError
for an unhashable value such as `result: []`. It also never checked
anything else the schema constrains (for example `bars`'s maxLength 400),
so an over-length bars value passed this check, ran the qualification
command(s), and was rejected only afterward by the pre-write
validate_receipt_shape() call -- after the side effect. Replace the
hand-rolled subset check with the same schema-driven validate_against_schema()
this module already uses everywhere else, run over the qualified_models[]
item subschema before any command executes; its equality-based enum check
never raises on a non-hashable value. Tests cover both failure modes,
asserting exit 2 (no traceback) and that no command ran.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Two P2/P3 findings against the CI wiring added earlier this branch:

- The verdict_flip_candidates.py | tee step had no `shell: bash`, so
  without pipefail a malformed-input exit 1 from the reporter would be
  swallowed by tee's own exit 0. Add `shell: bash`, matching the
  release_due step's existing pattern.
- "Check the gap crosswalk is current" and "...wave ledger is current"
  overstated what these --check steps prove: gap_crosswalk.py's build
  reads catalogs/landscape/*.json at a fixed historical commit
  (CURRENT_REV, 92bb279), and the wave ledger reads the checked-in
  crosswalk file, not the current catalogs. Both steps only prove the
  checked-in file still reproduces from its own fixed inputs; neither can
  detect that a layer re-recorded since CURRENT_REV made the crosswalk
  stale. Renamed to say so.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

P3 finding: the reproducibility of gap_crosswalk.py build --check and
gap_wave_ledger.py --check from their own fixed inputs (CURRENT_REV
92bb279 and the checked-in crosswalk, respectively) is not the same as
detecting that either has gone stale against the *current*
catalogs/landscape/*.json open_gaps. Section 4 now says so explicitly, so
a reader does not read the CI wiring as more than it is.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…ection

P3 finding: "until python3 scripts/hardware_profile.py runs on each host
and replaces them" implied the labelled-projection hosts[] entry
disappears once a host is measured. --record-host adds a separate, dated
native_proven entry alongside it; the projection entry is neither removed
nor overwritten. Reworded to say to prefer the measured entry once one
exists, not to expect the projection to vanish.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…n't make

P3 finding: 48471ea's decision block set memory=112GB but the evidence
string still said "the two records need reconciling by their owner; this
manifest does not choose the value" -- stale now that a decision exists.
Reworded to state the decision and point at the hardware-profiles.json
owner's own record of it (catalog PR #125) without this manifest claiming
to have updated that entry itself.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

python3 scripts/component_matrix.py --write, then
python3 scripts/new_host_grand_list.py --write, then
python3 scripts/evidence_manifest.py --write, plus re-registering
evidence/artifacts/host-upgrade-20260923/upgrade-evidence.json (the only
source file this round's fixes touched that manifests/evidence.json
already tracked a hash for). component-evidence-matrix.json was already
byte-identical (no new receipts this round). flip_rule_violations stayed 0.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Discovered while regenerating after rebasing onto catalog PR #125:
adoption/hardware-profiles.json's wsl-workstation-128gb-projected
workflow_concurrency_cap is now "min(16, 60 - 2) = 16 (nproc inside WSL is
the .wslconfig processors=60)". tier_value()'s rsplit("=", 1) took the
*last* '=' unconditionally, which lands inside that trailing explanatory
note ("processors=60)"), rendering the grand list's concurrency-cap column
as "60)" instead of "16". Strip one trailing, balanced parenthetical note
first (matching parens from the end, so it only touches a string that
actually ends in ")"); a string with several '=' but no trailing note
(the existing ecosystem_job_memory_max_gb style) still takes the result
after the last '=', unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Round-2 review follow-up (P2): the previous fix still ran
host_receipts.sanitize() over the whole serialized report and entry text.
sanitize() replaces every occurrence of $USER in the text it is given,
which still corrupted the *keys* of the new report/entry themselves: with
USER=ram, "effective_ram_gb" and "ram_gb_proc_meminfo" became
"effective_<user>_gb"/"<user>_gb_proc_meminfo"; with USER=gpu or
USER=cores, the bare "gpu"/"cores" keys became "<user>"; a preserved field
kept from a re-recorded entry (for example "ecosystem_job_memory_max_gb")
was equally at risk under USER=max. The grand list then silently dropped
RAM, GPU or cores for that host.

Add sanitize_values(), which walks the decoded report/entry the way
host_receipts.iter_receipt_strings() walks a receipt for its read-only
privacy scan, but only sanitizes each dict *value*/list *item* that is a
string -- never a dict *key* -- so JSON structure survives regardless of
$USER. Use it for both the report and the entry (including whatever an
existing entry's preserved fields are) in place of the prior
serialize-then-sanitize-the-whole-text approach.

Tests cover USER in {ram, max, gpu, cores, runner, mac}: the written
profile.json's and entry's own keys stay unchanged, and a key preserved
from a prior entry across a re-record survives too.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Round-2 review follow-up (P3): text ending in "(...)" with no separate
trailing note -- e.g. "cap = min(16, 30)" -- was treated the same as a
genuine explanatory note and stripped, returning "min" instead of
"min(16, 30)". Only drop a trailing "(...)" when it is set off by a space
(a real note, like "16 (nproc inside WSL is ...)"); one glued directly
onto a preceding token with no space (a function call, part of the result
expression itself) now stays. The two compose correctly: a genuine
trailing note after a parenthesized result is still dropped, leaving the
parenthesized result intact.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…lear reason

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>