Capability gate: retain scored results, enforce tool identity and directory-change forms (GPT-6 review repair); first host receipts with that review - #440

Scope

• What this PR changes:
  • It repairs the PR-H: E2E preregistration for the token-save-practice stack (frozen tasks, metrics, gate, procedure) #381 capability-gate harness. A GPT-6 cross-family review of its first host receipts returned needs_changes.
  • It publishes those first receipts with that review.
  • The re-recorded receipts follow in a separate PR and supersede the first ones.
• Base commit: 7fa3849c (Pre-push registry gate: run the three registry tests on every pushed commit #438), merged into the branch at e0e863ae; manifests/evidence.json re-registered per docs/lanes.md (the result equals git's automatic merge)
• Lane: lane:foundation
• Paths touched:
  • tools/capability-gate/: run_gate.py, assertions.js, m13_hooks.js, the jcodemunch and ai-memory configs, README.md;
  • tests/test_capability_gate.py;
  • docs/decisions/2026-09-27-capability-gate-harness.md (amendment);
  • the three first receipts under evidence/hosts/nativestack-5975wx-20260925/;
  • manifests/evidence.json (registration only) and the regenerated reports.

Why

The three receipts recorded at 55fc8d17 passed, but they claimed more than the harness verified. A read-only GPT-6 review (codex exec, gpt-6-astra at max) found this and appended its own verdict to each receipt, under its own identity, as codex_lane needs_changes.

SOTA sources

• promptfoo 0.123.1 (tag commit 34f74d34):
  • the openai:codex-sdk provider and its raw items (src/providers/openai/codex-sdk.ts:2430);
  • javascript assertions with assertion-level config;
  • the secret-field sanitizer (src/util/sanitizer.ts), read in the installed dist and confirmed by an eval.
• @openai/codex-sdk 0.153.4 (bundled): dist/index.d.ts defines CommandExecutionItem (command, aggregated_output, exit_code, status; no working directory) and McpToolCallItem (server, tool, arguments, result).
• jcodemunch-mcp 1.108.319: search_symbols returns id, file, line and signature per symbol, as observed through the MCP server on 2026-09-27.
• Each directory-changing tool's --help on the workstation, as listed in the P2 row above.
• docs/acceptance-evidence-policy.md: "Preserve the returned result" and "Confirm consequential claims through independent observations".

Evidence-class table

Local commands run

$ python3 -m unittest tests.test_capability_gate
Ran 27 tests ... OK
$ python3 scripts/component_matrix.py --write
{"flip_rule_violations": 0, "rows": 32, "status": "written"}
$ python3 scripts/new_host_grand_list.py --write
{"status": "written", "layers": 32, "winners": 66}
$ python3 scripts/validate.py
{"components": 69, "hashed_files": 7370, "profiles": 4, "receipts": 159, "status": "passed"}
$ GIT_NO_LAZY_FETCH=1 python3 scripts/host_receipts.py validate
{"receipts": 179, "status": "passed"}
$ git diff --check origin/main...HEAD   # clean

# smokes (host-private retained results; their sha256 is the 'results' field)
# at a0cbcb07 (first repair):
capability-gate ai-memory: PASS | gate 6/6 pass; ctrl-disabled 2/2 fail as predicted; ctrl-prompt 2/2 fail as predicted | 10 rows as expected, 10 distinct sessions | loki 10/10 rows match (8 MCP tool results) | results 07693e87eb3ea615de2c506b9a8501541df84290c078466b9ea5af5d2178fcfd | promptfoo 0.123.1, codex-cli 0.157.1, run 2026-09-27T22:15:54Z
capability-gate jcodemunch: PASS | gate 6/6 pass; ctrl-disabled 2/2 fail as predicted; ctrl-prompt 2/2 fail as predicted | 10 rows as expected, 10 distinct sessions | loki 10/10 rows match (8 MCP tool results) | results bcf89d5c0f918d475ace8f8530db9088b7b5aee536b490cb88beb59fb4ab1bb7 | promptfoo 0.123.1, codex-cli 0.157.1, run 2026-09-27T22:21:30Z
# at 622d54d1 (final harness):
capability-gate m13: PASS | gate 40/40 pass; ctrl-disabled 6/6 fail as predicted; ctrl-wrongroot 6/6 fail as predicted | 52 rows as expected, 52 distinct sessions | loki 52/52 rows match (236 MCP tool results) | results 89fed46b244d07bdb5ed731a838fc9201765645f6626f0d9afaaaebcf910a5a8 | promptfoo 0.123.1, codex-cli 0.157.1, run 2026-09-27T22:37:41Z
# final assertions.js re-applied to each retained results file, compared with the recorded component verdicts:
#   ai-memory 10 of 10 identical; jcodemunch 10 of 10 identical; m13 260 of 260 identical
#   (the earlier m13 smoke file, whose vars.token was [REDACTED], reproduced only 36 of 260: every own-token verdict failed)

Independent review

• The re-check. The GPT-6 reviewer resumed its own read-only thread to re-check the first repair at dc05ec64.
  • Resolved: P1, the P2 tool-identity finding and P3.
  • Partly resolved: the P2 directory-change finding. An attached -C still passed, so the verdict was RECHECK: not resolved.
  • It ran the 26 tests and the three config validations, and reported no other P0–P3 defect.
• After the re-check. 622d54d1 fixes that gap and the sentinel redaction found in verification.
• PR-head review. The same reviewer resumed its read-only thread on f025fb1e to review 622d54d1, and returned PRHEAD: ready.
  • The attached -C gap is closed: 24 direct checks rejected every form, and relative reads still pass.
  • The sentinel repair is sound: no vars.token reader remains, and the installed sanitizer redacts none of the harness's vars.
  • It found no new P0–P3 defect. It ran the 27 tests and the three config validations.
  • e0e863ae only merges Pre-push registry gate: run the three registry tests on every pushed commit #438 and re-registers the manifest.
• The review loop stays bounded. There was one review and one repair round. The PR-head review is the standing pre-merge cross-family check, not another repair round, and it found nothing to repair.
• The round-2 receipts will get fresh independent reviews covering the harness at the commit they record.

Host evidence

• The first receipts are host executions on nativestack-5975wx-20260925 (linux-wsl2-x86_64) at 55fc8d17.
• The smokes are not receipts:
  • ai-memory and jcodemunch ran at the first repair;
  • M13 ran at 622d54d1.

🤖 Generated with Claude Code