Skip to content

Record cross-family layer verdicts (blind Claude and Codex lanes, counterbalanced adjudication) - #79

Merged
seathatflowsinourveins merged 5 commits into
mainfrom
claude/codex-lane-20260922
Sep 23, 2026
Merged

seathatflowsinourveins merged 5 commits into
mainfrom
claude/codex-lane-20260922

Conversation

@seathatflowsinourveins

@seathatflowsinourveins seathatflowsinourveins commented Sep 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

This PR records the September 22 layer verdicts from two independent model-family lanes on the same stripped packets, plus the tooling each lane needed.

  • Claude lane (saved layer-verdict-lane workflow): each layer ran an Opus 5.5 proposer, two refuters and one revision.
  • Codex lane (codex_lane.py): one read-only codex exec call per layer at high effort.

Tooling

  • codex_lane.py passes Codex a strict-output copy of the lane schema. Codex strict mode rejects uniqueItems, $schema, $id, title and description. The copy drops those keywords but keeps property names and instance data, and record_verdicts.py still enforces every dropped rule.
  • lane_packets.py --withhold-labels drops the labels that revealed the incumbent (candidate and SOTA-component review_status, decision selection and review_status) from ledger-built packets. Manifest-built trading packets stay byte-identical.
  • record_verdicts.py enforces the counterbalanced adjudication rule:
    • an adjudicated winner is accepted only when the recorded judgments cover both presentation orders and all chose the same lane with no refuting vote;
    • a split is sealed as evidence, and the row stays pending_lanes with its tally.

Result

  • Recorded: 30 of 32 layers.
    • 20 layers are same_winner: 13 foundation and 7 trading.
    • 10 layers were decided by anonymous, order-swapped adjudication: Claude 7, Codex 3.
  • Contested: execution-broker and observability-hosting stay pending_lanes. Every judgment followed the presentation order, a 2-2 split over four judgments. An executed comparison must decide them.

Stated limits

  • Label exposure: the checkout the lanes read kept prior labels in catalog files. Every foundation return in both lanes lists at least one label-bearing file, and 19 of 20 per lane list decisions.json. Three trading returns in each lane list one. The lane packets and the scrubbed adjudication inputs are retained, so these claims can be checked.
  • Adjudicator family: the adjudicators are Opus, the same family as the Claude lane.
  • Process-wording cue: five Claude adjudication inputs keep wording that only the Claude lane's pipeline produces. Three of those layers went to Claude, one to Codex and one to neither.
  • Evidence strength: of the 30 recorded rows, 16 are native_proven, 7 local_integration, 4 synthetic and 3 source_review.

Review

  • Independent Opus review (fix_required: 2 major, 6 minor) and Codex review (1 major, 2 minor). All findings are addressed in ff417c7.
  • The five stale handbook summary rows are now guarded by tests/test_handbook_summary.py, which fails on exactly those rows of the old table.
  • Re-reviews of ff417c7:
    • Opus: fix_required. Its one major was that the label-exposure disclosure was incomplete.
    • Codex: no blocker or major, 4 minors.
    • All are resolved in 6581cfb. A final Opus verification of 6581cfb is recorded in the comments.

Verification (local, this head)

  • record_verdicts.py --check with the lane roots and adjudications: 0 rejections.
  • build_verdicts.py --check, scripts/landscape.py, scripts/validate.py, validate_convergence.py --all-recorded, build_ecosystem.py --check and trading_gates.py --check: all exit 0.
  • python3 -m unittest: 2067 tests OK, 324 skipped.
  • Guarded gitleaks on the branch range: 4 commits, no leaks.

🤖 Generated with Claude Code

…s for ledger-built packets

codex_lane.py passes Codex a copy of the lane schema without keywords its strict structured output rejects
(uniqueItems returned HTTP 400 invalid_json_schema with codex-cli 0.155.1); record_verdicts.py still enforces them.
lane_packets.py --withhold-labels drops candidate review_status and decision selection/review_status from
ledger-built packets: in the 2026-09-22 foundation packets confirmed_default and selection=default occur only on
selected candidates. Manifest-mode trading packets are left byte-identical. Default output is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
… counterbalanced adjudication

Both lanes judged the same 32 packets (lane_packets.py --trading-candidates manifest --withhold-labels) against a
checkout of main with every earlier verdict removed. Claude lane wf_dcc60c7a-e87: Opus 5.5 proposer, two refuters,
one revision per layer. Codex lane codex_lane.py: gpt-6-astra, high effort, read-only, 32 layers, 0 failed
attempts. Result: same_winner in 20 of 32 layers. The 12 disagreements were adjudicated from anonymized returns
(host paths and model names scrubbed; one Opus evidence-reviewer judge and two refuters), then re-judged with A/B
swapped; 10 layers chose the same lane in both orders with no refutation and are recorded from that lane's return
(Claude 7, Codex 3), each adjudication sealed under evidence/artifacts/layer-verdicts-20260922/adjudication/.
execution-broker and observability-hosting split 2-2 by presentation order over four judgments and stay
pending_lanes with both sets recorded. The handbook's production note, rerun steps and limits describe the method
and its limits (adjudicators share the Claude lane's model family). python3 -m unittest Ran 2052 OK.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ndness limits

record_verdicts.py now accepts an adjudicated winner only when the file's
judgments cover both presentation orders and all chose the same lane with no
refuting vote; a split is sealed and the row stays pending_lanes with its tally.
The 12 adjudication records now carry every judgment's order and pick, so the
execution-broker and observability-hosting 2-2 splits are checkable in-repo.

codex_lane.py keeps schema property names that match dropped keywords and keeps
instance data verbatim (the real lane schema's strict copy is unchanged); the
dry run says it writes nothing. lane_packets.py help text matches its behavior,
with a test for SOTA-component review_status.

Handbook: the five stale summary rows now match the ledger (guarded by
tests/test_handbook_summary.py); 30 of 32 recorded; both lanes described; the
foundation lanes' exposure to decisions.json selection labels, the residual
process-wording cue and the four-component observability difference are stated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…puts, exact guards

The label-exposure limit now covers every label-bearing catalog file the lanes
read (20 of 20 foundation returns per lane; 2 Claude and 3 Codex trading
returns), the full 54-decision selection breakdown, the packets' adopted flag
and what a blind checkout must strip. The 32 lane packets (SHA256SUMS matching
all 64 sealed returns) and the scrubbed adjudication inputs with their lane key
are retained, so the three adjudicator-visible process cues are checkable.

record_verdicts.py requires an explicit winner_lane key (tests for a missing
key, a boolean refuting_votes and non-list judgments). codex_lane.py keeps
names under dependentRequired and dependencies. The summary-table guard now
parses every name in a cell, so an unlisted extra name fails. The tools README
no longer says the recorded run used unwithheld packets.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…receipts

The retained adjudication inputs hold five cue-bearing Claude returns, not
three: instructions-skills, observation-inference and web-research went to
Claude, code-navigation to Codex, observability-hosting to neither. Trading
exposure counts the adaptive-paper receipt's decision (three returns per lane),
and the blind-checkout step also strips blueprint JSON labels.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins
seathatflowsinourveins merged commit 92bb279 into main Sep 23, 2026
7 of 8 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/codex-lane-20260922 branch September 23, 2026 00:02
seathatflowsinourveins added a commit that referenced this pull request Sep 23, 2026
… statuses, pinned source text

- ai-memory rebind: the first run's status/search reads went to the live server
  (default server_url), so they compared the live store with itself. Re-ran with
  a fresh restic restore and two disposable servers, each read sent to its own
  server; restore byte-identical, counts and top results equal.
- DVC restore: re-ran with live capture instead of a reconstructed log and added
  the executed baseline arm. DVC restores snapshot.json byte-identically;
  re-materializing changes snapshot_sha256 through ingested_at.
- financial-data offline regression -> local_integration; Dagu kill direction ->
  inconclusive (1-2 s window, before the 30 s/90 s thresholds).
- Ledger: eight gaps whose receipts did not address them move to not_settled;
  isolation[4] summary corrected; every gap carries its verbatim bdd04ca text
  and the source file hashes, since #79 rewrote all open_gaps.
- Receipts and a blueprint: username-bearing scratch paths removed.
- Tests: status recomputed from receipts unless a coordinator note says why;
  source text checked against the source revision; host-path scan over
  receipts and blueprints.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Sep 23, 2026
…l 32 layers (#81)

* Requalify worktrunk 0.79.0, probe its crash-kill cleanup, and run difft on a real multi-file range

- worktrunk-0-79-0-requalify: mirror the frozen 0.78.0 owned-worktree qualification
  (create/list/remove/dirty-refuse/cleanup) byte-for-byte against the pinned 0.79.0
  binary; passed end to end (git-github-automation gap 2).
- worktrunk-crash-cleanup-probe: 5 create-then-SIGKILL trials each for wt and native
  `git worktree`, using the critic-corrected `wt switch --create ... --format=json`
  invocation; all 10 kills landed before any on-disk worktree registration was
  written, so both tools' remove/prune paths correctly report nothing to clean up
  -- a genuine but narrower result than the mid-write crash case the gap asks about
  (workers gap 5, git-github-automation gap 3; settles_gap: partially, inconclusive).
- difftastic-larger-fixture: ran pinned difft 0.71.0 over all 20 files changed in
  native-agent-stack-token-practice-review 231077d..bdd04ca (json/md/py/html); 20/20
  exit 0, parsed, non-empty (git-github-automation gap 4).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Resolve storage-market unit gaps: DuckDB/financial-data regression, XNYS
multi-year holiday cross-check, sdk-env-baseline rebuild, DVC restore
comparison, delta-rs header run

storage-compute: reran tests.test_catalyst_dataset (14/14, byte-identical
repeat materialization) and tests.test_financial_data (16/16, offline
fail-closed contracts) in a freshly rebuilt, hash-checked SDK-lock
interpreter. SEC EDGAR retry recorded as needs_user_input per this unit's
binding instruction (no live EDGAR query attempted, even with a
project-URL-only User-Agent).

market-data-reference: extended gap 5's single-session exchange_calendars
spot check to a full 2016-2027 XNYS holiday/half-day cross-check against
pandas_market_calendars 5.4.0 (independent NYSE calendar implementation);
1 boundary-methodology disagreement found and explained (exchange_calendars
realized first_session clipping vs. the requested range start), zero
substantive rules disagreements.

identity-provenance: rebuilt adoption/sdk/requirements-linux-x86_64-py313.lock
in a new hash-checked uv prefix and ran the overturn_protocol baseline suite
(tests.test_nanosecond_replay + tests.test_point_in_time, 25/25, zero
failures/skips); installed DVC 3.67.1 into a new prefix and ran
init/add/status/checkout end to end over the project's own materialized
fixture, showing byte-identical restore after deletion versus the baseline
arm's own scripts, which grep confirms have no restore/backup mechanism;
installed deltalake 1.6.4 + pyarrow 25.0.1 (uv pip install form, per the
critic's correction) into a new prefix and ran the catalog's own delta-rs
write/read snippet, confirming it executes as documented.

All new tool prefixes are new, isolated installs under
~/.local/share/codex-ecosystem/tools/; no existing pinned install was
replaced. Evidence receipts sanitized of personal home paths and the
session scratchpad UUID per scripts/validate.py's hygiene checks;
manifests/evidence.json rehashed. Full python3 -m unittest suite: 1999
tests, OK (skipped=324, pre-existing/unrelated).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Add native evidence for the dagu-inflight-resume-after-kill and dagu-2170-checkpoint-parity gap checks

dagu-inflight-resume-after-kill (data-quality-orchestration gap 0): a
disposable two-step Dagu 2.16.6 DAG was started, SIGKILLed mid-step, and
recovery was attempted via both `dagu start --run-id <same>` and the
documented `dagu retry --run-id`. Neither could resume or restart the run:
the killed process left a stale on-disk unix-socket lock that SIGKILL never
cleans up, so Dagu reports the run as permanently "running"/"already
running" and refuses both recovery paths within the window tested. This
settles the gap with a stronger (negative) result than the base hypothesis:
not only is there no intra-step resume, the entire run becomes stuck.

dagu-2170-checkpoint-parity (scheduling-supervision gap 0): downloaded and
checksum-verified the official upstream Dagu v2.17.0 linux_amd64 release,
installed it to a new isolated prefix, and ran the SAME accepted
checkpoint/stop/retry job-recovery fixture (unmodified fixture.py) against
both the pinned 2.16.6 binary and 2.17.0. Both produced identical status
transitions and the exact same checkpoint sha256 as the previously accepted
2.16.6 record, so no regression or behaviour change was observed on 2.17.0
for this fixture.

Receipts follow the schema in evidence/artifacts/gap-resolution-20260922/;
manifests/evidence.json was updated and rehashed via
codex-ecosystem/state/grand-catalog-20260922/tools/rehash_evidence.py.
scripts/validate.py and the full unittest suite (1999 tests, OK,
skipped=318) both pass.

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* Execute the research-factors-ml/portfolio-risk gap-resolution checks

Rebuilds the receipt's hash-locked Python 3.13.15 environment offline
(uv --require-hashes against requirements.lock, 19 distributions matching
the accepted skfolio 1.2.9 receipt) in a disposable prefix and reruns:

- repo-local-full-test-rerun: the 12 focused research_evaluation tests
  still pass; the repo's full suite has grown from 214 to 1999 tests in
  three days and now depends on 'requests', which the bare pinned
  SDK_PYTHON interpreter lacks (6 unrelated errors in
  test_broad_universe_scan.py) while system python3 (matching the repo's
  actual CI interpreter) passes clean. Reproduced twice, deterministic.
- sklearn-timeseriessplit-matched-arm (c22): a new runner
  (blueprints/gap-resolution-20260922/sklearn-timeseriessplit/run_c22.py)
  executes scikit-learn's TimeSeriesSplit(gap=6) on the SAME frozen
  plan.json/evaluate.py helpers as the skfolio WalkForward receipt; the
  reserved-2021Q1 selection and its net-cost-proxy label match exactly,
  while development fold counts differ (19 vs 20, scikit-learn has no
  reduce_test equivalent) -- the first executed cross-library comparison
  for this layer.
- skf_walkforward_optimizer: a new runner
  (blueprints/gap-resolution-20260922/skf-walkforward-optimizer/run_optimizer.py)
  runs real skfolio MeanRisk and HierarchicalRiskParity optimizers over
  the same 252/63/6 WalkForward split and a reserved-2021Q1 fit/predict,
  producing actual portfolio weights and out-of-sample returns -- refutes
  the prior "no skfolio optimizer has been executed" claim for
  portfolio-risk gap 0.

Registers the three receipts and two helper scripts in
manifests/evidence.json and rehashes. scripts/validate.py and
python3 -m unittest pass.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record secrets-upstream unit checks: gitleaks history/dir scan, attestation recheck, upstream currency sweep

Runs the three gap-resolution-20260922 checks assigned to the secrets-upstream
unit (secrets-credentials, workers, document-retrieval, scheduling-supervision,
token-efficiency, evaluation-experiments, git-github-automation):

- gitleaks-repo-history-scan: guarded `gitleaks git` (615 commits, 1.27GB,
  45.6s) found one benign synthetic literal in tests/test_lane_packets.py;
  guarded `gitleaks dir` over the live worktree found none, with
  docs/ecosystem/index.html (13MB) skipped under the 2MB size cap, recorded
  as incomplete coverage.
- attestation-endpoint-recheck: gh api still returns 404 for the gitleaks
  8.30.1 linux_x64 asset and checksums-manifest digests, and the current
  release.yml has no attest-build-provenance step -- reconfirms the existing
  gap-4 finding unchanged.
- upstream-currency-sweep: one batched, read-only gh api snapshot across 14
  repositories (OpenHands software-agent-sdk, context-hub, dagu, headroom,
  phoenix, inspect_ai, qmd, poppler_mirror, markitdown, worktrunk, beads,
  gitleaks, syft, grype) per the critic's cross-layer grouping; reconfirms
  Dagu/Headroom/OpenHands are stale against upstream, confirms the inspect_ai
  pin-discrepancy text is accurate, and surfaces one new markitdown 0.1.7 vs
  0.1.8 currency gap.

Receipts and their preregistrations, commands, evidence classes and
limitations are under evidence/artifacts/gap-resolution-20260922/; the
currency-sweep helper script is under blueprints/gap-resolution-20260922/.
manifests/evidence.json files[] is rehashed accordingly.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Add gap-resolution evidence for token-efficiency and observation-inference (unit token-observation)

Four preregistered checks with retained native output: headroom-ai==0.38.0
installed into a new isolated uv-tool prefix confirms exact MCP
compress/retrieve recovery (sha256-matched) and the critic's finding that
there is no `headroom compress` CLI subcommand, without touching the pinned
0.37.0 install. RTK 0.49.0 proxy/recall exact-recovery and error-path
exit-code preservation hold on a second, independently constructed
git-fixture repo and command. ccusage 20.0.24 run over this host's real
Claude logs shows a session keyed by a workflow run id whose four token
counters exactly match an independently computed child-usage.mjs sum for
that same run, settling the combined child/subagent attribution half of
gap 4 (per-child/per-retry breakdown remains open). The offline
observability/alerting regression suite (21 tests) still passes, which
reconfirms but does not by itself close the named comparison gap against
alternative observation stacks.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Add retrieval-memory unit receipts: executed hybrid retrieval citation, isolated qmd bench attempt, ai-memory deletion-erasure probe, and 2.3.2 scope reread

Four preregistered checks for gap-resolution-20260922 (document-retrieval,
durable-memory, agents-models-workers layers):

- cite-executed-retrieval: reads agent-lab's two dated
  document-retrieval-preregistration-v2 runs (20260922T055714Z incomplete,
  20260922T060810Z complete) read-only; confirms a sealed-snapshot hybrid-vs-BM25
  comparison exists and shows hybrid ahead on recall_at_3/recall_at_1/mrr, with
  latency_p95_ms as the sole remaining closure blocker (decision D12).

- qmd-repeated-bench-isolated: built a disposable qmd 2.8.3 index
  (never registered in the live native-agent-stack-catalog index) over
  blueprints/us-equities and attempted the frozen 12-query bench fixture.
  Only query 1/12 was reached before hybrid mode's query-expansion step threw
  DisposedError in node-llama-cpp after ~5 minutes, and full mode stalled for
  a further ~8 minutes before the job was terminated; 3x cold + 3x warm
  repeats were not feasible given this non-completion under host load. Full
  crash trace and partial timings retained in the receipt.

- deletion-erasure-probe: confirmed ai-memory 2.3.2 delete-page is a logical
  delete, not byte erasure -- the deleted page's unique marker survives in
  db/memory.sqlite-wal even after a clean server shutdown. Also documents a
  process-safety finding: --data-dir alone does not isolate ai-memory CLI
  commands from an already-running server on the default port/AI_MEMORY_SERVER_URL;
  the first attempt inadvertently wrote to the live store and was immediately
  remediated (delete-page + purge-project --confirm) before rebuilding the
  probe on a genuinely isolated instance (distinct --bind port +
  AI_MEMORY_SERVER_URL).

- ai-memory-local-scope-reread: confirms the installed version is 2.3.2 (not
  the v2.3.1 the us-equities catalog row names), and that the CLI prints an
  explicit resolved workspace/project scope line on every call when a
  .ai-memory.toml marker is present. Config checked first for an LLM
  provider; none configured, so the auto-improve trial was correctly skipped
  and that decision recorded rather than silently omitted.

manifests/evidence.json: registered and rehashed the four new receipt paths.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Add the beads dead-worker lease-recovery probe receipt (gap-resolution-20260922)

Exercises bd 1.3.0's documented `bd reclaim` stale-lease recovery path in a
disposable scratch database: a live lease is correctly left untouched by
`bd reclaim --older-than 0s`, and once real elapsed time exceeds the
compiled 5-minute DefaultLeaseTTL, the claim reverts to open/ready and the
issue is claimable again. Moves catalogs/foundation/decisions.json
id=dependency-task-queue's "distributed lease correctness" limitation from
not_established to a single native execution.

Also records a genuine usage trap found along the way: combining bd update
--claim with an explicit --assignee (the triage's own proposed command
form) grants the lease to the CLI actor identity, not the stated assignee,
and left that pilot issue permanently unreclaimed past its true TTL --
root-caused by reading gastownhall/beads source at the exact pinned commit
(f45b249ce6b40ba62aecc03949e6371e8f7c79d8 == bd 1.3.0), not further
live experimentation on the same issue.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record promotion-gate fixture suite, frozen-fixture rerun and alpaca wire-serialization receipts (unit promotion-exec)

Three preregistered checks for data-quality-orchestration/execution-broker
gaps, per codex-ecosystem gap-triage.json and its critic corrections:

- promotion-gate-fixture-suite: ran tests/test_promotion_gate.py through the
  gate's own pinned isolated venv; all 18 tests pass including the previously
  skippable FixtureGateRuns class (0 skips), settling gaps 4/5.
- rerun-frozen-fixture-suite: re-ran all five pytest fixture_paths named in
  execution-broker's overturn_protocol (159 tests, OK, skipped=40, no
  regression); per critic correction this only reconfirms synthetic coverage
  and does not close peer_owned/RTH-gated gaps 2/6. Also surfaces a real
  finding: paper-receipt.json's declared runner_sha256 no longer matches
  paper_runner.py at HEAD (the runner was edited after that trial ran), and
  an unresolved discrepancy against the triage's cited prior test count
  (127 vs. this run's reproducible 150) for the same four-file combination.
- capture-alpaca-wire-serialization: extended sdk_probe.py's network-isolated
  pattern with a source-traced reproduction of requests' own
  PreparedRequest.prepare_body(json=...) call (the exact function alpaca-py's
  TradingClient.post/_request path invokes) to capture literal wire JSON
  bytes for 9 qty/limit_price/notional edge cases; confirms decimal-text loss
  (e.g. "100.10" -> 100.1) at the literal byte level, not just object field
  types, and that alpaca-py's SDK model performs no local rejection of
  zero/negative qty.

manifests/evidence.json updated with the 7 new files and rehashed via
codex-ecosystem's rehash_evidence.py; scripts/validate.py passes; full
python3 -m unittest suite stays OK (1999 tests, 324 skipped, no failures).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct difftastic fallback-detection bug and crash-probe overclaims

- difftastic-larger-fixture: the original run_difft.sh fallback check only
  matched hard-error strings, never difftastic's actual fallback header
  ("<path> --- Text" / "... --- n/m --- Text (exceeded
  DFT_BYTE_LIMIT|DFT_GRAPH_LIMIT)"), so every size-limited or
  Markdown-unsupported file was miscounted as a successful structural parse.
  Added reanalyze_reports.py, which reprocesses the same already-retained
  difft reports (no re-invocation of difft) with a corrected detection rule:
  15/20 files parsed structurally, 5/20 (both large JSON catalogs, both
  Markdown docs, the large HTML page) fell back to plain text. Corrected the
  receipt's files_parsed_ok/files_failed, settles_gap (false, not
  "partially"), and verdict_impact.direction (inconclusive, not
  "supports_incumbent"), keeping the original raw artifacts and adding the
  corrected ones alongside them.

- worktrunk-crash-cleanup-probe: corrected settles_gap from "partially" to
  false. An at-exec (Popen-return) SIGKILL lands before either wt or git
  worktree add has written any on-disk registration in 10/10 trials, so the
  probe never exercised the mid-write crash state either gap asks about and
  cannot discriminate the two tools' cleanup behavior. direction
  "inconclusive" was already correct. Also added a preregistration_timing
  field noting the preregistered text has no earlier timestamped file
  predating the raw artifacts (crash-probe-out/ and the receipt/commit are
  same-day, same-commit), so it documents the trial design's decision rule
  rather than a demonstrated advance prediction.

Registered the new blueprint script in manifests/evidence.json and reran
rehash_evidence.py (4 files rehashed, 0 missing). scripts/validate.py and
`python3 -m unittest` both pass (1999 tests, skipped=324).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

Co-Authored-By: Claude Sonnet 5 <[email protected]>

* Fix round: correct raw-artifact hash, out-of-ancestry attribution, evidence_class and OpenHands reconciliation gap in secrets-upstream receipts

Resolves four review findings on the secrets-upstream unit's committed
receipts without re-running any settled check:

- gitleaks-repo-history-scan.json: corrected the git-history.stderr.log
  raw_artifacts sha256 (it previously recorded dir-worktree.stderr.log's
  hash). Corrected the finding's attribution: commit 34fc51b is confirmed
  NOT an ancestor of this branch (`git merge-base --is-ancestor` fails,
  `git branch -a --contains` shows only sibling branches
  claude/gc-pr5-packets and claude/grand-catalog-pr5). Cited .gitleaks.toml's
  own prior HEAD-ancestry measurement (376 commits, 0 findings, recorded
  before this unit's branch point) instead of claiming git-history scanning
  was previously unmeasured, and explained the 615/542/491/738 commit-count
  spread as .gitleaks.toml's own documented non-stable-invariant behavior in
  this shared, concurrently active repository.

- upstream-currency-sweep.json (all six layer copies) and
  attestation-endpoint-recheck.json: reclassified evidence_class from
  native_proven to source_review, matching tools/sota-convergence/README.md's
  definition (read-only `gh api` metadata lookups install, build or run
  nothing) and distinguishing them from the gitleaks-repo-history-scan
  receipt, where gitleaks itself was actually executed.

- upstream-currency-sweep.json: added the previously omitted OpenHands
  software-agent-sdk revision-reconciliation sub-check (read-only gh api,
  preregistered before running). Result: the v1.49.2 tag resolves to a third,
  distinct commit (d128a786...); 856d99d48e4b11c70c5f1cab21e7830570dbc324 is
  a direct 7-commit ancestor of 9bc452ebf6b0093c531f25394c5ba5e9817ff910, so
  the two cataloged revisions are reconciled with each other on one lineage,
  though neither is current relative to HEAD or the latest release.

Rehashed manifests/evidence.json. `python3 scripts/validate.py` passed
(69 components, 1831 hashed files, 140 receipts). `python3 -m unittest`
passed (1999 tests, OK, skipped=318).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct beads-lease-recovery-probe's root-cause claim

The receipt, run.sh, and result/limitations text said combined `--claim
--assignee <name>` "grants the lease to the CLI actor identity rather
than the stated assignee". That is wrong: upstream source at the pinned
commit (f45b249ce6b40ba62aecc03949e6371e8f7c79d8) shows --claim arms a
lease for the actor, but the same call's --assignee value then runs
through the generic-update lease manager (ManageLeaseOnUpdate,
internal/storage/issueops/update.go), which deletes the lease row
whenever a generic update changes who holds an in_progress claim
(DeleteLeaseInTx). The pilot issue was left in_progress with no lease
row at all, and `bd reclaim`'s own query (ReclaimExpiredLeasesInTx,
internal/storage/issueops/lease.go) is an inner join against the
leases table, so an issue with no lease row can never be selected --
a permanent exclusion from recovery, not a transient miss at the
pilot's 0s-grace/~59s-past-TTL window.

Corrected the claim-response, reclaim-response, result, and
verdict_impact.explanation text in the receipt, and run.sh's comment
block, to state the confirmed mechanism. Added a new first limitation
recording that dead-worker recovery only covers issues claimed via
the bare lease-arming `--claim` verb, and that any issue whose
in_progress claim was established/transferred through a generic
update (including the triage's own proposed --claim --assignee form)
is permanently unreachable by reclaim -- to be recorded in
catalogs/foundation/decisions.json id=dependency-task-queue's
limitations text before that text is narrowed on this probe's
positive result. No check was re-run; only the receipt's causal
explanation and limitations were corrected against independently
re-verified upstream source at the same pinned commit.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct overstated dagu-inflight-resume-after-kill claims

Addresses three major findings from the coordinator's review of
dagu-inflight-resume-after-kill.json:

1. The receipt claimed the killed run was "permanently"/"indefinitely"
   stuck, but retries were only attempted ~1-2s after the SIGKILL. A
   static inspection of the pinned dagu-2.16.6 binary's own embedded
   config schema shows a lock_stale_threshold (default 30s) and a
   process-heartbeat stale_threshold (default 90s) that were never
   reached in the tested window. Removed the permanence language,
   changed settles_gap from true to "partially", and narrowed
   verdict_impact.explanation to the claim actually supported: no
   immediate/automatic resume after an unclean kill, not permanent
   unrecoverability. Added the config-schema inspection as a third
   command and a limitation naming the untested longer-delay/`dagu
   stop`/socket-deletion recovery paths, without re-running the
   kill/retry experiment itself.

2. The causal claim that Dagu's liveness check treats the stale
   socket file's "mere presence" as proof of a live run was stated as
   established fact. No test isolated the socket file from the
   separate per-run proc/heartbeat record observed under
   dagu-home/data/proc/inflight/... (pid + pidStartedAt) or the
   persisted history status. Relabelled this as an explicit,
   untested hypothesis in both the result and the raw-command
   excerpt.

3. The "preregistered" field's provenance could not be shown to
   predate execution: an untracked, abandoned earlier-draft script
   (run_inflight_kill.py, never committed, not part of the
   registered evidence) carried its own differently-ordered
   "preregistered" protocol comment shortly before the committed
   fixture.py was written, with no retained run output proving it
   executed. Added a limitation disclosing this so "preregistered"
   here is read as self-declared-in-receipt rather than externally
   verifiable, without fabricating a corrective commit ordering.

manifests/evidence.json rehashed via
codex-ecosystem/state/grand-catalog-20260922/tools/rehash_evidence.py.
scripts/validate.py passes; the full unittest suite (1999 tests,
OK, skipped=324) passes.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct retrieval-memory unit receipts (residue disclosure, numeric errors, version-citation misread)

Resolves four review findings against the retrieval-memory gap-resolution
receipts, without re-running any check to get a different outcome:

- deletion-erasure-probe: discloses residue in the live ai-memory store the
  original receipt omitted (four new commits in wiki/.git history that still
  recover the deleted page's full body via `git log --all -p`/`git show`; a
  leftover scope-manifest directory from purge-project; 2 lifecycle-hook
  observation rows in memory.sqlite containing probe tokens), all found via
  new read-only checks. Corrects the "clean shutdown" claim: the disposable
  server's log ends at "shutdown signal received; draining" with no
  completion line, .serve.lock.holder was left behind, and the WAL was never
  checkpointed. "Not erasure" conclusion stands; "fully remediated" framing
  is retracted.
- qmd-repeated-bench-isolated: fixes a fabricated "~1000x" bm25 slowdown
  claim (bm25 at 4ms was actually faster than the 5ms baseline, not slower),
  corrects the "full mode ran ~8 more minutes" claim to the ~192s the
  recorded duration_s actually supports, and discloses that this run's own
  concurrent disposable ai-memory serve process (embedding-model fetch)
  overlapped the bench window, not only unrelated peer sessions. Downgrades
  settles_gap to "no" and direction to "inconclusive" per the corrected,
  non-comparable, non-completing single run.
- ai-memory-local-scope-reread: retracts the false claim of three
  inconsistent ai-memory version citations. The "2.1.277"/"2.1.278" figures
  in gap 8's text are Claude Code client versions (confirmed against
  catalogs/landscape/foundation.json and the cited native-memory receipt),
  not ai-memory versions; only the single v2.3.1-named-vs-2.3.2-installed
  discrepancy is supported.

manifests/evidence.json hashes regenerated via rehash_evidence.py.

rtk proxy python3 scripts/validate.py -> status passed.
rtk proxy python3 -m unittest -> 1999 tests, OK (skipped=324).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct storage-market receipt overclaims (DVC direction, delta-rs settlement, post-hoc preregistration disclosure)

Corrects four major findings without re-running any check (the checks
themselves were not shown to be invalid, only their recorded claims):

- dvc-arm-restore-compare.json: direction was inverted. DVC (data-dvc)
  is catalogs/landscape/us-equities.json's identity-provenance winners[0]
  (evidence_class source_review); a successful DVC run supports that
  already-adopted candidate, not refutes_incumbent. settles_gap changed
  from true to "partially": gap 0 (DVC unexecuted) is settled, but gap 1's
  overturn_protocol metric (per-cutoff selection/SQL hashes equal to the
  baseline run; exit_code 2 rejection preserved) was not verified, only
  whole-file sha256 identity. Corrected the commands array to include the
  cp -a/chmod -R u+w/git config steps that actually ran but were omitted,
  and the grep command's real cwd/relative paths. Disclosed that the
  dvc-arm-restore-compare.log raw artifact was written via the Write tool
  after the run (a reconstructed transcript matching the real tool
  results, not a live capture), and that the baseline-arm "comparison"
  was only a static grep, not an executed delete-then-rematerialize run.

- delta-rs-header-run.json: a single-row smoke test of a non-incumbent
  "alternative" candidate was recorded as settles_gap: true and
  direction: supports_incumbent. Corrected to settles_gap: "partially"
  and direction: "not_applicable" -- running the alternative's own
  snippet is not a comparison against the incumbent.

- Disclosed, in all six receipts with commands that actually ran
  (dvc-arm-restore-compare, delta-rs-header-run, sdk-env-baseline,
  xnys-multiyear-holiday-check, duckdb-regression,
  financial-data-offline-regression), that the `preregistered` field was
  authored after the commands had already run and their results were
  known, so it is a post-hoc restatement rather than a true blind
  prediction. For xnys-multiyear-holiday-check specifically, disclosed
  the two earlier failed compute.py runs (exit 1, exchange_calendars
  start-date-bounds errors) and intervening script edits from a prior
  workflow attempt, which the original receipt omitted.

Re-ran manifests/evidence.json rehash after the edits; scripts/validate.py
and python3 -m unittest both pass (unittest: 1999 tests, OK, skipped=324).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Execute the recovery-portability restic/ai-memory/Qdrant gap checks (unit recovery)

The prior unit for this layer produced no receipts and no commit; its refusal
reasoning (treating recovery-portability as an unrecognized/injected layer_id)
was incorrect -- gap-triage.json's own triage[] entry for recovery-portability
lists both executable_checks with prerequisites_present:true and
could_change_verdict:true.

Executes both preregistered checks with retained evidence:
- restic-filesystem-semantics: restic 0.19.1 backup/restore round trip of an
  ACL/xattr/sparse/hardlink/symlink fixture, once to a local ext4 repo and once
  to a repo on /mnt/c (drvfs). The local round trip preserved every captured
  metadata field byte-for-byte; the drvfs round trip preserved content, symlink
  targets and hardlink grouping but dropped POSIX mode/ACL/xattrs (expected
  9p/drvfs limitation, now measured rather than assumed).
- local-ai-memory-qdrant-rebind: restic backup/restore of a read-only copy of
  the live ai-memory data dir and an API-taken Qdrant snapshot, rebound to a
  disposable ai-memory instance (new port) and a disposable Qdrant instance
  (pinned binary, scratch storage, new port). Restored reads (page/session
  counts, a fixed FTS query, a collection point count, a payload-filter search)
  matched pre-backup baselines exactly. Disposable processes were terminated
  and confirmed gone; live ai-memory/Qdrant state was left unmodified.

Both receipts settle their gaps partially (supports_incumbent): they narrow
open_gaps[2]/[3]/[4] for restic + pinned ai-memory/Qdrant from "untested" to
"measured, same-host, with drvfs permission-metadata loss now documented as a
limitation" rather than closing the needs_hardware sub-claims (independent
physical host, reboot/power-loss). Evidence class: native_proven.

Registers the 4 new files in manifests/evidence.json and rehashes via
tools/rehash_evidence.py. Paths and identifiers in both receipts are
normalized to this repo's publication-hygiene placeholders per
scripts/validate.py's PRIVATE_CONTENT rules.

Acceptance: scripts/validate.py passed (1826 hashed files, 140 receipts,
status passed). python3 -m unittest: 1999 tests, OK (skipped=324).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct mislabeled verdict_impact, tautological comparison claim, and mislabeled evidence_class in research-portfolio receipts

- skf_walkforward_optimizer.json: verdict_impact.direction corrected from
  refutes_incumbent to supports_incumbent. The receipt refutes the layer
  verdict's negative open_gaps[0] statement (missing evidence), not
  skfolio itself; closing that gap is evidence for the incumbent.
- sklearn-timeseriessplit-matched-arm.json: the preregistered deciding
  metrics (c)/(d) cannot differ between arms by construction, since both
  run_c22.py and evaluate.py derive the reserved window directly from
  plan.json rather than from either splitter's fold output. Corrected
  could_change_verdict to false, direction to inconclusive, dropped the
  incorrect portfolio-risk gap 6 mapping (that gap is about
  cvxportfolio-vs-skfolio, not a CV splitter), and reframed the result as
  an execution smoke test rather than an executed cross-library
  comparison. No commands were rerun; only the interpretation of the
  already-executed, already-hashed output was corrected.
- repo-local-full-test-rerun.json: evidence_class corrected from
  native_proven to local_integration per the handbook taxonomy (reruns
  the repo's own suite, not skfolio's upstream suite). Added a limitation
  noting the preregistered field was authored after the run outcomes
  were already known (no pre-run artifact exists for this check, unlike
  the other two receipts in this unit, whose preregistration lives in
  script docstrings written before their run commands executed); the
  underlying commands, exit codes, counts and raw logs are unaffected.

Re-ran manifests/evidence.json rehash after the edits; python3
scripts/validate.py passes (status: passed) and the full python3 -m
unittest suite is OK (skipped=324), matching the counts already on
record in repo-local-full-test-rerun.json.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix round: correct test-count reconciliation, pinned-runtime rerun, gap-5 evidence, evidence_class and a raw_artifacts hash (unit promotion-exec)

Addresses five reviewer findings against d0d8a78:
- rerun-frozen-fixture-suite: the '127 vs 150' discrepancy was a misreading
  of the triage citation (the other four files besides test_alpaca_paper sum
  to 127, matching this run exactly); also reruns the same five test files
  through the pinned adaptive-paper-20260921 runtime (alpaca-py 0.44.0
  installed) so the layer's verdict_overturn_when trigger is actually
  exercised instead of skipped (159/159 pass, 0 skips).
- promotion-gate-fixture-suite: adds the actual grep evidence for gap 5
  (paper runtime never imports pandera), which was previously claimed
  settled with no recorded command.
- capture-alpaca-wire-serialization: corrects the raw_artifacts sha256 for
  probe_wire.py to match the committed file and manifest; independently
  reproduced the recorded output from the committed script to confirm the
  run itself was valid.
- All three receipts: evidence_class corrected from native_proven to
  synthetic per the handbook definition (fixtures/offline reproduction
  only, no live workload or broker).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix headroom-0.38.0-recovery isolation and noop-fixture overclaim (fix round)

The prior receipt's MCP server ran without HEADROOM_WORKSPACE_DIR/
HEADROOM_OFFLINE/HEADROOM_TELEMETRY guards and wrote into the live
~/.headroom store (confirmed by mtime/content evidence in the finding),
and its small rag-note.md fixture only triggered Headroom's no-op
transform, so it never exercised real compression despite the receipt
claiming the check settled recovery.

Per the fix-round rule allowing one corrected re-run when the procedure
itself was wrong, this adds run_v2.sh: an isolated re-run (disposable
HEADROOM_WORKSPACE_DIR, HEADROOM_OFFLINE=1, HEADROOM_TELEMETRY=off) over
an owned synthetic MCP-tool-listing-shaped fixture
(mcp-tool-listing.json, generated by make_mcp_listing_fixture.py) large
enough to trigger a real compressing transform (router:mixed, 91.6%
reduction) with byte-exact recovery confirmed. Pre/post stat of
~/.headroom's three files shows unchanged mtimes, confirming this
corrected run did not touch the live store; the original run's
contamination is disclosed in the receipt and left in place pending
coordinator authorization to remove it. The historical run.sh and
original receipt content are retained (not deleted) with disclosure of
their defects. manifests/evidence.json is updated and rehashed for the
new/changed files.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Add vllm-0.30.0-uva-smoke receipt for semantic-rag/observation-inference

Installs vLLM 0.30.0 into a new isolated uv prefix (vllm-0.25.0/0.29.0
and the live 127.0.0.1:8231 embedding service untouched), starts a real
serve for the already-cached Nemotron-3-Embed-1B model on this WSL2/RTX
5090 host, and confirms the 0.29.0 "UVA is not available" startup crash
no longer occurs: upstream PR #56908 (merged in v0.30.0, released
2026-09-22) replaces it with a device-memory fallback. Server reached
readiness and answered one /v1/embeddings request end to end. Refutes
the observation-inference gap-4 needs_hardware blocker (no new hardware
was needed, only a network install) and settles semantic-rag gap 4.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Fix vllm-0.30.0-uva-smoke receipt overclaims (settles_gap, verdict_impact)

Corrects two Opus-critic findings on both identical receipt copies
(observation-inference and semantic-rag directories) without re-running
the check:

- settles_gap changed from true to "partially": the receipt disproves
  only the needs_hardware blocker's "needs different hardware" claim
  and answers semantic-rag gap 4's narrow tried/not-tried question. It
  does not settle observation-inference gap 4 as a whole -- that gap
  also covers the llama.cpp single-trial finding and OmniRoute's
  unsuccessful gateway tasks, and qualifying vLLM as an optional model
  route needs generative serving under task conditions, not just an
  embedding-model startup smoke test.
- verdict_impact corrected from could_change_verdict=true /
  direction="refutes_incumbent" to could_change_verdict=false /
  direction="not_applicable": the prior labels treated a gap-triage
  category (needs_hardware) as if it were a layer verdict. Neither
  layer's incumbent (SocratiCode+Qdrant+vLLM 0.25.0 for semantic-rag;
  the llama.cpp/vLLM/OmniRoute candidate-set status for
  observation-inference) is refuted or supported by this smoke test.
- purpose/result/limitations updated to state the narrower scope and
  flag that the preregistration's own framing conflated a triage
  category with a verdict.

Rehashed manifests/evidence.json for the two changed files.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Record the 2026-09-22 gap resolution: 41 receipts, a 315-gap ledger and the Headroom live-store cleanup

Merges eleven reviewed gap units (Dagu in-flight resume and 2.17.0 parity, Worktrunk requalification and crash probe, Beads lease recovery, promotion gate and execution fixtures, research/portfolio arms, retrieval and memory erasure, restic semantics and restore rebind, storage/market offline regressions with a second calendar source, secret-scan receipt and upstream currency sweep, Headroom/RTK/ccusage/observability checks, and the vLLM 0.30.0 smoke). Adds catalogs/landscape/gap-resolution-20260922.json (every open_gaps entry of the 32 rows with category, blocker, status and receipts: 7 settled, 33 advanced, 5 not settled, 270 open with a named blocker) and docs/gap-resolution-20260922.md. The ledger changes no verdict.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Sanitize host paths in the gap-resolution ledger

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Remove a duplicate evidence manifest entry introduced by the unit union

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Add consistency tests for the gap-resolution ledger

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Resolve the Codex review of the gap ledger: isolated re-runs, per-gap statuses, pinned source text

- ai-memory rebind: the first run's status/search reads went to the live server
  (default server_url), so they compared the live store with itself. Re-ran with
  a fresh restic restore and two disposable servers, each read sent to its own
  server; restore byte-identical, counts and top results equal.
- DVC restore: re-ran with live capture instead of a reconstructed log and added
  the executed baseline arm. DVC restores snapshot.json byte-identically;
  re-materializing changes snapshot_sha256 through ingested_at.
- financial-data offline regression -> local_integration; Dagu kill direction ->
  inconclusive (1-2 s window, before the 30 s/90 s thresholds).
- Ledger: eight gaps whose receipts did not address them move to not_settled;
  isolation[4] summary corrected; every gap carries its verbatim bdd04ca text
  and the source file hashes, since #79 rewrote all open_gaps.
- Receipts and a blueprint: username-bearing scratch paths removed.
- Tests: status recomputed from receipts unless a coordinator note says why;
  source text checked against the source revision; host-path scan over
  receipts and blueprints.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Mark two more gaps not_settled after the Codex follow-up review

agents-models-workers[5] (CLI-only check, gap is native MCP scope) and research-factors-ml[3] (repository-local rerun, gap is upstream skfolio tests).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: seathatflowsinourveins <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant