Repository navigation
Record cross-family layer verdicts (blind Claude and Codex lanes, counterbalanced adjudication) - #79
Merged
Conversation
…s for ledger-built packets codex_lane.py passes Codex a copy of the lane schema without keywords its strict structured output rejects (uniqueItems returned HTTP 400 invalid_json_schema with codex-cli 0.155.1); record_verdicts.py still enforces them. lane_packets.py --withhold-labels drops candidate review_status and decision selection/review_status from ledger-built packets: in the 2026-09-22 foundation packets confirmed_default and selection=default occur only on selected candidates. Manifest-mode trading packets are left byte-identical. Default output is unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
… counterbalanced adjudication Both lanes judged the same 32 packets (lane_packets.py --trading-candidates manifest --withhold-labels) against a checkout of main with every earlier verdict removed. Claude lane wf_dcc60c7a-e87: Opus 5.5 proposer, two refuters, one revision per layer. Codex lane codex_lane.py: gpt-6-astra, high effort, read-only, 32 layers, 0 failed attempts. Result: same_winner in 20 of 32 layers. The 12 disagreements were adjudicated from anonymized returns (host paths and model names scrubbed; one Opus evidence-reviewer judge and two refuters), then re-judged with A/B swapped; 10 layers chose the same lane in both orders with no refutation and are recorded from that lane's return (Claude 7, Codex 3), each adjudication sealed under evidence/artifacts/layer-verdicts-20260922/adjudication/. execution-broker and observability-hosting split 2-2 by presentation order over four judgments and stay pending_lanes with both sets recorded. The handbook's production note, rerun steps and limits describe the method and its limits (adjudicators share the Claude lane's model family). python3 -m unittest Ran 2052 OK. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…ndness limits record_verdicts.py now accepts an adjudicated winner only when the file's judgments cover both presentation orders and all chose the same lane with no refuting vote; a split is sealed and the row stays pending_lanes with its tally. The 12 adjudication records now carry every judgment's order and pick, so the execution-broker and observability-hosting 2-2 splits are checkable in-repo. codex_lane.py keeps schema property names that match dropped keywords and keeps instance data verbatim (the real lane schema's strict copy is unchanged); the dry run says it writes nothing. lane_packets.py help text matches its behavior, with a test for SOTA-component review_status. Handbook: the five stale summary rows now match the ledger (guarded by tests/test_handbook_summary.py); 30 of 32 recorded; both lanes described; the foundation lanes' exposure to decisions.json selection labels, the residual process-wording cue and the four-component observability difference are stated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…puts, exact guards The label-exposure limit now covers every label-bearing catalog file the lanes read (20 of 20 foundation returns per lane; 2 Claude and 3 Codex trading returns), the full 54-decision selection breakdown, the packets' adopted flag and what a blind checkout must strip. The 32 lane packets (SHA256SUMS matching all 64 sealed returns) and the scrubbed adjudication inputs with their lane key are retained, so the three adjudicator-visible process cues are checkable. record_verdicts.py requires an explicit winner_lane key (tests for a missing key, a boolean refuting_votes and non-list judgments). codex_lane.py keeps names under dependentRequired and dependencies. The summary-table guard now parses every name in a cell, so an unlisted extra name fails. The tools README no longer says the recorded run used unwithheld packets. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
…receipts The retained adjudication inputs hold five cue-bearing Claude returns, not three: instructions-skills, observation-inference and web-research went to Claude, code-navigation to Codex, observability-hosting to neither. Trading exposure counts the adaptive-paper receipt's decision (three returns per lane), and the blind-checkout step also strips blueprint JSON labels. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
force-pushed
the
claude/codex-lane-20260922
branch
from
September 22, 2026 23:55
562625b to
cd47862
Compare
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 23, 2026
… statuses, pinned source text - ai-memory rebind: the first run's status/search reads went to the live server (default server_url), so they compared the live store with itself. Re-ran with a fresh restic restore and two disposable servers, each read sent to its own server; restore byte-identical, counts and top results equal. - DVC restore: re-ran with live capture instead of a reconstructed log and added the executed baseline arm. DVC restores snapshot.json byte-identically; re-materializing changes snapshot_sha256 through ingested_at. - financial-data offline regression -> local_integration; Dagu kill direction -> inconclusive (1-2 s window, before the 30 s/90 s thresholds). - Ledger: eight gaps whose receipts did not address them move to not_settled; isolation[4] summary corrected; every gap carries its verbatim bdd04ca text and the source file hashes, since #79 rewrote all open_gaps. - Receipts and a blueprint: username-bearing scratch paths removed. - Tests: status recomputed from receipts unless a coordinator note says why; source text checked against the source revision; host-path scan over receipts and blueprints. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 23, 2026
…l 32 layers (#81) * Requalify worktrunk 0.79.0, probe its crash-kill cleanup, and run difft on a real multi-file range - worktrunk-0-79-0-requalify: mirror the frozen 0.78.0 owned-worktree qualification (create/list/remove/dirty-refuse/cleanup) byte-for-byte against the pinned 0.79.0 binary; passed end to end (git-github-automation gap 2). - worktrunk-crash-cleanup-probe: 5 create-then-SIGKILL trials each for wt and native `git worktree`, using the critic-corrected `wt switch --create ... --format=json` invocation; all 10 kills landed before any on-disk worktree registration was written, so both tools' remove/prune paths correctly report nothing to clean up -- a genuine but narrower result than the mid-write crash case the gap asks about (workers gap 5, git-github-automation gap 3; settles_gap: partially, inconclusive). - difftastic-larger-fixture: ran pinned difft 0.71.0 over all 20 files changed in native-agent-stack-token-practice-review 231077d..bdd04ca (json/md/py/html); 20/20 exit 0, parsed, non-empty (git-github-automation gap 4). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Resolve storage-market unit gaps: DuckDB/financial-data regression, XNYS multi-year holiday cross-check, sdk-env-baseline rebuild, DVC restore comparison, delta-rs header run storage-compute: reran tests.test_catalyst_dataset (14/14, byte-identical repeat materialization) and tests.test_financial_data (16/16, offline fail-closed contracts) in a freshly rebuilt, hash-checked SDK-lock interpreter. SEC EDGAR retry recorded as needs_user_input per this unit's binding instruction (no live EDGAR query attempted, even with a project-URL-only User-Agent). market-data-reference: extended gap 5's single-session exchange_calendars spot check to a full 2016-2027 XNYS holiday/half-day cross-check against pandas_market_calendars 5.4.0 (independent NYSE calendar implementation); 1 boundary-methodology disagreement found and explained (exchange_calendars realized first_session clipping vs. the requested range start), zero substantive rules disagreements. identity-provenance: rebuilt adoption/sdk/requirements-linux-x86_64-py313.lock in a new hash-checked uv prefix and ran the overturn_protocol baseline suite (tests.test_nanosecond_replay + tests.test_point_in_time, 25/25, zero failures/skips); installed DVC 3.67.1 into a new prefix and ran init/add/status/checkout end to end over the project's own materialized fixture, showing byte-identical restore after deletion versus the baseline arm's own scripts, which grep confirms have no restore/backup mechanism; installed deltalake 1.6.4 + pyarrow 25.0.1 (uv pip install form, per the critic's correction) into a new prefix and ran the catalog's own delta-rs write/read snippet, confirming it executes as documented. All new tool prefixes are new, isolated installs under ~/.local/share/codex-ecosystem/tools/; no existing pinned install was replaced. Evidence receipts sanitized of personal home paths and the session scratchpad UUID per scripts/validate.py's hygiene checks; manifests/evidence.json rehashed. Full python3 -m unittest suite: 1999 tests, OK (skipped=324, pre-existing/unrelated). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Add native evidence for the dagu-inflight-resume-after-kill and dagu-2170-checkpoint-parity gap checks dagu-inflight-resume-after-kill (data-quality-orchestration gap 0): a disposable two-step Dagu 2.16.6 DAG was started, SIGKILLed mid-step, and recovery was attempted via both `dagu start --run-id <same>` and the documented `dagu retry --run-id`. Neither could resume or restart the run: the killed process left a stale on-disk unix-socket lock that SIGKILL never cleans up, so Dagu reports the run as permanently "running"/"already running" and refuses both recovery paths within the window tested. This settles the gap with a stronger (negative) result than the base hypothesis: not only is there no intra-step resume, the entire run becomes stuck. dagu-2170-checkpoint-parity (scheduling-supervision gap 0): downloaded and checksum-verified the official upstream Dagu v2.17.0 linux_amd64 release, installed it to a new isolated prefix, and ran the SAME accepted checkpoint/stop/retry job-recovery fixture (unmodified fixture.py) against both the pinned 2.16.6 binary and 2.17.0. Both produced identical status transitions and the exact same checkpoint sha256 as the previously accepted 2.16.6 record, so no regression or behaviour change was observed on 2.17.0 for this fixture. Receipts follow the schema in evidence/artifacts/gap-resolution-20260922/; manifests/evidence.json was updated and rehashed via codex-ecosystem/state/grand-catalog-20260922/tools/rehash_evidence.py. scripts/validate.py and the full unittest suite (1999 tests, OK, skipped=318) both pass. Co-Authored-By: Claude Sonnet 5 <[email protected]> * Execute the research-factors-ml/portfolio-risk gap-resolution checks Rebuilds the receipt's hash-locked Python 3.13.15 environment offline (uv --require-hashes against requirements.lock, 19 distributions matching the accepted skfolio 1.2.9 receipt) in a disposable prefix and reruns: - repo-local-full-test-rerun: the 12 focused research_evaluation tests still pass; the repo's full suite has grown from 214 to 1999 tests in three days and now depends on 'requests', which the bare pinned SDK_PYTHON interpreter lacks (6 unrelated errors in test_broad_universe_scan.py) while system python3 (matching the repo's actual CI interpreter) passes clean. Reproduced twice, deterministic. - sklearn-timeseriessplit-matched-arm (c22): a new runner (blueprints/gap-resolution-20260922/sklearn-timeseriessplit/run_c22.py) executes scikit-learn's TimeSeriesSplit(gap=6) on the SAME frozen plan.json/evaluate.py helpers as the skfolio WalkForward receipt; the reserved-2021Q1 selection and its net-cost-proxy label match exactly, while development fold counts differ (19 vs 20, scikit-learn has no reduce_test equivalent) -- the first executed cross-library comparison for this layer. - skf_walkforward_optimizer: a new runner (blueprints/gap-resolution-20260922/skf-walkforward-optimizer/run_optimizer.py) runs real skfolio MeanRisk and HierarchicalRiskParity optimizers over the same 252/63/6 WalkForward split and a reserved-2021Q1 fit/predict, producing actual portfolio weights and out-of-sample returns -- refutes the prior "no skfolio optimizer has been executed" claim for portfolio-risk gap 0. Registers the three receipts and two helper scripts in manifests/evidence.json and rehashes. scripts/validate.py and python3 -m unittest pass. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Record secrets-upstream unit checks: gitleaks history/dir scan, attestation recheck, upstream currency sweep Runs the three gap-resolution-20260922 checks assigned to the secrets-upstream unit (secrets-credentials, workers, document-retrieval, scheduling-supervision, token-efficiency, evaluation-experiments, git-github-automation): - gitleaks-repo-history-scan: guarded `gitleaks git` (615 commits, 1.27GB, 45.6s) found one benign synthetic literal in tests/test_lane_packets.py; guarded `gitleaks dir` over the live worktree found none, with docs/ecosystem/index.html (13MB) skipped under the 2MB size cap, recorded as incomplete coverage. - attestation-endpoint-recheck: gh api still returns 404 for the gitleaks 8.30.1 linux_x64 asset and checksums-manifest digests, and the current release.yml has no attest-build-provenance step -- reconfirms the existing gap-4 finding unchanged. - upstream-currency-sweep: one batched, read-only gh api snapshot across 14 repositories (OpenHands software-agent-sdk, context-hub, dagu, headroom, phoenix, inspect_ai, qmd, poppler_mirror, markitdown, worktrunk, beads, gitleaks, syft, grype) per the critic's cross-layer grouping; reconfirms Dagu/Headroom/OpenHands are stale against upstream, confirms the inspect_ai pin-discrepancy text is accurate, and surfaces one new markitdown 0.1.7 vs 0.1.8 currency gap. Receipts and their preregistrations, commands, evidence classes and limitations are under evidence/artifacts/gap-resolution-20260922/; the currency-sweep helper script is under blueprints/gap-resolution-20260922/. manifests/evidence.json files[] is rehashed accordingly. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Add gap-resolution evidence for token-efficiency and observation-inference (unit token-observation) Four preregistered checks with retained native output: headroom-ai==0.38.0 installed into a new isolated uv-tool prefix confirms exact MCP compress/retrieve recovery (sha256-matched) and the critic's finding that there is no `headroom compress` CLI subcommand, without touching the pinned 0.37.0 install. RTK 0.49.0 proxy/recall exact-recovery and error-path exit-code preservation hold on a second, independently constructed git-fixture repo and command. ccusage 20.0.24 run over this host's real Claude logs shows a session keyed by a workflow run id whose four token counters exactly match an independently computed child-usage.mjs sum for that same run, settling the combined child/subagent attribution half of gap 4 (per-child/per-retry breakdown remains open). The offline observability/alerting regression suite (21 tests) still passes, which reconfirms but does not by itself close the named comparison gap against alternative observation stacks. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Add retrieval-memory unit receipts: executed hybrid retrieval citation, isolated qmd bench attempt, ai-memory deletion-erasure probe, and 2.3.2 scope reread Four preregistered checks for gap-resolution-20260922 (document-retrieval, durable-memory, agents-models-workers layers): - cite-executed-retrieval: reads agent-lab's two dated document-retrieval-preregistration-v2 runs (20260922T055714Z incomplete, 20260922T060810Z complete) read-only; confirms a sealed-snapshot hybrid-vs-BM25 comparison exists and shows hybrid ahead on recall_at_3/recall_at_1/mrr, with latency_p95_ms as the sole remaining closure blocker (decision D12). - qmd-repeated-bench-isolated: built a disposable qmd 2.8.3 index (never registered in the live native-agent-stack-catalog index) over blueprints/us-equities and attempted the frozen 12-query bench fixture. Only query 1/12 was reached before hybrid mode's query-expansion step threw DisposedError in node-llama-cpp after ~5 minutes, and full mode stalled for a further ~8 minutes before the job was terminated; 3x cold + 3x warm repeats were not feasible given this non-completion under host load. Full crash trace and partial timings retained in the receipt. - deletion-erasure-probe: confirmed ai-memory 2.3.2 delete-page is a logical delete, not byte erasure -- the deleted page's unique marker survives in db/memory.sqlite-wal even after a clean server shutdown. Also documents a process-safety finding: --data-dir alone does not isolate ai-memory CLI commands from an already-running server on the default port/AI_MEMORY_SERVER_URL; the first attempt inadvertently wrote to the live store and was immediately remediated (delete-page + purge-project --confirm) before rebuilding the probe on a genuinely isolated instance (distinct --bind port + AI_MEMORY_SERVER_URL). - ai-memory-local-scope-reread: confirms the installed version is 2.3.2 (not the v2.3.1 the us-equities catalog row names), and that the CLI prints an explicit resolved workspace/project scope line on every call when a .ai-memory.toml marker is present. Config checked first for an LLM provider; none configured, so the auto-improve trial was correctly skipped and that decision recorded rather than silently omitted. manifests/evidence.json: registered and rehashed the four new receipt paths. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Add the beads dead-worker lease-recovery probe receipt (gap-resolution-20260922) Exercises bd 1.3.0's documented `bd reclaim` stale-lease recovery path in a disposable scratch database: a live lease is correctly left untouched by `bd reclaim --older-than 0s`, and once real elapsed time exceeds the compiled 5-minute DefaultLeaseTTL, the claim reverts to open/ready and the issue is claimable again. Moves catalogs/foundation/decisions.json id=dependency-task-queue's "distributed lease correctness" limitation from not_established to a single native execution. Also records a genuine usage trap found along the way: combining bd update --claim with an explicit --assignee (the triage's own proposed command form) grants the lease to the CLI actor identity, not the stated assignee, and left that pilot issue permanently unreclaimed past its true TTL -- root-caused by reading gastownhall/beads source at the exact pinned commit (f45b249ce6b40ba62aecc03949e6371e8f7c79d8 == bd 1.3.0), not further live experimentation on the same issue. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Record promotion-gate fixture suite, frozen-fixture rerun and alpaca wire-serialization receipts (unit promotion-exec) Three preregistered checks for data-quality-orchestration/execution-broker gaps, per codex-ecosystem gap-triage.json and its critic corrections: - promotion-gate-fixture-suite: ran tests/test_promotion_gate.py through the gate's own pinned isolated venv; all 18 tests pass including the previously skippable FixtureGateRuns class (0 skips), settling gaps 4/5. - rerun-frozen-fixture-suite: re-ran all five pytest fixture_paths named in execution-broker's overturn_protocol (159 tests, OK, skipped=40, no regression); per critic correction this only reconfirms synthetic coverage and does not close peer_owned/RTH-gated gaps 2/6. Also surfaces a real finding: paper-receipt.json's declared runner_sha256 no longer matches paper_runner.py at HEAD (the runner was edited after that trial ran), and an unresolved discrepancy against the triage's cited prior test count (127 vs. this run's reproducible 150) for the same four-file combination. - capture-alpaca-wire-serialization: extended sdk_probe.py's network-isolated pattern with a source-traced reproduction of requests' own PreparedRequest.prepare_body(json=...) call (the exact function alpaca-py's TradingClient.post/_request path invokes) to capture literal wire JSON bytes for 9 qty/limit_price/notional edge cases; confirms decimal-text loss (e.g. "100.10" -> 100.1) at the literal byte level, not just object field types, and that alpaca-py's SDK model performs no local rejection of zero/negative qty. manifests/evidence.json updated with the 7 new files and rehashed via codex-ecosystem's rehash_evidence.py; scripts/validate.py passes; full python3 -m unittest suite stays OK (1999 tests, 324 skipped, no failures). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct difftastic fallback-detection bug and crash-probe overclaims - difftastic-larger-fixture: the original run_difft.sh fallback check only matched hard-error strings, never difftastic's actual fallback header ("<path> --- Text" / "... --- n/m --- Text (exceeded DFT_BYTE_LIMIT|DFT_GRAPH_LIMIT)"), so every size-limited or Markdown-unsupported file was miscounted as a successful structural parse. Added reanalyze_reports.py, which reprocesses the same already-retained difft reports (no re-invocation of difft) with a corrected detection rule: 15/20 files parsed structurally, 5/20 (both large JSON catalogs, both Markdown docs, the large HTML page) fell back to plain text. Corrected the receipt's files_parsed_ok/files_failed, settles_gap (false, not "partially"), and verdict_impact.direction (inconclusive, not "supports_incumbent"), keeping the original raw artifacts and adding the corrected ones alongside them. - worktrunk-crash-cleanup-probe: corrected settles_gap from "partially" to false. An at-exec (Popen-return) SIGKILL lands before either wt or git worktree add has written any on-disk registration in 10/10 trials, so the probe never exercised the mid-write crash state either gap asks about and cannot discriminate the two tools' cleanup behavior. direction "inconclusive" was already correct. Also added a preregistration_timing field noting the preregistered text has no earlier timestamped file predating the raw artifacts (crash-probe-out/ and the receipt/commit are same-day, same-commit), so it documents the trial design's decision rule rather than a demonstrated advance prediction. Registered the new blueprint script in manifests/evidence.json and reran rehash_evidence.py (4 files rehashed, 0 missing). scripts/validate.py and `python3 -m unittest` both pass (1999 tests, skipped=324). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Co-Authored-By: Claude Sonnet 5 <[email protected]> * Fix round: correct raw-artifact hash, out-of-ancestry attribution, evidence_class and OpenHands reconciliation gap in secrets-upstream receipts Resolves four review findings on the secrets-upstream unit's committed receipts without re-running any settled check: - gitleaks-repo-history-scan.json: corrected the git-history.stderr.log raw_artifacts sha256 (it previously recorded dir-worktree.stderr.log's hash). Corrected the finding's attribution: commit 34fc51b is confirmed NOT an ancestor of this branch (`git merge-base --is-ancestor` fails, `git branch -a --contains` shows only sibling branches claude/gc-pr5-packets and claude/grand-catalog-pr5). Cited .gitleaks.toml's own prior HEAD-ancestry measurement (376 commits, 0 findings, recorded before this unit's branch point) instead of claiming git-history scanning was previously unmeasured, and explained the 615/542/491/738 commit-count spread as .gitleaks.toml's own documented non-stable-invariant behavior in this shared, concurrently active repository. - upstream-currency-sweep.json (all six layer copies) and attestation-endpoint-recheck.json: reclassified evidence_class from native_proven to source_review, matching tools/sota-convergence/README.md's definition (read-only `gh api` metadata lookups install, build or run nothing) and distinguishing them from the gitleaks-repo-history-scan receipt, where gitleaks itself was actually executed. - upstream-currency-sweep.json: added the previously omitted OpenHands software-agent-sdk revision-reconciliation sub-check (read-only gh api, preregistered before running). Result: the v1.49.2 tag resolves to a third, distinct commit (d128a786...); 856d99d48e4b11c70c5f1cab21e7830570dbc324 is a direct 7-commit ancestor of 9bc452ebf6b0093c531f25394c5ba5e9817ff910, so the two cataloged revisions are reconciled with each other on one lineage, though neither is current relative to HEAD or the latest release. Rehashed manifests/evidence.json. `python3 scripts/validate.py` passed (69 components, 1831 hashed files, 140 receipts). `python3 -m unittest` passed (1999 tests, OK, skipped=318). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct beads-lease-recovery-probe's root-cause claim The receipt, run.sh, and result/limitations text said combined `--claim --assignee <name>` "grants the lease to the CLI actor identity rather than the stated assignee". That is wrong: upstream source at the pinned commit (f45b249ce6b40ba62aecc03949e6371e8f7c79d8) shows --claim arms a lease for the actor, but the same call's --assignee value then runs through the generic-update lease manager (ManageLeaseOnUpdate, internal/storage/issueops/update.go), which deletes the lease row whenever a generic update changes who holds an in_progress claim (DeleteLeaseInTx). The pilot issue was left in_progress with no lease row at all, and `bd reclaim`'s own query (ReclaimExpiredLeasesInTx, internal/storage/issueops/lease.go) is an inner join against the leases table, so an issue with no lease row can never be selected -- a permanent exclusion from recovery, not a transient miss at the pilot's 0s-grace/~59s-past-TTL window. Corrected the claim-response, reclaim-response, result, and verdict_impact.explanation text in the receipt, and run.sh's comment block, to state the confirmed mechanism. Added a new first limitation recording that dead-worker recovery only covers issues claimed via the bare lease-arming `--claim` verb, and that any issue whose in_progress claim was established/transferred through a generic update (including the triage's own proposed --claim --assignee form) is permanently unreachable by reclaim -- to be recorded in catalogs/foundation/decisions.json id=dependency-task-queue's limitations text before that text is narrowed on this probe's positive result. No check was re-run; only the receipt's causal explanation and limitations were corrected against independently re-verified upstream source at the same pinned commit. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct overstated dagu-inflight-resume-after-kill claims Addresses three major findings from the coordinator's review of dagu-inflight-resume-after-kill.json: 1. The receipt claimed the killed run was "permanently"/"indefinitely" stuck, but retries were only attempted ~1-2s after the SIGKILL. A static inspection of the pinned dagu-2.16.6 binary's own embedded config schema shows a lock_stale_threshold (default 30s) and a process-heartbeat stale_threshold (default 90s) that were never reached in the tested window. Removed the permanence language, changed settles_gap from true to "partially", and narrowed verdict_impact.explanation to the claim actually supported: no immediate/automatic resume after an unclean kill, not permanent unrecoverability. Added the config-schema inspection as a third command and a limitation naming the untested longer-delay/`dagu stop`/socket-deletion recovery paths, without re-running the kill/retry experiment itself. 2. The causal claim that Dagu's liveness check treats the stale socket file's "mere presence" as proof of a live run was stated as established fact. No test isolated the socket file from the separate per-run proc/heartbeat record observed under dagu-home/data/proc/inflight/... (pid + pidStartedAt) or the persisted history status. Relabelled this as an explicit, untested hypothesis in both the result and the raw-command excerpt. 3. The "preregistered" field's provenance could not be shown to predate execution: an untracked, abandoned earlier-draft script (run_inflight_kill.py, never committed, not part of the registered evidence) carried its own differently-ordered "preregistered" protocol comment shortly before the committed fixture.py was written, with no retained run output proving it executed. Added a limitation disclosing this so "preregistered" here is read as self-declared-in-receipt rather than externally verifiable, without fabricating a corrective commit ordering. manifests/evidence.json rehashed via codex-ecosystem/state/grand-catalog-20260922/tools/rehash_evidence.py. scripts/validate.py passes; the full unittest suite (1999 tests, OK, skipped=324) passes. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct retrieval-memory unit receipts (residue disclosure, numeric errors, version-citation misread) Resolves four review findings against the retrieval-memory gap-resolution receipts, without re-running any check to get a different outcome: - deletion-erasure-probe: discloses residue in the live ai-memory store the original receipt omitted (four new commits in wiki/.git history that still recover the deleted page's full body via `git log --all -p`/`git show`; a leftover scope-manifest directory from purge-project; 2 lifecycle-hook observation rows in memory.sqlite containing probe tokens), all found via new read-only checks. Corrects the "clean shutdown" claim: the disposable server's log ends at "shutdown signal received; draining" with no completion line, .serve.lock.holder was left behind, and the WAL was never checkpointed. "Not erasure" conclusion stands; "fully remediated" framing is retracted. - qmd-repeated-bench-isolated: fixes a fabricated "~1000x" bm25 slowdown claim (bm25 at 4ms was actually faster than the 5ms baseline, not slower), corrects the "full mode ran ~8 more minutes" claim to the ~192s the recorded duration_s actually supports, and discloses that this run's own concurrent disposable ai-memory serve process (embedding-model fetch) overlapped the bench window, not only unrelated peer sessions. Downgrades settles_gap to "no" and direction to "inconclusive" per the corrected, non-comparable, non-completing single run. - ai-memory-local-scope-reread: retracts the false claim of three inconsistent ai-memory version citations. The "2.1.277"/"2.1.278" figures in gap 8's text are Claude Code client versions (confirmed against catalogs/landscape/foundation.json and the cited native-memory receipt), not ai-memory versions; only the single v2.3.1-named-vs-2.3.2-installed discrepancy is supported. manifests/evidence.json hashes regenerated via rehash_evidence.py. rtk proxy python3 scripts/validate.py -> status passed. rtk proxy python3 -m unittest -> 1999 tests, OK (skipped=324). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct storage-market receipt overclaims (DVC direction, delta-rs settlement, post-hoc preregistration disclosure) Corrects four major findings without re-running any check (the checks themselves were not shown to be invalid, only their recorded claims): - dvc-arm-restore-compare.json: direction was inverted. DVC (data-dvc) is catalogs/landscape/us-equities.json's identity-provenance winners[0] (evidence_class source_review); a successful DVC run supports that already-adopted candidate, not refutes_incumbent. settles_gap changed from true to "partially": gap 0 (DVC unexecuted) is settled, but gap 1's overturn_protocol metric (per-cutoff selection/SQL hashes equal to the baseline run; exit_code 2 rejection preserved) was not verified, only whole-file sha256 identity. Corrected the commands array to include the cp -a/chmod -R u+w/git config steps that actually ran but were omitted, and the grep command's real cwd/relative paths. Disclosed that the dvc-arm-restore-compare.log raw artifact was written via the Write tool after the run (a reconstructed transcript matching the real tool results, not a live capture), and that the baseline-arm "comparison" was only a static grep, not an executed delete-then-rematerialize run. - delta-rs-header-run.json: a single-row smoke test of a non-incumbent "alternative" candidate was recorded as settles_gap: true and direction: supports_incumbent. Corrected to settles_gap: "partially" and direction: "not_applicable" -- running the alternative's own snippet is not a comparison against the incumbent. - Disclosed, in all six receipts with commands that actually ran (dvc-arm-restore-compare, delta-rs-header-run, sdk-env-baseline, xnys-multiyear-holiday-check, duckdb-regression, financial-data-offline-regression), that the `preregistered` field was authored after the commands had already run and their results were known, so it is a post-hoc restatement rather than a true blind prediction. For xnys-multiyear-holiday-check specifically, disclosed the two earlier failed compute.py runs (exit 1, exchange_calendars start-date-bounds errors) and intervening script edits from a prior workflow attempt, which the original receipt omitted. Re-ran manifests/evidence.json rehash after the edits; scripts/validate.py and python3 -m unittest both pass (unittest: 1999 tests, OK, skipped=324). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Execute the recovery-portability restic/ai-memory/Qdrant gap checks (unit recovery) The prior unit for this layer produced no receipts and no commit; its refusal reasoning (treating recovery-portability as an unrecognized/injected layer_id) was incorrect -- gap-triage.json's own triage[] entry for recovery-portability lists both executable_checks with prerequisites_present:true and could_change_verdict:true. Executes both preregistered checks with retained evidence: - restic-filesystem-semantics: restic 0.19.1 backup/restore round trip of an ACL/xattr/sparse/hardlink/symlink fixture, once to a local ext4 repo and once to a repo on /mnt/c (drvfs). The local round trip preserved every captured metadata field byte-for-byte; the drvfs round trip preserved content, symlink targets and hardlink grouping but dropped POSIX mode/ACL/xattrs (expected 9p/drvfs limitation, now measured rather than assumed). - local-ai-memory-qdrant-rebind: restic backup/restore of a read-only copy of the live ai-memory data dir and an API-taken Qdrant snapshot, rebound to a disposable ai-memory instance (new port) and a disposable Qdrant instance (pinned binary, scratch storage, new port). Restored reads (page/session counts, a fixed FTS query, a collection point count, a payload-filter search) matched pre-backup baselines exactly. Disposable processes were terminated and confirmed gone; live ai-memory/Qdrant state was left unmodified. Both receipts settle their gaps partially (supports_incumbent): they narrow open_gaps[2]/[3]/[4] for restic + pinned ai-memory/Qdrant from "untested" to "measured, same-host, with drvfs permission-metadata loss now documented as a limitation" rather than closing the needs_hardware sub-claims (independent physical host, reboot/power-loss). Evidence class: native_proven. Registers the 4 new files in manifests/evidence.json and rehashes via tools/rehash_evidence.py. Paths and identifiers in both receipts are normalized to this repo's publication-hygiene placeholders per scripts/validate.py's PRIVATE_CONTENT rules. Acceptance: scripts/validate.py passed (1826 hashed files, 140 receipts, status passed). python3 -m unittest: 1999 tests, OK (skipped=324). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct mislabeled verdict_impact, tautological comparison claim, and mislabeled evidence_class in research-portfolio receipts - skf_walkforward_optimizer.json: verdict_impact.direction corrected from refutes_incumbent to supports_incumbent. The receipt refutes the layer verdict's negative open_gaps[0] statement (missing evidence), not skfolio itself; closing that gap is evidence for the incumbent. - sklearn-timeseriessplit-matched-arm.json: the preregistered deciding metrics (c)/(d) cannot differ between arms by construction, since both run_c22.py and evaluate.py derive the reserved window directly from plan.json rather than from either splitter's fold output. Corrected could_change_verdict to false, direction to inconclusive, dropped the incorrect portfolio-risk gap 6 mapping (that gap is about cvxportfolio-vs-skfolio, not a CV splitter), and reframed the result as an execution smoke test rather than an executed cross-library comparison. No commands were rerun; only the interpretation of the already-executed, already-hashed output was corrected. - repo-local-full-test-rerun.json: evidence_class corrected from native_proven to local_integration per the handbook taxonomy (reruns the repo's own suite, not skfolio's upstream suite). Added a limitation noting the preregistered field was authored after the run outcomes were already known (no pre-run artifact exists for this check, unlike the other two receipts in this unit, whose preregistration lives in script docstrings written before their run commands executed); the underlying commands, exit codes, counts and raw logs are unaffected. Re-ran manifests/evidence.json rehash after the edits; python3 scripts/validate.py passes (status: passed) and the full python3 -m unittest suite is OK (skipped=324), matching the counts already on record in repo-local-full-test-rerun.json. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix round: correct test-count reconciliation, pinned-runtime rerun, gap-5 evidence, evidence_class and a raw_artifacts hash (unit promotion-exec) Addresses five reviewer findings against d0d8a78: - rerun-frozen-fixture-suite: the '127 vs 150' discrepancy was a misreading of the triage citation (the other four files besides test_alpaca_paper sum to 127, matching this run exactly); also reruns the same five test files through the pinned adaptive-paper-20260921 runtime (alpaca-py 0.44.0 installed) so the layer's verdict_overturn_when trigger is actually exercised instead of skipped (159/159 pass, 0 skips). - promotion-gate-fixture-suite: adds the actual grep evidence for gap 5 (paper runtime never imports pandera), which was previously claimed settled with no recorded command. - capture-alpaca-wire-serialization: corrects the raw_artifacts sha256 for probe_wire.py to match the committed file and manifest; independently reproduced the recorded output from the committed script to confirm the run itself was valid. - All three receipts: evidence_class corrected from native_proven to synthetic per the handbook definition (fixtures/offline reproduction only, no live workload or broker). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix headroom-0.38.0-recovery isolation and noop-fixture overclaim (fix round) The prior receipt's MCP server ran without HEADROOM_WORKSPACE_DIR/ HEADROOM_OFFLINE/HEADROOM_TELEMETRY guards and wrote into the live ~/.headroom store (confirmed by mtime/content evidence in the finding), and its small rag-note.md fixture only triggered Headroom's no-op transform, so it never exercised real compression despite the receipt claiming the check settled recovery. Per the fix-round rule allowing one corrected re-run when the procedure itself was wrong, this adds run_v2.sh: an isolated re-run (disposable HEADROOM_WORKSPACE_DIR, HEADROOM_OFFLINE=1, HEADROOM_TELEMETRY=off) over an owned synthetic MCP-tool-listing-shaped fixture (mcp-tool-listing.json, generated by make_mcp_listing_fixture.py) large enough to trigger a real compressing transform (router:mixed, 91.6% reduction) with byte-exact recovery confirmed. Pre/post stat of ~/.headroom's three files shows unchanged mtimes, confirming this corrected run did not touch the live store; the original run's contamination is disclosed in the receipt and left in place pending coordinator authorization to remove it. The historical run.sh and original receipt content are retained (not deleted) with disclosure of their defects. manifests/evidence.json is updated and rehashed for the new/changed files. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Add vllm-0.30.0-uva-smoke receipt for semantic-rag/observation-inference Installs vLLM 0.30.0 into a new isolated uv prefix (vllm-0.25.0/0.29.0 and the live 127.0.0.1:8231 embedding service untouched), starts a real serve for the already-cached Nemotron-3-Embed-1B model on this WSL2/RTX 5090 host, and confirms the 0.29.0 "UVA is not available" startup crash no longer occurs: upstream PR #56908 (merged in v0.30.0, released 2026-09-22) replaces it with a device-memory fallback. Server reached readiness and answered one /v1/embeddings request end to end. Refutes the observation-inference gap-4 needs_hardware blocker (no new hardware was needed, only a network install) and settles semantic-rag gap 4. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Fix vllm-0.30.0-uva-smoke receipt overclaims (settles_gap, verdict_impact) Corrects two Opus-critic findings on both identical receipt copies (observation-inference and semantic-rag directories) without re-running the check: - settles_gap changed from true to "partially": the receipt disproves only the needs_hardware blocker's "needs different hardware" claim and answers semantic-rag gap 4's narrow tried/not-tried question. It does not settle observation-inference gap 4 as a whole -- that gap also covers the llama.cpp single-trial finding and OmniRoute's unsuccessful gateway tasks, and qualifying vLLM as an optional model route needs generative serving under task conditions, not just an embedding-model startup smoke test. - verdict_impact corrected from could_change_verdict=true / direction="refutes_incumbent" to could_change_verdict=false / direction="not_applicable": the prior labels treated a gap-triage category (needs_hardware) as if it were a layer verdict. Neither layer's incumbent (SocratiCode+Qdrant+vLLM 0.25.0 for semantic-rag; the llama.cpp/vLLM/OmniRoute candidate-set status for observation-inference) is refuted or supported by this smoke test. - purpose/result/limitations updated to state the narrower scope and flag that the preregistration's own framing conflated a triage category with a verdict. Rehashed manifests/evidence.json for the two changed files. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Record the 2026-09-22 gap resolution: 41 receipts, a 315-gap ledger and the Headroom live-store cleanup Merges eleven reviewed gap units (Dagu in-flight resume and 2.17.0 parity, Worktrunk requalification and crash probe, Beads lease recovery, promotion gate and execution fixtures, research/portfolio arms, retrieval and memory erasure, restic semantics and restore rebind, storage/market offline regressions with a second calendar source, secret-scan receipt and upstream currency sweep, Headroom/RTK/ccusage/observability checks, and the vLLM 0.30.0 smoke). Adds catalogs/landscape/gap-resolution-20260922.json (every open_gaps entry of the 32 rows with category, blocker, status and receipts: 7 settled, 33 advanced, 5 not settled, 270 open with a named blocker) and docs/gap-resolution-20260922.md. The ledger changes no verdict. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Sanitize host paths in the gap-resolution ledger Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Remove a duplicate evidence manifest entry introduced by the unit union Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Add consistency tests for the gap-resolution ledger Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Resolve the Codex review of the gap ledger: isolated re-runs, per-gap statuses, pinned source text - ai-memory rebind: the first run's status/search reads went to the live server (default server_url), so they compared the live store with itself. Re-ran with a fresh restic restore and two disposable servers, each read sent to its own server; restore byte-identical, counts and top results equal. - DVC restore: re-ran with live capture instead of a reconstructed log and added the executed baseline arm. DVC restores snapshot.json byte-identically; re-materializing changes snapshot_sha256 through ingested_at. - financial-data offline regression -> local_integration; Dagu kill direction -> inconclusive (1-2 s window, before the 30 s/90 s thresholds). - Ledger: eight gaps whose receipts did not address them move to not_settled; isolation[4] summary corrected; every gap carries its verbatim bdd04ca text and the source file hashes, since #79 rewrote all open_gaps. - Receipts and a blueprint: username-bearing scratch paths removed. - Tests: status recomputed from receipts unless a coordinator note says why; source text checked against the source revision; host-path scan over receipts and blueprints. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Mark two more gaps not_settled after the Codex follow-up review agents-models-workers[5] (CLI-only check, gap is native MCP scope) and research-factors-ml[3] (repository-local rerun, gap is upstream skfolio tests). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: seathatflowsinourveins <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR records the September 22 layer verdicts from two independent model-family lanes on the same stripped packets, plus the tooling each lane needed.
layer-verdict-laneworkflow): each layer ran an Opus 5.5 proposer, two refuters and one revision.codex_lane.py): one read-onlycodex execcall per layer at high effort.Tooling
codex_lane.pypasses Codex a strict-output copy of the lane schema. Codex strict mode rejectsuniqueItems,$schema,$id,titleanddescription. The copy drops those keywords but keeps property names and instance data, andrecord_verdicts.pystill enforces every dropped rule.lane_packets.py --withhold-labelsdrops the labels that revealed the incumbent (candidate and SOTA-componentreview_status, decisionselectionandreview_status) from ledger-built packets. Manifest-built trading packets stay byte-identical.record_verdicts.pyenforces the counterbalanced adjudication rule:pending_laneswith its tally.Result
same_winner: 13 foundation and 7 trading.execution-brokerandobservability-hostingstaypending_lanes. Every judgment followed the presentation order, a 2-2 split over four judgments. An executed comparison must decide them.Stated limits
decisions.json. Three trading returns in each lane list one. The lane packets and the scrubbed adjudication inputs are retained, so these claims can be checked.native_proven, 7local_integration, 4syntheticand 3source_review.Review
fix_required: 2 major, 6 minor) and Codex review (1 major, 2 minor). All findings are addressed inff417c7.tests/test_handbook_summary.py, which fails on exactly those rows of the old table.ff417c7:fix_required. Its one major was that the label-exposure disclosure was incomplete.6581cfb. A final Opus verification of6581cfbis recorded in the comments.Verification (local, this head)
record_verdicts.py --checkwith the lane roots and adjudications: 0 rejections.build_verdicts.py --check,scripts/landscape.py,scripts/validate.py,validate_convergence.py --all-recorded,build_ecosystem.py --checkandtrading_gates.py --check: all exit 0.python3 -m unittest: 2067 tests OK, 324 skipped.🤖 Generated with Claude Code