Repository navigation
Grand catalogs carry each winner's evidence basis; ai-memory is the retained control, not a measured best - #269
Merged
seathatflowsinourveins merged 2 commits intoSep 25, 2026
Conversation
…le-memory basis with its evidence The user asked why ai-memory is the best in its field and required the grand catalogs to carry evidence. The catalog never claimed that, but its durable- memory rationale still said published scores do not show better retrieval on this project, although the 2026-09-25 matched LongMemEval-S run did. - foundation.json durable-memory: the rationale says ai-memory is the retained production control, not a measured best; attributes cross-client evidence to the 2026-09-21 lifecycle record, limits the WSL2 host receipts to what they ran; gives C3 (Mac 19b6429, qwen3-embedding:4b, reranker off) 0.570 recall_all@5 against four arms 0.170-0.251 higher (agentmemory with MiniLM, dense Qwen3-Embedding-4B, two BM25 arms); notes production is C4 under A14 (not run) and the workstation's 2.3.2/MiniLM was not measured; points at verdict_overturn_when. Two dated limitations: VelaNext retired (new decision record), so C4 and the confirmatory rerun need another host and a new amendment; open_gaps[0] superseded only in part. No verdict field or checked_at changes (verdict-review gate: 0 rows). - new_host_grand_list.py: the JSON carries each winner's ledger why_selected and evidence_refs; the Markdown gains a path-free legend (a winner is a recorded choice, not a best-in-field claim; 17 keep_but_compare, 14 retain, 1 adjust), so release_due's new-host set is unchanged. - blind_checkout.py: blind checkouts also remove the rendered matrix and grand-list Markdown, which name the same winners as their removed JSON. - docs/decisions/2026-09-25-retire-vela-velanext.md records the retirement. Independent review: needs_changes (6 findings), one repair round, then two one-line text fixes it named; all applied. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
Collaborator
Author
|
Full unit suite on head b04a7ea (uv Python 3.13 with jsonschema/PyYAML/numpy): Ran 5311 tests in 442.004s OK (skipped=613) |
seathatflowsinourveins
deleted the
claude/catalog-evidence-basis-20260925
branch
September 25, 2026 18:28
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 26, 2026
…ackets Codex post-merge review of #269, finding 1 (P2). The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the older #234 "Production now runs ai-memory 19b6429" entry and the #234 "keeps ai-memory as the production control" sentence named the incumbent's build, releases and role without a selection word next to a candidate name, so build_all_packets(..., withhold=True) kept them. Each sentence carrying those identifiers now calls the component the incumbent, a choice phrase the prose filter (lane_packets.py CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact. Verdict fields and checked_at are unchanged. A regression test builds the real withheld durable-memory packet (default and blind-export parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and "production control" are absent while the ledger still records them. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 26, 2026
…counts Codex post-merge review of #269, finding 2 (P3). The "What a winner means" legend said the JSON carries the layer's limitations and open gaps; the generator emits each winner's why_selected and evidence_refs and only gap counts. The legend now says so and points to the layer ledgers for the limitations and the open-gap text (without paths: the page is a new-host document). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 26, 2026
…ackets Codex post-merge review of #269, finding 1 (P2). The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the older #234 "Production now runs ai-memory 19b6429" entry and the #234 "keeps ai-memory as the production control" sentence named the incumbent's build, releases and role without a selection word next to a candidate name, so build_all_packets(..., withhold=True) kept them. Each sentence carrying those identifiers now calls the component the incumbent, a choice phrase the prose filter (lane_packets.py CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact. Verdict fields and checked_at are unchanged. A regression test builds the real withheld durable-memory packet (default and blind-export parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and "production control" are absent while the ledger still records them. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 26, 2026
…counts Codex post-merge review of #269, finding 2 (P3). The "What a winner means" legend said the JSON carries the layer's limitations and open gaps; the generator emits each winner's why_selected and evidence_refs and only gap counts. The legend now says so and points to the layer ledgers for the limitations and the open-gap text (without paths: the page is a new-host document). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 26, 2026
…ackets Codex post-merge review of #269, finding 1 (P2). The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the older #234 "Production now runs ai-memory 19b6429" entry and the #234 "keeps ai-memory as the production control" sentence named the incumbent's build, releases and role without a selection word next to a candidate name, so build_all_packets(..., withhold=True) kept them. Each sentence carrying those identifiers now calls the component the incumbent, a choice phrase the prose filter (lane_packets.py CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact. Verdict fields and checked_at are unchanged. A regression test builds the real withheld durable-memory packet (default and blind-export parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and "production control" are absent while the ledger still records them. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> (cherry picked from commit 418c0f0)
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Sep 26, 2026
…counts Codex post-merge review of #269, finding 2 (P3). The "What a winner means" legend said the JSON carries the layer's limitations and open gaps; the generator emits each winner's why_selected and evidence_refs and only gap counts. The legend now says so and points to the layer ledgers for the limitations and the open-gap text (without paths: the page is a new-host document). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> (cherry picked from commit caeaa4d)
seathatflowsinourveins
added a commit
that referenced
this pull request
Sep 26, 2026
…bootstrap reminder, blind-packet leak, retained markitdown evidence (#314) * Retain markitdown's installed-byte verifier, run directories and raw checker output Codex post-merge review of #291, finding 3. - verify_installed.py is retained byte-identical. Its 2026-09-25 stdout was never written to a file, so results.json says so (not_retained) and adds a 2026-09-26 read-only re-run on the production prefix (45 of 45 identical) plus a per-file listing (installed_files.py) of the wheel RECORD, wheel member and installed digests in RECORD form. - run-candidate/, run-baseline/ and run-postswitch2/ (converted documents, PDF stderr with the home directory replaced by ~, package listings) are retained; their files hash to the recorded values. run_arm.py kept only checker counts, so check_outputs.py is re-run on them (out/checker-rerun-20260926.out), and the independent verifier's own checker.out, legs logs and integrity check are retained. - results.json gains corrections, not_retained, steps and files; the receipt gains a dated limitation and regenerated provenance. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> (cherry picked from commit 9ec4c33) * Keep the durable-memory incumbent's build and releases out of blind packets Codex post-merge review of #269, finding 1 (P2). The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the older #234 "Production now runs ai-memory 19b6429" entry and the #234 "keeps ai-memory as the production control" sentence named the incumbent's build, releases and role without a selection word next to a candidate name, so build_all_packets(..., withhold=True) kept them. Each sentence carrying those identifiers now calls the component the incumbent, a choice phrase the prose filter (lane_packets.py CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact. Verdict fields and checked_at are unchanged. A regression test builds the real withheld durable-memory packet (default and blind-export parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and "production control" are absent while the ledger still records them. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> (cherry picked from commit 418c0f0) * Grand list legend: the JSON carries winners' basis and only open-gap counts Codex post-merge review of #269, finding 2 (P3). The "What a winner means" legend said the JSON carries the layer's limitations and open gaps; the generator emits each winner's why_selected and evidence_refs and only gap counts. The legend now says so and points to the layer ledgers for the limitations and the open-gap text (without paths: the page is a new-host document). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> (cherry picked from commit caeaa4d) * rtk hook exclusions: adopt #318's four-entry recipe and add rtk's own check to its reminder Reconciles this branch's Codex post-merge fixes for #291 (findings 1-2) and the #314 P2 with #318, which merged first. - Recipe: #318's four-entry exclude_commands block stays the one set of guidance in recipes/README.md, bootstrap.md step 4a, token-practice.md, the rtk pin note and the SOTA refresh record. The single regex this branch had tested ('^git(\s+\S+)*\s+show(\s+\S+)*\s+(:\S|[^\s-]\S*:)' with "diff") is kept only as dated evidence, labelled as an alternative that was tested (mitigation-20260926, readback-20260926). The recipe also explains rtk's anchoring (registry.rs:1540-1569, 1599-1604) and that explicit `rtk` commands bypass the exclusions (registry.rs:1690). - Reminder: rtk_config_reminder keeps #318's text check (the exact four-entry key, exactly once) and, once it passes, runs the installed "$bin_dir/rtk" hook check (timeout 30, stdin closed, the caller's environment) on `git show HEAD:x | tail -n 5`, `git -C . show --no-color HEAD:x | tail -n 5`, `diff a missing`, `git branch -a` and `git -C . branch`. Either check failing prints the reminder; a text pass with an rtk failure adds a hint naming the probe, because rtk ignores a TOML-valid file that does not deserialize (TrackingConfig, src/core/config.rs:152-158 at 1d87b8e7). Print-only. - Tests: all of #318's reminder tests are kept (the silent ones now run against a stub rtk that honours the file); new stub tests cover an exact text that rtk rewrites, the [tracking] counterexample, an unsupported subcommand, a rewritten branch probe, extra output, the #291 comments counterexample, the five probes and XDG_CONFIG_HOME reaching rtk. An optional real-binary test (skipped without an installed rtk 0.50.0) covers the four-entry block, the [tracking] rejection with and without history_days, the old two-entry line, the single-regex alternative and a missing file. - Evidence: combined-reminder-20260926.{sh,out} (hook check and the combined reminder on scratch configs; silent on this host's #318 block). The rtk receipt keeps #318's addenda untouched; its added limitation now references #318's recipe and the combined reminder; results.json gains a step, files and relabelled corrections; the receipt's provenance is regenerated. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> * Register the post-merge fixes' files and regenerate the grand list Hot-file commit (docs/lanes.md): manifests/evidence.json starts from main's copy (with #318's and #315's entries) and re-registers every changed or new file of this branch (indent 2, ensure_ascii); receipts[] mirrors the rtk and markitdown receipts' new limitations and keeps every other entry. component_matrix --write (no change) and new_host_grand_list --write regenerate the reports. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Co-authored-by: Scout <scout@local> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
4 tasks done
seathatflowsinourveins
pushed a commit
that referenced
this pull request
Oct 1, 2026
…, the arms without a result and upstream v2.5.2 The row's verdict fields (winners, alternatives, open_gaps) are the sealed 2026-09-22 wave and stay as recorded. The note follows the practice of #269 and #307: a dated limitations entry. It records that ai-memory is the source host's selection of record and the comparison's reference arm, that the merit winner is undetermined, which arms of the frozen harness have no result (C4, D2h, K1, M1, M2), that the 2026-09-25 result is D2 against C3 with the reranker off, that upstream published v2.5.2 on 2026-10-01, and that a verdict wave can name only an adopted candidate as winner. Co-Authored-By: Claude Fable 5.1 <[email protected]>
seathatflowsinourveins
added a commit
that referenced
this pull request
Oct 1, 2026
…puts; durable-memory note; landed #575 citations and four trading acceptance classes (#585) * Durable-memory landscape row: dated 2026-10-01 note on the merit rule, the arms without a result and upstream v2.5.2 The row's verdict fields (winners, alternatives, open_gaps) are the sealed 2026-09-22 wave and stay as recorded. The note follows the practice of #269 and #307: a dated limitations entry. It records that ai-memory is the source host's selection of record and the comparison's reference arm, that the merit winner is undetermined, which arms of the frozen harness have no result (C4, D2h, K1, M1, M2), that the 2026-09-25 result is D2 against C3 with the reranker off, that upstream published v2.5.2 on 2026-10-01, and that a verdict wave can name only an adopted candidate as winner. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Sweep inputs: date the sealed verdict fields and join the gap-wave ledgers onto the shown gaps An audit of the 32 frozen sweep inputs (the Codex catalog lane, 2026-10-01) found 14 winner pins behind the stack and gap texts that committed receipts had already answered. Both come from one cause: build_inputs.py copies winners, alternatives and the first five open_gaps from the row's sealed 2026-09-22 verdict, which only a new verdict wave re-records, and says nothing about their date. A repository-layer input now also carries verdict_checked_at (the row's checked_at), verdict_note (which fields are the sealed verdict; components_vs_upstream[].pin is the pin installed today) and open_gaps_followup: the gap-wave owner ledgers joined onto the shown gaps by index and exact text, with status, receipts present in the checkout, waves and ledgers. Existing fields keep their shape and no template changes, so the pinned prompt hashes stay. The summary line counts the ledger entries joined, dropped and beyond the five shown. Reference for the ledger reading: tools/sota-convergence/lane_packets.py gap_receipts_index. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Program record, decision 5: how a comparison's result reaches the landscape catalog A verdict wave can name only an adopted candidate as a layer's winner (a candidates[] entry with a disposition of selected or conditional; lane_packets.py build_candidate, landscape.py lane_winner_components), and a lane's return has no undetermined form. A comparison's result therefore enters catalogs/landscape/ through a candidate entry with an adopted disposition that cites the executed comparison, then a new verdict wave. The sealed 2026-09-22 verdict fields are not edited by hand; a dated limitations entry and the sweep input's dated fields carry what changed. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Architecture edition: the citations of merged #575 are landed sources; four trading acceptance classes corrected from committed receipts The 39 pending sources that named pull request 575 (37 in rows, 2 in the header) are plain source_path entries now that it merged as 75c83d8. DVC, pandera, MLflow and Inspect AI were rated none_recorded from the assessment's source review only; committed gap-wave2 receipts of 2026-09-23 record a run of each (the Codex lane's audit named them, the trading lane owner corrected its rating, and each receipt was read before its class changed). DVC, MLflow and Inspect AI become upstream_example_or_native_operation and pandera local_integration_check; the rows' classes follow (identity-provenance local_integration_check; data-quality-orchestration and evaluation-experiments structural_validation). The identity-provenance row names the four-store run as the comparison so far, not a closing one. Counts in the record: 130 winner entries, 14 none_recorded. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Architecture edition: the trading lane owner's note on identity-provenance (verbatim from its acknowledgement of #585) The note said the comparison needs a recorded DVC install first. The committed 2026-09-23 receipt records a native DVC run, so the owner replaced the sentence: the comparison needs a fresh DVC install on the target host and the point-in-time fixture; DVC's pin is the verdict's baseline 3.67.1 with no stack pin. Co-Authored-By: Claude Fable 5.1 <[email protected]> * Re-register this branch's changed files in manifests/evidence.json (hot-file protocol) Co-Authored-By: Claude Fable 5.1 <[email protected]> --------- Co-authored-by: Scout <scout@local> Co-authored-by: Claude Fable 5.1 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The user asked why ai-memory is the best in its field, and said the grand catalogs must carry the evidence.
The catalog never made a best-in-field claim; its durable-memory decision is
keep_but_compare. But it didn't carry its evidence well:What the evidence says (now in
catalogs/landscape/foundation.json, durable-memory)evidence/receipts/memory-landscape-lifecycle-20260921.json,docs/memory-landscape-maintenance.md).verdict_overturn_when. The memory-stack record's own replacement rule governs the Mac production setup.docs/decisions/2026-09-25-retire-vela-velanext.md). A15 fixed it as the confirmatory platform, so C4 and the rerun need another host, such as the workstation's RTX 4090, plus a new preregistration amendment.open_gaps[0]is superseded only in part: the representative recall and usefulness gap stands.checked_atchanged; the verdict-review gate reports 0 changed rows.The grand list carries each winner's recorded basis
catalogs/landscape/new-host-grand-list.jsonwinners now carry the ledger'swhy_selectedandevidence_refs, which is the basis the lane verdicts recorded.docs/new-host-grand-list.mdgains a short, path-free legend. It says a winner is a recorded choice, not a best-in-field claim, and gives the decision counts: 17keep_but_compare, 14retain, 1adjust. Keeping it path-free leavesscripts/release_due.py's new-host set unchanged; its output is identical on base and head.docs/{component-evidence-matrix,new-host-grand-list}.md. They name the same winners in every row as their JSON twins, which were already removed. The test fails without the fix.Review
An independent evidence-reviewer (Opus subagent, read-only) ran two rounds:
Verification (uv Python 3.13 with jsonschema/PyYAML/numpy, like CI; head b04a7ea on main ae3d3d3)
build_verdicts --check,verdict_review_gate --base origin/main(0 rows),audit_reports --check;landscape.py,validate_foundation.py,validate_catalogs.py,catalog_decisions --check;component_matrix --check,new_host_grand_list --check,build_ecosystem --check,evidence_manifest --check;validate.py.tests.test_new_host_grand_listandtests.test_blind_checkoutpass (76). The full suite before the repair round was 5311 OK; the rerun on this head is noted in a comment.Label:
lane:foundation. This PR touchesdocs/new-host-grand-list.md, a new-host document, so it is held until the v2026.09.25.1 release window closes.🤖 Generated with Claude Code