Skip to content

Grand catalogs carry each winner's evidence basis; ai-memory is the retained control, not a measured best - #269

Merged
seathatflowsinourveins merged 2 commits into
mainfrom
claude/catalog-evidence-basis-20260925
Sep 25, 2026
Merged

seathatflowsinourveins merged 2 commits into
mainfrom
claude/catalog-evidence-basis-20260925

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

Why

The user asked why ai-memory is the best in its field, and said the grand catalogs must carry the evidence.

The catalog never made a best-in-field claim; its durable-memory decision is keep_but_compare. But it didn't carry its evidence well:

  • The layer rationale still said published scores "do not demonstrate better retrieval … on this project", although our own 2026-09-25 matched run showed exactly that.
  • The confirmatory rerun was still assigned to VelaNext, which was retired today.
  • The new-host grand list named every winner with no recorded basis.

What the evidence says (now in catalogs/landscape/foundation.json, durable-memory)

  • Why it's retained. ai-memory is the retained production control, not a measured best. It already holds the adopted project knowledge and is the only candidate with retained native cross-client evidence:
    • capture and query from both Claude Code and Codex;
    • consolidation and maintenance (evidence/receipts/memory-landscape-lifecycle-20260921.json, docs/memory-landscape-maintenance.md).
  • What the WSL2 host receipts cover. Only the install and a scoped page write, search, read and delete.
  • Measured retrieval is weaker. LongMemEval-S on the Mac (descriptive, n = 470, recall_all@5): C3, the Mac's 19b6429 build with qwen3-embedding:4b and the reranker off, scored 0.570. Four arms scored 0.170–0.251 higher at Holm p 0.0001: agentmemory with MiniLM (+0.251), dense Qwen3-Embedding-4B (+0.183), and two BM25 arms (+0.177, +0.170).
  • Unmeasured configurations. Production under amendment A14 is C4, with the LLM reranker, which has not run. The workstation's 2.3.2 pin with MiniLM was not measured either.
  • How the verdict changes. Only through verdict_overturn_when. The memory-stack record's own replacement rule governs the Mac production setup.
  • Dated limitations:
    • VelaNext was retired (new docs/decisions/2026-09-25-retire-vela-velanext.md). A15 fixed it as the confirmatory platform, so C4 and the rerun need another host, such as the workstation's RTX 4090, plus a new preregistration amendment.
    • open_gaps[0] is superseded only in part: the representative recall and usefulness gap stands.
  • Nothing gated changed. No verdict field and no checked_at changed; the verdict-review gate reports 0 changed rows.

The grand list carries each winner's recorded basis

  • JSON. catalogs/landscape/new-host-grand-list.json winners now carry the ledger's why_selected and evidence_refs, which is the basis the lane verdicts recorded.
  • Markdown. docs/new-host-grand-list.md gains a short, path-free legend. It says a winner is a recorded choice, not a best-in-field claim, and gives the decision counts: 17 keep_but_compare, 14 retain, 1 adjust. Keeping it path-free leaves scripts/release_due.py's new-host set unchanged; its output is identical on base and head.
  • Blind checkouts now also remove docs/{component-evidence-matrix,new-host-grand-list}.md. They name the same winners in every row as their JSON twins, which were already removed. The test fails without the fix.

Review

An independent evidence-reviewer (Opus subagent, read-only) ran two rounds:

  • First round: needs_changes with six findings. The main ones:
    • the layer rationale was shown as the basis of trading winners, where it is shared domain text;
    • the WSL2 receipts were credited with lifecycle checks they did not run;
    • the rationale contradicted the row's overturn fields.
  • Repair round: all six were fixed.
  • Re-check: two one-line text errors remained, a stale code comment and the name of the governing overturn field. Both are fixed in this head, and the test is tightened as the reviewer suggested.
  • Codex: cross-family review was skipped because of the Codex usage limit until 2026-09-30.

Verification (uv Python 3.13 with jsonschema/PyYAML/numpy, like CI; head b04a7ea on main ae3d3d3)

  • Exit 0:
    • verdict build and gates: build_verdicts --check, verdict_review_gate --base origin/main (0 rows), audit_reports --check;
    • catalog validators: landscape.py, validate_foundation.py, validate_catalogs.py, catalog_decisions --check;
    • generated reports and manifest: component_matrix --check, new_host_grand_list --check, build_ecosystem --check, evidence_manifest --check;
    • validate.py.
  • Tests: tests.test_new_host_grand_list and tests.test_blind_checkout pass (76). The full suite before the repair round was 5311 OK; the rerun on this head is noted in a comment.

Label: lane:foundation. This PR touches docs/new-host-grand-list.md, a new-host document, so it is held until the v2026.09.25.1 release window closes.

🤖 Generated with Claude Code

Scout and others added 2 commits September 25, 2026 13:44
…le-memory basis with its evidence

The user asked why ai-memory is the best in its field and required the grand
catalogs to carry evidence. The catalog never claimed that, but its durable-
memory rationale still said published scores do not show better retrieval on
this project, although the 2026-09-25 matched LongMemEval-S run did.

- foundation.json durable-memory: the rationale says ai-memory is the
  retained production control, not a measured best; attributes cross-client
  evidence to the 2026-09-21 lifecycle record, limits the WSL2 host receipts
  to what they ran; gives C3 (Mac 19b6429, qwen3-embedding:4b, reranker off)
  0.570 recall_all@5 against four arms 0.170-0.251 higher (agentmemory with
  MiniLM, dense Qwen3-Embedding-4B, two BM25 arms); notes production is C4
  under A14 (not run) and the workstation's 2.3.2/MiniLM was not measured;
  points at verdict_overturn_when. Two dated limitations: VelaNext retired
  (new decision record), so C4 and the confirmatory rerun need another host
  and a new amendment; open_gaps[0] superseded only in part. No verdict
  field or checked_at changes (verdict-review gate: 0 rows).
- new_host_grand_list.py: the JSON carries each winner's ledger why_selected
  and evidence_refs; the Markdown gains a path-free legend (a winner is a
  recorded choice, not a best-in-field claim; 17 keep_but_compare, 14
  retain, 1 adjust), so release_due's new-host set is unchanged.
- blind_checkout.py: blind checkouts also remove the rendered matrix and
  grand-list Markdown, which name the same winners as their removed JSON.
- docs/decisions/2026-09-25-retire-vela-velanext.md records the retirement.

Independent review: needs_changes (6 findings), one repair round, then two
one-line text fixes it named; all applied.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
@seathatflowsinourveins seathatflowsinourveins added the lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers label Sep 25, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator Author

Full unit suite on head b04a7ea (uv Python 3.13 with jsonschema/PyYAML/numpy): Ran 5311 tests in 442.004s OK (skipped=613)

@seathatflowsinourveins
seathatflowsinourveins merged commit 89cf253 into main Sep 25, 2026
23 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/catalog-evidence-basis-20260925 branch September 25, 2026 18:28
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…ackets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…counts

Codex post-merge review of #269, finding 2 (P3). The "What a winner
means" legend said the JSON carries the layer's limitations and open
gaps; the generator emits each winner's why_selected and evidence_refs
and only gap counts. The legend now says so and points to the layer
ledgers for the limitations and the open-gap text (without paths: the
page is a new-host document).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…ackets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…counts

Codex post-merge review of #269, finding 2 (P3). The "What a winner
means" legend said the JSON carries the layer's limitations and open
gaps; the generator emits each winner's why_selected and evidence_refs
and only gap counts. The legend now says so and points to the layer
ledgers for the limitations and the open-gap text (without paths: the
page is a new-host document).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…ackets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 418c0f0)
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…counts

Codex post-merge review of #269, finding 2 (P3). The "What a winner
means" legend said the JSON carries the layer's limitations and open
gaps; the generator emits each winner's why_selected and evidence_refs
and only gap counts. The legend now says so and points to the layer
ledgers for the limitations and the open-gap text (without paths: the
page is a new-host document).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit caeaa4d)
seathatflowsinourveins added a commit that referenced this pull request Sep 26, 2026
…bootstrap reminder, blind-packet leak, retained markitdown evidence (#314)

* Retain markitdown's installed-byte verifier, run directories and raw checker output

Codex post-merge review of #291, finding 3.

- verify_installed.py is retained byte-identical. Its 2026-09-25 stdout
  was never written to a file, so results.json says so (not_retained)
  and adds a 2026-09-26 read-only re-run on the production prefix
  (45 of 45 identical) plus a per-file listing (installed_files.py) of
  the wheel RECORD, wheel member and installed digests in RECORD form.
- run-candidate/, run-baseline/ and run-postswitch2/ (converted
  documents, PDF stderr with the home directory replaced by ~, package
  listings) are retained; their files hash to the recorded values.
  run_arm.py kept only checker counts, so check_outputs.py is re-run on
  them (out/checker-rerun-20260926.out), and the independent verifier's
  own checker.out, legs logs and integrity check are retained.
- results.json gains corrections, not_retained, steps and files; the
  receipt gains a dated limitation and regenerated provenance.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 9ec4c33)

* Keep the durable-memory incumbent's build and releases out of blind packets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 418c0f0)

* Grand list legend: the JSON carries winners' basis and only open-gap counts

Codex post-merge review of #269, finding 2 (P3). The "What a winner
means" legend said the JSON carries the layer's limitations and open
gaps; the generator emits each winner's why_selected and evidence_refs
and only gap counts. The legend now says so and points to the layer
ledgers for the limitations and the open-gap text (without paths: the
page is a new-host document).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit caeaa4d)

* rtk hook exclusions: adopt #318's four-entry recipe and add rtk's own check to its reminder

Reconciles this branch's Codex post-merge fixes for #291 (findings 1-2)
and the #314 P2 with #318, which merged first.

- Recipe: #318's four-entry exclude_commands block stays the one set of
  guidance in recipes/README.md, bootstrap.md step 4a, token-practice.md,
  the rtk pin note and the SOTA refresh record. The single regex this
  branch had tested ('^git(\s+\S+)*\s+show(\s+\S+)*\s+(:\S|[^\s-]\S*:)'
  with "diff") is kept only as dated evidence, labelled as an alternative
  that was tested (mitigation-20260926, readback-20260926). The recipe
  also explains rtk's anchoring (registry.rs:1540-1569, 1599-1604) and
  that explicit `rtk` commands bypass the exclusions (registry.rs:1690).
- Reminder: rtk_config_reminder keeps #318's text check (the exact
  four-entry key, exactly once) and, once it passes, runs the installed
  "$bin_dir/rtk" hook check (timeout 30, stdin closed, the caller's
  environment) on `git show HEAD:x | tail -n 5`, `git -C . show
  --no-color HEAD:x | tail -n 5`, `diff a missing`, `git branch -a` and
  `git -C . branch`. Either check failing prints the reminder; a text
  pass with an rtk failure adds a hint naming the probe, because rtk
  ignores a TOML-valid file that does not deserialize (TrackingConfig,
  src/core/config.rs:152-158 at 1d87b8e7). Print-only.
- Tests: all of #318's reminder tests are kept (the silent ones now run
  against a stub rtk that honours the file); new stub tests cover an
  exact text that rtk rewrites, the [tracking] counterexample, an
  unsupported subcommand, a rewritten branch probe, extra output, the
  #291 comments counterexample, the five probes and XDG_CONFIG_HOME
  reaching rtk. An optional real-binary test (skipped without an
  installed rtk 0.50.0) covers the four-entry block, the [tracking]
  rejection with and without history_days, the old two-entry line, the
  single-regex alternative and a missing file.
- Evidence: combined-reminder-20260926.{sh,out} (hook check and the
  combined reminder on scratch configs; silent on this host's #318
  block). The rtk receipt keeps #318's addenda untouched; its added
  limitation now references #318's recipe and the combined reminder;
  results.json gains a step, files and relabelled corrections; the
  receipt's provenance is regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Register the post-merge fixes' files and regenerate the grand list

Hot-file commit (docs/lanes.md): manifests/evidence.json starts from
main's copy (with #318's and #315's entries) and re-registers every
changed or new file of this branch (indent 2, ensure_ascii); receipts[]
mirrors the rtk and markitdown receipts' new limitations and keeps every
other entry. component_matrix --write (no change) and
new_host_grand_list --write regenerate the reports.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Oct 1, 2026
…, the arms without a result and upstream v2.5.2

The row's verdict fields (winners, alternatives, open_gaps) are the sealed 2026-09-22 wave and stay as recorded.
The note follows the practice of #269 and #307: a dated limitations entry. It records that ai-memory is the source
host's selection of record and the comparison's reference arm, that the merit winner is undetermined, which arms of
the frozen harness have no result (C4, D2h, K1, M1, M2), that the 2026-09-25 result is D2 against C3 with the
reranker off, that upstream published v2.5.2 on 2026-10-01, and that a verdict wave can name only an adopted
candidate as winner.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
seathatflowsinourveins added a commit that referenced this pull request Oct 1, 2026
…puts; durable-memory note; landed #575 citations and four trading acceptance classes (#585)

* Durable-memory landscape row: dated 2026-10-01 note on the merit rule, the arms without a result and upstream v2.5.2

The row's verdict fields (winners, alternatives, open_gaps) are the sealed 2026-09-22 wave and stay as recorded.
The note follows the practice of #269 and #307: a dated limitations entry. It records that ai-memory is the source
host's selection of record and the comparison's reference arm, that the merit winner is undetermined, which arms of
the frozen harness have no result (C4, D2h, K1, M1, M2), that the 2026-09-25 result is D2 against C3 with the
reranker off, that upstream published v2.5.2 on 2026-10-01, and that a verdict wave can name only an adopted
candidate as winner.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Sweep inputs: date the sealed verdict fields and join the gap-wave ledgers onto the shown gaps

An audit of the 32 frozen sweep inputs (the Codex catalog lane, 2026-10-01) found 14 winner pins behind the stack
and gap texts that committed receipts had already answered. Both come from one cause: build_inputs.py copies
winners, alternatives and the first five open_gaps from the row's sealed 2026-09-22 verdict, which only a new verdict
wave re-records, and says nothing about their date.

A repository-layer input now also carries verdict_checked_at (the row's checked_at), verdict_note (which fields are
the sealed verdict; components_vs_upstream[].pin is the pin installed today) and open_gaps_followup: the gap-wave
owner ledgers joined onto the shown gaps by index and exact text, with status, receipts present in the checkout,
waves and ledgers. Existing fields keep their shape and no template changes, so the pinned prompt hashes stay.
The summary line counts the ledger entries joined, dropped and beyond the five shown. Reference for the ledger
reading: tools/sota-convergence/lane_packets.py gap_receipts_index.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Program record, decision 5: how a comparison's result reaches the landscape catalog

A verdict wave can name only an adopted candidate as a layer's winner (a candidates[] entry with a disposition of
selected or conditional; lane_packets.py build_candidate, landscape.py lane_winner_components), and a lane's return
has no undetermined form. A comparison's result therefore enters catalogs/landscape/ through a candidate entry with
an adopted disposition that cites the executed comparison, then a new verdict wave. The sealed 2026-09-22 verdict
fields are not edited by hand; a dated limitations entry and the sweep input's dated fields carry what changed.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Architecture edition: the citations of merged #575 are landed sources; four trading acceptance classes corrected from committed receipts

The 39 pending sources that named pull request 575 (37 in rows, 2 in the header) are plain source_path entries now
that it merged as 75c83d8. DVC, pandera, MLflow and Inspect AI were rated none_recorded from the assessment's
source review only; committed gap-wave2 receipts of 2026-09-23 record a run of each (the Codex lane's audit named
them, the trading lane owner corrected its rating, and each receipt was read before its class changed). DVC, MLflow
and Inspect AI become upstream_example_or_native_operation and pandera local_integration_check; the rows'
classes follow (identity-provenance local_integration_check; data-quality-orchestration and evaluation-experiments
structural_validation). The identity-provenance row names the four-store run as the comparison so far, not a
closing one. Counts in the record: 130 winner entries, 14 none_recorded.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Architecture edition: the trading lane owner's note on identity-provenance (verbatim from its acknowledgement of #585)

The note said the comparison needs a recorded DVC install first. The committed 2026-09-23 receipt records a native
DVC run, so the owner replaced the sentence: the comparison needs a fresh DVC install on the target host and the
point-in-time fixture; DVC's pin is the verdict's baseline 3.67.1 with no stack pin.

Co-Authored-By: Claude Fable 5.1 <[email protected]>

* Re-register this branch's changed files in manifests/evidence.json (hot-file protocol)

Co-Authored-By: Claude Fable 5.1 <[email protected]>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Fable 5.1 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lane:foundation Foundation lane: Claude/Codex setup, hosts, memory, RAG, research, workers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant