Skip to content

Record the memory-stack cross-family convergence: retain ai-memory, trial three systems and eight models, adopt none - #234

Merged
seathatflowsinourveins merged 1 commit into
mainfrom
claude/memory-stack-convergence-20260925
Sep 25, 2026
Merged

seathatflowsinourveins merged 1 commit into
mainfrom
claude/memory-stack-convergence-20260925

Conversation

@seathatflowsinourveins

Copy link
Copy Markdown
Collaborator

This records the September 25 cross-family convergence on the memory stack: which memory systems, embedders, rerankers, memory LLMs and local serving runtimes the one shared Claude Code and Codex memory should retain, trial, defer or reject. ai-memory stays the production control and nothing is adopted. No row makes a universal state-of-the-art claim, and nothing was installed, downloaded, promoted or reconfigured.

Method

  • Two frozen packets each went to two blind lanes. Neither lane saw the other's return or the catalogs.

    • The model screen covered embedders, rerankers and the memory LLM.
    • The memory-system screen covered systems to benchmark against ai-memory.
    • The Claude lanes ran on Opus 5.5 subagents with web tools.
    • The Codex lanes ran on GPT-6 Astra with live web search: 15 and 16 web_search items.
  • Three roles went to a blind GPT-6 Sol judge through codex exec in read-only mode, working from the brief alone. Each role ran in three orders, and a key maps the letters to the lanes:

    • a: the Codex return is A, listed first;
    • b: the Claude return is A, listed first;
    • c: the Codex return is A, and the Claude return is listed first.

    The judge never preferred the Codex return. Its only non-SPLIT answers came in order b, so the decisions rest on the candidate positions that hold in all three orders.

  • Two skeptic checks per row reviewed the coordinator's convergence table: 68 checks, 9 of them refuting, on 7 rows. The recorder applied every correction. The largest: Ollama did not serve every harness embedding arm, because the MiniLM arms ran in-process. Four next tests were also fixed: QMD, all-MiniLM-L6-v2, Qwen3-Embedding-8B/0.6B, and Nemotron 3.5 Lightning, whose added "unless H1 fails" gate was never preregistered. Every row citing LMEB now names the source: the LongMemEval task, nDCG@10, MTEB leaderboard API snapshot 2026-09-24. No decision changed.

  • Neutral measurements come from report-mac-descriptive.md, complete arms only. They are Mac-platform native_cli_execution, descriptive under A15.2.

    • C3 (ai-memory, production embedder, reranker off) is 0.570 recall_all@5 on the full track, n = 470.
    • Against C3: am-minilm +0.251 [+0.203, +0.299], dense-qwen3 +0.183, bm25-full +0.177 and A0 +0.170, each at Holm p 0.0001. am-keyless is +0.047 (p 0.52).
    • The layer decision still needs C4, production with the LLM reranker (A14). That is the next decision-changing test.
  • PREREGISTRATION.md amendments A13–A16 are cited. A16, which names the D2h, M2 and K1 arms, was added after the table, and C3 finished after it; both are dated addenda.

Outcome

catalogs/foundation/memory-stack-20260925.json holds 23 repository decisions, registered through catalog_decisions.py, and 20 non-repository decisions: Hugging Face models pinned by revision, Ollama tags, a production setting and one closed application.

Decision Items
Retain ai-memory 19b6429 (production control), Qwen3-Embedding-4B, ai-memory LLM listwise reranking with Qwen3.5-9B, Ollama 0.34.4, QMD 2.8.3. Inside the agentmemory trial: all-MiniLM-L6-v2, ms-marco-MiniLM-L-6-v2 and transformers.js.
Trial agentmemory 0.9.29, MemPalace 3.10.0 and Hindsight 0.10.1. Nemotron-3-Embed-8B/1B and harrier-oss-v1-0.6b. MemReranker-4B, KaLM-Reranker-V1-Small and ettin-reranker-400m-v1, in exploratory stage X only. Qwen3.6-35B-A3B and LFM2.5-2.6B. llama.cpp llama-server and MLX/mlx-lm. Every trial is isolated on VelaNext or in a trial profile.
Defer Attemory (a source and licence gate first), GBrain, Mem0 OSS, Graphiti/Zep, Supermemory, Cognee, claude-mem and MemOS. Also Qwen3-Embedding-8B/0.6B, Qwen3-Reranker-4B, Nemotron 3.5 Lightning, vLLM and LM Studio.
Reject for this role Honcho, OpenViking and Basic Memory (AGPL-3.0), the retired Letta server, Jina Reranker v3.5 (CC-BY-NC-4.0) and MemReader-4B-thinking.

Each row keeps its evidence class, its lanes and judges, a next decision-changing test and a rollback. A trial becomes a change only by passing both the C3 and C4 families (+5 pp, Holm) and then the memory-lane acceptance.

Registration and counts

  • The evidence is in evidence/artifacts/memory-stack-20260925/: 24 files, laid out like Record a cross-family discovery wave on local-model workloads beyond memory and RAG #223's record.
    • The packets, lane returns, judge briefs, key and verdicts are embedded verbatim beside parsed transcriptions.
    • convergence.json is the source-of-truth path that the coordinator's CLAUDE.md manifest names.
    • The directory also holds usage.json, provenance.json, experiment.json, README.md and SHA256SUMS.
    • Event streams, stderr, local paths and session identifiers are left out.
  • manifests/evidence.json registers the files, and the experiment is added to convergence_records.
  • In catalogs/landscape/foundation.json, the durable-memory layer (keep_but_compare) changes only in evidence_refs and limitations. No verdict field changes.
  • Repository identities go from 857 to 861. The new ones are AttemorySystem/attemory, MemTensor/MemOS, ml-explore/mlx and huggingface/transformers.js. README.md, catalogs/us-equities/manifest.json, saturation-audit.json and catalogs/foundation/README.md are updated.

Limits

  • There is one judge model, from the Codex lanes' family, and no refuter.
  • The Claude lanes' tool logs were not retained. The memory-system Codex lane read the ecosystem-catalog skill's instructions but ran no catalog query.
  • Mac figures are descriptive. C4 did not run on the Mac, and the confirmatory decisions come from the VelaNext rerun.
  • Pins were resolved after the lanes, through the GitHub and Hugging Face APIs. The Ollama manifest prefixes are the convergence writer's and were not re-checked.
  • Usage is known only for the Codex lanes and the judges. Whole-task usage is unknown.

Verification

  • These checks all exit 0 on the branch: validate.py (5,370 hashed files), validate_convergence.py --all-recorded (21 records; this one has 24 observations), landscape.py, catalog_decisions.py --check (861 identities), evidence_manifest.py --check, validate_catalogs.py, validate_foundation.py, build_verdicts.py --check, component_matrix.py --check, new_host_grand_list.py --check, build_ecosystem.py --check, saturation_ledger.py --check, audit_reports.py --check, host_receipts.py validate, gap_crosswalk.py build --check, gap_wave_ledger.py --check, the verdict-review gate against the base, shasum -c on the record's SHA256SUMS, and git diff --check.
  • Full python3 -m unittest on this Mac: 4,743 tests, 604 skipped and 2 failures. Both also fail on unchanged main here: test_adoption_bootstrap_macos.LlamaWrapperQuotingTests and test_new_host_grand_list.RecordHostToGrandListTests. The one failure this change caused, in test_lane_packets (a newcomer candidate named in shared layer prose), is fixed. CI runs the suite on Linux.

🤖 Generated with Claude Code

Two frozen packets (a model screen and a memory-system screen) went to blind Claude Opus 5.5
and Codex GPT-6 Astra lanes; three roles went to a blind GPT-6 Sol judge in three presentation
orders; the coordinator's convergence table set each row and two skeptic checks per row
reviewed it. The recorder applied every refuting check's correction, named the LMEB source
on every row citing it, and updated the rows C3 bears on from the Mac descriptive report
(C3 0.570 recall_all@5; the layer decision still needs C4).

The sealed record is evidence/artifacts/memory-stack-20260925/ (convergence.json is the source
of truth the CLAUDE.md manifest names). catalogs/foundation/memory-stack-20260925.json holds 23
registered repository decisions and 20 non-repository decisions: retain 9, trial 14, defer 14,
reject 6, adopt_within_scope none. Identities 857 -> 861. The durable-memory landscape layer
changes only in evidence_refs and limitations. Nothing was installed or promoted.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@seathatflowsinourveins seathatflowsinourveins changed the title Record the memory-stack cross-family convergence: retain ai-memory, trial three systems and nine models, adopt none Record the memory-stack cross-family convergence: retain ai-memory, trial three systems and eight models, adopt none Sep 25, 2026
@seathatflowsinourveins
seathatflowsinourveins merged commit 1fca818 into main Sep 25, 2026
23 of 26 checks passed
@seathatflowsinourveins
seathatflowsinourveins deleted the claude/memory-stack-convergence-20260925 branch September 25, 2026 07:14
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…ackets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…ackets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
seathatflowsinourveins pushed a commit that referenced this pull request Sep 26, 2026
…ackets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 418c0f0)
seathatflowsinourveins added a commit that referenced this pull request Sep 26, 2026
…bootstrap reminder, blind-packet leak, retained markitdown evidence (#314)

* Retain markitdown's installed-byte verifier, run directories and raw checker output

Codex post-merge review of #291, finding 3.

- verify_installed.py is retained byte-identical. Its 2026-09-25 stdout
  was never written to a file, so results.json says so (not_retained)
  and adds a 2026-09-26 read-only re-run on the production prefix
  (45 of 45 identical) plus a per-file listing (installed_files.py) of
  the wheel RECORD, wheel member and installed digests in RECORD form.
- run-candidate/, run-baseline/ and run-postswitch2/ (converted
  documents, PDF stderr with the home directory replaced by ~, package
  listings) are retained; their files hash to the recorded values.
  run_arm.py kept only checker counts, so check_outputs.py is re-run on
  them (out/checker-rerun-20260926.out), and the independent verifier's
  own checker.out, legs logs and integrity check are retained.
- results.json gains corrections, not_retained, steps and files; the
  receipt gains a dated limitation and regenerated provenance.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 9ec4c33)

* Keep the durable-memory incumbent's build and releases out of blind packets

Codex post-merge review of #269, finding 1 (P2).

The dated durable-memory limitations (the 19b6429/Mac/2.3.2/VelaNext
entry, the open_gaps[0] entry and the 2.4.0/2.4.1 cutover entry), the
older #234 "Production now runs ai-memory 19b6429" entry and the #234
"keeps ai-memory as the production control" sentence named the
incumbent's build, releases and role without a selection word next to a
candidate name, so build_all_packets(..., withhold=True) kept them.
Each sentence carrying those identifiers now calls the component the
incumbent, a choice phrase the prose filter (lane_packets.py
CHOICE_PHRASE, reduce_prose) drops; the ledger keeps every fact.
Verdict fields and checked_at are unchanged. A regression test builds
the real withheld durable-memory packet (default and blind-export
parameters) and asserts that 19b6429, 2.3.2, 2.4.0, 2.4.1 and
"production control" are absent while the ledger still records them.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit 418c0f0)

* Grand list legend: the JSON carries winners' basis and only open-gap counts

Codex post-merge review of #269, finding 2 (P3). The "What a winner
means" legend said the JSON carries the layer's limitations and open
gaps; the generator emits each winner's why_selected and evidence_refs
and only gap counts. The legend now says so and points to the layer
ledgers for the limitations and the open-gap text (without paths: the
page is a new-host document).

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
(cherry picked from commit caeaa4d)

* rtk hook exclusions: adopt #318's four-entry recipe and add rtk's own check to its reminder

Reconciles this branch's Codex post-merge fixes for #291 (findings 1-2)
and the #314 P2 with #318, which merged first.

- Recipe: #318's four-entry exclude_commands block stays the one set of
  guidance in recipes/README.md, bootstrap.md step 4a, token-practice.md,
  the rtk pin note and the SOTA refresh record. The single regex this
  branch had tested ('^git(\s+\S+)*\s+show(\s+\S+)*\s+(:\S|[^\s-]\S*:)'
  with "diff") is kept only as dated evidence, labelled as an alternative
  that was tested (mitigation-20260926, readback-20260926). The recipe
  also explains rtk's anchoring (registry.rs:1540-1569, 1599-1604) and
  that explicit `rtk` commands bypass the exclusions (registry.rs:1690).
- Reminder: rtk_config_reminder keeps #318's text check (the exact
  four-entry key, exactly once) and, once it passes, runs the installed
  "$bin_dir/rtk" hook check (timeout 30, stdin closed, the caller's
  environment) on `git show HEAD:x | tail -n 5`, `git -C . show
  --no-color HEAD:x | tail -n 5`, `diff a missing`, `git branch -a` and
  `git -C . branch`. Either check failing prints the reminder; a text
  pass with an rtk failure adds a hint naming the probe, because rtk
  ignores a TOML-valid file that does not deserialize (TrackingConfig,
  src/core/config.rs:152-158 at 1d87b8e7). Print-only.
- Tests: all of #318's reminder tests are kept (the silent ones now run
  against a stub rtk that honours the file); new stub tests cover an
  exact text that rtk rewrites, the [tracking] counterexample, an
  unsupported subcommand, a rewritten branch probe, extra output, the
  #291 comments counterexample, the five probes and XDG_CONFIG_HOME
  reaching rtk. An optional real-binary test (skipped without an
  installed rtk 0.50.0) covers the four-entry block, the [tracking]
  rejection with and without history_days, the old two-entry line, the
  single-regex alternative and a missing file.
- Evidence: combined-reminder-20260926.{sh,out} (hook check and the
  combined reminder on scratch configs; silent on this host's #318
  block). The rtk receipt keeps #318's addenda untouched; its added
  limitation now references #318's recipe and the combined reminder;
  results.json gains a step, files and relabelled corrections; the
  receipt's provenance is regenerated.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

* Register the post-merge fixes' files and regenerate the grand list

Hot-file commit (docs/lanes.md): manifests/evidence.json starts from
main's copy (with #318's and #315's entries) and re-registers every
changed or new file of this branch (indent 2, ensure_ascii); receipts[]
mirrors the rtk and markitdown receipts' new limitations and keeps every
other entry. component_matrix --write (no change) and
new_host_grand_list --write regenerate the reports.

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

---------

Co-authored-by: Scout <scout@local>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant