Repository navigation
docs: reflect the v4.20.0 LongMemEval-S measurement in the README, with its artifacts - #551
Conversation
The retrieval table cited only the v4.14.1 pre-tag run. It now shows v4.20.0 beside it: Recall@10 97.8%, MRR 0.9047, n=500, measured on the exact tagged tree (86251ab) with reproduce.sh --only longmemeval --no-ablation in an isolated ephemeral pgvector pg16 container, reranker loaded, consolidation disabled. The artifacts are committed under benchmarks/results/repro/20260909-v4.20.0-longmemeval-s/: longmemeval-s.json, MANIFEST.json and START_SNAPSHOT.json. The provenance paragraph states the delta against v4.14.1 (0.4 points of Recall@10, 0.012 of MRR) and the run's own floor check as the script reports it: Recall@10 within the 0.005 tolerance of the July floor, MRR 0.0093 below its floor, non-blocking by design, with the pointer to docs/agent-guidance.md on why main no longer clears those floors. The Verification lead-in is scoped to the v4.14.1 figures, since the v4.20.0 run has no ablation campaign behind it. The earlier full run on the same tree produced the same numbers before the host killed it and its JSON was lost; its log cannot be committed (.gitignore excludes benchmarks/results/**/*.log) and an absent artifact cannot be cited, so the leg was re-measured. Co-Authored-By: Claude Fable 5.1 <[email protected]>
The JSON's manifest.repro.git_dirty is false; the standalone MANIFEST.json has no such field, which is why the first cut said none existed. Found by the fresh-context review. Co-Authored-By: Claude Fable 5.1 <[email protected]>
|
ZETETIC-REVIEW: APPROVE Verdict on head e7d5bcf. An independent fresh-context reviewer read every number in the README paragraph against the committed artifacts, field by field: overall_recall10 0.978 exact; overall_mrr 0.9046547619047619 rounding to 0.905; the deltas against the committed v4.14.1 artifact, 0.4 points of Recall@10 and 0.01205 of MRR; consolidation_call_count 0; elapsed_s 1903.6. MANIFEST.json's git_sha equals the commit the README links and equals The reviewer's one finding, a NIT, is closed in e7d5bcf: the standalone MANIFEST.json has no dirty flag, but the JSON's embedded manifest.repro.git_dirty is false, so the README now states dirty=false from it and the PR body no longer claims the flag is absent. I verified that field myself before editing. The re-measurement reproduces the earlier killed run's figures to the third decimal (0.905 / 0.978), and matches the 2026-09-08 comparison on main, so v4.20.0 shows no regression against main under the committed 0.005 tolerance; it sits below the July absolute floors, which docs/agent-guidance.md already records. CI on this head: 24 pass, 4 skipping, zero failures. |
Summary
The README's retrieval table cited only the v4.14.1 pre-tag run. This adds the v4.20.0 figure beside it, backed by artifacts produced for this change on the exact tagged tree, and scopes the Verification lead-in so it no longer claims an ablation campaign behind a figure that has none.
The measurement
benchmarks/reproduce.sh --only longmemeval --no-ablation, run from a detached worktree at the v4.20.0 commit (86251ab), isolated ephemeralpgvector/pgvector:pg16container, reranker loaded, consolidation disabled, n=500.Artifacts committed under
benchmarks/results/repro/20260909-v4.20.0-longmemeval-s/:longmemeval-s.json(overall and per-category metrics, elapsed 1903.6 s, zero consolidation calls, embedded manifest),MANIFEST.json(git SHA, dataset sha256, image, embedding model revision, reranker state, Python 3.12.11),START_SNAPSHOT.json. The standalone MANIFEST.json has no dirty flag field, but the JSON's embeddedmanifest.repro.git_dirtyisfalse, and the README now states dirty=false from it; the first cut of this body claimed no flag existed, which the fresh-context review corrected.The run's own floor check: Recall@10 0.978 against the July floor 0.982 is within the 0.005 tolerance and passes; MRR 0.9047 against 0.914 is 0.0093 below and fails, which the script treats as non-blocking by design.
docs/agent-guidance.mdalready records thatmainno longer clears those floors and that the release gate is--no-regressionagainstorigin/main. LoCoMo and BEAM have no v4.20.0 figure; both legs of the earlier full run were killed by the host for memory pressure, and this change measures only the leg it cites.README changes
Why this run and not the earlier one
A full
reproduce.shrun on the same tree earlier today completed this leg with the same numbers (0.905 / 0.978) before the host killed it, and itslongmemeval-s.jsonwas then lost. The surviving run log could not be committed:.gitignoreexcludesbenchmarks/results/**/*.log, and citing an artifact absent from the tree is not allowed here. Re-measuring was the only honest route, and the first attempt at that exposed a script defect fixed separately in #550 (--only longmemeval-ssilently selected nothing).Type of change
Test plan
Every relative link resolves, including the three new artifact paths.
scripts/check_doc_claims.py,scripts/check_version_surfaces.py(17 sites): green.tests_py/scripts/test_codex_plugin_contract.pyandtest_version_surfaces.py: 33 pass.Table cells were asserted against the JSON in the script that edited the README (0.978 exact; MRR rounds to 0.905).
All existing tests pass.
Manual verification: link resolution and cell-to-artifact assertion.
Audit notes
Self-review plus the artifacts. A fresh-context review is requested before merge.
Reviewer checklist
🤖 Generated with Claude Code