Repository navigation
chore(bench): resolve stale-floors finding — isolated-instrument evidence, doc fixes, no floor changes - #121
Merged
Merged
Conversation
…caveat The v4.14.1 pre-tag guard measured LoCoMo against the live production Postgres store and read a floor FAIL (MRR 0.7984 < 0.800), even though the release's own regression criterion passed. Root cause investigated on chore/floors-rebaseline: CLAUDE.md:443 asserted "benchmarks use the production database," directly contradicting reproduce.sh's own design (an isolated ephemeral pgvector container, its documented single source of truth). Fixed the doc. Re-ran the full floor-relevant benchmark set at HEAD 5542aa7 (v4.14.1, 2 commits into main) through reproduce.sh's isolated instrument: LongMemEval passes comfortably (MRR 0.9174 vs floor 0.914). LoCoMo rep1 alone reads FAIL (MRR 0.7978), but 2 further isolated reps (0.8019, 0.8010) show same-commit noise of stdev 0.0022 against a 0.005 tolerance band — a ~2.3 signal-to-tolerance ratio, meaning a single unlucky run has a real chance of a false FAIL. The 3-run mean (0.8002) clears the floor, and is statistically indistinguishable from the historical isolated baseline across v4.10.0-v4.12.0 (mean 0.8020, delta -0.0017, within today's own measured stdev). No retrieval-code diff exists between those commits and HEAD (verified via git diff --stat). No floor-constant change: the data doesn't support one, and changing a gated number without evidence would itself violate the zetetic standard. Documented the noise caveat directly in check_floors() so a future pre-tag verifier reruns LoCoMo 3x and judges the mean instead of trusting rep1, matching how 4.14.0/4.14.1 were actually released. Evidence: benchmarks/results/repro/20260714-floors-rebaseline/ (curated) plus the raw timestamped reproduce.sh output dirs for the 4 runs. Co-Authored-By: Claude <[email protected]>
cdeust
added a commit
that referenced
this pull request
Jul 14, 2026
…th (#125) Root CLAUDE.md had grown to 463 lines and drifted from PRIVACY.md on storage (falsely asserted "PostgreSQL MANDATORY, No SQLite"). Extract the module inventory and MCP tool tables to docs/module-inventory.md and docs/mcp-tools.md (loaded via @-references), correcting the documented module counts against the actual tree (measured 2026-07-14: core 208, infrastructure 77, shared 25, handlers 131 vs the stale "108/21/11" prose) while preserving every previously-documented module description. Rewrite CLAUDE.md itself around five verifiable sections: Build & Test (with the ruff-double-check and consolidation-OFF traps), Architecture (3 sentences + @-refs), Code Style (3 greppable rules, including an honest note on 3 pre-existing core->infrastructure violations found during this pass, not newly introduced), What NOT to do (3 dated-incident prohibitions), and a condensed Scientific Implementation Standard. Benchmark score table dropped from CLAUDE.md as pure duplication of README.md/docs/arxiv-*/main.tex; the reproduce.sh-vs-production-DB distinction from PR #121 is preserved verbatim as the release-gate instrument of record. Fixes #114 Co-authored-by: Claude <[email protected]>
pull Bot
referenced
this pull request
in asleekgeek/Cortex
Jul 15, 2026
Cuts 13 commits ahead of v4.14.1: core→infrastructure layer-violation cleanup via reverse-DI ports-and-adapters (cdeust#135), check_setup MCP tool + /preflight command (#115, #119, cdeust#133), XDG-aware embedding cache_folder (cdeust#132), project settings catalogue + headless CI regimes (cdeust#131), devcontainer with prewarmed models (#129), optional OTLP telemetry (#128), stale docs/ module-count fixes (cdeust#130, #121), CLAUDE.md refactor (#125), Windows postInstall dispatch fix (#117). No retrieval-path code touched. Pre-tag guard on the exact release tree (git_sha 018c76d, isolated ephemeral pgvector container via benchmarks/reproduce.sh --no-ablation): LongMemEval MRR 0.9166/R@10 0.9820 (matches v4.14.1, no regression); LoCoMo 3-run mean MRR 0.8009/R@10 0.9146 (vs v4.14.1's 3-run mean 0.7984/0.9142, an improvement within noise); BEAM MRR 0.5437/R@10 0.7139 (within v4.14.1's established intra-day variance band). All 4 gated floors PASS (tolerance 0.005) — unlike v4.14.1 where LoCoMo MRR failed the absolute floor on its 3-run mean, 4.14.2's mean clears it by -0.0041. Evidence: benchmarks/results/repro/20260715-v4.14.2-pretag/. Co-Authored-By: Claude <[email protected]>
Closed
7 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Résout le finding « floors stales » ouvert à la release 4.14.1 (dérogation LoCoMo).
Verdict mesuré : pas de dérive réelle. Sur l'instrument isolé (reproduce.sh, conteneur pgvector éphémère) : LoCoMo moyenne 3 runs 0.8002 ± 0.0022 vs floor effectif 0.800 → PASS ; LongMemEval 0.9174/0.982 PASS confortable ; l'historique isolé committé (v4.10→v4.12 : 0.8014-0.8024) est à −0.0017 de la mesure du jour, inférieur à l'écart-type intra-commit (0.0022) — indiscernable du bruit ; et
git diff 1352f167..5542aa71 -- mcp_server/ne touche aucun code de retrieval.Deux défauts procéduraux corrigés, aucune constante modifiée (la donnée ne justifie pas de re-baseline — en changer sans preuve violerait la discipline zététique) :
CLAUDE.md:443affirmait « Benchmarks use the production database » — cause racine de la garde pré-tag 4.14.1 exécutée sur le mauvais instrument (dérive intra-journée ±0.003 de la DB de prod, dossier 20260714-v4.14.1-pretag). Corrigé.check_floors()gate un run unique alors que le bruit intra-commit mesuré (σ=0.0022) donne un ratio signal/tolérance ~2,3 — un rep1 malchanceux (0.7978 FAIL) peut échouer quand la statistique de release (moyenne ×3 : PASS) est saine. Caveat documenté au-dessus de la fonction avec les chiffres + recommandation (juger LoCoMo sur la moyenne de 3 reps).Dossier de preuve commité :
benchmarks/results/repro/20260714-floors-rebaseline/(5 JSONs) + manifests bruts (sha git, versions libs, hash reranker).Refs #120 (cas-d'école, item floors).
🤖 Generated with Claude Code
https://claude.ai/code/session_01R5Z1wfiBpADgENq6pB6E2i