fix(benchmarks): make floor check non-blocking, add --no-regression gate - #503

Summary

main itself no longer clears the published FLOOR_* thresholds in benchmarks/reproduce.sh: PR #492 measured LongMemEval MRR 0.904988 (< floor 0.914), LoCoMo MRR 3-run mean 0.779868 (< floor 0.805) and LoCoMo Recall@10 0.889506 (< floor 0.915), all via this same script on main@6ec76a0e. Every PR built on top of main therefore inherited an already-failing gate it did not cause and could not pass — this is the structural issue currently blocking the perf/green-* PR stack (#492→#502).

Closes the two issues named for this: rounding and the floor-vs-regression conflation.

Type of change

• Bug fix (non-breaking change which fixes an issue)

Changes

1. Rounding. check_floors now rounds got to the same 4-decimal precision it prints before computing delta/status, so the printed verdict always matches what a reader sees on screen instead of comparing at a hidden higher precision.
2. Floor check is now non-blocking. Same disposition as the pyright ratchet (issue type-debt: +79 pyright errors accrued on main since the 568 baseline (PRs #181/#183/#185/#186) #188: "raising the floor would just hide the debt" — no rebaseline, keep the drift visible without failing the build). check_floors still prints PASS/FAIL per metric; it no longer exit 1s.
3. New --no-regression / --baseline-ref <ref> gate (benchmarks/lib/bench_regression.sh, sourced by reproduce.sh): runs the same benchmark suite against a baseline ref (default origin/main) in an isolated git worktree, inside the same already-running ephemeral container (harnesses self-clean, so a second run against the same DB is safe), then fails only if HEAD is worse than its own baseline by more than the existing 0.005 (0.5pp) tolerance. This is the gate a PR should use to prove it doesn't regress — instead of lowering FLOOR_* to match wherever main currently sits.
4. CLAUDE.md and CHANGELOG.md updated to point at --no-regression as the PR-blocking gate.

Test plan

• bash -n benchmarks/reproduce.sh benchmarks/lib/bench_regression.sh — syntax OK.

• Extracted and ast.parse()'d every embedded Python heredoc in both files — OK.

• Sourced bench_regression.sh standalone and confirmed REGRESSION_TOLERANCE/BASELINE_REF defaults resolve correctly.

• Did not run a full reproduce.sh --no-regression pass here (needs Docker + ~1-2h wall time per PR perf(capture): reject bounded writes before loading embeddings #492's own measured durations, ~1700-2050s per benchmark, doubled). This should be exercised by a maintainer with the isolated container before relying on it to gate a release.

• All existing tests pass. (no Python production code touched; nothing under mcp_server/ changed)

• New tests added for new behavior. (bash harness script, not covered by pytest; verified via syntax/AST checks above)

• Manual verification of any UI / CLI / MCP-tool behavior changes. (CLI flag behavior traced by hand; not executed end-to-end — see caveat above)

Audit notes

• No engineering/genius review cycle run for this change (single-file bash + doc change, no mcp_server/ code touched).
• Outstanding: an actual --no-regression run against a live Docker daemon has not been executed in this session (no Docker available here). Flagging for a maintainer to run once before relying on it for a release decision.

Coding-standards compliance

• §2.2 N/A — no layer-dependency code touched.
• §3.2 N/A — bash, not TypeScript.
• §4.1 No file > 500 lines (bench_regression.sh is ~150 lines; reproduce.sh is bash, not gated by the Python-only craftsmanship AST check).
• §4.4 N/A.
• §7 Local reasoning preserved — each function has a single responsibility, factored the same way bench_container.sh already is.
• §8 Every numeric constant sourced: REGRESSION_TOLERANCE reuses FLOOR_TOLERANCE's own provenance (design §8, benchmarks/results/a3_longmemeval_post_refactor.md); the floor drift numbers are sourced to PR perf(capture): reject bounded writes before loading embeddings #492 and measured commit 6ec76a0e.
• §9 No dead code.

Breaking changes

None. check_floors behavior changes from blocking to non-blocking (a relaxation, not a tightening); --no-regression is strictly additive/opt-in.

Reviewer checklist

• CHANGELOG.md updated under [Unreleased].
• Documentation updated (CLAUDE.md FLOOR_* pointer).
• No secrets / credentials / PII in the diff.
• CI passes on the latest commit — this repo has no automated CI job that runs reproduce.sh (it's a manual, Docker-requiring release gate); the standard lint/typecheck/test CI jobs should be unaffected since no mcp_server/ code changed.

🤖 Generated with Claude Code

https://claude.ai/code/session_01StMBvNd7eVJGtpnC2zNsx1

Generated by Claude Code