Skip to content

Commit 9574c31

Browse files
committed
fix(test): hand the benchmark index to the caller on request
The evaluation plan (docs/EVALUATION_PLAN.md §7) indexes a language with benchmark-index.sh and then answers graph questions against that index from its own MCP session. With a run-private runtime (#1696) the index was gone before that session could start, so the handoff is now explicit and opt-in. With CBM_BENCH_KEEP_RUNTIME set, a SUCCESSFUL run stops its daemon, leaves the private root in place and records the paths that reach it in <results>/<lang>/runtime-root.txt (sourceable: CBM_BENCH_RUNTIME_ROOT, CBM_RUNTIME_DIR, CBM_CACHE_DIR). Ownership of that root, including its removal, passes to the caller. A failed run cleans up regardless: there is no index worth keeping and nothing may leak. Nothing changes in the default path or in the three timing files. §7's skeleton sets the flag, sources the file before the graph session and removes the root in step 8 instead of the live-store *.db files. The contract test drives a kept run and requires the root to survive, the handoff file to be sourceable, and its paths to be the ones the product processes actually used; the unflagged run still asserts the root is gone. The kept-run assertions fail against the previous head. Signed-off-by: Anton Standrik <[email protected]>
1 parent 8f0ff31 commit 9574c31

3 files changed

Lines changed: 53 additions & 2 deletions

File tree

‎docs/EVALUATION_PLAN.md‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -283,8 +283,13 @@ for lang in $ALL_LANGS; do # ALL_LANGS = full 159-name list
283283

284284
# --- step 2: cold index in the main channel, TIMED (key metric) ---
285285
t0=$(now_ms)
286+
CBM_BENCH_KEEP_RUNTIME=1 \
286287
scripts/benchmark-index.sh ~/.local/bin/codebase-memory-mcp "$lang" /tmp/bench/"$lang" /tmp/eval-results
287288
index_ms=$(( $(now_ms) - t0 )) # clone+index wall-clock → manifest + report (§5)
289+
# The harness indexes into a run-private root (#1696). CBM_BENCH_KEEP_RUNTIME leaves that
290+
# root behind on success and records its paths; the graph session in steps 4-7 must be
291+
# started with these CBM_RUNTIME_DIR / CBM_CACHE_DIR, and step 8 removes the root.
292+
set -a; . /tmp/eval-results/"$lang"/runtime-root.txt; set +a
288293

289294
# --- step 3: record per-type histograms (zeros back-filled) ---
290295
# node-types.json, edge-types.json (every label + all 32 edge types, zeros kept, §7 below)
@@ -293,7 +298,7 @@ for lang in $ALL_LANGS; do # ALL_LANGS = full 159-name list
293298
# + per-language report + (deferred, blind) judge ---
294299

295300
# --- step 8: delete THIS language's index so the next is cold, then mark done ---
296-
rm -f ~/.cache/codebase-memory-mcp/*.db
301+
rm -rf -- "$CBM_BENCH_RUNTIME_ROOT"
297302
manifest_mark_done "$lang" "$index_ms"
298303
done
299304
```

‎scripts/benchmark-index.sh‎

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,28 @@ RESULTS_DIR="${4:?}"
1515
# shellcheck source=test-runtime.sh
1616
source "$(dirname "${BASH_SOURCE[0]}")/test-runtime.sh"
1717
cbm_test_runtime_init
18-
trap 'cbm_test_runtime_cleanup "$BINARY"' EXIT
18+
19+
# The evaluation plan (docs/EVALUATION_PLAN.md §7) indexes a language here and
20+
# then answers graph questions against that index from its own MCP session.
21+
# A run-private root would be gone before that session starts, so the caller
22+
# may ask for it to be kept: after a SUCCESSFUL run the harness stops its
23+
# daemon, leaves the root in place, records the paths that reach it in
24+
# <results>/<lang>/runtime-root.txt (sourceable), and ownership of the root —
25+
# including its removal — passes to the caller. A failed run cleans up
26+
# regardless: there is no index worth keeping, and nothing must leak. Never on
27+
# by default, or an unattended run accumulates one root per language.
28+
bench_finish() {
29+
local rc=$?
30+
if [ "$rc" -eq 0 ] && [ -n "${CBM_BENCH_KEEP_RUNTIME:-}" ] && [ -d "${OUT:-}" ]; then
31+
"$BINARY" daemon stop >/dev/null 2>&1 || true
32+
printf 'CBM_BENCH_RUNTIME_ROOT=%q\nCBM_RUNTIME_DIR=%q\nCBM_CACHE_DIR=%q\n' \
33+
"$CBM_TEST_RUNTIME_ROOT" "$CBM_RUNTIME_DIR" "$CBM_CACHE_DIR" > "$OUT/runtime-root.txt"
34+
echo " $LANG: runtime kept at $CBM_TEST_RUNTIME_ROOT; paths in $OUT/runtime-root.txt" >&2
35+
return 0
36+
fi
37+
cbm_test_runtime_cleanup "$BINARY"
38+
}
39+
trap bench_finish EXIT
1940

2041
# Resolve symlinks
2142
REPO=$(cd "$REPO" && pwd -P)

‎tests/test_benchmark_runtime_isolation_contract.sh‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,31 @@ for metric in setup-time total-time index-time; do
9090
fail "benchmark-index did not record $metric.txt"
9191
done
9292

93+
# The evaluation plan indexes a language and then reads that index from its own
94+
# MCP session (docs/EVALUATION_PLAN.md §7). Asked to keep the runtime, a
95+
# successful run must leave its root behind and record, sourceably, the paths
96+
# that reach it — and they must be the paths the product processes actually
97+
# used. Unasked, the root is gone (asserted above).
98+
KEEP_LOG="$WORKDIR/keep-environment.log"
99+
CBM_CACHE_DIR="$CALLER_CACHE" \
100+
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \
101+
CBM_BENCH_ENV_PROBE="$KEEP_LOG" \
102+
CBM_BENCH_KEEP_RUNTIME=1 \
103+
"$ROOT/scripts/benchmark-index.sh" "$ENV_PROBE" keep "$REPO" "$WORKDIR/results" \
104+
> "$WORKDIR/keep.out" 2>&1 || true
105+
HANDOFF="$WORKDIR/results/keep/runtime-root.txt"
106+
[[ -s "$HANDOFF" ]] || fail "benchmark-index was asked to keep its runtime but recorded no handoff"
107+
KEPT_ROOT=$(bash -c '. "$1" && printf "%s" "${CBM_BENCH_RUNTIME_ROOT-}"' _ "$HANDOFF")
108+
KEPT_RUNTIME=$(bash -c '. "$1" && printf "%s" "${CBM_RUNTIME_DIR-}"' _ "$HANDOFF")
109+
KEPT_CACHE=$(bash -c '. "$1" && printf "%s" "${CBM_CACHE_DIR-}"' _ "$HANDOFF")
110+
[[ -n "$KEPT_ROOT" && -d "$KEPT_ROOT" && ! -L "$KEPT_ROOT" ]] ||
111+
fail "benchmark-index did not keep its runtime root: ${KEPT_ROOT:-<none>}"
112+
[[ -d "$KEPT_ROOT/cache" && -d "$KEPT_ROOT/runtime" ]] ||
113+
fail "the kept root $KEPT_ROOT lost its cache or runtime directory"
114+
grep -qF -- "${KEPT_CACHE}"$'\t'"${KEPT_RUNTIME}" "$KEEP_LOG" ||
115+
fail "runtime-root.txt does not name the runtime and cache the product processes used"
116+
rm -rf -- "$KEPT_ROOT"
117+
93118
SEARCH_LOG="$WORKDIR/search-environment.log"
94119
CBM_CACHE_DIR="$CALLER_CACHE" \
95120
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \

0 commit comments

Comments
 (0)