Skip to content

Commit 23333bc

Browse files
committed
test(soak): real multi-hour #581 soak (query-leak mode) + fix soak timeout cap
- _soak.yml: timeout-minutes was 30 (soak-quick) / 45 (asan) while nightly passes duration_minutes=240 — every 'nightly 4h soak' was silently KILLED at 30 min and never ran multi-hour. Raise to 300/60 so the soak can actually complete. - soak-test.sh: add CBM_SOAK_MODE=query-leak (default unchanged). It indexes once then hammers read-only tools (search_graph/query_graph/trace_path/ get_code_snippet/search_code) with NO reindex/mutation — so index_repository's cbm_mem_collect never runs to sweep the query-only leak #581 implicates. The existing RSS ceiling/slope/ratio checks become the #581 detector. - soak.yml: workflow_dispatch (duration_minutes, mode) + push to qa/soak-** ; builds the prod binary and runs the soak with timeout = duration + 60, on ubuntu + macos. Pushing a qa/soak-* branch starts a real multi-hour run. Signed-off-by: Martin Vogel <[email protected]>
1 parent c66719a commit 23333bc

3 files changed

Lines changed: 154 additions & 22 deletions

File tree

‎.github/workflows/_soak.yml‎

Lines changed: 20 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,12 @@ jobs:
4747
cc: cc
4848
cxx: c++
4949
runs-on: ${{ matrix.os }}
50-
timeout-minutes: 30
50+
# BUG FIX: this was hard-coded to 30, but the caller (nightly-soak.yml)
51+
# passes duration_minutes: 240. GitHub killed the job at 30 min, so the
52+
# "4h nightly soak" was SILENTLY TRUNCATED to 30 min and never once ran
53+
# multi-hour. Budget must always exceed the passed duration; 300 covers
54+
# the 240-min nightly with headroom (build + analysis + idle phases).
55+
timeout-minutes: 300
5156
steps:
5257
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
5358
- name: Install deps (Linux)
@@ -67,7 +72,10 @@ jobs:
6772

6873
soak-quick-windows:
6974
runs-on: windows-latest
70-
timeout-minutes: 30
75+
# BUG FIX (same 30→240 mismatch as soak-quick above): the caller passes
76+
# duration_minutes: 240, so a 30-min cap truncated the nightly soak here
77+
# too. 300 covers the 240-min nightly with headroom.
78+
timeout-minutes: 300
7179
steps:
7280
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
7381
- uses: msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2
@@ -125,7 +133,12 @@ jobs:
125133
cc: cc
126134
cxx: c++
127135
runs-on: ${{ matrix.os }}
128-
timeout-minutes: 45
136+
# ASan soak runs a FIXED 15-min soak (hard-coded below, NOT driven by
137+
# inputs.duration_minutes), but the ASan-instrumented build is slow and
138+
# leak reporting adds teardown time. 60 keeps the budget comfortably above
139+
# the 15-min run so it is never truncated. (Same class of bug as the
140+
# soak-quick 30→240 mismatch above — keep the timeout above the run length.)
141+
timeout-minutes: 60
129142
steps:
130143
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
131144
- name: Install deps (Linux)
@@ -150,7 +163,10 @@ jobs:
150163
soak-asan-windows:
151164
if: ${{ inputs.run_asan }}
152165
runs-on: windows-latest
153-
timeout-minutes: 45
166+
# FIXED 15-min soak (hard-coded below). MSYS2/Wine + ASan build is the
167+
# slowest path; 60 keeps the budget well above the run length so it is
168+
# never truncated.
169+
timeout-minutes: 60
154170
steps:
155171
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
156172
- uses: msys2/setup-msys2@66cd2cce69caa17b53920067426061ca1de3a884 # v2

‎.github/workflows/soak.yml‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
# Real multi-hour soak — #581 query-only memory-leak reproducer.
2+
#
3+
# WHY THIS EXISTS (separate from _soak.yml / nightly-soak.yml):
4+
# The nightly path was structurally incapable of running a real long soak:
5+
# 1. nightly-soak.yml passes duration_minutes: 240, but _soak.yml's
6+
# soak-quick / soak-asan jobs hard-cap `timeout-minutes: 30` (45 for
7+
# ASan). GitHub kills the job at 30 min → the "4h" soak NEVER ran past
8+
# 30 min. (Fixed in _soak.yml too, but this workflow guarantees the
9+
# right budget for the long #581 run.)
10+
# 2. scripts/soak-test.sh's default mode reindexes every 2 min;
11+
# index_repository triggers cbm_mem_collect (mimalloc page return),
12+
# which sweeps the query-only leak — masking #581 even on a long run.
13+
# This workflow drives CBM_SOAK_MODE=query-leak, which never reindexes
14+
# and never mutates files, so the leak can accumulate and be detected
15+
# by soak-test.sh's RSS slope / ratio / ceiling analysis.
16+
#
17+
# NON-GATING: workflow_dispatch + push to qa/soak-** only. Never a required
18+
# check, never blocks a merge.
19+
#
20+
# CRITICAL: timeout-minutes = duration + 60. A 240-min soak gets ~300 min.
21+
name: Soak (multi-hour #581)
22+
23+
on:
24+
workflow_dispatch:
25+
inputs:
26+
duration_minutes:
27+
description: 'Soak duration in minutes (default: 240 = 4h)'
28+
type: number
29+
default: 240
30+
mode:
31+
description: 'Soak mode (query-leak = #581 detector, no reindex/mutate)'
32+
type: choice
33+
options: ['default', 'query-leak']
34+
default: 'query-leak'
35+
# Iteration convenience: pushing a qa/soak-** branch starts a real run from
36+
# that branch's own copy of this file (no main merge needed). Non-gating.
37+
push:
38+
branches: ['qa/soak-**']
39+
40+
permissions:
41+
contents: read
42+
43+
jobs:
44+
soak:
45+
strategy:
46+
fail-fast: false
47+
matrix:
48+
# Limited to the two platforms where soak-test.sh's RSS reading works
49+
# well, to bound cost (no windows / no arm here).
50+
include:
51+
- os: ubuntu-latest
52+
cc: gcc
53+
cxx: g++
54+
- os: macos-14
55+
cc: cc
56+
cxx: c++
57+
runs-on: ${{ matrix.os }}
58+
# duration + 60 min headroom. On push (no inputs) duration defaults to 240,
59+
# so the budget is 300 min — NOT the 30 min that silently truncated nightly.
60+
timeout-minutes: ${{ fromJSON(inputs.duration_minutes || '240') + 60 }}
61+
steps:
62+
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
63+
64+
- name: Install deps (Linux)
65+
if: startsWith(matrix.os, 'ubuntu')
66+
run: sudo apt-get update && sudo apt-get install -y zlib1g-dev python3 git
67+
68+
- name: Build (prod binary)
69+
run: scripts/build.sh CC=${{ matrix.cc }} CXX=${{ matrix.cxx }}
70+
71+
- name: Soak
72+
env:
73+
# On push events there are no inputs → fall back to shell defaults
74+
# (240 min / query-leak) so a qa/soak-** push runs the real #581 soak.
75+
CBM_SOAK_MODE: ${{ inputs.mode || 'query-leak' }}
76+
DURATION_MINUTES: ${{ inputs.duration_minutes || '240' }}
77+
run: scripts/soak-test.sh build/c/codebase-memory-mcp "${DURATION_MINUTES}"
78+
79+
- name: Upload metrics
80+
if: always()
81+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
82+
with:
83+
name: soak-${{ matrix.os }}-${{ inputs.mode || 'query-leak' }}
84+
path: soak-results/
85+
retention-days: 14

‎scripts/soak-test.sh‎

Lines changed: 49 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,20 @@ DURATION_MIN="${2:?Usage: soak-test.sh <binary> <duration_minutes>}"
2020
SKIP_CRASH="${3:-}"
2121
BINARY=$(cd "$(dirname "$BINARY")" && pwd)/$(basename "$BINARY")
2222

23+
# Soak mode selector.
24+
# default = original mixed workload (queries + mutations + periodic reindex
25+
# + crash-recovery). Unchanged from before this env var existed.
26+
# query-leak = #581 detector. After the initial index, NEVER reindex and NEVER
27+
# mutate files, so the mimalloc page-return path (cbm_mem_collect,
28+
# triggered by index_repository) is never invoked and cannot sweep
29+
# a query-only leak. Phase 3 then hammers a variety of READ tools
30+
# (search_graph / query_graph / trace_path / get_code_snippet /
31+
# search_code) to exercise the query-only store-open + WAL + alloc
32+
# paths the bug report implicates. The RSS slope/ratio/ceiling
33+
# analysis below is the leak detector. The crash-recovery phase is
34+
# skipped in this mode because it reindexes (which would mask #581).
35+
CBM_SOAK_MODE="${CBM_SOAK_MODE:-default}"
36+
2337
RESULTS_DIR="soak-results"
2438
mkdir -p "$RESULTS_DIR"
2539

@@ -33,7 +47,7 @@ echo "timestamp,tool,duration_ms,exit_code" > "$LATENCY_CSV"
3347

3448
DURATION_S=$((DURATION_MIN * 60))
3549

36-
echo "=== soak-test: binary=$BINARY duration=${DURATION_MIN}m ==="
50+
echo "=== soak-test: binary=$BINARY duration=${DURATION_MIN}m mode=${CBM_SOAK_MODE} ==="
3751

3852
# ── Helper: generate realistic test project (~200 files) ─────────
3953

@@ -287,22 +301,36 @@ while [ "$(date +%s)" -lt "$END_TIME" ]; do
287301
NOW=$(date +%s)
288302
CYCLE=$((CYCLE + 1))
289303

290-
# Queries every 2 seconds
291-
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*compute.*\"}"
292-
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"compute\",\"direction\":\"both\"}"
293-
294-
# File mutation every 2 minutes
295-
if [ $((NOW - LAST_MUTATE)) -ge 120 ]; then
296-
echo "# mutation at cycle $CYCLE $(date)" >> "$SOAK_PROJECT/src/main.py"
297-
git -C "$SOAK_PROJECT" add -A 2>/dev/null
298-
git -C "$SOAK_PROJECT" -c user.email=test@test -c user.name=test commit -q -m "cycle $CYCLE" 2>/dev/null || true
299-
LAST_MUTATE=$NOW
300-
fi
301-
302-
# Full reindex every 2 minutes (compressed — simulates 15min real interval)
303-
if [ $((NOW - LAST_REINDEX)) -ge 120 ]; then
304-
mcp_call index_repository "{\"repo_path\":\"$SOAK_PROJECT\"}"
305-
LAST_REINDEX=$NOW
304+
if [ "$CBM_SOAK_MODE" = "query-leak" ]; then
305+
# ── #581 query-only leak mode ────────────────────────────────
306+
# Pure read-query hammering: no mutation, no reindex — so
307+
# cbm_mem_collect (mimalloc page return) is NEVER triggered and
308+
# cannot sweep a query-only leak. Hammer a VARIETY of read tools to
309+
# exercise the store-open + WAL + alloc paths the report implicates.
310+
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*Handle.*\"}"
311+
mcp_call query_graph "{\"project\":\"$PROJ_NAME\",\"query\":\"MATCH (n) RETURN n.name LIMIT 25\"}"
312+
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"handle_1\",\"direction\":\"both\"}"
313+
mcp_call get_code_snippet "{\"project\":\"$PROJ_NAME\",\"qualified_name\":\"handle_1\"}"
314+
mcp_call search_code "{\"project\":\"$PROJ_NAME\",\"pattern\":\"def \"}"
315+
else
316+
# ── default mode (unchanged) ─────────────────────────────────
317+
# Queries every 2 seconds
318+
mcp_call search_graph "{\"project\":\"$PROJ_NAME\",\"name_pattern\":\".*compute.*\"}"
319+
mcp_call trace_path "{\"project\":\"$PROJ_NAME\",\"function_name\":\"compute\",\"direction\":\"both\"}"
320+
321+
# File mutation every 2 minutes
322+
if [ $((NOW - LAST_MUTATE)) -ge 120 ]; then
323+
echo "# mutation at cycle $CYCLE $(date)" >> "$SOAK_PROJECT/src/main.py"
324+
git -C "$SOAK_PROJECT" add -A 2>/dev/null
325+
git -C "$SOAK_PROJECT" -c user.email=test@test -c user.name=test commit -q -m "cycle $CYCLE" 2>/dev/null || true
326+
LAST_MUTATE=$NOW
327+
fi
328+
329+
# Full reindex every 2 minutes (compressed — simulates 15min real interval)
330+
if [ $((NOW - LAST_REINDEX)) -ge 120 ]; then
331+
mcp_call index_repository "{\"repo_path\":\"$SOAK_PROJECT\"}"
332+
LAST_REINDEX=$NOW
333+
fi
306334
fi
307335

308336
# Collect diagnostics every 10 seconds (5 cycles)
@@ -324,8 +352,11 @@ IDLE_CPU=$(ps -o %cpu= -p "$SERVER_PID" 2>/dev/null | tr -d ' ' || echo "0")
324352
echo "OK: idle CPU=${IDLE_CPU}%"
325353

326354
# ── Phase 5: Crash recovery test ────────────────────────────────
355+
# Skipped in query-leak mode: crash recovery re-indexes (Phase 5 calls
356+
# index_repository), which triggers cbm_mem_collect and would mask the #581
357+
# query-only leak the whole run is trying to surface.
327358

328-
if [ "$SKIP_CRASH" != "--skip-crash-test" ]; then
359+
if [ "$SKIP_CRASH" != "--skip-crash-test" ] && [ "$CBM_SOAK_MODE" != "query-leak" ]; then
329360
echo "--- Phase 5: crash recovery ---"
330361

331362
# Kill server mid-operation, restart, verify clean index

0 commit comments

Comments
 (0)