Skip to content

Commit 132460f

Browse files
committed
fix: cbm_mem_rss() undercounts on Linux, blinding the ceiling/backpressure
Discovered while calibrating tests/mem_ceiling_abort.sh: a synthetic large-file index genuinely reached ~2.2-2.75GB RSS (confirmed via ps and mi_process_info's own peak_rss field, which read correctly), yet cbm_mem_rss() reported ~4MB at the exact same moments, so neither the new hard ceiling nor the PRE-EXISTING advisory budget/backpressure (pass_parallel.c:559, cbm_mem_over_budget) ever saw real memory pressure during concurrent large-file parsing. Root cause: vendored/mimalloc/src/prim/unix/prim.c's _mi_prim_process_info() never sets pinfo->current_rss on Linux — only peak_rss (via getrusage's ru_maxrss, a monotonic high-water mark). current_rss silently falls back to mimalloc's own internal committed-page counter, which this project deliberately tunes low (mi_option_arena_eager_commit=0 + purge_decommits=1 + purge_delay=0, set in cbm_mem_init to reduce upfront memory) — so on Linux, "current RSS" was never actually current RSS, it was a low-biased mimalloc-internal metric that happened to be nonzero often enough to defeat the existing `current_rss > 0` ASan-only fallback guard. Fix: on Linux, cbm_mem_rss() now prefers os_rss() (/proc/self/statm, already implemented, unaffected by mimalloc's internal accounting) as the PRIMARY source, falling back to mimalloc's value only if /proc is unavailable. macOS/Windows unaffected (mimalloc's primitives set current_rss correctly there via task_info/GetProcessMemoryInfo). Verified via a calibrated 40x8MB synthetic large-file index on IO: before the fix, a 2100MB forced ceiling never tripped despite RSS reaching 2.75GB; after the fix, the same fixture aborts cleanly at rss_mb=2135 against ceiling_mb=2100 with the expected diagnostic dump. Full ASan test suite re-run clean (0 FAIL) after the fix — the pre-existing test_incremental.c:302 RSS-delta flake (a live FastAPI clone that had grown past its 2048MB assumption) did not recur, consistent with more accurate RSS reporting also improving the existing backpressure's effectiveness. tests/mem_ceiling_abort.sh: fixed two harness bugs found while calibrating this — (1) `mode=full` index_repository does not force a re-parse when file hashes are unchanged from a prior run against the same repo_path (routes to incremental.noop), so every run needing a genuine full extraction now resets the target's cached .db first; (2) the forced test ceiling is now calibrated against a directly-probed peak RSS for the exact fixture shape (2100MB, between the 2048MB floor and an observed ~2.2-2.75GB peak) instead of an unverified guess.
1 parent d5feb6e commit 132460f

2 files changed

Lines changed: 103 additions & 42 deletions

File tree

‎src/foundation/mem.c‎

Lines changed: 29 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -146,13 +146,41 @@ void cbm_mem_init(double ram_fraction) {
146146
}
147147

148148
size_t cbm_mem_rss(void) {
149+
#if defined(__linux__) && !defined(_WIN32)
150+
/* Linux: mimalloc's _mi_prim_process_info() (vendored/mimalloc/src/
151+
* prim/unix/prim.c) never sets pinfo->current_rss on Linux — only
152+
* peak_rss (via getrusage's ru_maxrss, a high-water mark). current_rss
153+
* silently falls back to mimalloc's OWN committed-page counter
154+
* (mi_process_info()'s pinfo.current_commit default), which this
155+
* project deliberately tunes low via mi_option_arena_eager_commit=0 +
156+
* mi_option_purge_decommits=1 + mi_option_purge_delay=0 (cbm_mem_init,
157+
* above) to reduce upfront memory. Combined, "current_rss" on Linux
158+
* can read near-zero (observed: 4MB) while true RSS is multiple GB —
159+
* a small-but-NONZERO value that defeated the `current_rss > 0`
160+
* ASan-only fallback guard below, silently blinding both this
161+
* function's callers (cbm_mem_over_budget backpressure AND the
162+
* enforcing ceiling in this same file) to real memory pressure during
163+
* concurrent large-file parsing — the exact mechanism the 2026-07-01
164+
* incident diagnosed. os_rss() (/proc/self/statm) is unaffected by
165+
* mimalloc's internal accounting and is authoritative OS-reported RSS
166+
* on every Linux build regardless of allocator tuning, so it is the
167+
* PRIMARY source here, not a last-resort fallback. Verified via a
168+
* calibrated 40x8MB synthetic large-file index on IO: mi_process_info
169+
* current_rss=4MB, actual RSS (ps/proc)=~2.2GB, os_rss()=~2.2GB. */
170+
size_t rss = os_rss();
171+
if (rss > 0) {
172+
return rss;
173+
}
174+
/* Extremely unlikely (/proc unavailable) — fall through to mimalloc. */
175+
#endif
149176
size_t current_rss = 0;
150177
size_t peak_rss = 0;
151178
mi_process_info(NULL, NULL, NULL, &current_rss, &peak_rss, NULL, NULL, NULL);
152179
if (current_rss > 0) {
153180
return current_rss;
154181
}
155-
/* Fallback for ASan builds (MI_OVERRIDE=0) */
182+
/* Fallback for ASan builds (MI_OVERRIDE=0) and any other platform
183+
* where mimalloc's current_rss is unavailable/zero. */
156184
return os_rss();
157185
}
158186

‎tests/mem_ceiling_abort.sh‎

Lines changed: 74 additions & 41 deletions
Original file line numberDiff line numberDiff line change
@@ -13,9 +13,10 @@
1313
# Row 7 — A normal full index of the real repo does NOT abort (healthy
1414
# path preserved; anti-false-positive guard for R3).
1515
# Row 9 — CBM_MEM_CEILING_MB override adjusts the abort threshold: forced
16-
# low, the HEALTHY real-repo index now aborts (proves the knob
17-
# bites); unset uses the default and completes; invalid warns and
18-
# falls back to the default (completes).
16+
# below a large synthetic index's actual RSS, that index now
17+
# aborts (proves the knob bites); unset uses the default and
18+
# completes; invalid warns and falls back to the default
19+
# (completes).
1920
# Row 10 (real-environment half) — both the abort and no-abort verdicts
2021
# are read against the REAL RSS of a real process, not an
2122
# injected value.
@@ -26,6 +27,15 @@
2627
# abort during extract precedes any DB write. This script asserts no .db
2728
# file (partial or otherwise) is left behind by an aborted run.
2829
#
30+
# IMPORTANT — incremental indexing gotcha: `index_repository` with
31+
# mode=full does NOT force a full re-parse when file hashes are unchanged
32+
# from a prior run against the same repo_path (it routes to
33+
# incremental.noop). Every run below that needs a genuine full extraction
34+
# pass (i.e. any run whose PURPOSE is to exercise cbm_max_file_bytes() /
35+
# cbm_mem_abort_if_over_ceiling() on real file reads) deletes the target's
36+
# .db first. Skipping that reset is a silent false-pass, not a crash — the
37+
# process still exits 0 (it did no real work), so watch for that pattern.
38+
#
2939
# Usage: bash tests/mem_ceiling_abort.sh
3040
# Exit 0 on success, non-zero on failure. SLOW (generates GB-scale fixtures) —
3141
# intended for the IO build host, not routine `make test`.
@@ -51,6 +61,19 @@ fi
5161

5262
project_name() { printf '%s' "$1" | sed 's#^/##; s#[^A-Za-z0-9._-]#-#g'; }
5363

64+
db_path_for() {
65+
local repo="$1"
66+
printf '%s/%s.db' "$CACHE_DIR" "$(project_name "$repo")"
67+
}
68+
69+
# Force the NEXT index_repository call against $1 to do a genuine full
70+
# extraction (not an incremental no-op) by wiping its cached .db/.tmp.
71+
reset_index_state() {
72+
local repo="$1" db
73+
db="$(db_path_for "$repo")"
74+
rm -f "$db" "${db}.tmp"
75+
}
76+
5477
# Run `cli index_repository` against $1, with env overrides from the
5578
# remaining args (NAME=VALUE pairs), capturing exit code + stderr.
5679
# Sets: LAST_EXIT, LAST_LOG (path).
@@ -64,13 +87,9 @@ run_index() {
6487
LAST_EXIT=$?
6588
}
6689

67-
db_path_for() {
68-
local repo="$1"
69-
printf '%s/%s.db' "$CACHE_DIR" "$(project_name "$repo")"
70-
}
71-
7290
# ── Row 7 + row 10 (no-abort half): healthy real-repo index, default ceiling ──
7391
echo "[mem_ceiling] Row 7/10: healthy real-repo index must NOT abort (default ceiling)..." >&2
92+
reset_index_state "$ROOT"
7493
run_index "$ROOT"
7594
if [[ "$LAST_EXIT" -ne 0 ]]; then
7695
echo "[mem_ceiling] FAIL [row7]: healthy real-repo index aborted (exit=$LAST_EXIT) under default ceiling" >&2
@@ -83,6 +102,7 @@ rm -f "$LAST_LOG"
83102

84103
# ── Row 9 (unset leg): explicit unset also completes ─────────────────────
85104
echo "[mem_ceiling] Row 9 (unset): explicit CBM_MEM_CEILING_MB unset uses default, completes..." >&2
105+
reset_index_state "$ROOT"
86106
run_index "$ROOT"
87107
if [[ "$LAST_EXIT" -ne 0 ]]; then
88108
echo "[mem_ceiling] FAIL [row9-unset]: index aborted with ceiling env unset (exit=$LAST_EXIT)" >&2
@@ -94,6 +114,7 @@ rm -f "$LAST_LOG"
94114

95115
# ── Row 9 (invalid leg): non-numeric override warns + falls back, completes ──
96116
echo "[mem_ceiling] Row 9 (invalid): non-numeric CBM_MEM_CEILING_MB falls back to default..." >&2
117+
reset_index_state "$ROOT"
97118
run_index "$ROOT" CBM_MEM_CEILING_MB="not-a-number"
98119
if [[ "$LAST_EXIT" -ne 0 ]]; then
99120
echo "[mem_ceiling] FAIL [row9-invalid]: index aborted with invalid ceiling env (exit=$LAST_EXIT)" >&2
@@ -108,56 +129,68 @@ else
108129
fi
109130
rm -f "$LAST_LOG"
110131

111-
# ── Row 9 (low override leg) + Row 6 + R4: force ceiling low, healthy repo now aborts ──
112-
echo "[mem_ceiling] Row 6/9(low)/R4: CBM_MEM_CEILING_MB forced to the floor (2048MB) — healthy repo must now abort with a diagnostic dump..." >&2
113-
db_before="$(db_path_for "$ROOT")"
114-
rm -f "$db_before" "${db_before}.tmp"
115-
run_index "$ROOT" CBM_MEM_CEILING_MB="2048"
132+
# ── Row 6 + Row 9(low) + R4: synthetic large-file repro forces cumulative
133+
# RSS past a ceiling set BELOW its expected peak but ABOVE the 2048MB
134+
# floor (mem_ceiling_floor_applies_on_tiny_ram in test_mem.c already
135+
# pins the floor's own arithmetic in-process; this proves the knob bites
136+
# on a REAL run — a ceiling below the floor would just get clamped up
137+
# to the floor and never trip, so the fixture must genuinely exceed the
138+
# CHOSEN ceiling, not the floor). Sized well past what a healthy 587-file
139+
# real repo would ever reach (~1-1.3GB peak per the diagnosis), so a
140+
# false negative here is a genuine "doesn't bite" signal, not sizing
141+
# noise. ──────────────────────────────────────────────────────────────
142+
CEILING_TEST_MB=2100
143+
echo "[mem_ceiling] Row 6/9(low)/R4: generating a large-file repro sized to exceed a ${CEILING_TEST_MB}MB forced ceiling..." >&2
144+
WORKDIR="$(mktemp -d /tmp/cbm_mem_ceiling_repro.XXXXXX)"
145+
# 40 tiny + 40 large (~8MB each = ~320MB source) — mirrors the ORIGINAL
146+
# diagnosis repro shape (600 tiny + 40 large ~7.6MB files -> 8.8GB on IO)
147+
# on file COUNT and per-file size (tiny-file count trimmed 600->40; tiny
148+
# files barely contribute RSS, so this doesn't change the mechanism).
149+
# CALIBRATED by direct probe on IO/CBM_WORKERS=8 with NO ceiling override:
150+
# this exact shape peaks at rss_mb=2198 (parallel.extract.mem log). The
151+
# floor (CBM_MEM_CEILING_FLOOR_MB, mem.c) is 2048MB — a ceiling below that
152+
# gets clamped UP to the floor and would never trip (this is what silently
153+
# broke the first two attempts at 3000MB, and an earlier 20x15MB/3000MB
154+
# combination that just didn't reach 3000MB at all). 2100MB sits strictly
155+
# between the floor (2048) and the calibrated peak (2198), so it is NOT
156+
# floor-clamped and WILL trip against this exact fixture shape on this
157+
# exact host. Re-calibrate CEILING_TEST_MB (or the gen call) if run on a
158+
# host with materially different allocator/CPU behaviour.
159+
bash "$GEN" "$WORKDIR" 40 40 8
160+
reset_index_state "$WORKDIR"
161+
run_index "$WORKDIR" CBM_MEM_CEILING_MB="$CEILING_TEST_MB" CBM_WORKERS="8"
162+
db_synth="$(db_path_for "$WORKDIR")"
116163
if [[ "$LAST_EXIT" -eq 0 ]]; then
117-
echo "[mem_ceiling] FAIL [row6/row9-low]: index did NOT abort with ceiling forced to the floor (proves the knob doesn't bite, or the real repo genuinely never nears 2GB on this host — re-run with a smaller floor override or bigger synthetic fixture below)" >&2
164+
echo "[mem_ceiling] FAIL [row6/row9-low]: synthetic fixture did NOT trip a ${CEILING_TEST_MB}MB forced ceiling — either the knob doesn't bite, or the fixture is undersized for this host's allocator behaviour (IO: 31GB/8 cores). Widen n_large/size_mb in the gen call above if this needs to be load-bearing on a beefier host." >&2
118165
FAILURES=$((FAILURES + 1))
119166
else
120167
echo "[mem_ceiling] index aborted as expected (exit=$LAST_EXIT)" >&2
121168
if ! grep -q "mem.ceiling.abort" "$LAST_LOG"; then
122169
echo "[mem_ceiling] FAIL [row6]: no mem.ceiling.abort diagnostic dump in output" >&2
170+
tail -10 "$LAST_LOG" >&2
123171
FAILURES=$((FAILURES + 1))
124172
elif ! grep -qE "file=\S+ phase=\S+ rss_mb=[0-9]+ ceiling_mb=[0-9]+" "$LAST_LOG"; then
125173
echo "[mem_ceiling] FAIL [row6]: diagnostic dump missing file/phase/rss_mb/ceiling_mb fields" >&2
126-
tail -5 "$LAST_LOG" >&2
174+
grep "mem.ceiling.abort" "$LAST_LOG" >&2
127175
FAILURES=$((FAILURES + 1))
128176
else
129177
echo "[mem_ceiling] PASS [row6]: diagnostic dump present with file+phase+rss" >&2
178+
grep "mem.ceiling.abort" "$LAST_LOG" >&2
130179
fi
131180
fi
132181
# R4: an aborted run must leave no (partial) .db file — the abort happens
133-
# strictly before dump_and_persist_hashes()/cbm_gbuf_dump_to_sqlite().
134-
if [[ -f "$db_before" || -f "${db_before}.tmp" ]]; then
135-
echo "[mem_ceiling] FAIL [R4]: aborted run left a .db (or .db.tmp) file — store-integrity violation: $db_before" >&2
136-
FAILURES=$((FAILURES + 1))
137-
else
138-
echo "[mem_ceiling] PASS [R4]: aborted run left no .db file (extract-phase abort precedes any SQLite write)" >&2
139-
fi
140-
rm -f "$LAST_LOG" "$db_before" "${db_before}.tmp"
141-
142-
# ── Row 6 (synthetic large-file variant): generate a fixture that exceeds
143-
# a forced-low ceiling purely from cumulative parse RSS, independent of
144-
# whether the real repo happens to be big enough on this host ─────────
145-
echo "[mem_ceiling] Row 6 (synthetic): generating large-file repro to force the ceiling from cumulative RSS..." >&2
146-
WORKDIR="$(mktemp -d /tmp/cbm_mem_ceiling_repro.XXXXXX)"
147-
bash "$GEN" "$WORKDIR" 20 8 8 # 20 tiny + 8 large (~8MB each = ~64MB source; parse working
148-
# sets multiply this several-fold across concurrent workers)
149-
db_synth="$(db_path_for "$WORKDIR")"
150-
rm -f "$db_synth" "${db_synth}.tmp"
151-
run_index "$WORKDIR" CBM_MEM_CEILING_MB="2048" CBM_WORKERS="8"
152-
if [[ "$LAST_EXIT" -eq 0 ]]; then
153-
echo "[mem_ceiling] WARN [row6-synth]: synthetic fixture did not trip the forced ceiling on this host (IO's 31GB may absorb 8x8MB files even at 8 workers) — not counted as a failure since the real-repo variant above already proved the abort path; widen n_large/size_mb if this needs to be load-bearing" >&2
154-
else
155-
if grep -q "mem.ceiling.abort" "$LAST_LOG"; then
156-
echo "[mem_ceiling] PASS [row6-synth]: synthetic large-file repro tripped the ceiling with a diagnostic dump" >&2
157-
else
158-
echo "[mem_ceiling] FAIL [row6-synth]: synthetic fixture aborted (exit=$LAST_EXIT) but with no diagnostic dump" >&2
182+
# strictly before dump_and_persist_hashes()/cbm_gbuf_dump_to_sqlite(). Only
183+
# meaningful when the run actually aborted (LAST_EXIT != 0) — a completed
184+
# run legitimately leaves a .db behind, that is not a violation.
185+
if [[ "$LAST_EXIT" -ne 0 ]]; then
186+
if [[ -f "$db_synth" || -f "${db_synth}.tmp" ]]; then
187+
echo "[mem_ceiling] FAIL [R4]: aborted run left a .db (or .db.tmp) file — store-integrity violation: $db_synth" >&2
159188
FAILURES=$((FAILURES + 1))
189+
else
190+
echo "[mem_ceiling] PASS [R4]: aborted run left no .db file (extract-phase abort precedes any SQLite write)" >&2
160191
fi
192+
else
193+
echo "[mem_ceiling] SKIP [R4]: run did not abort, nothing to check (see row6/row9-low failure above)" >&2
161194
fi
162195
rm -f "$LAST_LOG" "$db_synth" "${db_synth}.tmp"
163196

0 commit comments

Comments
 (0)