You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 132460f
Browse filesBrowse the repository at this point in the historyBrowse files
fix: cbm_mem_rss() undercounts on Linux, blinding the ceiling/backpressure
Discovered while calibrating tests/mem_ceiling_abort.sh: a synthetic
large-file index genuinely reached ~2.2-2.75GB RSS (confirmed via ps and
mi_process_info's own peak_rss field, which read correctly), yet
cbm_mem_rss() reported ~4MB at the exact same moments, so neither the new
hard ceiling nor the PRE-EXISTING advisory budget/backpressure
(pass_parallel.c:559, cbm_mem_over_budget) ever saw real memory pressure
during concurrent large-file parsing.
Root cause: vendored/mimalloc/src/prim/unix/prim.c's
_mi_prim_process_info() never sets pinfo->current_rss on Linux — only
peak_rss (via getrusage's ru_maxrss, a monotonic high-water mark).
current_rss silently falls back to mimalloc's own internal committed-page
counter, which this project deliberately tunes low
(mi_option_arena_eager_commit=0 + purge_decommits=1 + purge_delay=0, set
in cbm_mem_init to reduce upfront memory) — so on Linux, "current RSS" was
never actually current RSS, it was a low-biased mimalloc-internal metric
that happened to be nonzero often enough to defeat the existing
`current_rss > 0` ASan-only fallback guard.
Fix: on Linux, cbm_mem_rss() now prefers os_rss() (/proc/self/statm,
already implemented, unaffected by mimalloc's internal accounting) as the
PRIMARY source, falling back to mimalloc's value only if /proc is
unavailable. macOS/Windows unaffected (mimalloc's primitives set
current_rss correctly there via task_info/GetProcessMemoryInfo).
Verified via a calibrated 40x8MB synthetic large-file index on IO:
before the fix, a 2100MB forced ceiling never tripped despite RSS
reaching 2.75GB; after the fix, the same fixture aborts cleanly at
rss_mb=2135 against ceiling_mb=2100 with the expected diagnostic dump.
Full ASan test suite re-run clean (0 FAIL) after the fix — the
pre-existing test_incremental.c:302 RSS-delta flake (a live FastAPI clone
that had grown past its 2048MB assumption) did not recur, consistent with
more accurate RSS reporting also improving the existing backpressure's
effectiveness.
tests/mem_ceiling_abort.sh: fixed two harness bugs found while
calibrating this — (1) `mode=full` index_repository does not force a
re-parse when file hashes are unchanged from a prior run against the same
repo_path (routes to incremental.noop), so every run needing a genuine
full extraction now resets the target's cached .db first; (2) the forced
test ceiling is now calibrated against a directly-probed peak RSS for the
exact fixture shape (2100MB, between the 2048MB floor and an observed
~2.2-2.75GB peak) instead of an unverified guess.
echo"[mem_ceiling] FAIL [row6/row9-low]: index did NOT abort with ceiling forced to the floor (proves the knob doesn't bite, or the real repo genuinely never nears 2GB on this host — re-run with a smaller floor override or bigger synthetic fixture below)">&2
164
+
echo"[mem_ceiling] FAIL [row6/row9-low]: synthetic fixture did NOT trip a ${CEILING_TEST_MB}MB forced ceiling — either the knob doesn't bite, or the fixture is undersized for this host's allocator behaviour (IO: 31GB/8 cores). Widen n_large/size_mb in the gen call above if this needs to be load-bearing on a beefier host.">&2
118
165
FAILURES=$((FAILURES +1))
119
166
else
120
167
echo"[mem_ceiling] index aborted as expected (exit=$LAST_EXIT)">&2
121
168
if! grep -q "mem.ceiling.abort""$LAST_LOG";then
122
169
echo"[mem_ceiling] FAIL [row6]: no mem.ceiling.abort diagnostic dump in output">&2
echo"[mem_ceiling] WARN [row6-synth]: synthetic fixture did not trip the forced ceiling on this host (IO's 31GB may absorb 8x8MB files even at 8 workers) — not counted as a failure since the real-repo variant above already proved the abort path; widen n_large/size_mb if this needs to be load-bearing">&2
154
-
else
155
-
if grep -q "mem.ceiling.abort""$LAST_LOG";then
156
-
echo"[mem_ceiling] PASS [row6-synth]: synthetic large-file repro tripped the ceiling with a diagnostic dump">&2
157
-
else
158
-
echo"[mem_ceiling] FAIL [row6-synth]: synthetic fixture aborted (exit=$LAST_EXIT) but with no diagnostic dump">&2
182
+
# strictly before dump_and_persist_hashes()/cbm_gbuf_dump_to_sqlite(). Only
183
+
# meaningful when the run actually aborted (LAST_EXIT != 0) — a completed
184
+
# run legitimately leaves a .db behind, that is not a violation.
185
+
if [[ "$LAST_EXIT"-ne 0 ]];then
186
+
if [[ -f"$db_synth"||-f"${db_synth}.tmp" ]];then
187
+
echo"[mem_ceiling] FAIL [R4]: aborted run left a .db (or .db.tmp) file — store-integrity violation: $db_synth">&2
159
188
FAILURES=$((FAILURES +1))
189
+
else
190
+
echo"[mem_ceiling] PASS [R4]: aborted run left no .db file (extract-phase abort precedes any SQLite write)">&2
160
191
fi
192
+
else
193
+
echo"[mem_ceiling] SKIP [R4]: run did not abort, nothing to check (see row6/row9-low failure above)">&2
0 commit comments