index_repository: OOM kill instead of degradation — nested .gitignore below another .gitignore is dropped, and the memory budget is advisory (mem.backpressure.futile → soft_overshoot) #1973

Version

codebase-memory-mcp dev — built from source at ae5de7fe (main, 2026-08-31).
Also reproduced with the 0.10.8 release archive (linux-amd64-portable).

Platform

Linux (x64) — Ubuntu 26.04, 24 cores, 31 GB RAM, gcc 15.2, scripts/build.sh (no UI).

Install channel

Built from source (and GitHub release archive).

Binary variant

standard

What happened, and what did you expect?

index_repository on a repository that contains a nested .gitignore below a directory that has its own .gitignore does not just index too much — it is killed by the OOM killer, and nothing usable is reported to the caller.

Two separate defects compound here:

1. Discovery ignores the deeper .gitignore. try_load_nested_gitignore() in src/discover/discover.c:826 early-returns as soon as the walk frame already carries a local_gi, and local_gi is inherited by child frames. So only the shallowest nested .gitignore on a path is ever loaded. This is the same root cause reported in #530 (§2 "Cascading .gitignore Files Ignored in Subdirectories"); I am filing separately because the consequence below is not covered there and is much worse than "too many files get scanned". The code is unchanged on current main:

static cbm_gitignore_t *try_load_nested_gitignore(const walk_frame_t *frame) {
    if (frame->local_gi || frame->prefix[0] == '\0') {
        return NULL;   // any ancestor .gitignore disables every deeper one
    }

An empty ancestor .gitignore is enough to disable the deeper one.

2. The memory budget is advisory, so over-collection turns into an OOM kill rather than degradation. In src/pipeline/pass_parallel.c:774-781, once backpressure has spun PP_BACKPRESSURE_MAX_SPINS times and the process is still over budget, it logs mem.backpressure.futile action=soft_overshoot once and then keeps admitting work. There is no cap, no shedding, and no abort — RSS then grows by roughly an order of magnitude past the budget until the kernel kills the process.

Expected: exceeding the configured budget should degrade (shed work, cap concurrency, or fail the index with a clear error), not overshoot ~9x and die. CBM_MEM_BUDGET_MB reads like a limit but does not act as one.

3. The failure is invisible to the caller. The CLI surfaces only:

error: daemon-backed CLI execution failed
Failed index_repository (23983 ms)

No per-project log is written (only .worker-log-* survives, and only if you know to look). In one run cli --progress printed nothing at all for 5 minutes before the kill. cbm-daemon.log ends at daemon.start.

Reproduction

Fully synthetic, no proprietary code. This mimics the standard Laravel layout, where storage/.gitignore exists and storage/<subdir>/.gitignore contains * — so every Laravel project with cached artifacts on disk hits this.

T=/tmp/cbm-oom-repro; rm -rf "$T"; mkdir -p "$T/storage/dump" "$T/app"; cd "$T"
git init -q .
printf 'node_modules/\n' > .gitignore
: > storage/.gitignore                      # empty ancestor .gitignore is enough
printf '*\n' > storage/dump/.gitignore      # git honours this; cbm does not
printf '<?php\nfunction appFn() { return 1; }\n' > app/x.php

python3 - <<'PY'
import json
def nest(d):
    if d == 0: return {"leaf": "x"*60, "n": 12345}
    return {"lvl%d_%d" % (d, i): nest(d-1) for i in range(3)}
blob = json.dumps({"__meta": {"route": "/api/v1/thing"}, "data": [nest(5) for _ in range(3)]})
for n in range(5000):                       # 5000 x ~72 KB deeply nested JSON
    open("storage/dump/%05d.json" % n, "w").write(blob)
PY

git check-ignore -q storage/dump/00000.json && echo "git DOES ignore these files"

CBM_CACHE_DIR="$T/cache" codebase-memory-mcp cli index_repository --repo-path "$T"

Result: the process grows past the budget and is OOM-killed (12.5 GB anon-RSS inside a 12 GB cgroup, ~10 s). Run it under systemd-run --user --scope -p MemoryMax=12G so it kills only itself.

Expected: ~2 nodes — app/x.php and the project — since git ignores everything under storage/dump/.

A softer variant of the same repro shows the discovery bug without the kill: use flat 26 KB JSON payloads instead of the nested ones, and indexing succeeds with 1,020,008 nodes / 1,020,006 edges built entirely from files git ignores, out of a repository whose only source file is two lines of PHP.

Real-world impact (a Laravel app, ~5,300 PHP source files): discovery reported files=25638 where the repo has 6,682 git-visible files; 5,404 of the extras were storage/debugbar/*.json dumps. Indexing was OOM-killed at 25 GB anon-RSS / 38 GB virtual after 26 s wall (5m28s CPU across 24 cores), twice, on both 0.10.8 and main. Adding a one-line .cbmignore with storage/ makes the same repository index in 9 s with 32,108 nodes / 174,369 edges — so nothing else about the repo is unusual.

Logs

level=info  msg=pipeline.discover files=25638 elapsed_ms=1215
level=info  msg=pipeline.mode mode=parallel workers=24 files=25638
level=info  msg=parallel.extract.start files=25638 workers=24
level=info  msg=parallel.extract.retention retain_sources=true total_mb=347 per_file_mb=32
level=info  msg=parallel.mem.budget total_mb=2779 per_worker_mb=115
level=warn  msg=mem.pressure.warn rss_mb=2798 budget_mb=2779 pct=100
level=warn  msg=mem.backpressure.futile action=soft_overshoot
level=warn  msg=index.retain_capped path=storage/debugbar/<id>.json bytes=98669
... 5451 x "parallel.extract.file.start", only 24 x "parallel.extract.file.done" ...

kernel: Memory cgroup out of memory: Killed process (codebase-memory)
        total-vm:38258628kB, anon-rss:25088216kB
systemd: run-*.scope: Consumed 5min 28.032s CPU time over 25.717s wall clock time, 24G memory peak.

CBM_MEM_BUDGET_MB=4096 was set explicitly in one run and accepted (mem.init budget_mb=4096 source=CBM_MEM_BUDGET_MB); peak RSS was unaffected.

Diagnostics trajectory

Not captured — the process dies within ~10-26 s of the extract phase, and CBM_DIAGNOSTICS output is owned by the daemon that goes down with it. Happy to run it if you tell me how you want it captured across the kill; the synthetic repro above reproduces the whole thing in under a minute.

Project scale

Synthetic repro: 5,000 ignored files, 1 source file → 1,020,008 nodes / 1,020,006 edges (flat variant) or OOM (nested variant).
Real repo: 6,682 git-visible files, 25,638 discovered; 32,108 nodes / 174,369 edges once storage/ is excluded.

Confirmations

• Searched existing issues: Bug Fixes & Improvements: Windows Stdio hangs, Cascading Gitignore, UTF-8 Snippets #530 (§2) already identifies the same discovery root cause, and .gitignore (non repo root) gaps and overrides #510 / index_repository: root-anchored .gitignore patterns are applied to nested paths, silently excluding tracked source #1205 cover different .gitignore paths (subfolder indexing and root-anchored patterns). None of them cover the memory-budget overshoot or the OOM outcome, which is what makes this unrecoverable rather than merely noisy.
• The reproduction is fully synthetic; no proprietary code or logs are included.