cbm_subprocess_spawn pays O(RLIMIT_NOFILE) close() calls per spawn (~0.8 s at nofile=524288) — use close_range() #1484

Version

codebase-memory-mcp dev (source build, main @ 0d6f26a)

Platform

Linux (x64)

Install channel

Built from source

Binary variant

standard

What happened, and what did you expect?

On any machine whose soft open-files limit is high, every cbm_subprocess_spawn burns hundreds of milliseconds between fork and exec: cbm_posix_child_exec (src/foundation/subprocess.c) closes inherited descriptors with for (fd = 3; fd < max_fd; fd++) close(fd), where max_fd = sysconf(_SC_OPEN_MAX) capped at 1,048,576. With nofile=524288 (Debian 13's default hard limit, and the soft limit inside agent harnesses that raise it, e.g. Claude Code's shell) that is ~524K syscalls ≈ 0.7–0.85 s per spawn — measured: a standalone probe spawning the documented ignoring-tree script sees the child's first shell command execute 716–842 ms after spawn, while the script itself takes 4 ms.

Deterministic test fallout: subprocess_quiet_timeout_kills_ignoring_tree fails every run (FAIL tests/test_subprocess.c:510: ASSERT(ready)) because the fixture waits only 500 ms for the pid file and arms a 750 ms quiet timeout — the tree is SIGKILLed before it ever writes its pids. Secondary effect: each failed run leaks an orphaned cbm-parent/cbm-grandchild tree that ignores SIGTERM (cleanup only knows pids read from the pid file, which never appeared).

Expected: spawn cost independent of RLIMIT_NOFILE — close_range(3, ~0U, 0) on Linux ≥ 5.9 (glibc ≥ 2.34), closefrom() on the BSDs, falling back to the current loop (or /proc/self/fd iteration) elsewhere. This also affects production spawn-heavy paths, not just tests.

Reproduction

1. Code: this repository @ 0d6f26a (no indexing involved — subprocess supervision only).
2. Command: ulimit -Sn 524288 && scripts/test.sh --suites subprocess (any Linux where the hard limit allows it; Debian 13 ships hard nofile 524288, so a shell that raises its soft limit — e.g. agent harnesses do — hits this out of the box).
3. Result: subprocess_quiet_timeout_kills_ignoring_tree FAIL tests/test_subprocess.c:510: ASSERT(ready) — deterministic, 4/4 runs. Expected: green, as with ulimit -Sn 1024 (verified: 29/29 pass).

Isolation evidence (standalone probe linking src/foundation/subprocess.c, quiet timeout disabled, same ignoring-tree script as the fixture):

• pid file appears 842 ms after cbm_subprocess_spawn returns; with date markers inside the script, the child's first command runs at +716 ms and the whole script needs only 4 ms more — the entire delay sits between fork and the shell reaching the script, i.e. the close() loop.
• plain /bin/sh -c '/bin/sh -c true' in the same shell: 7 ms — the environment itself spawns fast.
• with the quiet timeout armed (750 ms) the tree is killed before the echo, so wait_for_tree_pids (500 ms budget) never succeeds and the orphaned tree survives SIGKILL cleanup (pids unknown), ignoring TERM forever.

Suggested fix: replace the loop in cbm_posix_child_exec with close_range(STDERR_FILENO + 1, ~0U, 0) when available (Linux ≥ 5.9 / glibc ≥ 2.34, FreeBSD 12.2+), keeping the loop as fallback. Optionally also relax the fixture budgets, but with close_range the current budgets are fine.

Logs

$ ulimit -Sn
524288
$ build/c/test-runner subprocess
  subprocess_quiet_timeout_kills_ignoring_tree             FAIL tests/test_subprocess.c:510: ASSERT(ready)
  28 passed, 1 failed

$ ulimit -Sn 1024 && build/c/test-runner subprocess
  29 passed

standalone probe (quiet_timeout_ms=0, same spawn options as the fixture):
  [  0 ms] spawn rc=0
  [  0 ms] iter=0 fopen errno=2 (No such file or directory)
  ...
  [500 ms] TIMEOUT waiting for pidfile        <- fixture's wait_for_tree_pids budget
  [842 ms] pid file: 638002 638009            <- write finally lands

script-side date markers relative to spawn:
  start: +716 ms   (first command of the -c script)
  trap:  +717 ms
  bg:    +719 ms
  echo:  +720 ms   (script body itself: 4 ms)

leaked trees after failed runs (ignore TERM, reparented to init):
  359572 /bin/sh -c trap '' TERM; /bin/sh -c '...' cbm-grandchild & ... wait cbm-parent /tmp/cbm-subprocess-tree-mTiXJj
  459335 ... /tmp/cbm-subprocess-tree-0Uix7h
  459358 ... /tmp/cbm-subprocess-tree-Q83ZXg

Diagnostics trajectory (memory / performance / leak issues)

Project scale (if relevant)

No response

Confirmations

• I searched existing issues and this is not a duplicate.
• My reproduction uses shareable code (a dummy snippet or a public OSS repository), not proprietary code.