app-server brokers never self-terminate → orphaned process/RAM leak (34 chains, 272 procs, ~2.2GB)

Bug: app-server brokers never self-terminate → orphaned process/RAM leak

Repo: openai/codex-plugin-cc
Version: codex plugin 1.0.6
File: scripts/app-server-broker.mjs (+ context in scripts/lib/broker-lifecycle.mjs, scripts/session-lifecycle-hook.mjs)

Summary

The shared app-server broker has no idle timeout. It exits only on an explicit
broker/shutdown RPC, SIGTERM, or SIGINT. Its only automatic reaper is the
SessionEnd hook, which is fragile. When that hook fails to run, the detached
broker (child.unref()) leaks forever, holding a Codex app-server child plus
its MCP/worker subprocesses.

Observed in the wild: 34 orphaned broker chains, 272 processes, ~2.2 GB RSS,
0 clients connected. 16 chains pointed at already-deleted git worktrees.

Root cause

Two gaps compound:

Broker never self-exits. In app-server-broker.mjs, socket.on("close")
just does sockets.delete(socket) + clearSocketOwnership(socket). There is
no check for "no clients left" and no idle timer. A broker with zero clients
lives indefinitely.
The reaper is best-effort and misses common cases. SessionEnd in
session-lifecycle-hook.mjs is the intended reaper, keyed per-cwd via
loadBrokerSession(cwd). It fails to reap when:
- Claude is hard-killed / crashes → hook never runs.
- The 5s hook timeout (hooks.json) is exceeded on a busy machine → hook killed.
- The worktree is deleted before SessionEnd → loadBrokerSession(cwd)
  resolves to gone state, returns null, no shutdown is sent → orphan.
  (This matches the 16 deleted-worktree chains.)

Because brokers are spawned detached and keyed to a cwd, any hook miss orphans a
broker that nothing will ever clean up.

Fix

Give the broker an idle self-exit. It is safe: the companion opens one socket
per CodexAppServerClient.connect() and closes it when the job ends
(BrokerCodexAppServerClient.initialize/close in lib/app-server.mjs), so a
broker at zero sockets with no active request/stream is genuinely idle. Killing
an idle broker only costs a cold respawn on the next job — ensureBrokerSession
already checks endpoint readiness and respawns if dead. This makes the broker
self-healing regardless of whether any hook fires, covering hard-kills and
deleted worktrees alike.

Threshold defaults to 15 min, overridable via CODEX_COMPANION_BROKER_IDLE_MS.

Patch (scripts/app-server-broker.mjs)

Add idle state after const sockets = new Set();:

  const IDLE_TIMEOUT_MS = Number(process.env.CODEX_COMPANION_BROKER_IDLE_MS ?? 15 * 60 * 1000);
  let idleTimer = null;
  let serverRef = null;

  function isIdle() {
    return sockets.size === 0 && !activeRequestSocket && !activeStreamSocket;
  }

  function clearIdleTimer() {
    if (idleTimer) {
      clearTimeout(idleTimer);
      idleTimer = null;
    }
  }

  function armIdleTimer() {
    clearIdleTimer();
    if (!(IDLE_TIMEOUT_MS > 0) || !serverRef || !isIdle()) {
      return;
    }
    idleTimer = setTimeout(async () => {
      if (!isIdle()) {
        return;
      }
      await shutdown(serverRef);
      process.exit(0);
    }, IDLE_TIMEOUT_MS);
    if (typeof idleTimer.unref === "function") {
      idleTimer.unref();
    }
  }

Cancel the timer on connect (in net.createServer callback, after sockets.add(socket)):

    clearIdleTimer();

Re-arm on disconnect (in both socket.on("close") and socket.on("error"),
after clearSocketOwnership(socket)):

      armIdleTimer();

Arm at startup and hold the server ref (replace server.listen(listenTarget.path);):

  serverRef = server;
  server.listen(listenTarget.path, () => {
    armIdleTimer();
  });

Optional hardening (not required if idle-exit lands)

Make the SessionEnd hook timeout longer than 5s, or move broker teardown off
the hook's critical path, so slow machines still reap on clean exit.
On SessionStart, sweep os.tmpdir() for stale cxc-*/broker.pid whose pid is
dead (leftover session dirs). Redundant once idle-exit ships, but cleans state.

Can I work on this?

Confirming the issue.

Additional downstream impact observed on plugin 1.0.4 (Linux, codex-cli 0.145.0): the leaked chains don't just cost RAM — they can wedge fresh codex invocations system-wide.

Setup: one evening of agent-driven review calls left 9 orphaned node → codex app-server chains, all reparented to systemd --user (one chain per plugin invocation; spawn times matched our agent calls minute-for-minute). The orphans stayed active: CPU time kept accruing and the shared ~/.codex sqlite stores (logs_2.sqlite had grown to 245 MB) were being written continuously.

Impact 1 — codex exec startup deadlock. While the orphans were churning, a fresh codex exec --sandbox read-only ... hung indefinitely (17+ min): single thread parked in futex_do_wait, 0 CPU total, no TCP sockets, and — notably — zero rows written to logs_2.sqlite, i.e. it blocked at/before store initialization. Not reproducible after killing the orphans; three control runs with identical flags/prompt/env completed in seconds. Smells like contention on the shared sqlite stores rather than anything network-side.

Impact 2 — review results never delivered. An adversarial review dispatched through the plugin ran to apparent completion inside an app-server session (visible in the log DB), but the wrapper agent never surfaced a verdict — consistent with requests landing on a stale broker no live session drains.

Workaround we use: kill orphan chains (node wrappers whose PPID is systemd/1 — never interactive TUIs), and wrap direct CLI calls in timeout. An idle-timeout self-exit in the broker (as proposed here) would eliminate both impacts at the source.

#563 is an open PR against this: the broker releases its idle codex app-server child and restarts it on demand, and Unix cleanup is changed from the current single-pid SIGTERM to identity-verified process-tree termination, so the detached helpers that outlive the app-server are reaped rather than reparented.

Worth flagging for anyone else working this issue: the second half matters independently of the idle timeout. close() in lib/app-server.mjs tree-kills only on win32; the Unix path signals one pid, so a broker that shuts down cleanly can still leave the helper subprocesses behind. The PR write-up documents where the guarantee stops, including the setsid-after-scan case it does not cover.

Independent confirmation on macOS, plus two details that affect the proposed fixes.

Environment

Plugin openai-codex/codex 1.0.4 (marketplace commit 807e03a); the two relevant
files are byte-identical on main (db52e28), so this is still open at HEAD.
macOS 15/26, Apple Silicon + Intel, Node 22/26, 5 machines, heavy task --cwd <repo>
delegation from Claude Code sessions.

Observed (one-time measurement, 2026-07-30 ~13:00 local, aggregated over 5 hosts
via ps -Ao pid=,ppid=,pgid=,etime=,rss=,args= filtered in Python — not pgrep -f, see below):
23 broker processes, 56 processes in total including their codex app-server children,
~2.1 GB summed RSS, zero clients connected. Oldest broker: 4 d 15 h elapsed
(corroborated by the birth time of its leftover $TMPDIR/cxc-* session dir).
The largest single tree summed to ~3.9 GB RSS on one host (RSS summing overcounts
shared pages, so treat that as an upper bound). All 23 were idle: no TCP/UDP sockets,
no CPU movement beyond timer ticks (10 ms per 8 s), no queued/running jobs in the
matching state.json.

Detail 1 — the SessionEnd hook is not the missing kill.
session-lifecycle-hook.mjs:109 does pass killProcess: terminateProcessTree.
The gap is that it is keyed by cwd (loadBrokerSession(cwd), :82-84), so it only ever
reaps the broker of the session workspace root. Every broker started via
--cwd <other repo> is registered under a different state dir and is never visited.
That is #380; worth stating explicitly here because "SessionEnd doesn't kill" is
the wrong summary and sends fixes in the wrong direction.

Detail 2 — the pid-file is not a usable handle for any reaper.
teardownBrokerSession (broker-lifecycle.mjs:174) only kills
if (Number.isFinite(pid) && killProcess), but unlinks the pid-file unconditionally
right after (:182-184). Since the production caller
app-server.mjs:340 → ensureBrokerSession(cwd, { env: options.env }) passes no
killProcess (this is exactly PR #566), the normal outcome is a live broker whose
pid-file is already gone. Consequence for the fixes discussed in this thread: a
pid-file- or session-dir-based reaper systematically misses the oldest orphans.
The reliable identity is broker.json under
~/.claude/plugins/data/codex-openai-codex/state/<slug>-<hash>/, plus argv from ps.

Two measurement traps that cost us a day, in case others script around this:

pgrep -f app-server-broker excludes its own ancestors on BSD/macOS, so a check
run from inside a codex process tree silently misses the very broker that spawned it;
ps | grep overcounts by the pipeline. Read ps -Ao …args= and filter in-process.
ppid=1 is not an orphan indicator: brokers are started detached: true + unref()
(broker-lifecycle.mjs:59-70), so even a 5-minute-old healthy broker has ppid 1.

Workaround we run now (external, so plugin updates don't clobber it): a reaper
every 30 min that terminates only brokers which are (a) missing their socket file,
(b) whose --cwd no longer exists, or (c) idle beyond a threshold — guarded against
live jobs (checking that the recorded worker pid is actually alive; we found
status: "running" records whose workers had been dead for a month), open sockets,
real CPU movement, and linked worktrees. Shutdown via the broker/shutdown RPC over
the unix socket succeeded for all 22 brokers we terminated — SIGTERM was never needed,
which suggests a self-exit path would be equally clean.

Happy to provide the anonymized process dumps if useful.

Thank you. This is strong independent confirmation, and both implementation details match the root cause addressed by #563 (now at 4e8884b).

The fix does not depend on cwd-scoped SessionEnd enumeration, pid files, session directories, or PPID 1. Each broker, stable session owner, and app-server child is published with PID + process-start identity in a plugin-global immutable registry before activation. SessionStart scans every registered broker across cwd boundaries; a dead owner makes the registered tree eligible, while live owners or malformed evidence remain report-only.

The latest commit also durably appends post-activation child observations before request results are exposed, so independently regrouped helpers remain identity-backed cleanup targets, and it writes a terminal record after verified zero-residual cleanup so completed registries are not retried forever. The end-to-end regression creates a helper after activation in its own process group, crashes the app-server, verifies the helper is reclaimed before replacement, and the full suite passes 192/192.

Your PPID and pgrep cautions are especially useful. The implementation does not classify PPID 1 as abandonment; eligibility comes from registered owner identity/liveness. A retest of #563 against the multi-cwd workload you described would be very valuable.

Another independent confirmation on macOS at HEAD, plus a root-cause detail I haven't seen stated in this thread: there are two separate mechanisms that break the state-file lookup, and only one of them involves symlinks. Both make SessionEnd resolve a different state directory than the one the spawn wrote, so the broker is never found to be torn down.

Environment

Plugin openai-codex/codex 1.0.4, marketplace commit 807e03a
macOS 26.5.1, arm64, Node v24.16.0, git 2.50.1, codex-cli 0.145.0
Workload: headless plugin invocations driven from Claude Code sessions, many of them in short-lived git worktrees that are deleted at the end of the session

Measurement

Taken just now, roughly one week after a manual sweep cleared this machine to zero, so this is pure re-accumulation over ~7 days:

brokers: 25 | dead --cwd: 25 | reparented to pid 1: 25
summed RSS: 0.60 GB
oldest: 02-23:17:18 elapsed (71.3 h)
age buckets: {'<1h': 2, '18h': 2, '19h': 4, '20h': 3, '21h': 6, '1d': 4, '2d': 4}
distinct dead cwds: 12

25 brokers across 12 distinct working directories, so roughly 2 per directory - consistent with the single-slot state file being overwritten when a directory spawns more than one broker over its life. 25 matching stale cxc-* session directories under $TMPDIR. Every one reparented to pid 1.

Enumerated from ps -Ao pid=,ppid=,etime=,rss=,args= filtered in Python rather than pgrep -f, and elapsed times parsed as [[DD-]HH:]MM:SS - a naive string sort over that field silently reports a 3-day-old process as younger than a 34-minute-old one.

The earlier sweep on the same machine found 235 brokers at 2.06 GB, oldest 28 days, spread over ~90 distinct working directories, 226 of which pointed at a directory that no longer existed.

The two divergence mechanisms

resolveStateDir(cwd) in lib/state.mjs keys the state directory on <basename>-<sha256(canonicalWorkspaceRoot)[:16]>, where the root comes from resolveWorkspaceRoot(cwd) and is then canonicalised with fs.realpathSync.native. Both of those steps fall back silently when the directory is gone, and they fail in different ways:

A. The git-toplevel fallback. Symlink-independent, fires on any cwd that is not itself the repository root. resolveWorkspaceRoot calls ensureGitRepository(cwd) and, on throw, returns cwd unchanged. While the directory exists that call returns the toplevel; once it is gone the call throws and you get the cwd instead. Both the slug and the hash change.

B. The realpath fallback. Fires only when the path has a symlinked component. fs.realpathSync.native(workspaceRoot) throws on a missing path and the code falls back to the non-canonical string, so the sha256 input differs from the one used at spawn time. On macOS this catches anything under $TMPDIR, since /var/folders/... is a symlink to /private/var/folders/....

Isolating experiment - same script, one run under $TMPDIR and one under a path with no symlink component, resolving the state dir before and after deleting the directory:

# under $TMPDIR (symlinked)
subdir   : repo-09b2caecd26ebddb -> sub-273bc10b58f6e89b   DIVERGED
worktree : wt-eac720c64148774e   -> wt-039d7fc91beb1e45    DIVERGED

# under a non-symlinked path
subdir   : repo-f16ac2794153246c -> sub-2bc4dbecdd39c4ba   DIVERGED
worktree : wt-6482b1395b610526   -> wt-6482b1395b610526    SAME

So a worktree at a non-symlinked path is the only configuration that still resolves correctly after deletion. A cwd one level below the repository root never does, deleted or not, regardless of symlinks - mechanism A alone is enough. That is worth knowing for anyone testing a fix: a repro that only ever uses the repository root as cwd on a non-symlinked path will look like it passes.

This also seems to me like an argument for the shape #563 already takes. Publishing PID plus process-start identity to a plugin-global registry and scanning it across cwd boundaries sidesteps both mechanisms, because neither the path nor its canonical form is load-bearing any more. I've read the description of that PR rather than its diff, so treat that as a reading of the approach and not a review of it.

Interim mitigation

For anyone hitting this before a fix lands, the rule that worked here was: kill only brokers whose --cwd no longer exists, and skip any whose cwd cannot be parsed rather than killing it. A live session's directory still exists, so the rule cannot touch one. On the 235-process sweep that killed 229 and spared 6, reclaiming 1.94 GB, with no session disrupted.

One gotcha worth repeating: capture the pid list up front and iterate that static list. Running the loop inside a shell pipeline truncated it, and only 16 of 229 were killed on the first attempt.

This is a mitigation, not a fix - and it stays necessary even after one, since a crash, a kill -9, or a reboot skips session-end teardown entirely no matter how correct that teardown becomes.

Confirming this on macOS 26.6 (arm64, Node 24), plugin 1.0.3 and current main (db52e28) — the two broker files are byte-identical between them, so this reproduces on every released version.

I want to add a second, independent mechanism that I don't think this thread has named yet. The framing here is "no idle timeout + fragile SessionEnd reaper", and both are real. But there is also a spawn race that creates orphans no reaper of any design can reach, because their pids are never written down.

The race

ensureBrokerSession (plugins/codex/scripts/lib/broker-lifecycle.mjs:113) is check-then-act with no mutual exclusion:

const existing = loadBrokerSession(cwd);
if (existing && (await isBrokerEndpointReady(existing.endpoint))) return existing;
// ...
const child = spawnBrokerProcess({ ... });
const ready = await waitForBrokerEndpoint(endpoint, options.timeoutMs ?? 2000);
saveBrokerSession(cwd, session);   // last writer wins

N concurrent invocations from one workspace all observe "no ready broker", all spawn one, and all but the last writer of broker.json become unreachable. Their pids exist in no file on disk. handleSessionEnd reads a single record via loadBrokerSession, so even a perfectly reliable SessionEnd hook only ever reaps the one winner.

Reproduction — deterministic, no credentials, no `codex` binary

A stub stands in for app-server-broker.mjs, so this needs nothing but the repo:

import fs from "node:fs";
import os from "node:os";
import path from "node:path";
import { execFileSync } from "node:child_process";
import { ensureBrokerSession, loadBrokerSession }
  from "./plugins/codex/scripts/lib/broker-lifecycle.mjs";

const root = fs.mkdtempSync(path.join(os.tmpdir(), "cxrace-"));
const workspace = path.join(root, "workspace");
fs.mkdirSync(workspace);
execFileSync("git", ["init", "-q", "-b", "main"], { cwd: workspace });
process.env.CLAUDE_PLUGIN_DATA = root;

const tally = path.join(root, "spawned.log");
const stub = path.join(root, "stub.mjs");
fs.writeFileSync(stub, `import fs from "node:fs";
import net from "node:net";
const argv = process.argv.slice(2), at = f => argv[argv.indexOf(f) + 1];
fs.appendFileSync(${JSON.stringify(tally)}, process.pid + "\\n");
fs.writeFileSync(at("--pid-file"), String(process.pid));
net.createServer(() => {}).listen(at("--endpoint").replace(/^unix:/, ""));
setTimeout(() => process.exit(0), 120000);
`);

await Promise.all(Array.from({ length: 8 }, () =>
  ensureBrokerSession(workspace, { scriptPath: stub })));

const spawned = fs.readFileSync(tally, "utf8").trim().split("\n").map(Number);
const recorded = loadBrokerSession(workspace);
console.log(`spawned=${spawned.length} recorded=1 orphaned=${
  spawned.filter(p => p !== recorded.pid).length}`);

for (const pid of spawned) { try { process.kill(pid, "SIGKILL"); } catch {} }
fs.rmSync(root, { recursive: true, force: true });

spawned=8 recorded=1 orphaned=7

(The last two lines matter — without them the reproduction leaks the eight stubs it just created, which is rather the point.)

Eight callers, eight brokers, one record. Reproduces on the first run, every run.

Two smaller findings from chasing this

A plain SIGTERM to a broker does not reliably stop it. Its handler awaits appClient.close(), which awaits the codex app-server child's exit; signalling the broker alone leaves it hung. Signalling the process group works — brokers are spawned detached: true, so each leads its own group, and terminateProcessTree already does the right thing. Worth knowing for anyone scripting a cleanup: kill <pid> will look like it did nothing.

A broker carries no record of the environment it was born under. The codex app-server child inherits its environment at spawn and holds it for life, while later callers reuse it over a socket and never have their own environment consulted. broker.json has no environment fingerprint, so a caller cannot tell whether the broker it is about to reuse was started under an environment compatible with its own. This turned a deterministic credential fault into two days of ghost hunting for me: same command, same second, callers that reused a healthy broker succeeded while callers that spawned their own failed. It is adjacent to #382 but not the same claim, so I have not filed it separately — say the word if it deserves its own issue.

Note on ordering, for anyone fixing this

The leak currently masks environment faults. Reap the orphans first and every invocation spawns fresh, so a bad environment reaches codex every time and an intermittent failure becomes a constant one. Nothing has broken at that point — the mask has come off.

Happy to open a PR for the lock if that's wanted; I have one written with regression tests that fail on main and pass with it. I held off because #563, #566 and #580 are already in flight over the same three files and I would rather not add a fifth.

@maximsht

Independent confirmation on Windows — the first in this thread, I think, and the platform detail matters for #563. Also adding a magnitude measurement ~4x the original report, and the mechanism that explains the difference.

Environment

Windows 10 Pro 19045, Node 22.14, plugin openai-codex/codex 1.0.4 (marketplace commit 807e03ac).
Confirmed still open at HEAD: scripts/app-server-broker.mjs is byte-identical between 1.0.4 and main (db52e28). git log 80c31f9..db52e28 -- plugins/codex/scripts/app-server-broker.mjs returns 0 commits, so neither 1.0.5 nor 1.0.6 touches it.
Workload: Claude Code sessions in a multi-pane WezTerm setup delegating to the plugin, one day of use.

Measurement

Killing only the orphaned broker chains (nothing else):

	before	after
`node.exe` processes	214	78
total processes	890	576
free RAM	12.85 GB	22.38 GB

~9.5 GB reclaimed on one workstation, against the ~2.2 GB / 272 procs in the original report.

Why it is that much larger: the cost is the app-server child's MCP fleet, not the broker

One cwd's chain had spawned its 9-server MCP fleet 11 times in ~15 hours — 00:37, 01:22, 01:57, 02:28, 11:03, 11:54, 12:21, 12:22, 12:34, 15:11, 15:52 — 99 MCP servers, roughly 9 GB. The 11 fleets were statistically indistinguishable from one another: 13-18 s CPU, ~805 MB, ~2200 handles each. Each one initialized fully and was then abandoned.

Two consequences:

The leak is per plugin invocation, not per session — matching @maximsht's one-chain-per-invocation observation.
On an MCP-heavy configuration the per-orphan cost is ~800 MB, not ~8 MB. That multiplier is the whole difference between 2.2 GB and 9.5 GB; it is not a different bug.

Idleness confirmed rather than assumed: the two surviving orphan brokers (started 00:37 and 12:25, serving pipes cxc-bvUlfZ-codex-app-server and cxc-Wf5FrG-codex-app-server) showed CPU deltas of 0.00-0.02 s across an 8 s sample — no work at all, consistent with "0 clients connected".

Identifying orphans safely on Windows

Worth stating because a blanket sweep is not safe here — brokers are shared between concurrent sessions (#540, #526), so killing every broker kills live work.

The reliable discriminator was ancestry, not name or age: a broker is an orphan iff its own immediate parent is dead. On this setup a live pane's CLI roots at sh.exe, whereas a plugin-spawned chain roots at node.exe, so parent-liveness separates the two cleanly. Sweeping on that rule produced no false positives — all 6 live pane CLIs survived.

Windows and #563

Reading the PR, several of its mechanisms are explicitly gated off on win32:

ownershipSnapshot / procIdentity are set to null on win32, so a Windows broker publishes no process identity into the registry;
refreshOwnership() and startUnexpectedExitCleanup() early-return on win32;
gatedBrokerChild (the idle app-server child release) is disabled on win32.

What Windows keeps is the terminateProcessTree call on the normal-close path — which is precisely the path that does not run in the hard-kill / crash / deleted-cwd case that produces these orphans. The new broker-ownership.mjs and registered-broker-reaper.mjs themselves look platform-neutral, but on Windows they would be reasoning over registrations with no identity attached.

Is Windows intended to be in scope for #563, or is it better tracked as a follow-up? Happy to test a Windows path against this workload if it helps — the reproduction here is reliable and the signal is large.

Follow-up to my earlier comment: we investigated the "silent empty result" failure mentioned there (an invocation at 6-way parallelism that printed "Turn started", then exited 0 with empty stdout) and traced it to a different mechanism than the spawn race. Two findings, both reproduced against plugin 1.0.3 with stubs only (no credentials, no real codex binary):

1. A turn's completion promise is never rejected on transport loss — this is the silent empty result.

captureTurn awaits state.completion (scripts/lib/codex.mjs:600). The promise's rejectCompletion is captured (codex.mjs:299-315, typed at line 20) but never called from anywhere in the plugin — the rejection path was designed and left unwired. handleExit (scripts/lib/app-server.mjs:162-175) rejects only in-flight JSON-RPC requests; once a turn is streaming, pending is empty, so a broker socket or app-server child dying mid-turn settles nothing. The event loop drains (the inferred-completion timer is unref'd), node exits 0, and stdout is empty — a lost result indistinguishable from success.

Reproduced 20/20 deterministically on both transports: broker socket destroyed mid-turn, and a direct-spawn app-server exiting mid-turn. No concurrency required — parallelism just raises the odds of a transport death (e.g. via the orphan accumulation this issue tracks). A turn that reaches any terminal event always writes stdout (the answer, JSON, or "Codex did not return a final message."), so exit 0 + zero bytes is an exact signature of this path.

Fix shape: wire the already-captured rejectCompletion — e.g. race state.completion against the client's exitPromise (app-server.mjs:70-72) — so transport loss becomes a loud error instead of a clean empty exit.

2. Single-target notification routing can deliver one client's answer to another — and this interacts with the spawn-lock fix.

routeNotification (scripts/app-server-broker.mjs:84-100) sends every notification to one global target (activeRequestSocket ?? activeStreamSocket), and the guard at line 92 releases stream ownership on any turn/completed carrying no threadId, regardless of whose stream is running. Demonstrated against the genuine broker with two clients: client A's turn/completed and final answer were delivered to client B, which discarded them; a real launcher in A's seat hangs indefinitely. Caveat: the trigger (a turn/completed without threadId) was injected by a stub app-server — I have not measured whether real codex-cli traffic emits one.

The interaction worth flagging: today the spawn race gives concurrent callers mostly-private brokers, so this routing path is rarely exercised. The per-workspace spawn lock (in the fix I offered earlier, and in the open PRs here) makes broker sharing the norm — which is correct, but it turns this defect from accidentally-rare into routinely reachable. I'd suggest landing per-client (thread-keyed) notification routing in the same change as the lock, or the lock trades a process leak for lost answers.

Happy to share the stub harnesses for either reproduction if useful.

Another data point for this, from a different machine — and one dimension that I think is missing from the current framing.

The orphaned brokers are not idle. They spin.

Every report here (including #652) frames this as a RAM leak. On my machine the CPU cost was the larger problem:


Orphaned broker chains	10 (30 processes: node wrapper → app-server → code-mode-host)
RSS	871 MB
Age	57–83 hours
Accumulated CPU time	~3100 s per process (~52 min each)
Live CPU draw	5 of them sampled at 2 CPU-seconds over 10 wall-seconds = 20% of one core → ~40% of a core for all 10

Sampling method, in case it's useful for reproducing:

a=$(ps -o times= -p <pid1> <pid2> ... | tr -d ' ' | paste -sd+ | bc); sleep 10
b=$(ps -o times= -p <pid1> <pid2> ... | tr -d ' ' | paste -sd+ | bc)
echo "$((b-a)) CPU-seconds in 10s"

On a laptop that is continuous battery drain from sessions that ended days ago, which I think argues for a shorter default than 15 minutes — or at least makes the idle timeout worth prioritising above a pure-RAM reading of the bug.

Environment: Fedora 44, plugin 1.0.6.

Confirming the deleted-worktree correlation. Same as the sixteen chains you saw: my leaked brokers' --cwd pointed at git worktrees I had removed after merging. That is a reliable, cheap signal — a broker whose --cwd no longer exists provably has no session to return to.

Reproduces immediately. A single /codex:review run left a fresh orphaned broker (60 MB) that was still running 30 minutes later, after the review had completed.

Two traps I hit writing a local stopgap reaper, in case they're useful for the fix design:

1. ppid=1 is not evidence a broker is abandoned. The broker is spawned detached: true + child.unref() in lib/broker-lifecycle.mjs, so being reparented to init is the normal state for a healthy broker, and it is reused across sessions via broker.json. I built the first version of my reaper on ppid=1 + an age threshold, and measured a broker that was actively serving an in-flight review while having ppid=1 and being several hours old. That reaper would have killed it. Anything that garbage-collects on "orphaned" needs a stronger signal than parentage.

2. Careful if the idle check is based on connected clients. #652 proposes self-exit "after ten minutes with no connected client". When I measured a live, in-use broker, ss -x showed zero peers on its broker.sock — while an unrelated idle broker showed one. I don't claim that proves the broker had no client; more likely my socket-level inspection is the wrong lens. But it does suggest the liveness signal should come from the broker's own protocol state (last request timestamp, in-flight turn count) rather than from OS-level socket inspection, or the timeout risks killing brokers mid-work.

The two signals I ended up trusting, both cheap and both false-positive-free in testing:

the broker's --cwd no longer exists → the workspace is gone
broker.json for that workspace records a different pid → this one is a superseded predecessor

That is deliberately conservative: it leaves a leaked broker running when its workspace still exists. A proper idle timeout in the broker itself is the real fix, which is why I'd rather see #543 land than keep maintaining a reaper outside the plugin.

Additional confirmation (macOS, plugin 1.0.4)

This reproduces the broker leak described here in a real Claude Code workload.

Observed locally:

28 detached app-server-broker.mjs processes, all PPID=1, with --cwd pointing at per-task git worktrees.
Their trees included 32 codex app-server processes and 128 Notion MCP processes; summed matching RSS was approximately 3.3 GB (RSS sum, so shared pages may be counted more than once).
The broker Unix sockets existed, but lsof showed only the listening broker for each socket and no connected client.
The broker source still has no idle timer: socket close only removes the socket from the set and clears ownership.
The upstream main branch currently reports plugin version 1.0.6 and still declares a 5-second SessionEnd timeout.

The local Claude transcripts also contain at least five occurrences of:

SessionEnd hook [node "${CLAUDE_PLUGIN_ROOT}/scripts/session-lifecycle-hook.mjs" SessionEnd] failed: Hook cancelled

This matches the failure chain in the issue: SessionEnd sends broker/shutdown, which waits for app-server cleanup; when Claude cancels the 5-second hook, the detached broker and its child tree remain alive. The absence of broker-side idle self-termination then makes the leak permanent.

The relevant upstream files are:

plugins/codex/hooks/hooks.json
plugins/codex/scripts/session-lifecycle-hook.mjs
plugins/codex/scripts/app-server-broker.mjs

This is additional field evidence for #543 rather than a separate issue. #474 covers the related request to increase the SessionEnd timeout.

Another data point, plus a trigger that I don't think is called out above: Claude Code subagents running with isolation: "worktree".

Environment: macOS 26 (Darwin 25.6), Apple Silicon, plugin codex 1.0.6 at marketplace commit db52e28 (current main), Node 23.9, Claude Code 2.1.273.

Observed on 2026-09-16: 39 orphaned broker chains, all reparented to launchd (ppid 1), 0 clients on every broker.sock. Each chain held codex app-server + the computer-use plugin + node_repl + SkyComputerUseClient + a @playwright/mcp server, for a total of 258 node processes and ~4.8 GB RSS; 1-minute load average was 18–24 on an otherwise idle machine. 12 of the 39 pointed at git worktrees that had already been deleted (2–7 days old). The other 27 pointed at .claude/worktrees/agent-* directories and were all under 1 hour old, spawned by a single Claude Code session.

Why subagents make this worse: when the main session dispatches a subagent with isolation: "worktree", the subagent's cwd is the new worktree, so ensureBrokerSession(cwd) spawns a fresh broker keyed to that worktree. SessionEnd never fires for a subagent, and when the main session ends the hook looks up broker.json for the main cwd only (#380). Net effect: one leaked chain per worktree subagent, so a session that fans out a dozen subagents leaks a dozen chains in an afternoon, with no crash or hard-kill involved.

Workaround we deployed (until an idle timeout lands, cf. #457 / #580 / #680): a user-level hook on SessionStart, SubagentStop and SessionEnd that lists app-server-broker.mjs serve processes and sends SIGTERM to the ones that are (a) older than 5 minutes and (b) have zero clients on their unix socket (lsof -U, ignoring the broker's own fd). The broker's SIGTERM handler already tears down the app-server tree cleanly, and ensureBrokerSession respawns on the next call, so an idle kill costs one cold start and nothing else. It also removes cxc-* temp dirs whose broker.pid is dead. Happy to share the script if useful.

Confirms the diagnosis in the issue body: an idle self-exit inside the broker is the fix that survives every hook-miss path, including this one.

Same thing here on Windows 11, and it has a second consequence beyond the RAM. The broker makes a git worktree undeletable.

I'm on Claude Code 2.1.278 with plugin 1.0.4 and now 1.0.6, and the broker script is byte for byte the same in both. Here's what I measured on 2026-09-19.

A Claude Code worktree session (claude -w tldr-skill) calls Codex once. ensureBrokerSession spawns node app-server-broker.mjs serve --cwd C:/.../.claude/worktrees/tldr-skill, detached and unref'd. The session exits and Claude Code removes the clean worktree at exit with git worktree remove. On Windows that folder is the broker's working directory, so git deletes the checkout and the .git file and then fails on the folder itself. The registration gets pruned but the branch and the empty folder stay behind.

The SessionEnd hook never reaped the broker. Its broker.json under CLAUDE_PLUGIN_DATA\state\tldr-skill-\ is still there, so teardownBrokerSession and clearBrokerSession never ran. Four hours later the broker (pid 60652, parent gone) was still alive with zero clients, holding the empty folder. I had a second one from three days earlier holding another worktree's folder the same way.

Looking at app-server-broker.mjs, there's no exit path of its own. socket.on("close") just clears ownership, there's no check for the last client leaving, no idle timer, and nothing watching the process that started it. So any miss by the SessionEnd hook leaves it forever, and a Claude Code worktree session removing its cwd at exit is a miss every time.

A few things would fix this at the source. Exit when the last client disconnects and nothing reconnects within a few minutes, which is the patch proposed above. Or watch the owning Claude process and exit when it's gone. And either way, don't spawn the broker with cwd set to the workspace. It already gets --cwd as an argument, so spawning it from os.tmpdir() means a leaked broker no longer pins the folder.

For now our worktree removal script ends any process whose command line names the folder before it runs git worktree remove, which works but shouldn't be needed.

Another data point, plugin 1.0.6 on macOS 26.6 (Darwin 25.6), Claude Code driving codex-companion.mjs task … through the codex-rescue subagent, one Claude Code session:

After ~20 task invocations over a day (each a separate review job), ps -Axo pid,ppid,command | awk '$2==1' showed 20 orphaned node …/codex/1.0.6/scripts/app-server-broker.mjs processes, ages 1 h – 20 h, 192 MB RSS total, none attached to a live job.
The session was still open the whole time, so the SessionEnd hook never had a chance to run; nothing else reaps them.
A plain kill (SIGTERM) cleared all 20 cleanly, so the shutdown path itself works — it is just never invoked.

An idle timeout in the broker (exit after N minutes without a client) would have covered this case without relying on the hook.

Same here on macOS 26.6, plugin 1.0.6 (Claude Code 2.1.278): 269 orphaned processes in 47 broker chains (app-server-broker.mjs + codex app-server + cua-repl + codex-code-mode-host), all reparented to PID 1, 11.9 GB RSS total, oldest alive for 1d16h, with only 4 Claude sessions open. Killed them by hand. Interim workaround until an idle-timeout PR (#680, #580) lands: a Claude Code SessionEnd hook that kills those four process names older than 3 h. +1 for merging an idle timeout in the broker itself.

Another data point, plugin 1.0.6 on macOS 26.6 (Darwin 25.6), Claude Code 2.1.280/2.1.283, several Claude Code sessions each calling codex-companion.mjs task in the background.

After about 3 days: 121 brokers, all reparented to launchd, with about 3,400 processes under them and about 32 GB RSS on a 48 GB machine. Each chain holds codex app-server, codex-code-mode-host, two ChatGPT cua_node processes and one copy of every MCP server in the user's Codex config. That is 368 copies of each Python MCP server, plus 163 open ssh sessions from an ssh-launched MCP server.

30 of the 34 brokers still running at the time of checking had a throwaway working directory (a mkdtemp dir holding task.md/output.md). SessionEnd only stops the broker for the session's own cwd (#767), so a broker keyed to a temp dir is never stopped. #753 (ensureBrokerSession clearing a live broker's state without killing it) fits the rest.

Workaround we run: an hourly job that sends SIGTERM to every broker older than 2 hours, and to every process under it, skipping any broker whose cwd matches a running codex-companion.mjs job. An idle timeout in app-server-broker.mjs, so the broker exits after N minutes with no connected clients, would fix all of these cases.

Another data point, and a case the SessionEnd-based fixes (#605, #660) can't help with: the owning sessions never ended.

Setup: plugin 1.0.6, codex-cli 0.161.0, Claude Code 2.1.300, Linux VM with 8 vCPUs. Claude Code sessions run in tmux under an agent orchestrator. They stay open for days, and many of them run the review gate via codex-companion.mjs review / adversarial-review.

What we found: 36 broker chains (35 codex app-server processes plus their node wrappers, about 5.2 GB RSS in the codex processes alone), all reparented to systemd --user.

35 of the 36 belonged to a Claude Code session that was still alive. We matched each broker's BERTH_SESSION / CLAUDE_CODE_SESSION_ID environment to a running claude process.
One session held 4 brokers, one per worktree cwd it had reviewed in.
Only 1 was a true orphan: its session was gone and its worktree was deleted.
Ages ranged from 20 minutes to 2.5 days.

CPU: every idle codex app-server drew about 4% of a core, constantly, with no client or job attached. Measured from /proc/<pid>/stat utime+stime deltas over 10 s. Together that was about 240% of one core, or 30% of the machine. That was more than all 46 Claude Code sessions combined, five of which were actively working.

Why SessionEnd doesn't cover it: the hook works fine here; it just never fires, because the sessions are long-lived by design. The broker's lifetime is tied to the session's, and a session holds one broker per cwd, so brokers pile up over its life. The plugin also has no way to tell an open-but-idle session from an active one. An idle timeout in the broker itself (#108, and PRs #680, #580, #457) is the only one of the proposed fixes that addresses this case. ensureBrokerSession() already starts a new broker when the endpoint is dead, so a short timeout costs little: one cold start on the next review.

Workaround: stopping every idle broker with SIGTERM was clean. I sent SIGTERM to each broker and its node/codex children together, and nothing needed SIGKILL. I haven't yet run a review in one of those sessions afterwards; going by ensureBrokerSession(), it should simply start a new broker. Afterwards: 35 → 3 codex processes, 5.2 GB → 0.7 GB, and the box went from about 11% to 67% idle.

Same leak on 1.0.4, macOS: 48 broker chains up to 19 days old, 309 processes, ~2.25 GB RSS, with no live Claude session older than a day. One thing I didn't see called out above: each orphaned app-server also keeps its Codex Computer Use client (SkyComputerUseClient) alive — 117 of them here. On a 24 GB machine this pushed swap to 15.5/16 GB; stopping the orphans freed ~5.7 GB of swap. +1 for landing an idle timeout (#680 / #580).

cx-4 [6c32f3] (codex): independently reproduced the idle broker leak on plugin 1.0.6 at db52e28f4d9ded852ab3942cea316258ae4ef346 using a fake Codex executable, then ran the existing PR #814 candidate 4ab2ee43fc10e90f9d60a8d4a129d2cd3044ad21 on macOS/Node 22.23.3. Installed source: 1 test, 0 pass, 1 fail, 0 skip. Candidate broker/process suites: 28 tests, 28 pass, 0 fail, 0 skip. No live model requests or signals to existing sessions.

Baseline log SHA-256: 6979fdf02610aed4b5c93a61e64b2922fa74738549fd0c7cd14a445864cc653b. Candidate log SHA-256: 514acfc24d9608ce90583f5de2df033ec023bff4838b5f5878172df96ee04a14.

A safety constraint for external cleanup: brokers are intentionally detached, so age and PPID=1 cannot establish abandonment. The current broker has no atomic idle-only shutdown RPC, and the state loader converts unreadable/malformed job state into an empty default. The upstream candidate checks connected clients, pending requests, active turns and queued/running jobs inside the broker, and preserves unreadable job state as a refusal. These local tests support evaluating that existing candidate without duplicating its owner's work. They do not establish production readiness or fix shared-session shutdown (#612).

Agent-Tool: codex

Same on macOS, plugin 1.0.6. Found 55 orphaned app-server-broker.mjs (ppid=1, oldest 17 days). The leak is amplified by MCP servers configured in Codex: each orphaned broker kept its codex app-server plus its MCP children (playwright-mcp, computer-use node processes, node_repl) alive — 1,491 processes / ~18 GB RSS total. Killing the broker trees dropped node memory from 21.5 GB to 2.9 GB. +1 for an idle timeout that tears down the whole process tree (#680 / #814).

Data point for the live broker side of this issue, plus a small mitigation I haven't seen proposed here.

Idle timeouts don't cover it. The idle-timeout PRs (#680, #814, #660) only fire when a broker has no clients. A long-lived Claude Code session that calls Codex more often than the timeout keeps its broker alive, and every review/task leaves its thread loaded: the broker never sends thread/unsubscribe, so the app-server keeps one full MCP server set per thread. That is #706, and #707 fixes it. On plugin 1.0.6 / macOS, one broker alive for 1 d 17 h held 35 idle threads (thread/loaded/list, all idle): 281 processes, ~14 GB.

Verification. Sending thread/unsubscribe for those 35 threads through the running broker (no restart) unloaded them about 60 s later (thread/closed). The app-server went from 281 processes to 1, and node/python processes on the machine went from 422/130 to 199/51. With a local broker patch equivalent to the core of #707 (thread owners per socket, unsubscribe on last-owner close, subagent children found via parentThreadId), a finished task now leaves 0 loaded threads and 0 MCP processes about 70 s after it ends.

Complementary mitigation: start review threads without MCP servers. Reviews rarely need the user's MCP servers, and thread/start accepts a per-thread config, so the shared app-server can keep them for tasks while review threads skip them:

// lib/codex.mjs: buildThreadParams() passes options.config through to thread/start;
// runAppServerReview() and the adversarial review's runAppServerTurn() set it.
const REVIEW_THREAD_CONFIG = { mcp_servers: { "<server>": { enabled: false } } };
const thread = await startThread(client, cwd, { ...reviewOptions, config: REVIEW_THREAD_CONFIG });

Verified on a real adversarial-review: the disabled server never starts for that thread, the others do, and the review still finds the bug planted in the test repo. It is the per-thread variant of the app-server-wide override suggested in #509, and it limits what any remaining leak can hold.

Plugin 1.0.6, Linux. One workstation running a fleet of Claude Code lanes, each in its own git worktree, each running codex-companion.mjs review once per candidate commit.

One datum I don't see stated above: the leak does not need an unclean exit. A review pass that completes and returns its verdict normally still leaves its broker running, and because each worktree is a new workspace key, each lane's pass spawns a new broker. The count therefore grows with the number of passes, not with crashes. SessionEnd only visits the broker for the session's own cwd (Sanexio and francocapra describe this above), so it would not reach these either; on this workstation the plugin was not enabled in Claude Code as of 6 Oct, so nothing sent broker/shutdown at all.

Measured here:

23 Sep 2026: 45 stranded broker groups (165 processes), the oldest 16 days old. Killing them freed about 5.8 GB of RAM plus swap, taken from free -m before and after. A review pass run during that diagnosis returned its verdict and left its own broker alive afterwards.
30 Sep 2026: 64 live brokers dating back to 23 Sep, 39 of them with a working directory that no longer existed (deleted worktrees).

I agree with the idle timeout proposed in #108 as the fix that covers these cases, and have nothing to add on the design.

@vaskockorovski

Reproduced @vaskockorovski's per-worktree case on macOS, and measured it with both fixes applied locally: thread release on client disconnect (as in #707) plus a 15-minute no-client idle shutdown (as in #680 / #814).

Test: throwaway repo, two worktrees, one adversarial-review --wait per worktree, each completing normally (verdict returned, exit 0), then git worktree remove on the second one.

When	Brokers from the test	Descendants per broker
before	0	—
after review in worktree 1	1	5
after review in worktree 2	2	5
after removing worktree 2	2	5
+10 min	2	1 (threads released, MCP servers gone, app-server left)
+17 min	0	—

So the three observations hold on 1.0.6: a clean exit leaves the broker alive, every worktree gets its own broker, and removing the worktree does not stop it. With the two changes together, the count is bounded by the worktrees used in the last 15 minutes:

releasing threads when a client socket closes tears down the per-thread MCP servers within about a minute;
the idle timer only checks whether any client socket is open, not the cwd or loaded threads, so a broker whose worktree was deleted still exits on its own.

Neither change alone is enough. Without thread release, a broker reused more often than the timeout keeps piling up threads (my previous comment). Without the idle timer, the per-worktree brokers from this comment never exit.

Navigation Menu

Repository navigation

app-server brokers never self-terminate → orphaned process/RAM leak (34 chains, 272 procs, ~2.2GB) #543

Bug: app-server brokers never self-terminate → orphaned process/RAM leak

Summary

Root cause

Fix

Patch (scripts/app-server-broker.mjs)

Optional hardening (not required if idle-exit lands)

Environment

Measurement

The two divergence mechanisms

Interim mitigation

The race

Reproduction — deterministic, no credentials, no `codex` binary

Two smaller findings from chasing this

Note on ordering, for anyone fixing this

Environment

Measurement

Why it is that much larger: the cost is the app-server child's MCP fleet, not the broker

Identifying orphans safely on Windows

Windows and #563

Additional confirmation (macOS, plugin 1.0.4)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

app-server brokers never self-terminate → orphaned process/RAM leak (34 chains, 272 procs, ~2.2GB) #543

Description

Bug: app-server brokers never self-terminate → orphaned process/RAM leak

Summary

Root cause

Fix

Patch (scripts/app-server-broker.mjs)

Optional hardening (not required if idle-exit lands)

Activity

HoneyTyagii commented on Jul 23, 2026

ghbaud commented on Jul 24, 2026

maximsht commented on Jul 25, 2026

ccheng555 commented on Jul 28, 2026

Sanexio commented on Jul 30, 2026

ccheng555 commented on Jul 30, 2026

aaronmeza commented on Aug 3, 2026

Environment

Measurement

The two divergence mechanisms

Interim mitigation

scotthamilton77 commented on Aug 4, 2026

The race

Reproduction — deterministic, no credentials, no codex binary

Two smaller findings from chasing this

Note on ordering, for anyone fixing this

wolverin0 commented on Aug 6, 2026

Environment

Measurement

Why it is that much larger: the cost is the app-server child's MCP fleet, not the broker

Identifying orphans safely on Windows

Windows and #563

scotthamilton77 commented on Aug 15, 2026

Polsemaker commented on Aug 20, 2026

t3ta commented on Sep 7, 2026

Additional confirmation (macOS, plugin 1.0.4)

francocapra commented on Sep 16, 2026

tehfiend commented on Sep 19, 2026

drakulavich commented on Sep 20, 2026

Fairmont77 commented on Sep 22, 2026

gus752 commented on Sep 27, 2026

Stener1 commented on Oct 15, 2026

kevvykevwin commented on Oct 24, 2026

adoosh-afk commented on Oct 24, 2026

rampagege commented on Nov 2, 2026

Morghot42 commented on Nov 3, 2026

vaskockorovski commented on Nov 6, 2026

Morghot42 commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Reproduction — deterministic, no credentials, no `codex` binary