SessionEnd cleanup fails when review cwd ≠ session cwd — broker.json looked up by cwd-hash, leaving orphan brokers even on graceful /quit

Summary

handleSessionEnd cleans up the broker by calling loadBrokerSession(input.cwd), but broker.json is stored under a cwd-derived hash (resolveStateDir(cwd)). When the broker was spawned for a different cwd than the session's cwd, the lookup misses, endpoint is null, no broker/shutdown is sent, and the broker is orphaned — even when the session exits gracefully (/quit, SIGTERM).

This is distinct from #108 (which attributes orphans to the hook not firing on crash/kill). Here the hook fires normally (exit 0) but cannot locate the broker.

Why this happens in normal use

The broker cwd is decided by resolveCommandCwd (codex-companion.mjs:149):

options.cwd ? path.resolve(process.cwd(), options.cwd) : process.cwd()

→ flows to connect(cwd) → ensureBrokerSession(cwd) → broker --cwd and broker.json location.

A very common workflow breaks this:

Session starts in the main tree (session cwd = /repo).
A review runs against a git worktree — either cd worktree && node companion … or node companion … --cwd worktree → broker cwd = worktree.
/quit → SessionEnd input.cwd = /repo → loadBrokerSession(/repo) looks at the /repo hash, never finds the worktree broker.json → broker survives.

Reviewing a worktree's uncommitted working tree requires broker cwd = worktree, so for a main-tree session this mismatch is structural, not accidental.

Reproduction (verified)

# broker registered for cwd=A
ensureBrokerSession("/tmp/probe")            # broker --cwd=/tmp/probe, alive

# SessionEnd with matching cwd  -> broker is killed (OK)
echo '{"cwd":"/tmp/probe","session_id":"s1","hook_event_name":"SessionEnd"}' \
  | node session-lifecycle-hook.mjs SessionEnd   # exit 0, broker DEAD

# SessionEnd with mismatched cwd -> hook exits 0 but broker SURVIVES
echo '{"cwd":"/tmp","session_id":"s2","hook_event_name":"SessionEnd"}' \
  | node session-lifecycle-hook.mjs SessionEnd   # exit 0, broker ALIVE  <-- bug

Observed in the wild: every live session had cwd in the main tree, while every orphan broker's --cwd pointed at a worktree — a perfect mismatch pattern.

Root-cause fix (preferred)

Decouple broker cleanup from cwd:

A. Track brokers by session, not cwd-hash. Record the owning session_id in broker.json (and/or a global broker index). handleSessionEnd resolves brokers by input.session_id and tears down all of them regardless of cwd. This fixes the graceful-exit mismatch directly.
B. Parent-session liveness watchdog in the broker. The broker records the owning session PID and periodically checks process.kill(pid, 0); if the owner is gone it self-terminates. This additionally covers SIGKILL / crash / OOM, where SessionEnd never fires at all.

A handles graceful exits; B handles abnormal exits. Together they remove the cwd dependency entirely.

Fallback (defense-in-depth, not a root fix)

An idle timeout (#108) as a last-resort net if both A and B fail. Useful but secondary — it lets brokers linger for the timeout window and masks the real ownership/lookup defect.

Broker process not cleaned up on session exit — no idle timeout #108 — broker not cleaned up on session exit / no idle timeout (overlapping symptom, different attributed root cause; the idle-timeout there is the fallback above)
Same-cwd parallel /codex:* races on jobs.json + broker.json: data loss + orphan brokers #286 — same-cwd parallel races on broker.json (opposite case: same cwd)
Background task hangs forever in "queued" when --cwd is a git worktree (detached worker dies silently; no timeout) #367 — background task hangs in a worktree cwd (worktree-specific, different symptom)

Environment

Plugin openai-codex/codex v1.0.4
Linux; reproduced directly against session-lifecycle-hook.mjs / broker-lifecycle.mjs

Confirming #380 in the wild, with a git worktree remove amplifier plus scale/symptom data that may help prioritize.

Setup: a repo that cuts a fresh git worktree per task and removes it right after merge. Codex reviews run against the worktree working tree, so the broker's --cwd is the worktree path — exactly the cwd ≠ session-cwd mismatch described here.

The amplifier: after merge, git worktree remove deletes the broker's --cwd. So beyond the SessionEnd lookup missing (this issue), the broker is now keyed to a path that no longer exists — it can never be located for shutdown and never be reused (ensureBrokerSession only reuses a reachable endpoint for the same cwd-hash, and that cwd is gone). Every merged worktree leaves a permanent orphan, recurring on every task.

Scale observed (macOS, plugin v1.0.4, ~26 days): 98 orphaned app-server-broker.mjs serve + 375 codex app-server processes, all holding ~/.codex/logs_2.sqlite open → the logs DB had grown to 384 MB with an active WAL.

Downstream symptom: once enough orphans accumulate, adversarial-review began intermittently dying at startup with:

[codex] Starting Codex task thread.
failed to load configuration: No such file or directory (os error 2)

So the accumulation doesn't just waste resources — it eventually breaks the review path (looks like contention on shared per-session state / the logs DB). Killing the orphans and rotating the logs DB cleared it immediately; a fresh run rebuilt a ~2 MB DB.

This reinforces the #108 idle-timeout ask as the right fix: in the deleted-worktree case, neither a corrected cwd-hash lookup nor session-liveness (#450) can help — the owning path is gone — so a broker-side idle self-terminate is the only thing that can reap it.

Corroborating on Linux (plugin v1.0.4, worktree-per-task workflow, task --cwd <worktree> background jobs rather than reviews): 69 orphaned app-server-broker.mjs processes accumulated over a few weeks, every one keyed to a since-removed worktree path — same mismatch pattern as above. Two additions to the thread:

1. The deleted-worktree case is still reachable for graceful shutdown — idle timeout is not the only remedy.

in the deleted-worktree case, neither a corrected cwd-hash lookup nor session-liveness (#450) can help — the owning path is gone — so a broker-side idle self-terminate is the only thing that can reap it.

This turns out not to be so: resolveStateDir falls back to hashing the raw path string when the directory no longer resolves (realpathSync/git-repo resolution fail), so loadBrokerSession(<deleted-path>) still finds broker.json and the broker can be shut down cleanly. Verified directly: spawned a broker via ensureBrokerSession(dir), rm -rf dir, then ran loadBrokerSession(dir) → sendBrokerShutdown(endpoint) → teardownBrokerSession({..., killProcess: terminateProcessTree}) → clearBrokerSession(dir) — the broker exits and state is cleaned, directory long gone. (That sequence, targeted at the workdir instead of the session cwd, is exactly what handleSessionEnd already does; it's how we're working around this today, run right before git worktree remove.)

So fix A (session-scoped broker registry) would cover deleted worktrees on graceful exit, and an idle timeout is defense-in-depth for crash paths rather than the only option.

2. Feature ask: a supported caller-facing shutdown verb.

The companion CLI currently has no way to stop a broker (cancel interrupts the turn and kills the job worker's process tree, but the broker is spawned detached + unref() and isn't in it). Callers that manage worktrees know exactly when a workdir's broker becomes garbage — at git worktree remove — but the only way to act on that today is importing lib/broker-lifecycle.mjs internals as above. A supported

codex-companion.mjs shutdown --cwd <dir>

(or broker stop) would make deterministic cleanup possible for any worktree-based workflow, independent of — and complementary to — the SessionEnd-side fixes proposed here.

+1. Corroborating the mismatch pattern in normal use on macOS v1.0.6 — among the ~260 orphaned app-server-broker.mjs (all PPID=1) I just cleaned up, the survivors consistently pointed at socket/cwd dirs (cxc-*) different from the session cwd, exactly the "hook fires but can't locate the broker" class described here rather than the never-fires case in #108. Decoupling cleanup from the cwd-hash (option A) + a parent-session liveness watchdog (option B) would cover both. The idle-timeout in #490 is a useful net for the survivors, but as you note it only masks the lookup defect.

@GeoffreyDjof

Correction on the "PPID=1" qualifier I used above — it's the wrong discriminator for the broker process, as @GeoffreyDjof lays out in #108 (and I've since confirmed on my own box): because spawnBrokerProcess uses detached: true + unref() and the spawner (codex-companion.mjs) is a one-shot that exits immediately, a broker is reparented to launchd at birth. A perfectly healthy broker serving a live session reads PPID=1. So "PPID=1" alone does not separate leaked from active — SIGTERM'ing that cohort would kill in-flight sessions.

The ~260 count in my earlier comment is still real, but the proof has to come from elsewhere: the survivors pointed at cxc-* socket/cwd dirs whose --cwd no longer exists on disk (a deleted worktree can't be under review by anyone — false positives impossible, and it reads --cwd off the process cmdline so it doesn't depend on PID liveness at all). That's the same sound signal @GeoffreyDjof converges on. The mismatch/lookup defect this issue is about is unchanged; just flagging the ppid framing in my prior note as unreliable.

Still present in 1.0.6, and I have field data plus two sub-mechanisms that I don't think are on this issue yet.

The lookup key is the git repository root, not the cwd

This sharpens the framing here, and it explains why some "different cwd" cases don't leak. resolveStateDir runs the cwd through resolveWorkspaceRoot → ensureGitRepository before hashing (lib/state.mjs:29-40, lib/workspace.mjs:3-9), so the key is the repository root of the cwd.

Measured on 1.0.6 with a real linked worktree:

shared checkout     -> .../state/main-7e72ebf75b469c52
linked worktree     -> .../state/side-d4945553b1ed5388
subdir of worktree  -> .../state/side-d4945553b1ed5388   <- same key

Consequences:

A session sitting in a subdirectory of the broker's cwd resolves to the same key and is reclaimed correctly. So "cwd differs" alone does not predict the leak.
A linked worktree is its own repository root, so a main-tree session delegating into a worktree mismatches unconditionally — which is your "structural, not accidental" point, now with the mechanism pinned down.
Because the key is evaluated at session end, a session that delegates to N worktrees can only ever reclaim the last one. (Same corollary as the comment on Broker process not cleaned up on session exit — no idle timeout #108.)

Reproduction, extending yours — no live broker needed. Plant a broker.json with "pid": null under the worktree's state dir and drive the hook directly:

case	cwd at session end	session dir	result
1	shared checkout	removable	`exit=0`, no output, record still present
2	the worktree	removable	`exit=0`, no output, record removed
3	the worktree	one leftover file in it	`exit=0`, no output, record removed, directory still there

Cases 1 and 2 are byte-identical in exit status and output, which is the observability half of this bug (also reported as #636).

Sub-mechanism A: the env-var fallback could not kill even if it were reachable

handleSessionEnd falls back to CODEX_COMPANION_APP_SERVER_ENDPOINT when the lookup misses (session-lifecycle-hook.mjs:87-93). That object carries only endpoint, pidFile and logFile — no pid and no sessionDir. teardownBrokerSession guards its kill with Number.isFinite(pid) (lib/broker-lifecycle.mjs:174), so this path can only send a shutdown request; it can never terminate the process.

#636 separately establishes that the variable is never exported to hooks, so the branch is dead code today. Both facts point the same way: it should either carry pid/sessionDir or be removed, because as written it cannot finish the job it exists for.

Sub-mechanism B: the record is cleared whether or not the broker died

Inside teardownBrokerSession only the kill is conditional (:174). Everything that makes the process findable is removed unconditionally: pid file (:182-184), log (:186-188), socket (:190-199), session dir (:201-207). Then handleSessionEnd calls clearBrokerSession(cwd) unconditionally (:113).

So on any path where the kill does not happen — no pid in the record, the fallback above, a signal that does not land — the end state is a live broker whose pid file, socket and record have all been deleted. Record-driven cleanup can never reach it, and neither can an age-based sweep, because the files it would inspect are gone. This is the same defect one call-site over from the one PR #566 addresses.

Suggested ordering: clear the record last, and only once the process is confirmed gone. A stale record is recoverable; a missing one is not.

Field data

Two machines, both on 1.0.5/1.0.6:

	machine A	machine B
live brokers	38 (oldest started 42 days earlier)	7, plus 3 with no record at all
resident memory	~3.9 GB	~305 MB
leftover records	—	121

The distribution on machine B matches this issue's thesis exactly: 121 of 121 leftover records were keyed to a short-lived working directory, and not one was keyed to a long-lived repository root. The eight clean state directories were the repository roots and the worktrees where sessions had actually ended.

The three record-less live brokers on machine B are sub-mechanism B's end state observed in the wild. I can't trace what removed their records, so I'm offering that as an observed end state rather than a causal claim.

Reclaiming them, 8 of 8 stopped on the broker/shutdown RPC alone — no SIGTERM needed. Worth noting for any reaper: going straight to a forced kill leaves the socket and the record behind and creates more of the orphans described above.

For cross-reference, the self-termination half of this (no idle timeout, no parent-death watch; 1.0.6's app-server-broker.mjs is 252 lines with zero occurrences of idle/ppid/parent/watchdog/heartbeat) is #108 / #543, with PR #490 proposing the idle timer.

Adding an impact data point not yet on this issue (plugin 1.0.5, macOS). Each leaked broker keeps its codex app-server children and their 64 KiB pipe buffers. Once enough accumulate (~50 idle brokers holding ~24 MiB of 64 KiB pipe buffers in our case), the kernel's global pipe-buffer limit is exceeded and every new pipe on the host stays at 512 bytes. Unrelated programs that write more than a few hundred bytes into a pipe that isn't drained concurrently then block in write() and hang. Sending the plugin's own graceful broker/shutdown to the idle leaked brokers restored normal 64 KiB pipes immediately.

Another data point, still reproducing on 1.0.4 (and the cleanup path is unchanged in 1.0.6):

Workflow: Claude Code desktop app, session cwd is a parent workspace, /codex:adversarial-review --cwd <git worktree of a child repo> run many times a day, worktrees removed after each task.
Result: app-server-broker.mjs serve trees accumulate with ppid 1 — 68 brokers, up to 13 days old, ~2,900 processes on 2026-10-03, and 63 brokers, 1–4 days old on 2026-10-11. Each holds a codex app-server plus its MCP children. macOS hits kern.maxprocperuid (2666) and every unrelated command fails with fork failed: resource temporarily unavailable; memory pressure makes the machine unusable before that.
Two causes compound: SessionEnd looks the broker up by the session cwd (this issue), and desktop sessions are often closed without SessionEnd firing at all (app-server brokers leak permanently when SessionEnd doesn't fire #605). Brokers whose --cwd directory has since been deleted are never reused either, so nothing ever reaches them.

Workaround we run until one of the idle-timeout PRs (#680, #580, #457, #652) lands: a launchd job every 5 minutes that kills a broker tree when its --cwd no longer exists, or when it is >20 min old, used <1 s CPU since the previous pass, and no codex-companion.mjs process is running. An in-broker idle timeout would make that unnecessary.

Confirming this on 1.0.6 (codex-cli 0.162.1, Node 24.22, macOS 27 / Apple silicon, 16 GB). Here it caused a kernel watchdog panic.

Git worktrees trigger this all the time. Each linked worktree resolves to its own workspace root, so it gets its own broker.json and its own broker. A common workflow starts Claude Code at the repo root and has review agents run codex-companion.mjs review from inside .worktrees/<branch>. SessionEnd then calls loadBrokerSession(input.cwd) on the repo root, finds no broker (or only the root's), and every worktree broker is left running. 141 of my 154 state dirs under $CLAUDE_PLUGIN_DATA/state/ still have a broker.json, almost all named after worktrees. Reboots inflate that count, but it shows the pattern.

Impact. Every leftover broker keeps a codex app-server running with all of the user's configured MCP servers (for me: npx @playwright/mcp@latest, node_repl, codex-code-mode-host). The macOS jetsam report from just before the panic showed 28 of these trees, about 300 processes holding roughly 10 GB resident, some of them 5+ days old. When a heavy test run started on top of that, the compressor filled up and watchdogd stopped checking in.

Minimal repro (no model calls):

P=~/.claude/plugins/cache/openai-codex/codex/1.0.6/scripts
T=$(mktemp -d); export CLAUDE_PLUGIN_DATA=$T/data; mkdir -p $CLAUDE_PLUGIN_DATA
git init -q $T/repo && git -C $T/repo commit -q --allow-empty -m init
git -C $T/repo worktree add -q .worktrees/feat -b feat
cat > $T/spawn.mjs <<EOF
import { ensureBrokerSession } from "$P/lib/broker-lifecycle.mjs";
console.log((await ensureBrokerSession(process.argv[2], {})).pid);
EOF
PID=$(node $T/spawn.mjs $T/repo/.worktrees/feat)
echo "{\"cwd\":\"$T/repo\",\"session_id\":\"x\"}" | node $P/session-lifecycle-hook.mjs SessionEnd
ps -p $PID >/dev/null && echo "LEAKED: broker $PID still running after SessionEnd"
kill $PID

With the session cwd set to the worktree instead, the same hook stops the broker.

Possible fix: record each broker the session creates in session-scoped state, keyed by CODEX_COMPANION_SESSION_ID (it's already exported at SessionStart), not only per workspace. Then SessionEnd can tear down all of them whatever its own cwd is. An idle timeout in the broker (#543) would cover the cases where SessionEnd never fires (#605).

Workaround I'm using for now: a user-level SessionEnd hook that sends SIGTERM to any running app-server-broker.mjs whose --cwd appears in the ending session's transcript (including subagent transcripts), or whose --cwd no longer exists. It reads the live process list rather than broker.json, so it avoids #743. It does have the trade-off in #803.

Same on 1.0.6 / Linux: sessions start in ~, reviews run in repos and worktrees, so SessionEnd never finds broker.json. Found 5 live brokers (oldest 17 days) and 16 stale records.

Workaround that has held up: a SessionStart hook that kills a broker when the CODEX_COMPANION_SESSION_ID in its env matches no ~/.claude/sessions/<pid>.json with a live pid and its unix socket has no connected peer (/proc/net/unix, state 03). The second check matters because the broker is shared per workspace (#803). It also removes broker.json records whose pid is no longer a broker (#743). Recording the owning session ID in broker.json and running this check on SessionStart would cover #380, #605 and most of #543.

Another data point on 1.0.6 / macOS, same root cause.

Workflow: I always start Claude Code in my home directory and cd into different project repos during a session before running /codex:review or /codex:rescue. Each project gets its own broker (state/<project>-<hash>/broker.json), but SessionEnd only looks up input.cwd (the home directory), so none of them are ever shut down.

What accumulated in about 7 days: 5 live brokers, idle for 49–158 hours, with 0 queued/running jobs. Each one held a codex app-server, and each app-server kept one stdio MCP server per thread. One of my MCP servers is an SSH bridge, so that alone came to 46 ssh-mcp processes, each holding an SSH connection to the same host.

On working around it outside the plugin: I tried an external reaper (stop brokers with no jobs, idle for over 6 hours, and not answering busy on a probe RPC). A Codex review of it pointed out that a client can still start a request between the last check and the SIGTERM, and that broker.json can be replaced between being read and being renamed. A reaper outside the broker can't close those races, so I went back to monitoring and cleaning up by hand.

An upstream fix — tracking which brokers a session started and shutting those down at SessionEnd, or an idle timeout in the broker itself as proposed in #543 — would be much safer than anything users can do from outside.

Navigation Menu

Repository navigation

SessionEnd cleanup fails when review cwd ≠ session cwd — broker.json looked up by cwd-hash, leaving orphan brokers even on graceful /quit #380

Summary

Why this happens in normal use

Reproduction (verified)

Root-cause fix (preferred)

Fallback (defense-in-depth, not a root fix)

Related

Environment

The lookup key is the git repository root, not the cwd

Sub-mechanism A: the env-var fallback could not kill even if it were reachable

Sub-mechanism B: the record is cleared whether or not the broker died

Field data

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

SessionEnd cleanup fails when review cwd ≠ session cwd — broker.json looked up by cwd-hash, leaving orphan brokers even on graceful /quit #380

Description

Summary

Why this happens in normal use

Reproduction (verified)

Root-cause fix (preferred)

Fallback (defense-in-depth, not a root fix)

Related

Environment

Activity

MountainsCalling-me commented on Jul 10, 2026

sufhayes7 commented on Jul 15, 2026

axisrow commented on Jul 16, 2026

axisrow commented on Jul 18, 2026

h2suzuki commented on Aug 23, 2026

The lookup key is the git repository root, not the cwd

Sub-mechanism A: the env-var fallback could not kill even if it were reachable

Sub-mechanism B: the record is cleared whether or not the broker died

Field data

kasnol commented on Oct 4, 2026

pdrivom commented on Oct 11, 2026

skaffen commented on Oct 20, 2026

davidomisi commented on Oct 20, 2026

wtrchmr commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions