Repository navigation
[Bug]: Codex subagent child threads and stuck-active root threads are never unsubscribed, so their MCP servers live until the app-server exits #21354
Description
Before submitting
- I searched existing issues and did not find a duplicate.
- I included enough detail to reproduce or investigate the problem.
Area
apps/server
Steps to reproduce
Related, not duplicates: #15979 (closed, fixed by #16917) covered idle root/delegated threads. This report is what is still left after that fix on a build that includes it.
Observed on a long-lived shared codex app-server (macOS, T3 Code Nightly 0.0.46-nightly.20261014.3118, codex-cli 0.163.0, several stdio MCP servers configured, Codex threads that spawn their own subagents, plus a few long-running root threads):
- Run the above for about a day.
- Count the Codex thread files the app-server holds open (
lsof -p <app-server pid>, rollout.jsonlfiles) and the app-server's descendant processes. - Compare with T3's own record of those threads (
statev2.sqlite) and with thethread/unsubscribecalls in T3's provider log.
Model-free check of the Codex side (no model calls): start codex app-server with a stub stdio MCP server, thread/start, then thread/unsubscribe. The stub exits only after Codex emits thread/closed, about 30 minutes later with the default delay. Skip the unsubscribe and the stub lives as long as the app-server does. So the cleanup has to come from T3, as #16917 found.
Expected behavior
- Codex-native subagent child threads (
source: subagent) are unsubscribed on the same idle rule as root threads, so their MCP servers stop. - A root thread that T3 marks
activebut that has produced no events for a long time eventually times out or is reconciled, so it does not hold its MCP servers forever.
Actual behavior
After about 24 hours, one app-server held 100 loaded Codex threads, 393 descendant processes and about 1.5 GB. T3's database knew 98 of the 100:
| Threads | Count | T3 status | Idle for |
|---|---|---|---|
Codex subagent children (source: subagent) |
97 (95 found in T3) | idle |
5.7 h to 24 h |
| T3 root threads | 3 | active, no pending background work |
16 h to 19 h |
- Root threads that went idle follow the fix(server): unload Codex threads left idle on the shared app-server #16917 path correctly:
thread/unsubscribe, thenthread/closedabout 30 minutes later, and their MCP servers are gone. T3's retained log shows 76 unsubscribe calls; none is for any of the 100 open threads. The log is rotated and partial, so this is evidence, not proof. - From reading
ProviderSessionManager.tsonmain(not instrumented): the idle-unload timer is keyed by the attached T3 thread (idleThreadUnloads, set inholdThreadLoaded) and tracks one provider thread per T3 thread. A Codex-native child thread is never attached as a T3 thread, so no timer exists for it. - Likewise from source reading:
unloadIdleThreadskips a thread with a busy turn and defers whilehasPendingBackgroundWorkForThreadis true, so a thread that T3 believes is running is never unloaded. That may explain the 3activeroots; I did not confirm why they areactive. - 32 MCP sets (one
node_repleach) were alive, started 5.7 h to 21.8 h ago and none younger. Three start in the same second as the three root threads' files; eleven start in the same second as a subagent thread file; fourteen match no thread-file creation time (a thread resumed later would not). Thread count and set count are not one-to-one, so this shows who holds some sets, not all.
Not proven: which of the 100 threads owns each remaining set, and whether the 3 active roots are truly stuck or are long turns that T3 mis-tracks. No thread was opened or changed to gather this.
Impact
Major degradation or frequent failure
Version or commit
Desktop app bundle 0.0.46-nightly.20261014.3118 (includes #16917 / eba0521). Source references are main as read on 2026-10-18.
Environment
macOS, T3 Code Nightly desktop, Codex app-server codex-cli 0.163.0, shared Codex provider session.
Logs or stack traces
No raw logs attached. Counts above come from lsof on the app-server, ps for descendants, a read-only read of statev2.sqlite, and T3's provider log.
Workaround
Restarting the T3 server (and so the codex app-server) releases everything. In a stub-server check, archiving a thread freed its MCP servers at once.
Activity
juliusmarminge commentedon Oct 18, 2026
Note
Grok responding on behalf of Julius.
Accepting the subagent half as a bug. On main, the Codex adapter's
unloadThreadsendsthread/unsubscribeonly for the root native thread, and the idle unload timer inProviderSessionManageris keyed by attached T3 threads, so Codex-native subagent children are never unsubscribed. Open PR #20048 makesunloadThreadunsubscribe the whole subagent subtree, which covers this half.The three roots stuck
activeare a separate question. Unloading is deliberately skipped while a turn is busy, and #20048 notes that a wedged turn whose interrupt times out stays active natively. The report doesn't show why those turns never completed, so we can't act on that part yet. If it happens again, these would help:codex-interrupt-timeout.1kuna commentedon Oct 24, 2026
Title: Additional retention measurement and owner-aware disposal visibility for delegated MCP stacks
Problem: A passive census found 20 MCP/helper stacks with 275 descendants and 13.50 GiB summed RSS under the T3-owned Codex app-server. Of 19 time-matched stacks, 10 aligned with completed delegated tasks. The inspected projection did not expose a definitive process-to-provider-thread ownership map. These are retained stacks, not proven orphaned processes: a final worker answer can intentionally leave background work running, and a reusable provider thread may remain loaded after a turn ends. The absence of ownership and disposal status prevents distinguishing legitimate retention from missed unloads.
Reproduction or observation: Use a stub stdio MCP server and a delegated task with no background work. Record server births, child completion, subscription/unsubscribe events,
thread/closed, and descendant exit. Repeat for a provider-native child and for a child with intentionally live background work. Observe T3's state projection separately from provider terminal evidence. The audit itself used process-tree and lifecycle-time comparisons; it did not run this controlled test or terminate processes.Proposed behavior: Expose the owning task/provider thread, MCP server identity, process tree, last actual tool call, in-flight work, subscription state, retention reason, and pending/completed disposal state. On task completion, release idle task-owned resources through the provider's supported unsubscribe/unload lifecycle. Keep resources needed by live commands, requests, or background work until that work finishes, then release them. Reconcile terminal provider events with T3's projected state so a stale Running flag cannot retain resources indefinitely. Disposing a session must preserve durable transcript/resume state.
Why native: T3 knows delegated-task ownership and Codex owns its transports; they can coordinate safe unloads. Process-name/age-based cleanup cannot tell an idle server from a live background job, and polling agents cannot supply the missing ownership map.
braden808 commentedon Oct 27, 2026
Additional macOS incident on T3 Code Beta 0.1.2 / Codex 0.164.0, October 26, 2026.
A retained 21:37 MDT process snapshot has 782 direct children of the T3-owned Codex app-server, including 774 helpers: 259
cua_node, 259node_repl, and 256cua-driverMCP processes. Their summed RSS was 4,474.8 MiB (summed RSS, not unique physical memory). The app-server had been running since approximately 18:10 MDT. A subsequent read verified the T3 → Node launcher → Codex app-server parent chain; no processes were terminated to collect this evidence.The host operator reported approximately 30 GB swap, Data-volume free space declining from 111 GB to 61 GB, and load above 200. Those are the operator's incident measurements, not new measurements by this source investigation. The raw snapshot contains unrelated private command arguments and is retained locally rather than attached.
Current
main(f6a9315d19284422e2f2718e487c16acda4ba24e) still has root-onlythread/unsubscribeinCodexAdapterV2.unloadThread; its registered native subagents are not included. This matches the accepted native-subagent source defect here. I found the existing fix in #20048 and am preparing that fix locally rather than opening a duplicate implementation PR.The process census does not establish ownership of each helper or prove that every turn leaks. The separate documented T3/Codex idle delays, live background work, and roots still marked active need to remain distinct. The fix must preserve active work, independently loaded children, shared app-server lifetime, and durable resume history. This incident does not establish actual macOS teardown of the proposed fix; that still needs verification on an isolated runtime before deployment.
Krarilotus commentedon Nov 9, 2026
Additional Windows observation on official T3 Code 0.1.7 (
322f6544dfad0a3c8a0c8cbecf2c7f7874b14268), 9 November 2026. Reporting here rather than opening a duplicate; #16917 is already in this version, while #20048 remains open.After roughly 15 T3-owned Codex delegates since a 10:58 CET restart, the operator's read-only census at 13:22 found 37 direct children of the shared Codex app-server:
node.exe, about 1.1 GB summed working set;node_repl.exe;codex-code-mode-host.exe.They ranged from 10:58 to 13:19 start times. The operator reports that helper counts accumulate as delegates are used and the machine becomes sluggish. On this 64 GB machine the same snapshot had 17 GB free and 21% CPU, with other processes/providers also active. Those figures do not establish how much of the slowdown this retention causes.
A subsequent read-only check confirmed the same app-server/start time and counted 30
node.exe, 15node_repl.exe, one code-mode host, plus the observation command itself. No processes were killed to collect this.Limits: these are process-tree counts, not a complete process-to-native-thread ownership trace. We have not established which helpers belong to completed T3 delegates versus provider-native subagents, live background work or legitimate warm sessions. No controlled unsubscribe/
thread/closed/OS-exit test was run here. Consequently this is additional Windows impact evidence, not proof that every helper is orphaned or that #20048 closes every observed case. The already-accepted native-subtree bug and separately stuck-active roots should remain distinct.The server reports 0.1.7 and the latest stable release is still 0.1.7. We are retaining the official build rather than installing a local patch; the relevant upstream fix remains #20048.
1kuna commentedon Nov 10, 2026
Additional supported-lifecycle test on macOS, 10 November 2026, T3 Code Beta 0.1.7 / codex-cli 0.164.0. The same T3-owned Codex app-server remained running throughout (about 27.5 hours uptime).
We selected only completed T3-delegated Codex children belonging to one orchestrator. Each had a final report, no active run and zero pending requests. Other owners' threads were untouched.
Measurements from
ps(summed RSS, not unique physical memory) andsysctl vm.swapusage:Node-family here means executables named node or node_repl, plus npm launchers whose process title is rewritten. This definition is consistent across the table; host totals include unrelated live work.
The first batch initially showed no obvious disposal. At ~110 seconds, direct children fell to 879 and apparent repeated stacks fell from 110 xcodebuildmcp npm launchers to 109 (the final sample also had 109 launchers, 545 node and 218 node_repl direct children). That suggests one delayed stack exit, but this census does not establish its owning thread or causation. After that partial drop, we archived the remaining eligible children as the test plan specified. All 183 archive states were read back and verified; all remained completed with no active run or pending request. No further stack reduction appeared during the next minute.
We also tried the exposed settle action on an archived completed child. T3 rejected it because the thread was archived; no settle action succeeded, and we did not bypass that precondition. The exposed MCP tools offered no dedicated idle provider-session close/unload action.
Result: archiving completed T3 delegates did not provide enough immediate resource relief to resolve this incident. The apparent one-stack reduction is partial evidence, not proof that all remaining helpers are orphaned or that archiving never disposes resources. Longer Codex disposal delays, provider-native children, and other live owners remain possible factors. No processes were killed and no application or provider was restarted. No raw logs attached.
juliusmarminge commentedon Nov 12, 2026
I tried to reproduce this on the current build and could not.
Commit 3bd7b04 (origin/main). (1) New adapter test 'unsubscribes native subagent threads when their root thread unloads' in apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts. It replays a Codex root turn that spawns one subAgentActivity child, lets the child and the root complete, then calls runtime.unloadThread on the root. On main the only thread/unsubscribe sent is {threadId:'unload-root'}; the child 'unload-child' is never unsubscribed, so the test fails. With a throwaway edit that also unsubscribes the threads tracked in subagentThreads, the test passes (edit reverted, not committed). (2) Real codex app-server (codex-cli 0.167.2) driven over JSON-RPC with no T3 involved: thread/start, then a turn telling the agent to spawn one subagent. The child thread appeared (subAgentActivity started, then completed) and thread/loaded/list listed both root and child. A root-only thread/unsubscribe returned 'unsubscribed', and 3 s later both threads were still loaded. That is expected, because Codex closes a thread only after a delay of about 30 minutes. A 34-minute run to see whether the child outlives the root's thread/closed was still going when this result was written, so its outcome is not recorded. I did not run the full T3 app through a 30-minute idle unload: the idle timeout (DEFAULT_IDLE_TIMEOUT_MS, 30 min) cannot be configured at runtime. So
reproducedis false under the strict 'real run of the app' rule, though the failing test exercises the real adapter code path. The 'stuck-active roots' half was not reproduced. A maintainer already said it is a separate question that needs more data.Closing as not planned. If it still happens for you, reply with the exact steps and the version you are on and it will be looked at again.
Krarilotus commentedon Nov 12, 2026
Could you append the existing result of the 34-minute Codex experiment mentioned in the closure note, if that run completed?
The note records that the current-main adapter test fails with root-only unsubscribe and passes with the temporary subtree-unsubscribe change, but the full-app idle-unload case was not run and the delayed provider-close result was still pending. Whether the native child remained loaded after the root's
thread/closedhelps distinguish a missing T3 subtree unsubscribe from Codex's own delayed cleanup.I have no new current-build full-app reproduction to add beyond the qualified 9 November Windows observation, and I am not treating those process counts as proof that every helper is orphaned. The separately unproven stuck-active-root case remains separate.
#20048 was subsequently closed unmerged as backlog cleanup, with an invitation to refresh and reopen; that closure does not establish that its change landed. I am asking only for the outcome of the already-started experiment, not a new run or automatic reopening.