Sign up
View stargazersSign up

[Bug]: Shared Codex app-server backlogs or dies silently; thread/start hangs for hours and new Codex threads never start#24300

opened22d agobyJJulius Marminge

Summary

T3 runs one shared Codex app-server per environment (provider-session:provider-instance:codex:shared). On long-running servers, that process's message stream falls hours behind or stops entirely. When that happens, thread/start requests wait hours or never get an answer. New Codex threads then sit at "Workspace ready" with the provider thread stuck in pending, and no turn ever starts. T3 doesn't detect any of this: there is no timeout, no restart, and no error in the UI.

A second case: if the app-server process is killed (for example by stopping its systemd scope), T3 keeps routing new threads to the dead session. The only trace is CodexAppServerTransportError: Codex App Server transport operation 'read-process-exit-status' failed from ProviderSessionManager.unloadIdleThread.

Environment

  • T3 0.1.2-preview.20261022.3511, Linux server (headless, 4 vCPU / 16 GB)
  • codex-cli 0.163.0
  • Several Codex threads at once (up to 4 leads, each delegating step tasks), with the server up 1–2 days

Evidence (from ~/.t3/userdata/logs/provider/events.*.log)

thread/start latency on one server, request (outgoing id) to its response:

  • 2026-10-22 08:15–08:19: 0 s
  • 2026-10-22 09:50: 938 s
  • 2026-10-22 10:30: 1510 s
  • 2026-10-22 18:37 onward: 13,000–69,000 s (4–19 h)
  • from 2026-10-23 18:06: no response at all (11 requests)

Example: request id 88 was sent at 2026-10-23T11:53:51.883Z, and the response arrived at 2026-10-23T20:43:19.075Z.

Notification lag: receive time minus the app-server's own emittedAtMs.

  • About 20 min on 10-22.
  • 2–5 h on 10-23 (for example, 7,000–9,000 s throughout 10:25–11:34).

The app-server emits on time, but the messages reach T3 hours later. That points at the shared stdio stream or T3's reader, not at model latency. The process itself is idle (0% CPU).

Across 40 servers running the same workload, counting thread/start requests since 2026-10-23:

  • 61/61 unanswered on two servers.
  • 21/26 unanswered on another.
  • 46–55 answered late (over 5 min) on two more.
  • Five servers whose app-server had been killed: every request unanswered.

Reconnecting... 5/5 errors show up in the same stream around the stalls.

Expected

  • A thread/start (and turn start) that gets no response within a bounded time fails the run with a visible error, instead of hanging.
  • T3 detects when the shared app-server is dead or backlogged (process exit, or message lag past a threshold) and restarts it. It doesn't keep routing new threads to it.
  • Ideally, one busy or wedged thread can't stall thread/start for every other thread in the environment.

Workaround

Restart t3code.service, which starts a fresh app-server.

Rrafiki2704d ago
Synced from GitHub

Still happening on stable 0.1.7 (322f6544df) on Windows, which already includes #24600, #26188 and #27924. This instance differs from the original report in one useful way: the app-server's stdout stayed open and its notifications arrived on time. The process stopped answering requests, while T3's reader kept up.

Setup

  • Windows 11 Pro build 26300, AMD Ryzen 9 9950X3D, 125 GB RAM. Load during the incident was about 20% CPU, with 34 GB RAM free.
  • T3 Code desktop 0.1.7 running as an always-on server. Used remotely from a Mac and a phone through T3 Connect (linked environment, "approval-required" access).
  • codex-cli 0.166.0 (WinGet). One shared app-server (provider-session:provider-instance:codex:shared) for every Codex thread in the environment.
  • The workload is heavy orchestration. Lead threads delegate to up to 20 concurrent delegate_task children (gpt-6-luna, gpt-6.1-sol), and several leads run at once. The environment has 528 threads, and statev2.sqlite is 2.3 GB.
  • The app-server was started at 22:58 local after a reboot. It had been up about an hour when it stopped answering.

Evidence (~/.t3/userdata/logs/provider/events.*.log, UTC)

12026-11-10T23:52:43Z id 158 thread/unsubscribe -> answered in 7 ms # last answered request
22026-11-11T00:00:08Z id 159 thread/resume (excludeTurns: true) # never answered
32026-11-11T08:15:12Z id 160 turn/steer (user: "U stuck?!") # never answered
42026-11-11T08:29:43Z id 161 thread/start (brand-new thread) # never answered
52026-11-11T08:33 .. 09:49 incoming skills/changed every ~5 min, emittedAtMs == receive time
  • After id 158 at 23:52, no incoming message carried an id. Only skills/changed notifications arrived, all on time.
  • The app-server process (PID 6256) was alive and idle. Its CPU time moved 0.016 s in 5 s. It held 553 OS threads and 3,849 handles, and had about 50 sets of MCP children under it (node, node_repl, 1password-mcp, cmd).
  • Nothing reached the UI. Every Codex thread on the machine sat at "starting" or "Working" for about 10 hours. A new thread showed "Workspace ready" and never produced a turn.
  • Network was fine. DNS and HTTPS to api.openai.com and chatgpt.com answered from the same host.

Recovery

Stopping PID 6256 made T3 spawn a new app-server within seconds. The new thread waiting since 08:29 answered at 09:53, and the thread whose thread/resume had hung resumed. Its running children were marked failed, which is correct. One lead thread stayed stuck on its old run. I'm filing that as a separate issue, because it happens after the process is replaced and needs a different fix.

What would have helped

The original report already asks for a bounded timeout on thread/start, thread/resume and turn requests that fails the run visibly, and for restarting the shared app-server when it stops answering. This case adds one thing: a liveness check can't rely on stdout closing or on notification lag. Here both looked healthy. It has to check that requests are still being answered, for example by pinging with a cheap request and treating no response within N seconds as dead.

I can send a PR with a request timeout and a test using a fake app-server that keeps sending notifications but never answers a request. Would you take it?

Rrafiki2704d ago
Synced from GitHub

Follow-up: here's where it happens in the code, with a test that fails on main (8d236998d0).

Where it happens

Nothing puts a time limit on a request to the app-server. A request ends only when Codex answers or the connection terminates.

  • packages/effect-codex-app-server/src/protocol.ts:497-530: request registers a pending entry, enqueues the frame and then calls Deferred.await(deferred) (line 524) with no timeout. The only ways out are handleResponse (290-308) and failAllPending (197-205). failAllPending runs from handleTermination (207-229), which fires only on stdout end, a read failure or a write failure (447-495). That's the path #24600, #26188 and #27924 fixed. In this incident none of those happened.
  • CodexAdapterV2.ts: thread/resume (6971-6986 and 7336-7357, the excludeTurns: true path that hung here), thread/start (7304-7316), turn/start (7023), turn/steer (7500) and initialize (1816-1846) all wait without a deadline. ProviderSessionManager.ts:1985-2045 and ProviderTurnStartService.ts:1268 add none either.
  • Only interruptTurn has one (ProviderSessionManager.ts:2140-2153, 15 s). Its comment says: "A provider starved of memory can stop answering, so no caller may wait on it unbounded." Resume, start and steer don't follow that rule.
  • Liveness of the shared app-server is detected only through client.awaitTermination (CodexAdapterV2.ts:1858-1870). That requires the process to exit.

On T3's side, one hung request doesn't block other threads' requests. The second test below shows another thread's thread/start answered while thread/resume hangs. So in this incident ids 160 and 161 went unanswered because Codex itself stopped answering.

One thing I couldn't tell from the logs: outgoing frames are logged when they're enqueued (protocol.ts:237-248), before they're written. If Codex had stopped reading stdin, the writer would block (_internal/stdio.ts:25-28) and the log would look the same. Either way, nothing in T3 notices.

Failing test

The test goes into apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts, next to the process-exit tests. The fake app-server answers initialize and thread/start, never answers thread/resume, and keeps sending skills/changed every 5 minutes while the TestClock advances 60 minutes.

Test code
ts
1describe("CodexAdapterV2 unanswered app-server request", () => {
2 // A live shared app-server whose stdout stays open and keeps sending
3 // notifications, but which never answers one request. Every other request is
4 // answered from the replay preamble with the request's own id.
5 const makeSilentResumeSession = (scenario: string) =>
6 Effect.gen(function* () {
7 const nativeThreadId = `native-${scenario}`;
8 const input = yield* Queue.unbounded<Uint8Array, Cause.Done>();
9 const emit = (frame: unknown) =>
10 Queue.offer(input, new TextEncoder().encode(JSON.stringify(frame) + "
11"));
12 const resumeSent = yield* Deferred.make<void>();
13 const preamble = codexReplayPreamble({ nativeThreadId, nativeTurnId: "unused", prompt: "" });
14 const client = yield* CodexClient.make(
15 Stdio.make({
16 args: Effect.succeed([]),
17 stdin: Stream.fromQueue(input),
18 stdout: () =>
19 Sink.forEach((chunk: string | Uint8Array) =>
20 Effect.gen(function* () {
21 const frame = JSON.parse(
22 typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk),
23 ) as { readonly id?: number; readonly method: string };
24 if (frame.method === "thread/resume") {
25 // Codex accepted the request and never replies.
26 yield* Deferred.succeed(resumeSent, undefined);
27 return;
28 }
29 const response = preamble.find(
30 (entry) =>
31 entry.type === "emit_inbound" &&
32 entry.label === frame.method &&
33 Predicate.isObject(entry.frame) &&
34 "id" in entry.frame,
35 );
36 if (response?.type === "emit_inbound" && Predicate.isObject(response.frame)) {
37 yield* emit({ ...response.frame, id: frame.id });
38 }
39 }),
40 ),
41 stderr: () => Sink.drain,
42 }),
43 );
44 const adapter = yield* CodexAdapterV2.makeCodexAdapterV2({
45 instanceId: CodexAdapterV2.CODEX_DEFAULT_INSTANCE_ID,
46 settings: DEFAULT_CODEX_SETTINGS,
47 environment: {},
48 clientFactory: { open: () => Effect.succeed(client) },
49 digest: yield* Digest.Digest,
50 fileSystem: yield* FileSystem.FileSystem,
51 idAllocator: yield* IdAllocator.IdAllocatorV2,
52 serverConfig: yield* ProviderReplayHarness.makeReplayServerConfig(scenario).pipe(
53 Effect.orDie,
54 ),
55 });
56 const threadId = ThreadId.make(`thread-${scenario}`);
57 const runtime = yield* adapter.openSession({
58 threadId,
59 providerSessionId: ProviderSessionId.make(`session-${scenario}`),
60 modelSelection: CODEX_TEST_MODEL_SELECTION,
61 runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
62 });
63 const providerThread = yield* runtime.ensureThread({
64 threadId,
65 modelSelection: CODEX_TEST_MODEL_SELECTION,
66 runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
67 });
68 const resume = yield* runtime
69 .resumeThread({
70 providerThread,
71 modelSelection: CODEX_TEST_MODEL_SELECTION,
72 runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
73 })
74 .pipe(Effect.forkScoped);
75 yield* Deferred.await(resumeSent);
76 // The process stays healthy-looking: it keeps notifying every 5 minutes.
77 const keepNotifying = (minutes: number) =>
78 Effect.gen(function* () {
79 for (let elapsed = 0; elapsed < minutes; elapsed += 5) {
80 yield* emit({ method: "skills/changed", params: {} });
81 yield* TestClock.adjust("5 minutes");
82 }
83 });
84 return { runtime, resume, keepNotifying, scenario };
85 });
86
87 it.effect("fails thread/resume within a bounded time while the app-server stays alive", () =>
88 Effect.gen(function* () {
89 const { resume, keepNotifying } = yield* makeSilentResumeSession("silent-resume");
90
91 yield* keepNotifying(60);
92
93 // Without a request deadline the resume is still pending after an hour, so
94 // the run sits in "starting" with no error until the process is killed.
95 const exit = resume.pollUnsafe();
96 assert.isDefined(exit, "thread/resume is still waiting for a response after 60 minutes");
97 assert.isTrue(exit !== undefined && Exit.isFailure(exit));
98 const error = exit !== undefined && Exit.isFailure(exit) ? Cause.squash(exit.cause) : null;
99 assert.nestedPropertyVal(error, "_tag", "ProviderAdapterResumeThreadError");
100 }).pipe(
101 Effect.scoped,
102 Effect.provide(
103 Layer.mergeAll(
104 IdAllocator.layer,
105 McpProviderSessions.layer,
106 NodeServices.layer,
107 NodeCrypto.layer,
108 ),
109 ),
110 ),
111 );
112
113 it.effect("keeps serving other threads' requests while one request is unanswered", () =>
114 Effect.gen(function* () {
115 const { runtime, resume } = yield* makeSilentResumeSession("silent-resume-concurrent");
116 const otherThreadId = ThreadId.make("thread-silent-resume-concurrent-other");
117
118 // T3 does not serialize requests on the shared client: a second thread's
119 // thread/start is written and its response routed while resume hangs.
120 const other = yield* runtime.ensureThread({
121 threadId: otherThreadId,
122 modelSelection: CODEX_TEST_MODEL_SELECTION,
123 runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
124 });
125 assert.equal(other.appThreadId, otherThreadId);
126 assert.isUndefined(resume.pollUnsafe());
127 }).pipe(
128 Effect.scoped,
129 Effect.provide(
130 Layer.mergeAll(
131 IdAllocator.layer,
132 McpProviderSessions.layer,
133 NodeServices.layer,
134 NodeCrypto.layer,
135 ),
136 ),
137 ),
138 );
139});
1$ cd apps/server && npx vp test run src/orchestration-v2/Adapters/CodexAdapterV2.test.ts -t "unanswered app-server request"
2× fails thread/resume within a bounded time while the app-server stays alive 19ms
3AssertionError: thread/resume is still waiting for a response after 60 minutes: expected undefined to not equal undefined
4Tests 1 failed | 1 passed | 151 skipped (153)

Fix I have locally

  • A withRunRequestDeadline in CodexAdapterV2.ts applied to thread/resume, thread/start, turn/start and turn/steer. The limit is 2 minutes, chosen without measurements and open to discussion.
  • When the deadline passes, it fails the run with a visible error (Codex did not answer thread/resume within 2 minutes, surfaced as ProviderAdapterResumeThreadError). It also fails the session's event stream with ProviderAdapterEventStreamError, which the process-exit tests already use to retire the shared app-server.
  • I kept the deadline out of the protocol layer, because some requests, such as command/exec, are legitimately long.

With this change both tests pass, the full CodexAdapterV2.test.ts file passes (153 tests), and effect-tsgo typecheck for apps/server is clean.

Not verified yet:

  • That ProviderSessionManager then respawns the app-server, end to end with a real Codex binary.
  • Whether initialize should get the same deadline.
  • The full test suite.

I'm happy to open the PR if you want it.

SSacredAI1d ago
Synced from GitHub

Another cause of unanswered thread/start, with a repro that does not need T3: Codex blocks forever on a locked Linux Secret Service keyring.

Setup: T3 Code nightly 0.1.10-nightly.20261113.4656, systemd user service on headless Linux x64 (Ubuntu; only the GDM greeter on seat0, no desktop login). codex-cli 0.169.0, an OAuth MCP server (Sentry), default mcp_oauth_credentials_store.

What happened: both shared app-servers (codex_personal, codex) answered initialize, then never answered thread/resume (id 2) or thread/start (ids 2, 3, 4, 6, 7) between 02:18 and 02:34 UTC. Each app-server was idle at 0% CPU, with open connections to the session D-Bus. Codex threads hung with no error in the UI. In the same window:

  • codex mcp list failed; text generation keeps its MCP servers { reason: 'timeout' }, and its codex exec children stayed hung
  • orchestration-v2.driver-session.detach-unload-failed: TimeoutError: Operation timed out after '10s'

Cause: the GNOME keyring login collection was locked: busctl --user get-property org.freedesktop.secrets /org/freedesktop/secrets/collection/login org.freedesktop.Secret.Collection Locked → b true. Codex asks the Secret Service for the MCP OAuth token. The keyring needs an unlock prompt that cannot appear, so the call never returns.

Repro:

  1. Headless Linux, gnome-keyring running, login collection locked.
  2. Codex config with an OAuth MCP server.
  3. codex exec -c mcp_oauth_credentials_store=auto "Reply with exactly: ok" hangs (killed at 45 s). With -c mcp_oauth_credentials_store=file it answers in 5 s. codex mcp list behaves the same way (hangs vs. 0.01 s).

Workaround: add mcp_oauth_credentials_store = "file" to ~/.codex/config.toml, then restart the T3 server to replace the stuck app-servers. On shadow-home installs this can lead to #26811 through .credentials.json.

This is one more reason for the time limit on thread/start proposed above: T3 had no sign that Codex was stuck.

Posted by Claude Code (Opus 5.5) via T3 Code triage.