[Bug]: Shared Codex app-server backlogs or dies silently; thread/start hangs for hours and new Codex threads never start #24300

Activity

1. juliusmarminge

   added

   bugSomething is broken or behaving incorrectly.Something is broken or behaving incorrectly.

   on Oct 24, 2026
2. rafiki270 commented on Nov 11, 2026

   

   rafiki270

   on Nov 11, 2026

   More actions

   Still happening on stable 0.1.7 (322f6544df) on Windows, which already includes #24600, #26188 and #27924. This instance differs from the original report in one useful way: the app-server's stdout stayed open and its notifications arrived on time. The process stopped answering requests, while T3's reader kept up.

   Setup

   • Windows 11 Pro build 26300, AMD Ryzen 9 9950X3D, 125 GB RAM. Load during the incident was about 20% CPU, with 34 GB RAM free.
   • T3 Code desktop 0.1.7 running as an always-on server. Used remotely from a Mac and a phone through T3 Connect (linked environment, "approval-required" access).
   • codex-cli 0.166.0 (WinGet). One shared app-server (provider-session:provider-instance:codex:shared) for every Codex thread in the environment.
   • The workload is heavy orchestration. Lead threads delegate to up to 20 concurrent delegate_task children (gpt-6-luna, gpt-6.1-sol), and several leads run at once. The environment has 528 threads, and statev2.sqlite is 2.3 GB.
   • The app-server was started at 22:58 local after a reboot. It had been up about an hour when it stopped answering.

   Evidence (~/.t3/userdata/logs/provider/events.*.log, UTC)

   2026-11-10T23:52:43Z  id 158 thread/unsubscribe -> answered in 7 ms   # last answered request
   2026-11-11T00:00:08Z  id 159 thread/resume  (excludeTurns: true)       # never answered
   2026-11-11T08:15:12Z  id 160 turn/steer     (user: "U stuck?!")        # never answered
   2026-11-11T08:29:43Z  id 161 thread/start   (brand-new thread)         # never answered
   2026-11-11T08:33 .. 09:49  incoming skills/changed every ~5 min, emittedAtMs == receive time
   

   • After id 158 at 23:52, no incoming message carried an id. Only skills/changed notifications arrived, all on time.
   • The app-server process (PID 6256) was alive and idle. Its CPU time moved 0.016 s in 5 s. It held 553 OS threads and 3,849 handles, and had about 50 sets of MCP children under it (node, node_repl, 1password-mcp, cmd).
   • Nothing reached the UI. Every Codex thread on the machine sat at "starting" or "Working" for about 10 hours. A new thread showed "Workspace ready" and never produced a turn.
   • Network was fine. DNS and HTTPS to api.openai.com and chatgpt.com answered from the same host.

   Recovery

   Stopping PID 6256 made T3 spawn a new app-server within seconds. The new thread waiting since 08:29 answered at 09:53, and the thread whose thread/resume had hung resumed. Its running children were marked failed, which is correct. One lead thread stayed stuck on its old run. I'm filing that as a separate issue, because it happens after the process is replaced and needs a different fix.

   What would have helped

   The original report already asks for a bounded timeout on thread/start, thread/resume and turn requests that fails the run visibly, and for restarting the shared app-server when it stops answering. This case adds one thing: a liveness check can't rely on stdout closing or on notification lag. Here both looked healthy. It has to check that requests are still being answered, for example by pinging with a cheap request and treating no response within N seconds as dead.

   I can send a PR with a request timeout and a test using a fake app-server that keeps sending notifications but never answers a request. Would you take it?
3. rafiki270 commented on Nov 11, 2026

   

   rafiki270

   on Nov 11, 2026

   More actions

   Follow-up: here's where it happens in the code, with a test that fails on main (8d236998d0).

   Where it happens

   Nothing puts a time limit on a request to the app-server. A request ends only when Codex answers or the connection terminates.

   • packages/effect-codex-app-server/src/protocol.ts:497-530: request registers a pending entry, enqueues the frame and then calls Deferred.await(deferred) (line 524) with no timeout. The only ways out are handleResponse (290-308) and failAllPending (197-205). failAllPending runs from handleTermination (207-229), which fires only on stdout end, a read failure or a write failure (447-495). That's the path fix(codex): bound pending requests after app-server stdout closes #24600, fix(server): Codex works again after its app-server is killed #26188 and fix(codex): a failed write to the app-server no longer leaves requests hanging #27924 fixed. In this incident none of those happened.
   • CodexAdapterV2.ts: thread/resume (6971-6986 and 7336-7357, the excludeTurns: true path that hung here), thread/start (7304-7316), turn/start (7023), turn/steer (7500) and initialize (1816-1846) all wait without a deadline. ProviderSessionManager.ts:1985-2045 and ProviderTurnStartService.ts:1268 add none either.
   • Only interruptTurn has one (ProviderSessionManager.ts:2140-2153, 15 s). Its comment says: "A provider starved of memory can stop answering, so no caller may wait on it unbounded." Resume, start and steer don't follow that rule.
   • Liveness of the shared app-server is detected only through client.awaitTermination (CodexAdapterV2.ts:1858-1870). That requires the process to exit.

   On T3's side, one hung request doesn't block other threads' requests. The second test below shows another thread's thread/start answered while thread/resume hangs. So in this incident ids 160 and 161 went unanswered because Codex itself stopped answering.

   One thing I couldn't tell from the logs: outgoing frames are logged when they're enqueued (protocol.ts:237-248), before they're written. If Codex had stopped reading stdin, the writer would block (_internal/stdio.ts:25-28) and the log would look the same. Either way, nothing in T3 notices.

   Failing test

   The test goes into apps/server/src/orchestration-v2/Adapters/CodexAdapterV2.test.ts, next to the process-exit tests. The fake app-server answers initialize and thread/start, never answers thread/resume, and keeps sending skills/changed every 5 minutes while the TestClock advances 60 minutes.

   Test code

   describe("CodexAdapterV2 unanswered app-server request", () => {
     // A live shared app-server whose stdout stays open and keeps sending
     // notifications, but which never answers one request. Every other request is
     // answered from the replay preamble with the request's own id.
     const makeSilentResumeSession = (scenario: string) =>
       Effect.gen(function* () {
         const nativeThreadId = `native-${scenario}`;
         const input = yield* Queue.unbounded<Uint8Array, Cause.Done>();
         const emit = (frame: unknown) =>
           Queue.offer(input, new TextEncoder().encode(JSON.stringify(frame) + "
   "));
         const resumeSent = yield* Deferred.make<void>();
         const preamble = codexReplayPreamble({ nativeThreadId, nativeTurnId: "unused", prompt: "" });
         const client = yield* CodexClient.make(
           Stdio.make({
             args: Effect.succeed([]),
             stdin: Stream.fromQueue(input),
             stdout: () =>
               Sink.forEach((chunk: string | Uint8Array) =>
                 Effect.gen(function* () {
                   const frame = JSON.parse(
                     typeof chunk === "string" ? chunk : new TextDecoder().decode(chunk),
                   ) as { readonly id?: number; readonly method: string };
                   if (frame.method === "thread/resume") {
                     // Codex accepted the request and never replies.
                     yield* Deferred.succeed(resumeSent, undefined);
                     return;
                   }
                   const response = preamble.find(
                     (entry) =>
                       entry.type === "emit_inbound" &&
                       entry.label === frame.method &&
                       Predicate.isObject(entry.frame) &&
                       "id" in entry.frame,
                   );
                   if (response?.type === "emit_inbound" && Predicate.isObject(response.frame)) {
                     yield* emit({ ...response.frame, id: frame.id });
                   }
                 }),
               ),
             stderr: () => Sink.drain,
           }),
         );
         const adapter = yield* CodexAdapterV2.makeCodexAdapterV2({
           instanceId: CodexAdapterV2.CODEX_DEFAULT_INSTANCE_ID,
           settings: DEFAULT_CODEX_SETTINGS,
           environment: {},
           clientFactory: { open: () => Effect.succeed(client) },
           digest: yield* Digest.Digest,
           fileSystem: yield* FileSystem.FileSystem,
           idAllocator: yield* IdAllocator.IdAllocatorV2,
           serverConfig: yield* ProviderReplayHarness.makeReplayServerConfig(scenario).pipe(
             Effect.orDie,
           ),
         });
         const threadId = ThreadId.make(`thread-${scenario}`);
         const runtime = yield* adapter.openSession({
           threadId,
           providerSessionId: ProviderSessionId.make(`session-${scenario}`),
           modelSelection: CODEX_TEST_MODEL_SELECTION,
           runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
         });
         const providerThread = yield* runtime.ensureThread({
           threadId,
           modelSelection: CODEX_TEST_MODEL_SELECTION,
           runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
         });
         const resume = yield* runtime
           .resumeThread({
             providerThread,
             modelSelection: CODEX_TEST_MODEL_SELECTION,
             runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
           })
           .pipe(Effect.forkScoped);
         yield* Deferred.await(resumeSent);
         // The process stays healthy-looking: it keeps notifying every 5 minutes.
         const keepNotifying = (minutes: number) =>
           Effect.gen(function* () {
             for (let elapsed = 0; elapsed < minutes; elapsed += 5) {
               yield* emit({ method: "skills/changed", params: {} });
               yield* TestClock.adjust("5 minutes");
             }
           });
         return { runtime, resume, keepNotifying, scenario };
       });
   
     it.effect("fails thread/resume within a bounded time while the app-server stays alive", () =>
       Effect.gen(function* () {
         const { resume, keepNotifying } = yield* makeSilentResumeSession("silent-resume");
   
         yield* keepNotifying(60);
   
         // Without a request deadline the resume is still pending after an hour, so
         // the run sits in "starting" with no error until the process is killed.
         const exit = resume.pollUnsafe();
         assert.isDefined(exit, "thread/resume is still waiting for a response after 60 minutes");
         assert.isTrue(exit !== undefined && Exit.isFailure(exit));
         const error = exit !== undefined && Exit.isFailure(exit) ? Cause.squash(exit.cause) : null;
         assert.nestedPropertyVal(error, "_tag", "ProviderAdapterResumeThreadError");
       }).pipe(
         Effect.scoped,
         Effect.provide(
           Layer.mergeAll(
             IdAllocator.layer,
             McpProviderSessions.layer,
             NodeServices.layer,
             NodeCrypto.layer,
           ),
         ),
       ),
     );
   
     it.effect("keeps serving other threads' requests while one request is unanswered", () =>
       Effect.gen(function* () {
         const { runtime, resume } = yield* makeSilentResumeSession("silent-resume-concurrent");
         const otherThreadId = ThreadId.make("thread-silent-resume-concurrent-other");
   
         // T3 does not serialize requests on the shared client: a second thread's
         // thread/start is written and its response routed while resume hangs.
         const other = yield* runtime.ensureThread({
           threadId: otherThreadId,
           modelSelection: CODEX_TEST_MODEL_SELECTION,
           runtimePolicy: CODEX_TEST_RUNTIME_POLICY,
         });
         assert.equal(other.appThreadId, otherThreadId);
         assert.isUndefined(resume.pollUnsafe());
       }).pipe(
         Effect.scoped,
         Effect.provide(
           Layer.mergeAll(
             IdAllocator.layer,
             McpProviderSessions.layer,
             NodeServices.layer,
             NodeCrypto.layer,
           ),
         ),
       ),
     );
   });

   $ cd apps/server && npx vp test run src/orchestration-v2/Adapters/CodexAdapterV2.test.ts -t "unanswered app-server request"
   × fails thread/resume within a bounded time while the app-server stays alive 19ms
   AssertionError: thread/resume is still waiting for a response after 60 minutes: expected undefined to not equal undefined
   Tests  1 failed | 1 passed | 151 skipped (153)
   

   Fix I have locally

   • A withRunRequestDeadline in CodexAdapterV2.ts applied to thread/resume, thread/start, turn/start and turn/steer. The limit is 2 minutes, chosen without measurements and open to discussion.
   • When the deadline passes, it fails the run with a visible error (Codex did not answer thread/resume within 2 minutes, surfaced as ProviderAdapterResumeThreadError). It also fails the session's event stream with ProviderAdapterEventStreamError, which the process-exit tests already use to retire the shared app-server.
   • I kept the deadline out of the protocol layer, because some requests, such as command/exec, are legitimately long.

   With this change both tests pass, the full CodexAdapterV2.test.ts file passes (153 tests), and effect-tsgo typecheck for apps/server is clean.

   Not verified yet:

   • That ProviderSessionManager then respawns the app-server, end to end with a real Codex binary.
   • Whether initialize should get the same deadline.
   • The full test suite.

   I'm happy to open the PR if you want it.
4. SacredAI commented on Nov 14, 2026

   

   SacredAI

   on Nov 14, 2026

   More actions

   Another cause of unanswered thread/start, with a repro that does not need T3: Codex blocks forever on a locked Linux Secret Service keyring.

   Setup: T3 Code nightly 0.1.10-nightly.20261113.4656, systemd user service on headless Linux x64 (Ubuntu; only the GDM greeter on seat0, no desktop login). codex-cli 0.169.0, an OAuth MCP server (Sentry), default mcp_oauth_credentials_store.

   What happened: both shared app-servers (codex_personal, codex) answered initialize, then never answered thread/resume (id 2) or thread/start (ids 2, 3, 4, 6, 7) between 02:18 and 02:34 UTC. Each app-server was idle at 0% CPU, with open connections to the session D-Bus. Codex threads hung with no error in the UI. In the same window:

   • codex mcp list failed; text generation keeps its MCP servers { reason: 'timeout' }, and its codex exec children stayed hung
   • orchestration-v2.driver-session.detach-unload-failed: TimeoutError: Operation timed out after '10s'

   Cause: the GNOME keyring login collection was locked:
   busctl --user get-property org.freedesktop.secrets /org/freedesktop/secrets/collection/login org.freedesktop.Secret.Collection Locked → b true.
   Codex asks the Secret Service for the MCP OAuth token. The keyring needs an unlock prompt that cannot appear, so the call never returns.

   Repro:

   1. Headless Linux, gnome-keyring running, login collection locked.
   2. Codex config with an OAuth MCP server.
   3. codex exec -c mcp_oauth_credentials_store=auto "Reply with exactly: ok" hangs (killed at 45 s). With -c mcp_oauth_credentials_store=file it answers in 5 s. codex mcp list behaves the same way (hangs vs. 0.01 s).

   Workaround: add mcp_oauth_credentials_store = "file" to ~/.codex/config.toml, then restart the T3 server to replace the stuck app-servers. On shadow-home installs this can lead to #26811 through .credentials.json.

   This is one more reason for the time limit on thread/start proposed above: T3 had no sign that Codex was stuck.

   ---

   Posted by Claude Code (Opus 5.5) via T3 Code triage.