[Bug]: A steering restart whose Codex turn died with the shared app-server never settles; the run stays "starting" and blocks the queue #34271

Activity

1. rafiki270 commented on Nov 11, 2026

   

   rafiki270

   on Nov 11, 2026

   Author

   More actions

   Follow-up: I traced the server state and the code, and the mechanism is narrower than the report above says. The run wasn't stuck because the app-server exited. It was stuck in a steering restart, and that restart can't complete once the app-server behind the old turn has been replaced. Below are a corrected timeline, a test that fails on main (9ef52de293), and a small fix.

   Corrected timeline

   From statev2.sqlite on the affected machine (outbox, run attempts and projection tables), in UTC:

   • 08:15 and 08:29: two provider-turn.steer effects were sent to the app-server that had stopped answering ([Bug]: Shared Codex app-server backlogs or dies silently; thread/start hangs for hours and new Codex threads never start #24300).

   • 08:33:37: a third message was sent with a claudeAgent model selection. The provider changed, so T3 forced a steering restart instead of steering:

     • Attempt 1 became superseded.
     • Attempt 2 was created pending on claudeAgent (startedAt: null).
     • The run went back to starting.

     This is why the thread's providerInstanceId shows claudeAgent.

   • 09:52:50: the app-server was killed, and T3 spawned a new one.

   • 09:52:53–55: both steers failed after 5 attempts.

   • 09:52:58: the provider-turn.restart effect (sessionTransition: detach) failed after 5 attempts with this error, and nothing settled the run:
     ProviderAdapterProtocolError: Provider turn …01a1c7c5… is not active and cannot be interrupted.

   • 09:55:23: the user pressed Stop. Run 25 became interrupted, and runs 26–30 were cancelled. The old Codex provider turn row is still running.

   Cause (origin/main 9ef52de293)

   1. A provider change forces a restart. Orchestrator.ts:~5405-5420 (selectionMustApplyNow → forceRestart) decides it. Orchestrator.ts:5745-5756 puts the run back to starting with a new pending attempt. :5911-5928 enqueues provider-turn.restart.
   2. The restart first interrupts the old turn (ProviderTurnControlService.ts:292-295), without requestRuntimeRestart. The shared session id now resolves to the respawned process. Because that session is live, the no-live-session shortcut at :277-290 is skipped.
   3. The new process doesn't know the old turn. CodexAdapterV2.ts:7603-7607 returns "is not active and cannot be interrupted". interruptAndAwaitTerminal treats that as a failure, so the restart never reaches detach and start.
   4. Retries don't help. EffectWorker.ts:42-57 treats "is not active" as already stopped, but only for provider-turn.interrupt. Restarts are excluded on purpose.
   5. When retries run out, nothing settles the run. EffectWorker.ts:816-833 marks the outbox row failed and stops there. Nothing like settleStartFailure runs for a restart, so the run stays starting and blocks the queue.
   6. The cleanup from fix(server): Codex works again after its app-server is killed #26188 can't reach it. That cleanup (RunExecutionService.ts:~1690-1702) writes only to a run that is running on its original attempt. This run was starting on attempt 2.
   7. Secondary: boundExecution (EffectWorker.ts:585-600) puts a time limit only on delegated-tasks.stop. So the restart also sat on turn/interrupt to the hung app-server from 08:33 until the kill.

   Failing test

   Added to apps/server/src/orchestration-v2/SelectionRestart.integration.test.ts, reusing makeRestartAdapter and ProviderReplayHarness.layerWithRegistry(..., { runEffectWorker: false }).

   The adapter's interruptTurn behaves the way Codex does after a respawn: it succeeds with requestRuntimeRestart and otherwise fails with "is not active". The test does the following:

   1. Starts a turn.
   2. Sends a restart_active message with a different model.
   3. Queues a follow-up with queue_after_active.
   4. Drains the worker's 5 retries with TestClock.

   It then checks that the restarted run settles and the queued run starts.

   Test code

   import * as TestClock from "effect/testing/TestClock";
   
   // A Codex app-server that dies mid-turn is respawned under the same shared provider
   // session id. The respawned process never saw the old native turn, so the restart's
   // interrupt is answered with "is not active" and no terminal event ever arrives.
   it.effect(
     "a steering restart whose turn died with its app-server settles the run and frees the queue",
     () =>
       Effect.scoped(
         Effect.gen(function* () {
           const name = "selection-restart-provider-exited";
           const cwd = yield* checkpointWorkspace(name);
           const threadId = ThreadId.make(`thread:${name}`);
           const state = yield* Ref.make<RestartAdapterState>({
             activeTurn: null,
             opened: [],
             started: [],
             closedSessionCount: 0,
             failedReplacementOpen: true,
           });
           const interruptCalls = yield* Ref.make(0);
           const base = makeRestartAdapter(state);
           const adapter: ProviderAdapter.ProviderAdapterV2["Service"] = {
             ...base,
             openSession: (input) =>
               base.openSession(input).pipe(
                 Effect.map((session) => ({
                   ...session,
                   interruptTurn: (interrupt: ProviderAdapter.ProviderAdapterV2InterruptInput) =>
                     Effect.gen(function* () {
                       yield* Ref.update(interruptCalls, (count) => count + 1);
                       // Same contract as CodexAdapterV2.interruptTurn for a turn it does not own.
                       if (interrupt.requestRuntimeRestart === true) return;
                       return yield* new ProviderAdapter.ProviderAdapterProtocolError({
                         driver,
                         detail: `Provider turn ${interrupt.providerTurnId} is not active and cannot be interrupted.`,
                       });
                     }),
                 })),
               ),
           };
   
           const result = yield* Effect.gen(function* () {
             const orchestrator = yield* Orchestrator.OrchestratorV2;
             const worker = yield* EffectWorker.OrchestrationEffectWorkerV2;
             const running = yield* orchestrator.streamDomainEvents.pipe(
               Stream.filter(
                 (event) =>
                   event.type === "provider-turn.updated" && event.payload.status === "running",
               ),
               Stream.take(1),
               Stream.runDrain,
               Effect.forkScoped({ startImmediately: true }),
             );
             yield* orchestrator.dispatch({
               type: "thread.create",
               createdBy: "user",
               creationSource: "web",
               commandId: CommandId.make(`${name}:create`),
               threadId,
               projectId: ProjectId.make(`project:${name}`),
               title: name,
               modelSelection: initialSelection,
               runtimeMode: "full-access",
               interactionMode: "default",
               branch: null,
               worktreePath: cwd,
             });
             yield* orchestrator.dispatch({
               type: "message.dispatch",
               createdBy: "user",
               creationSource: "web",
               commandId: CommandId.make(`${name}:first`),
               threadId,
               messageId: MessageId.make(`${name}:first`),
               text: "first",
               attachments: [],
               modelSelection: initialSelection,
               dispatchMode: { type: "start_immediately" },
             });
             yield* worker.drain();
             yield* Fiber.join(running);
             const runId = (yield* orchestrator.getThreadProjection(threadId)).runs[0]!.id;
   
             // The app-server exits here; its replacement no longer knows the running turn.
             yield* orchestrator.dispatch({
               type: "message.dispatch",
               createdBy: "user",
               creationSource: "web",
               commandId: CommandId.make(`${name}:steer`),
               threadId,
               messageId: MessageId.make(`${name}:steer`),
               text: "switch models and continue",
               attachments: [],
               modelSelection: replacementSelection,
               dispatchMode: { type: "restart_active", targetRunId: runId },
             });
             yield* orchestrator.dispatch({
               type: "message.dispatch",
               createdBy: "user",
               creationSource: "web",
               commandId: CommandId.make(`${name}:queued`),
               threadId,
               messageId: MessageId.make(`${name}:queued`),
               text: "queued follow-up",
               attachments: [],
               modelSelection: replacementSelection,
               dispatchMode: { type: "queue_after_active" },
             });
   
             // Give the worker every retry the outbox allows (5 attempts with backoff).
             yield* worker.drain();
             yield* Effect.forEach(
               [1, 2, 3, 4],
               () => TestClock.adjust("1 minute").pipe(Effect.andThen(worker.drain())),
               { discard: true },
             );
             return {
               projection: yield* orchestrator.getThreadProjection(threadId),
               interruptCalls: yield* Ref.get(interruptCalls),
             };
           }).pipe(
             Effect.provide(
               ProviderReplayHarness.layerWithRegistry(
                 { name },
                 ProviderAdapterRegistry.layerSingle(adapter),
                 { runEffectWorker: false },
               ),
             ),
           );
   
           const [restarted, queued] = result.projection.runs;
           assert.include(
             ["completed", "failed", "interrupted"],
             restarted?.status,
             `the restarted run must settle once its provider turn is gone (status ${restarted?.status}, ${result.interruptCalls} interrupt attempts)`,
           );
           assert.include(
             ["running", "completed"],
             queued?.status,
             `the queued run must leave the queue (status ${queued?.status})`,
           );
         }),
       ),
   );

   $ cd apps/server && vp test run src/orchestration-v2/SelectionRestart.integration.test.ts -t "died with its app-server"
   AssertionError: the restarted run must settle once its provider turn is gone (status starting, 5 interrupt attempts): expected [ Array(3) ] to include 'starting'
   Tests  1 failed | 8 skipped (9)
   

   The test fails every run, after 278 ms. It reproduces the production sequence: 5 "is not active" attempts, and then the run stays starting.

   Fix I have locally

   In ProviderTurnControlService.interruptAndAwaitTerminal, I run interruptTurn through Effect.result. When the failure chain is "is not active" (a new helper, isTurnNotActiveFailure), it logs a warning and continues, the same way the no-live-session path does. The restart then detaches and starts the new turn, so the user's message is delivered.

   The change is limited to the interrupt step. A failed start whose error says "is not active" still isn't swallowed, which keeps the rule in the comment at EffectWorker.ts:42.

   Checks with the fix:

   • SelectionRestart.integration.test.ts: 9 of 9 pass, three runs in a row.
   • ProviderTurnControlService.test.ts, EffectWorker.test.ts and SteeringCompletion.integration.test.ts: 60 of 60 pass.
   • typecheck, lint, fmt --check and the pre-commit hooks pass.

   Two follow-ups would make this sturdier. I'd keep each as its own PR unless you prefer otherwise:

   • When provider-turn.restart runs out of retries, settle the starting run as failed with a visible error, the way settleStartFailure does.
   • Mark the orphaned old provider turn interrupted.

   Not verified:

   • Against a real Codex binary with a real process kill.
   • The actual codex → claudeAgent switch. The test uses a model change on one instance, which takes the same restart path.
   • The full test suite, or anything on Windows.

   I'm happy to open the PR for the interrupt fix and its test if you want it.
2. rafiki270

   changed the title [-][Bug]: A run in flight when the shared Codex app-server exits stays "starting" forever and blocks every queued run on its thread[/-] [+][Bug]: A steering restart whose Codex turn died with the shared app-server never settles; the run stays "starting" and blocks the queue[/+] on Nov 11, 2026
3. juliusmarminge commented on Nov 11, 2026

   

   juliusmarminge

   on Nov 11, 2026

   Member

   More actions

   @rafiki270 Could you share the server log lines (or the orchestration_v2_effect_outbox rows) for run 25's thread between 09:52:49 and 09:52:56, especially any 'Orchestration effect execution failed' entries with effectType provider-turn.restart or provider-turn.start, and say whether the first completion notice that reached the lead thread was sent with Claude (the thread's current selection) or Codex? That shows whether a restart steer put run 25 back into starting and then failed for good against the dead session.