Skip to content

[Bug]: Run never settles when a Codex turn ends with an error during a steer; interrupt/stop hang and every queued run blocks behind it #24743

Closed

Description

@juliusmarminge

Summary

A Codex turn that ends with a provider error while T3 is steering input into it leaves the T3 run active forever. Every later message, scheduled tick, and webhook run then queues behind it. Interrupt is accepted but changes nothing, and a Stop (t3_thread_interrupt with stop: true) never returns. Only restarting the T3 server clears it.

Environment

  • T3 0.1.2-preview.20261022.3516, Linux server
  • Codex provider (gpt-6.1-sol), long-running dispatcher thread that delegates tasks and gets "Delegated task … finished" notices steered into its active turn

Sequence (provider event log, 2026-10-24 UTC)

23:02:11.789  item/started    mcpToolCall t3-code.task_status
23:02:12.376  outgoing        {"id":307,"method":"turn/steer","params":{"expectedTurnId":"01a17024-…", "input":[{"text":"Delegated task … finished"…}]}}
23:02:12.477  incoming        {"id":307,"result":{"turnId":"01a17024-…"}}
23:02:12.478  item/started + item/completed   userMessage (the steered notice)
23:02:12.946  thread/status/changed  {"status":{"type":"systemError"}}
23:02:12.946  error           "exceeded retry limit, last status: 429 Too Many Requests" (responseTooManyFailedAttempts)
23:02:12.947  turn/completed  {"turn":{"id":"01a17024-…", "items":[], …}}
23:02:18–25   item/completed for three commandExecutions and the mcpToolCall that were in flight

After this:

  • The T3 thread still showed status: running with activeRunId = that run, and no items were added for 6 hours.
  • In-flight items showed failed in T3 at 23:02:12.000Z, even though Codex later reported them completed.
  • 50+ runs queued behind it, and none started.
  • t3_thread_interrupt (no stop) returned interrupt_requested twice, hours apart, and nothing changed.
  • t3_thread_interrupt with stop: true hung for about 40 minutes without returning.
  • systemctl --user restart t3code.service cleared it: activeRunId became null and the queue could run again.

Expected

  • turn/completed after systemError (or any provider error) settles the run as failed, even when a steer landed in the same turn.
  • Interrupt and Stop on a run whose provider turn has already ended settle it immediately, instead of waiting for a provider acknowledgement that can never come.
  • Ideally, a watchdog fails a run that has had no provider activity for a long time while its provider reports no active turn.

Related: https://git.cafe/pingdotgg/t3code/issues/24300 (shared Codex app-server backlog/death not detected).

Activity

  1. added
    bugSomething is broken or behaving incorrectly.
    on Oct 25, 2026
  2. juliusmarminge commented on Oct 25, 2026

    @juliusmarminge
    MemberAuthor

    Tested Linux web/server on 2fd28bf938 with Codex 0.163.0. A normal steer completed. With a real delegated child finishing during the parent's shell call, a schema-valid replay of systemError → non-retryable 429 error → failed turn/completed settled the parent as failed immediately; it did not remain active. This injected the provider failure rather than exhausting an upstream account. The focused adapter replay also passed on main.

    Please provide the full decoded error and turn/completed JSON from 23:02:12.946–.947, including turn.status, turn.error, and codexErrorInfo, plus the incident's codex --version. Prompt text and tool arguments can be redacted. The abbreviated terminal frame in the report omits the fields needed to check whether it was accepted by the protocol decoder. An invalid error shape in an initial replay was dropped and left a run active, so that attempt was discarded.

    Replay transcript and persisted result · Replay driver

    Delegated completion followed by a settled failed parent

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is broken or behaving incorrectly.needs more infoInitial triage showed no bug. Awaiting more info

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions