Bug type

Behavior bug (incorrect state without crash) — session wedge / message loss.

Summary

On 2026.5.28, a Codex app-server session can wedge indefinitely with the agent silently not replying — the same activeAppServerTurnRequests > 0 watchdog-gating mechanism reported and "completed"-closed in #82681 (and related to #84076). The two trigger-agnostic mitigations proposed in #82681 are still not present in 2026.5.28, so the wedge still reproduces.

What's new here vs #82681: #82681 explicitly ruled out rate-limiting as its trigger (its account/rateLimits/updated was a benign periodic notification). This report is the rate-limit-triggered variant — a genuine HTTP 429 burst against codex/gpt-5.5 leaves the shared app-server client unresponsive, after which in-flight turns never settle, the turn counter never returns to zero, and the gated idle watchdogs never fire. Because the client is shared, one poisoned app-server stalls multiple unrelated sessions at once (we observed an interactive Telegram turn and a scheduled cron turn wedged simultaneously, both only released when the client was force-closed by a process restart).

Environment

• openclaw: 2026.5.28 (e932160), npm i -g openclaw
• Node: v22.22.2
• OS: Linux aarch64 (kernel 6.17), systemd user service (openclaw-gateway.service)
• Agent backend: codex/gpt-5.5 via ACP/codex app-server (OAuth); codex sidecar @openai/codex-linux-arm64 0.134.0
• agents.defaults.model.fallback: none configured (so a Codex failure is not masked by a fallback)
• acp.maxConcurrentSessions: 4; shared Codex app-server client across cron + channel sessions
• Dist file for this build: dist/run-attempt-QNNU1VbX.js; source region extensions/codex/src/app-server/attempt-turn-watches.ts

Symptom (observed)

1. A burst of HTTP 429 Too Many Requests from codex/gpt-5.5 (145 over ~1h in our logs; embedded run failover decision … reason=rate_limit … next=none).
2. Shortly after, an interactive Telegram turn is dispatched, emits embedded_run:started, and then makes zero further progress. The bot shows a brief "typing…" then never replies.
3. Diagnostics: stalled session … state=processing … reason=active_work_without_progress … lastProgress=embedded_run:started lastProgressAge=169s … recovery=none. No idle timeout ever fires.
4. A cron turn on the same app-server is simultaneously stuck (durationMs=118722 when finally killed) — confirming the wedge is at the shared client / process level, not per-session.
5. Recovery: only a gateway restart (which force-closes the client → both stuck turns reject with codex app-server client closed before turn completed) or, for the sticky-across-restart case from Telegram session wedges indefinitely when codex turn-completion idle watchdog is gated by stuck activeAppServerTurnRequests counter #82681, gateway call sessions.delete. There is no automatic self-recovery.

Root cause — confirmed from 2026.5.28 source

extensions/codex/src/app-server/attempt-turn-watches.ts:

• The completion- and terminal-idle watchdogs both bail out when a request is nominally in flight. In 2026.5.28 this guard is present on both the schedule and the fire paths of the terminal-idle watchdog:

function scheduleTerminalIdleWatch() {
  clearTerminalIdleTimer();
  if (params.isCompleted() || params.signal.aborted || !terminalIdleWatchArmed
      || params.getActiveAppServerTurnRequests() > 0) return;   // <-- gate
  ...
}
function fireTerminalIdleTimeout() {
  if (params.isCompleted() || params.isTerminalTurnNotificationQueued() || params.signal.aborted
      || !terminalIdleWatchArmed
      || params.getActiveAppServerTurnRequests() > 0) return;    // <-- gate
  ...
}

Consequence: while a turn request is in flight, terminal-idle is never even scheduled — so it cannot act as the last-resort hard cap precisely when an in-flight request hangs.

• activeAppServerTurnRequests is incremented around the app-server turn request and decremented in a finally. That balances correctly only if the awaited request settles. When the app-server stops responding (post-429), the awaited promise never settles → finally never runs → the counter is stuck > 0 → all the counter-gated watchdogs are suppressed forever.

• Idle-timeout constants are still hardcoded (unchanged from Telegram session wedges indefinitely when codex turn-completion idle watchdog is gated by stuck activeAppServerTurnRequests counter #82681):

const CODEX_TURN_COMPLETION_IDLE_TIMEOUT_MS = 6e4;            // 60s
const CODEX_TURN_ASSISTANT_COMPLETION_IDLE_TIMEOUT_MS = 1e4;  // 10s
const CODEX_TURN_TERMINAL_IDLE_TIMEOUT_MS = 30 * 6e4;         // 30 min

• turnTerminalIdleTimeoutMs is still not exposed in the config schema (only turnCompletionIdleTimeoutMs is), so operators cannot lower the 30-minute worst-case cap.

Status of #82681's proposed fixes in 2026.5.28

Hypothesized (not yet confirmed from source)

• That the main app-server turn request is dispatched without a hard timeoutMs (the transport applies a timeout only when options.timeoutMs is set). This would explain why a never-responding app-server produces a never-settling await. Supporting observable: the documented ~60s/30-min idle defaults exist, yet nothing fired for 118–169s.
• That the one un-gated watchdog (attempt-idle) is not armed for the pre-first-notification window, so it can't cover a turn that hangs before producing any output. (armAttemptIdleWatch() appears to have a single, activity-tied call site.)

Proposed fix (in priority order)

1. Relax the activeAppServerTurnRequests > 0 gate on the terminal-idle watchdog (both scheduleTerminalIdleWatch and fireTerminalIdleTimeout). Terminal-idle is the last-resort hard cap; a hung in-flight request is exactly what it must catch. Because it is idleness-based (resets on any activity notification), relaxing the gate aborts only turns that have been completely silent for the full window — it does not kill long-but-active turns. The resulting abort rejects the pending request → finally runs → the counter resets → the session self-heals without a restart or sessions.delete. This is trigger-agnostic (covers 429 poisoning, hung handlers, app-server desync alike).
2. Expose turnTerminalIdleTimeoutMs in the config schema and lower the default (e.g. 3–5 min). On its own this is ineffective (the gate suppresses the watchdog regardless of value); combined with (1) it turns a silent multi-minute/30-min hang into prompt, tunable self-recovery.

Optionally, as defense-in-depth: a hard ceiling timeoutMs on the main turn request, and/or arming the un-gated attempt-idle watch for the in-flight pre-output window.

Reproduction

1. Single Codex app-server (codex/gpt-5.5 via ACP), no model fallback, shared across a chat channel + scheduled jobs.
2. Drive enough concurrent Codex turns to hit the account's 429 rate limit (e.g. several scheduled jobs firing together).
3. During/just after the 429 burst, send an interactive message. Observe embedded_run:started with no further progress, recovery=none, and no idle timeout firing.
4. Confirm a second session (e.g. a cron turn) on the same app-server is also stuck. Only a restart (or sessions.delete for the sticky case) recovers.

Happy to provide full redacted journald traces and to test a patched build.