Bug: agent silently stops responding when codex turn enters terminal-idle with stuck activeAppServerTurnRequests counter

TL;DR

When a Telegram (and likely any channel-routed) inbound triggers a heavier agent turn — typically with one or more dynamic tool calls — the agent occasionally enters a state where the codex app-server turn never receives its turn/completed notification AND the activeAppServerTurnRequests counter does not return to zero. Both turnCompletionIdleTimeoutMs and turnTerminalIdleTimeoutMs watchdogs bail out via the activeAppServerTurnRequests > 0 guard, so the turn never aborts. From the user's perspective the bot silently stops replying; openclaw reports the channel as healthy. The session must be deleted via gateway call sessions.delete to recover. The wedge is sticky across systemctl restart openclaw.

Environment

• openclaw version: 2026.5.12 (f066dd2) (from CLI banner)
• Install method: sudo npm install -g openclaw@latest
• Path: /usr/lib/node_modules/openclaw/dist/
• Node: 22.22.2
• Host: Ubuntu Linux x86_64, systemd-managed service
• Agent backend: openai-codex OAuth (Codex sidecar @openai/codex-linux-x64)
• Channels: Telegram (long-polling) + Discord
• Memory backend: qmd
• Session scope: session.dmScope: per-channel-peer

Symptom

1. User sends a Telegram message (>~200 chars, or any message that triggers tool calls).
2. Bot receives the inbound (logged via gateway/channels/telegram/inbound) and dispatches it to the agent session agent:main:telegram:direct:<USER_ID>.
3. Agent starts processing and emits some normal notifications (item/agentMessage/delta, etc.), then stops emitting.
4. No reply is ever sent. No error logged. No timeout fires.
5. Subsequent inbounds from the same user queue behind the dead "processing" turn (session.dmScope: per-channel-peer) and are also silently dropped.
6. openclaw channels status shows Telegram as running, connected with in:Nm ago updating but out:Ym ago frozen. openclaw health event loop is healthy.
7. systemctl restart openclaw does NOT recover — the wedged session entry survives in ~/.openclaw/agents/main/sessions/sessions.json and the same dead-turn state is loaded back. The bot will refuse to process new inbounds for that session indefinitely.
8. Only fix: openclaw gateway call sessions.delete --params '{"key":"agent:main:telegram:direct:<USER_ID>"}'. After deletion, the next inbound creates a fresh session and works — until the next wedge.

Smoking-gun log line

A [diagnostic] liveness warning consistently appears ~20-30s after the wedge starts, with the work descriptor naming the most recently processed notification:

[diagnostic] liveness warning: reasons=event_loop_delay
  interval=39s eventLoopDelayP99Ms=34.6 eventLoopDelayMaxMs=18438.2
  eventLoopUtilization=0.567 cpuCoreRatio=0.625
  active=1 waiting=0 queued=1
  phase=channels.telegram.start-account
  recentPhases=sidecars.main-session-recovery:7ms,sidecars.restart-sentinel:33ms,
    post-attach.update-sentinel:10ms,sidecars.session-locks:40ms,
    sidecars.model-prewarm:921ms,post-ready.maintenance:29ms
  work=[
    active=agent:main:telegram:direct:<USER_ID>(processing/tool_call,q=1,age=20s
      last=codex_app_server:notification:account/rateLimits/updated)
    queued=agent:main:telegram:direct:<USER_ID>(processing/tool_call,q=1,age=20s
      last=codex_app_server:notification:account/rateLimits/updated)
  ]

The last=codex_app_server:notification:account/rateLimits/updated substring is reliably present in every wedge I've observed.

NOT the cause (ruled out)

• Auth: openclaw agent --agent main -m "ping..." returns PONG in seconds. LLM and codex backend are healthy. Direct agent calls never wedge.
• Rate limiting: no HTTP 429, no quota/throttl/too many log entries. The account/rateLimits/updated notification is a normal periodic status update, NOT a rate-limit-exceeded signal.
• Telegram side: getMe and sendMessage succeed; pending_update_count=0; no webhook conflict.
• Stale session route migration: openclaw doctor --fix was run; routes are clean.
• Token / channel config: validated; bot polls successfully.
• Orphan processes: pgrep confirms a single openclaw process owns port 18789.

Root-cause hypothesis (from source reading)

File: /usr/lib/node_modules/openclaw/dist/run-attempt-DValQTsj.js

Three relevant idle watchdogs are defined with these hardcoded defaults (lines 2105-2107):

const CODEX_TURN_COMPLETION_IDLE_TIMEOUT_MS = 6e4;           // 60s
const CODEX_TURN_ASSISTANT_COMPLETION_IDLE_TIMEOUT_MS = 1e4; // 10s
const CODEX_TURN_TERMINAL_IDLE_TIMEOUT_MS = 30 * 6e4;        // 30 min (!)

Both fireTurnCompletionIdleTimeout (line 2711) and fireTurnTerminalIdleTimeout (line 2740) bail without firing when activeAppServerTurnRequests > 0:

const fireTurnTerminalIdleTimeout = () => {
    if (completed || runAbortController.signal.aborted
        || !turnTerminalIdleWatchArmed
        || activeAppServerTurnRequests > 0) return;    // <-- the escape hatch
    ...
};

The counter is incremented at line 2933 inside an addRequestHandler callback and decremented in a finally block at line 3056. As long as the finally always runs, the counter is balanced. But there are at least three code paths that can leave the counter elevated under pathological conditions:

1. Request handlers that await a promise that never settles — mcpServer/elicitation/request, item/tool/requestUserInput, isCodexAppServerApprovalRequest(...) (lines 2945-2974) — none of these have their own timeout wrappers. If the awaited handler hangs, finally waits forever.
2. Race between increment and a synchronous throw before the try block — very unlikely but theoretically possible.
3. Unhandled rejection inside the inner await that bypasses the catch but doesn't fully unwind — needs verification.

handleDynamicToolCallWithTimeout (line 3469) DOES enforce a 30s default timeout for tool calls via Promise.race([..., timeoutPromise]), so the item/tool/call path is protected. But the other request methods listed in (1) are NOT wrapped in a timeout race.

Why the 30-min terminal-idle constant matters

Even if the counter eventually returns to zero, the upper-bound wedge time is 30 minutes before the terminal-idle timeout fires. This is far too long for an interactive chat assistant. The constant is not exposed via config (only turnCompletionIdleTimeoutMs has a schema entry in client-DRJokO_e.d.ts:397).

Suggested fixes

1. Wrap mcpServer/elicitation/request, item/tool/requestUserInput, and approval-request handlers in Promise.race with a configurable per-request timeout, mirroring handleDynamicToolCallWithTimeout. This is the actual root-cause fix.
2. Expose turnTerminalIdleTimeoutMs via config schema (and consider lowering the default to something like 5 min). This caps the worst-case wedge duration even when the bigger fix isn't in.
3. Consider removing the activeAppServerTurnRequests > 0 bailout from the terminal-idle watchdog specifically — terminal idle is a last-resort hard cap and should fire regardless of whether requests are nominally in-flight, since a hung request is precisely what creates the wedge.
4. Make session state recovery on systemctl restart openclaw drop in-flight turn state, so a restart actually serves as a recovery action rather than re-loading the dead state.

Workaround currently deployed

User-level systemd timer (~/.config/systemd/user/openclaw-telegram-watchdog.timer, every 15s) runs ~/.local/bin/openclaw-telegram-watchdog.sh, which:

• Parses openclaw channels status for in: and out: ages of the Telegram channel
• If a recent inbound (within 7 min) has been waiting >45s with no fresh outbound, calls openclaw gateway call sessions.delete --params '{"key":"agent:main:telegram:direct:<USER_ID>"}'
• Logs to /tmp/openclaw-telegram-watchdog.log

Worst-case recovery time: ~60s (15s timer + 45s threshold). Not great UX but the bot no longer goes silent for hours.

Reproduction recipe

1. Configure openclaw with a Telegram channel, session.dmScope: per-channel-peer, and an agent backed by openai-codex OAuth (Codex sidecar).
2. Enable a tool that takes more than a few seconds occasionally (e.g., memory_search via qmd, or any tool that hits an external service).
3. From the configured Telegram user, send a substantial message (200+ chars) that prompts the agent to invoke at least one dynamic tool.
4. Observe openclaw channels status over the next ~60s: in: updates with the new inbound, but out: does not.
5. Send a second message. It will not be processed.
6. Tail journalctl for the smoking-gun log line above.

Wedge frequency in my environment: roughly 1 in 2-3 longer inbounds. Short greetings ("hi", "thanks") almost never trigger it.

Questions for the maintainer

• Is there a config-key path to override turnTerminalIdleTimeoutMs that I missed?
• Are there known issue tracker entries for this failure mode? Happy to attach observations to an existing thread if one exists.
• Would you accept a PR to (a) wrap the non-tool-call request handlers in Promise.race and (b) expose turnTerminalIdleTimeoutMs via config schema?

Happy to provide additional traces, instrumented logs, or test against a patched build.