You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Channel inbound dispatchers (Telegram, Slack, WhatsApp, etc.) currently invoke the agent method through agent-via-gateway with no effective wall-clock deadline on the turn. The cross-process call uses a no-timeout shape and waits for settlement (tool call OR text block) indefinitely. When a model deliberates for many seconds without emitting either, the runtime keeps reporting cli_live:stream_progress ticks but no settlement event ever arrives, the gateway's event loop wedges, and downstream systems (cron isolated-setup-timeout watchdog, supervisor) start firing restarts that interact poorly with sessions.json restart-recovery — see #86239 + #95288 for the production failure mode this creates.
This issue requests a configurable per-lane wall-clock deadline at the channel-dispatcher boundary, modeled after the existing cron isolated_agent_setup_timeout pattern.
Asymmetry with the cron path
The cron path already has equivalent protection:
Isolated cron lanes have an isolated_agent_setup_timeout watchdog with a structured cancellation path.
When the timeout fires, it logs reason: "cron.isolated_agent_setup_timeout" and triggers a clean teardown (which today happens to ALSO trigger a SIGUSR1 restart, but that's a separate concern — see fix: restart gateway after isolated cron setup timeout #89055 + the restart-recovery family).
Channel lanes (Telegram, Slack, WhatsApp, etc.) have no equivalent. The result is that a single no-settlement turn on a channel lane can:
Wedge the event loop for the full duration of the model's deliberation (observed: 559 s on 2026.6.9 c645ec4).
Stage 1 — narrow change. Add an optional dispatcher.channelTurnTimeoutMs config at the channel-dispatcher level (or per-channel under channels.<provider>.turnTimeoutMs). When set:
Start a timer when the dispatcher forwards an inbound message to the agent method.
The timer cancels on settlement (tool call OR text block emitted).
If the timer fires first: emit a structured cancellation, log reason: "channel.turn_no_settlement_timeout", deliver a user-visible message to the channel ("Turn aborted: no settlement within {ms}ms; please retry"), and DO NOT mark the session entry as abortedLastRun=true.
Default: 300000 (5 minutes) — long enough to avoid clipping legitimate long turns, short enough to keep the failure window bounded.
The "do NOT mark for restart-recovery" semantic is load-bearing — it's what distinguishes this from a SIGUSR1-driven restart, and it's what breaks the death-loop pattern in #86239 + #95288. The lane should recover by simply waiting for the next inbound message.
Stage 2 — broader change (separate issue if accepted). Consider migrating channel lanes to command-payload routing (analogous to what cron lanes already do internally). This addresses the underlying impedance mismatch between the dispatcher's request/response shape and the streaming agentTurn surface, but is materially larger work. Treating it as a follow-up to Stage 1 lets the cheap fix ship without coupling to the harder one.
Behavioral contract (Stage 1)
Condition
Today
Proposed
Turn emits tool call within N seconds
Continues normally
Continues normally — timer cancelled
Turn emits text block within N seconds
Continues normally
Continues normally — timer cancelled
Turn hangs ≥ N seconds with no settlement
Hangs indefinitely; event loop starvation
Aborted cleanly; user notified; session NOT marked for restart-recovery
Timer fires while a tool call is mid-execution
n/a
Treat as settled (tool calls are already bounded by their own timeouts)
Channel disabled via config
n/a
Timer is off; behavior unchanged
Coordination with existing timeout work
The cron isolated_agent_setup_timeout watchdog is structurally similar but operates at a different layer (lane setup, not in-flight turn). The new channel timeout would be the in-flight equivalent.
agent-via-gateway uses a no-timeout default for the agent method invocation. The new channel timeout can be implemented as a wrapping cancellation, not as a change to the gateway timeout itself — keeps the change narrow and avoids regressing tool calls that legitimately run long.
For operators who already run a separate harness watchdog tailing the journal: this proposal does not change the gateway-restart contract. It only adds a clean per-turn ceiling.
Why this matters now
#86239 (closed cant_reproduce, but see my comment with a fresh repro on 2026.6.9 c645ec4) and #95288 (open, near-twin) both describe the same death-loop pattern. The root chain is:
Channel main-session turn hangs with no settlement.
This issue addresses (1). Fixing (1) cleanly would prevent the chain from starting in most cases — operators wouldn't need restart-recovery quarantine to bail them out, because there'd be no wedge to recover from in the first place.
Repro and references
Production occurrence: 2026-06-21 on 2026.6.9 c645ec4. Two concurrent Telegram lane main-session turns (DM + group topic) hung 559 s each.
Field-level evidence: cli_live:stream_progress ticks every ~30s through the entire hang; eventLoopDelayP99Ms climbed to ~68 s.
One timer at the channel dispatcher (one location per provider, or one shared point if there's a unified inbound shape).
One settlement listener that cancels the timer.
One config knob (channels.<provider>.turnTimeoutMs).
One log line with a stable reason for operator observability.
One user-visible message on timeout.
If this shape is acceptable in principle, happy to scope a PR. If the right answer is the broader Stage 2 (channel command-payload migration) instead, happy to discuss that separately — I'd suggest landing Stage 1 first because it ships in days, and Stage 2 is a multi-day refactor with conversational-shape concerns.
Sanity question
Is there an existing dispatcher-level timeout config that I missed? NO_GATEWAY_TIMEOUT_MS exists at the agent-via-gateway boundary and is intentionally a no-timeout (so tool calls can run long). I don't see a counterpart at the channel-inbound layer. If the right knob already exists, please point me at it and I'll close this.
Environment
2026.6.9(c645ec4)cliBackends.claude-cli(Anthropic Claude Code CLI)agents.list[main], modelclaude-cli/claude-opus-4-7systemd --user,Restart=always,StartLimitBurst=5,StartLimitIntervalSec=60Summary
Channel inbound dispatchers (Telegram, Slack, WhatsApp, etc.) currently invoke the
agentmethod throughagent-via-gatewaywith no effective wall-clock deadline on the turn. The cross-process call uses a no-timeout shape and waits for settlement (tool call OR text block) indefinitely. When a model deliberates for many seconds without emitting either, the runtime keeps reportingcli_live:stream_progressticks but no settlement event ever arrives, the gateway's event loop wedges, and downstream systems (cron isolated-setup-timeout watchdog, supervisor) start firing restarts that interact poorly withsessions.jsonrestart-recovery — see #86239 + #95288 for the production failure mode this creates.This issue requests a configurable per-lane wall-clock deadline at the channel-dispatcher boundary, modeled after the existing cron
isolated_agent_setup_timeoutpattern.Asymmetry with the cron path
The cron path already has equivalent protection:
isolated_agent_setup_timeoutwatchdog with a structured cancellation path.reason: "cron.isolated_agent_setup_timeout"and triggers a clean teardown (which today happens to ALSO trigger a SIGUSR1 restart, but that's a separate concern — see fix: restart gateway after isolated cron setup timeout #89055 + the restart-recovery family).Channel lanes (Telegram, Slack, WhatsApp, etc.) have no equivalent. The result is that a single no-settlement turn on a channel lane can:
2026.6.9 c645ec4).Proposed shape
Stage 1 — narrow change. Add an optional
dispatcher.channelTurnTimeoutMsconfig at the channel-dispatcher level (or per-channel underchannels.<provider>.turnTimeoutMs). When set:agentmethod.reason: "channel.turn_no_settlement_timeout", deliver a user-visible message to the channel ("Turn aborted: no settlement within {ms}ms; please retry"), and DO NOT mark the session entry asabortedLastRun=true.300000(5 minutes) — long enough to avoid clipping legitimate long turns, short enough to keep the failure window bounded.The "do NOT mark for restart-recovery" semantic is load-bearing — it's what distinguishes this from a SIGUSR1-driven restart, and it's what breaks the death-loop pattern in #86239 + #95288. The lane should recover by simply waiting for the next inbound message.
Stage 2 — broader change (separate issue if accepted). Consider migrating channel lanes to command-payload routing (analogous to what cron lanes already do internally). This addresses the underlying impedance mismatch between the dispatcher's request/response shape and the streaming agentTurn surface, but is materially larger work. Treating it as a follow-up to Stage 1 lets the cheap fix ship without coupling to the harder one.
Behavioral contract (Stage 1)
Coordination with existing timeout work
isolated_agent_setup_timeoutwatchdog is structurally similar but operates at a different layer (lane setup, not in-flight turn). The new channel timeout would be the in-flight equivalent.agent-via-gatewayuses a no-timeout default for theagentmethod invocation. The new channel timeout can be implemented as a wrapping cancellation, not as a change to the gateway timeout itself — keeps the change narrow and avoids regressing tool calls that legitimately run long.Why this matters now
#86239 (closed
cant_reproduce, but see my comment with a fresh repro on2026.6.9 c645ec4) and #95288 (open, near-twin) both describe the same death-loop pattern. The root chain is:StartLimitBurstis exhausted.Three independent layers each contribute:
This issue addresses (1). Fixing (1) cleanly would prevent the chain from starting in most cases — operators wouldn't need restart-recovery quarantine to bail them out, because there'd be no wedge to recover from in the first place.
Repro and references
2026.6.9 c645ec4. Two concurrent Telegram lane main-session turns (DM + group topic) hung 559 s each.cli_live:stream_progressticks every ~30s through the entire hang;eventLoopDelayP99Msclimbed to ~68 s.sessions.jsonedit (striprestartRecovery*fields),systemctl --user reset-failed openclaw-gateway, restart.sessions.jsonrecovery hardening), fix: restart gateway after isolated cron setup timeout #89055 (open PR; restart-on-isolated-setup-timeout, which would amplify the death-loop family if it lands before this issue is addressed).Scope sanity check (Stage 1 only)
The narrow change is:
channels.<provider>.turnTimeoutMs).reasonfor operator observability.If this shape is acceptable in principle, happy to scope a PR. If the right answer is the broader Stage 2 (channel command-payload migration) instead, happy to discuss that separately — I'd suggest landing Stage 1 first because it ships in days, and Stage 2 is a multi-day refactor with conversational-shape concerns.
Sanity question
Is there an existing dispatcher-level timeout config that I missed?
NO_GATEWAY_TIMEOUT_MSexists at theagent-via-gatewayboundary and is intentionally a no-timeout (so tool calls can run long). I don't see a counterpart at the channel-inbound layer. If the right knob already exists, please point me at it and I'll close this.