-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Telegram ingress spooled update retry loop freezes gateway on non-retryable errors #92980
Copy link
Copy link
Open
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
Title: Telegram ingress spooled update retry loop freezes gateway on non-retryable errors
What happened
OpenClaw gateway became completely unresponsive for ~10 minutes. No messages were processed — the gateway appeared frozen to the user. Linux itself was fine (no OOM, no kernel panic, no CPU stall, disks OK).
Root cause
A Telegram spooled update (old/stale update
441315747, message6918) hit a non-recoverable error:ERR_MODULE_NOT_FOUNDfor a stale bundled JS file (directive-handling.impl-BTE_v-hO.jsimported byget-reply-BpFiu3Nn.js) that didn't exist anymore in the current npm package. The Telegram ingress spool handler treated this as retryable and entered an infinite loop:ERR_MODULE_NOT_FOUND(non-recoverable — the file will never appear)This repeated from 13:24:56 to 13:34:52 UTC at high frequency (multiple times per second), saturating logs and making the gateway useless until an external reboot cleared in-memory state.
A separate initial trigger was
Requested agent harness "codex" is not registered, which also should not cause an infinite retry cycle.Proposed fix
In the Telegram spooled update processing path (
monitor-pollingruntime):ERR_MODULE_NOT_FOUNDin spooled updates should be non-retryable — if the required module doesn't exist, it won't appear on retry. Send to dead-letter immediately.Missing/unknown agent harness errors should be non-retryable — same logic: if the harness isn't registered, it won't register itself on retry.
Poison retry limit for spooled updates — if any spooled update fails N times with the same error signature, dead-letter it regardless of error type. Suggested default: 3 consecutive same-signature failures.
Configurable via env var — e.g.,
OPENCLAW_TELEGRAM_SPOOLED_POISON_RETRY_LIMITso operators can adjust.Success should reset the failure counter — once an update is processed successfully, the counter for that update is cleared.
Environment
Impact
Additional context
A local workaround patch was applied to the installed
monitor-polling.runtime-*.jsbundle by treating the two error types above asnon-retryableand adding a poison counter. This confirms the fix works: after restart, the spool queue was empty and no retry loop occurred.Residual related issues found:
openclaw doctor)messagecapability from Telegram-routed agents