Skip to content

Telegram ingress spooled update retry loop freezes gateway on non-retryable errors #92980

Description

@rickrvo

Title: Telegram ingress spooled update retry loop freezes gateway on non-retryable errors

What happened

OpenClaw gateway became completely unresponsive for ~10 minutes. No messages were processed — the gateway appeared frozen to the user. Linux itself was fine (no OOM, no kernel panic, no CPU stall, disks OK).

Root cause

A Telegram spooled update (old/stale update 441315747, message 6918) hit a non-recoverable error: ERR_MODULE_NOT_FOUND for a stale bundled JS file (directive-handling.impl-BTE_v-hO.js imported by get-reply-BpFiu3Nn.js) that didn't exist anymore in the current npm package. The Telegram ingress spool handler treated this as retryable and entered an infinite loop:

  1. Process spooled update
  2. ERR_MODULE_NOT_FOUND (non-recoverable — the file will never appear)
  3. Release for retry
  4. Pick up again → repeat

This repeated from 13:24:56 to 13:34:52 UTC at high frequency (multiple times per second), saturating logs and making the gateway useless until an external reboot cleared in-memory state.

A separate initial trigger was Requested agent harness "codex" is not registered, which also should not cause an infinite retry cycle.

Proposed fix

In the Telegram spooled update processing path (monitor-polling runtime):

  1. ERR_MODULE_NOT_FOUND in spooled updates should be non-retryable — if the required module doesn't exist, it won't appear on retry. Send to dead-letter immediately.

  2. Missing/unknown agent harness errors should be non-retryable — same logic: if the harness isn't registered, it won't register itself on retry.

  3. Poison retry limit for spooled updates — if any spooled update fails N times with the same error signature, dead-letter it regardless of error type. Suggested default: 3 consecutive same-signature failures.

  4. Configurable via env var — e.g., OPENCLAW_TELEGRAM_SPOOLED_POISON_RETRY_LIMIT so operators can adjust.

  5. Success should reset the failure counter — once an update is processed successfully, the counter for that update is cleared.

Environment

  • OpenClaw version: latest npm as of 2026-06-14
  • Node.js: v24.15.0
  • OS: Linux 6.8.0-124-generic
  • Channel: Telegram (direct chat)
  • Gateway service: user-level systemd unit

Impact

  • Gateway completely unavailable for any message processing until external intervention (process killed / system rebooted)
  • No automatic recovery possible — errors were non-recoverable but classified as retryable
  • Log pollution from rapid retry cycle

Additional context

A local workaround patch was applied to the installed monitor-polling.runtime-*.js bundle by treating the two error types above as non-retryable and adding a poison counter. This confirms the fix works: after restart, the spool queue was empty and no retry loop occurred.

Residual related issues found:

  • Conflicting Codex plugin install metadata (reported by openclaw doctor)
  • Orphan/duplicate gateway service definitions
  • Tool policy stripping message capability from Telegram-routed agents

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions