Skip to content

[Bug]: Poisoned Telegram spooled update retries every 500ms forever and blocks all later messages in its lane #98774

Description

@obviyus

Bug type

Behavior bug (incorrect output/state without crash)

Summary

A Telegram spooled update that keeps failing is retried forever with no backoff and no attempts cap, re-claiming every ~500ms; because the drain claims the lowest update id per lane, the poison row permanently blocks every later message in that chat/topic.

What happened

The ingress-spool drain (extensions/telegram/src/polling-session.ts at eb417fa206e) treats only two error classes as non-retryable (:156-177) and applies retry backoff only to "reply session initialization conflicted" errors (:179-195, added in 13ecca5408c for that one instance instead of the general mechanism). Every other failed-retryable outcome releases the claim for immediate retry (:812-853) on the ~500ms drain interval. The queue persists attempts/lastAttemptAt (src/channels/message/ingress-queue.ts:645) but nothing consumes them as a cap.

Any deterministic failure — e.g. Telegram getFile persistently returning 5xx for one file, or any exception thrown inside a debounce/media-group flush (bot-handlers.runtime.ts:638-653 settles participants failed-retryable for every error class) — hot-loops ~2×/sec indefinitely: constant claim/release write transactions, log spam, and a permanently blocked lane.

Two directly associated spool defects (same subsystem, fixed together):

  1. Swallowed post-claim errors lose replayed messages: the catch in handleInboundMessageLike (bot-handlers.runtime.ts:3403-3423) never records a failed-retryable result, so a transient store read error (TelegramPairingStoreReadError) during restart replay marks the update completed and deletes the spool row — the same loss window [Bug]: Telegram inbound documents silently dropped if received during gateway restart/upgrade window #98076 closed for media fetch, still open for store reads.
  2. Completed rows are deleted instead of complete()-tombstoned (telegram-ingress-spool.ts:287-289): the queue's tombstone + enqueue-time duplicate detection exists but is unused, so a crash-window refetch re-dispatches an already-processed update — dispatch dedupe covers only plain messages, so a replayed callback query re-executes its side effect.

Expected behavior

Retryable spooled failures back off exponentially using the persisted attempts/lastAttemptAt, dead-letter to the existing failed tombstones after a bounded number of attempts (unblocking the lane), transient store errors on replay requeue instead of silently deleting, and completed rows tombstone so refetched duplicates are not re-dispatched.

Evidence

Version

main @ eb417fa206e

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingclawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:data-lossCan lose, corrupt, or silently drop user/session/config data.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.maintainerMaintainer-authored PR

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions