-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[Bug]: Poisoned Telegram spooled update retries every 500ms forever and blocks all later messages in its lane #98774
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingSomething isn't workingclawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:data-lossCan lose, corrupt, or silently drop user/session/config data.Can lose, corrupt, or silently drop user/session/config data.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maintainerMaintainer-authored PRMaintainer-authored PR
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.bugSomething isn't workingSomething isn't workingclawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.ClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:data-lossCan lose, corrupt, or silently drop user/session/config data.Can lose, corrupt, or silently drop user/session/config data.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.maintainerMaintainer-authored PRMaintainer-authored PR
Type
Fields
Priority
None yet
Bug type
Behavior bug (incorrect output/state without crash)
Summary
A Telegram spooled update that keeps failing is retried forever with no backoff and no attempts cap, re-claiming every ~500ms; because the drain claims the lowest update id per lane, the poison row permanently blocks every later message in that chat/topic.
What happened
The ingress-spool drain (
extensions/telegram/src/polling-session.tsateb417fa206e) treats only two error classes as non-retryable (:156-177) and applies retry backoff only to "reply session initialization conflicted" errors (:179-195, added in13ecca5408cfor that one instance instead of the general mechanism). Every otherfailed-retryableoutcome releases the claim for immediate retry (:812-853) on the ~500ms drain interval. The queue persistsattempts/lastAttemptAt(src/channels/message/ingress-queue.ts:645) but nothing consumes them as a cap.Any deterministic failure — e.g. Telegram
getFilepersistently returning 5xx for one file, or any exception thrown inside a debounce/media-group flush (bot-handlers.runtime.ts:638-653settles participantsfailed-retryablefor every error class) — hot-loops ~2×/sec indefinitely: constant claim/release write transactions, log spam, and a permanently blocked lane.Two directly associated spool defects (same subsystem, fixed together):
handleInboundMessageLike(bot-handlers.runtime.ts:3403-3423) never records afailed-retryableresult, so a transient store read error (TelegramPairingStoreReadError) during restart replay marks the update completed and deletes the spool row — the same loss window [Bug]: Telegram inbound documents silently dropped if received during gateway restart/upgrade window #98076 closed for media fetch, still open for store reads.deleted instead ofcomplete()-tombstoned (telegram-ingress-spool.ts:287-289): the queue's tombstone + enqueue-time duplicate detection exists but is unused, so a crash-window refetch re-dispatches an already-processed update — dispatch dedupe covers only plain messages, so a replayed callback query re-executes its side effect.Expected behavior
Retryable spooled failures back off exponentially using the persisted
attempts/lastAttemptAt, dead-letter to the existingfailedtombstones after a bounded number of attempts (unblocking the lane), transient store errors on replay requeue instead of silently deleting, and completed rows tombstone so refetched duplicates are not re-dispatched.Evidence
extensions/telegram/src/polling-session.ts:156-195,812-853(retry policy),src/channels/message/ingress-queue.ts:645,682-752(unused attempts +complete())13ecca5408c(session-init backoff only),6528912e909(stalled claim recovery),14a6cd9ee67/f9d7afda100([Bug]: Telegram inbound documents silently dropped if received during gateway restart/upgrade window #98076 media-only durable retry)Version
main @
eb417fa206e