Skip to content

fix(telegram): back off, dead-letter, and tombstone spooled updates so poison messages cannot block or duplicate#98776

Merged
obviyus merged 4 commits into
mainfrom
codex/fix-telegram-spool
Jul 1, 2026
Merged

fix(telegram): back off, dead-letter, and tombstone spooled updates so poison messages cannot block or duplicate#98776
obviyus merged 4 commits into
mainfrom
codex/fix-telegram-spool

Conversation

@obviyus

@obviyus obviyus commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Closes #98774

What Problem This Solves

Fixes an issue where a Telegram spooled update that keeps failing would retry every ~500ms forever with no backoff and no attempts cap — and because the spool drain claims the lowest update id per conversation lane, the poison row permanently blocked every later message in that chat/topic. Two directly associated spool defects are fixed with it: a transient store read error during restart replay silently deleted the spooled message (the same loss window #98076 closed for media, still open for store reads), and successfully processed rows were deleted instead of tombstoned, so a crash-window Telegram refetch could re-dispatch an already-processed update — including re-executing callback-query side effects.

Why This Change Was Made

Three commits, in dependency order. First, the one-off session-init-conflict backoff (from 13ecca5408c) is generalized: all retryable spooled failures now back off exponentially from the queue's persisted attempts/lastAttemptAt (1s base, 3m cap) and dead-letter to the existing failed tombstones after 8 attempts, unblocking the lane. Second, TelegramPairingStoreReadError in the inbound handler now records a failed-retryable result so spooled replays are requeued instead of deleted; the user-facing "please try again" apology is suppressed for replays (they retry durably) and kept for live updates (acked without retry). Third, the success path uses the ingress queue's complete() tombstones (30d/1000 retention, pruned on the existing enqueue-side prune) so enqueue-time duplicate detection absorbs refetches; a redrain-ordering fix ensures lane guards clear before a requested immediate redrain.

User Impact

One malformed or persistently failing message can no longer silently freeze an entire Telegram chat or topic — it retries with backoff, then dead-letters with a warning log operators can find. Messages that hit transient store errors during a gateway restart are retried instead of lost, without apology spam. Buttons and commands no longer risk double-execution after a crash-window refetch.

Evidence

  • Regression tests added: dead-letter after max attempts unblocks the lane and later updates in the same lane process (polling-session.test.ts, 72 pass); transient pairing-store error on spooled replay requeues instead of deleting, with no repeated apologies (bot.create-telegram-bot.test.ts, 113 pass); completed-then-refetched update is detected as duplicate and not re-dispatched, at both queue and drain level (telegram-ingress-spool.test.ts, 14 pass).
  • Validation: pnpm tsgo:extensions pass; node scripts/run-oxlint.mjs on touched files pass; repo autoreview (branch mode vs origin/main) clean.
  • Combined-merge validation with the sibling PR (codex/fix-telegram-cache-transport): conflict-free octopus merge, pnpm tsgo:extensions pass, 403/403 tests across all six touched test files.
  • Constants chosen: max attempts 8, retry base 1s, retry cap 3m, completed tombstone TTL 30d / max 1000 (matching the existing failed tombstone bounds).

@openclaw-barnacle openclaw-barnacle Bot added channel: telegram Channel integration: telegram size: M maintainer Maintainer-authored PR labels Jul 1, 2026
@clawsweeper

clawsweeper Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper review: did not complete due to Codex infrastructure failure. Reviewed July 1, 2026, 6:27 PM ET / 22:27 UTC.

Summary
Review failed before ClawSweeper could summarize the requested change.

PR surface: Source +50, Tests +179. Total +229 across 6 files.

Reproducibility: unclear. The review failed before ClawSweeper could establish a reproduction path.

Review metrics: none identified.

Merge readiness
Not assessed.
Failure reason: timeout.

This is a ClawSweeper/Codex infrastructure failure, not a PR readiness or patch-quality verdict.
Keep any merge decision on the normal maintainer review path until ClawSweeper can complete a fresh review.

Risk before merge

  • [P1] No close action taken because the review did not complete.

Maintainer options:

  1. Decide the mitigation before merge
    Retry the Codex review after fixing the execution failure.
  2. Pause or close
    Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

  • Review did not complete, so no work-lane recommendation was made.
Review details

Best possible solution:

Retry the Codex review after fixing the execution failure.

Do we have a high-confidence way to reproduce the issue?

Unclear. The review failed before ClawSweeper could establish a reproduction path.

Is this the best way to solve the issue?

Unclear. Retry the review first so ClawSweeper can evaluate the actual issue and fix direction.

AGENTS.md: unclear because the file could not be read completely.

Codex review notes: model internal, reasoning high; reviewed against eb417fa206e6.

Evidence reviewed

PR surface:

Source +50, Tests +179. Total +229 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 3 64 14 +50
Tests 3 192 13 +179
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 6 256 27 +229

What I checked:

  • failure reason: timeout.
  • codex failure detail: Codex review failed for this PR: Codex process timed out after 1200000ms.
  • codex stderr: No stderr captured.
  • codex stdout: {"type":"thread.started","thread_id":"019f1fb9-27fb-7420-b531-9502cf3616d3"}.
  • process error code: ETIMEDOUT.

Likely related people:

  • unknown: Codex failed before it could trace repository history. (role: review did not complete; confidence: low)
How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@obviyus
obviyus force-pushed the codex/fix-telegram-spool branch from 9b36c21 to 9d7c633 Compare July 1, 2026 22:30
@obviyus
obviyus merged commit 55d31be into main Jul 1, 2026
87 checks passed
@obviyus
obviyus deleted the codex/fix-telegram-spool branch July 1, 2026 22:32
@obviyus

obviyus commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Landed via rebase onto main.

  • Scoped tests: node scripts/run-vitest.mjs extensions/telegram/src/telegram-ingress-spool.test.ts extensions/telegram/src/polling-session.test.ts extensions/telegram/src/bot.create-telegram-bot.test.ts (200 passed); node scripts/run-oxlint.mjs extensions/telegram/src/polling-session.ts extensions/telegram/src/polling-session.test.ts; git diff --check origin/main..HEAD before merge; .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main clean after fixing the accepted retry-limit finding.
  • Changelog: release-owned; not edited.
  • Land commit: 9d7c633
  • Merge commit: 55d31be

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: telegram Channel integration: telegram maintainer Maintainer-authored PR size: M

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Poisoned Telegram spooled update retries every 500ms forever and blocks all later messages in its lane

1 participant