Skip to content

fix(telegram): recover stalled ingress spool claims#97118

Merged
vincentkoc merged 6 commits into
mainfrom
fix/telegram-spool-progress-lease
Jun 27, 2026
Merged

fix(telegram): recover stalled ingress spool claims#97118
vincentkoc merged 6 commits into
mainfrom
fix/telegram-spool-progress-lease

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What Problem This Solves

Telegram isolated polling can leave updates permanently claimed while the ingress spool drain is wedged. The claim refresh loop continued advancing claimed_at even when no drain cycle completed, and stale PID-reused claims could block later same-lane updates. That matches the reported channel_ingress_events state: pending Telegram updates accumulated while claimed rows stayed live.

This PR consolidates the queue-native claim fix from #96912 with the stalled-drain lease fix, so there is one canonical Telegram drain path instead of mixed claim implementations.

Why This Change Was Made

  • Use the shared ingress queue's atomic claimNext path with bounded candidate snapshots and same-lane blocking.
  • Preserve Telegram update ordering while allowing unrelated lanes to continue.
  • Recover stale claims when a process PID is reused, without treating a genuinely live foreign owner as stale.
  • Gate claim refresh on a shared successful-drain heartbeat, with fail-closed behavior after the grace window.
  • Stop renewing claims and supersede the Telegram reply fence when the drain heartbeat expires or refresh is lost.
  • Keep the queue selector behavior generic while recovery remains scoped to Telegram, the only current production consumer of this drain path.

The queue-native commits retain Dallin Romney's authorship in this branch history; the final commit removes the obsolete current-process fail-on-timeout path so PID-reused work is replayed instead of dead-lettered.

User Impact

This affects Telegram Bot API isolated polling ingress and its SQLite spool only. Stalled Telegram spool claims can become recoverable, PID-reused claims no longer strand a lane, and healthy drain cycles continue refreshing active claims. No direct behavior change is intended for Discord, Slack, Signal, WhatsApp's journal-only queue usage, or non-spooled Telegram webhook delivery.

Evidence

  • Focused regression proof: 99 tests passed across polling-session, telegram-ingress-spool, and shared ingress-queue suites.
  • Local formatting and oxlint checks passed for all six changed files.
  • Testbox-through-Crabbox: check:changed passed on tbx_01kw33f6bjngpjv0z5c1vtxzqv (Actions run 28270524234).
  • Autoreview: clean; no accepted/actionable findings.
  • Mantis native Telegram Desktop proof passed for candidate bd5d6a072cffdd2072814f148e499f15fbb6523e; proof job 83823345394, run 28291142592, artifact 7925262302, evidence comment: fix(telegram): recover stalled ingress spool claims #97118 (comment).
  • Crabbox regression artifact from the original stalled-drain reproduction remains recorded in the prior evidence: run_1a86842eb325.

Related Work

@openclaw-barnacle openclaw-barnacle Bot added channel: telegram Channel integration: telegram size: M maintainer Maintainer-authored PR labels Jun 26, 2026
@clawsweeper

clawsweeper Bot commented Jun 26, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 27, 2026, 11:59 AM ET / 15:59 UTC.

Summary
The PR changes Telegram isolated polling ingress to recover stalled SQLite spool claims by using queue-native claimNext, adding selector options, lane blocking, PID-reuse recovery, a 30-minute claim lease, and focused regression tests.

PR surface: Source +147, Tests +353. Total +500 across 6 files.

Reproducibility: yes. The source path is clear and the Mantis Telegram Desktop before/after proof shows the baseline stuck behind claimed rows and the candidate draining queued messages.

Review metrics: 2 noteworthy metrics.

  • SDK-visible claimNext selector options: 4 added. orderBy, scanLimit, candidateIds, and deriveLaneKey are exposed through the queue type used by plugin/runtime callers.
  • Telegram stale-claim recovery lease: 30-minute lease added. The lease changes the recovery boundary for durable Telegram ingress claims after stalls or PID reuse.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #90945
Summary: This PR is the current maintainer fix candidate for the Telegram SQLite ingress stale-claim deadlock cluster, superseding #96912 and building on #96962 while leaving webhook durability distinct.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster ✨ media proof bonus
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] The four new claimNext selector options are exported through the plugin SDK queue type, so they become public plugin API once released.
  • [P1] The 30-minute stale-claim lease and drain-health fail-closed behavior change when Telegram ingress work is replayed, protected, or left for a live owner.
  • [P1] This PR is scoped to Telegram isolated polling and does not resolve non-spooled webhook durability or a gateway-wide startup stale-claim sweep.

Maintainer options:

  1. Land with accepted lease and API contract (recommended)
    Maintainers can land this once they are comfortable accepting the SDK-visible queue selector shape and 30-minute Telegram recovery lease as the intended contract.
  2. Narrow selector exposure before release
    If the selector options should not become plugin API yet, keep the behavior internal to Telegram or document and test the SDK contract before merging.
  3. Keep broader recovery separate
    If maintainers want gateway-wide stale-claim startup recovery, leave that to the canonical follow-up rather than expanding this targeted Telegram drain fix.

Next step before merge

  • No ClawSweeper repair job is needed; maintainer review should decide whether to accept the SDK-visible selectors and recovery lease before merge.

Security
Cleared: No concrete security or supply-chain concern was found; the diff changes TypeScript Telegram/queue runtime and tests without workflows, dependencies, secrets, or package resolution changes.

Review details

Best possible solution:

Land after maintainer acceptance of the SDK-visible queue selectors and Telegram recovery lease, while keeping webhook durability and broader startup recovery tracked separately.

Do we have a high-confidence way to reproduce the issue?

Yes. The source path is clear and the Mantis Telegram Desktop before/after proof shows the baseline stuck behind claimed rows and the candidate draining queued messages.

Is this the best way to solve the issue?

Yes for the targeted Telegram isolated-polling drain fix; queue-native claiming removes the split claim path while heartbeat-gated refresh addresses wedged drain renewal without broadening into webhook or all-channel recovery.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f857e8d66eb3.

Label changes

Label justifications:

  • P1: The PR addresses Telegram ingress deadlocks that can leave accepted user messages blocked until manual or automatic recovery.
  • merge-risk: 🚨 compatibility: The diff expands SDK-visible queue selection options and changes durable recovery behavior for upgraded Telegram installations.
  • merge-risk: 🚨 message-delivery: The touched path decides whether queued Telegram updates drain, block, replay, or duplicate after stalled claims.
  • merge-risk: 🚨 session-state: Incorrect claim recovery or lane blocking could leave Telegram session queue state stale, replayed, or associated with the wrong active handler.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (recording): Mantis native Telegram Desktop before/after recording proof shows the real stalled-ingress recovery path and the final PR head only adds follow-up test coverage after that candidate proof.
  • proof: sufficient: Contributor real behavior proof is sufficient. Mantis native Telegram Desktop before/after recording proof shows the real stalled-ingress recovery path and the final PR head only adds follow-up test coverage after that candidate proof.
  • proof: 🎥 video: Contributor real behavior proof includes video or recording evidence. Mantis native Telegram Desktop before/after recording proof shows the real stalled-ingress recovery path and the final PR head only adds follow-up test coverage after that candidate proof.
  • mantis: telegram-visible-proof: Mantis should capture Telegram visible proof. The PR changes visible Telegram delivery after stalled ingress recovery, and the required Telegram Desktop proof has already been supplied.
Evidence reviewed

PR surface:

Source +147, Tests +353. Total +500 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 3 248 101 +147
Tests 3 369 16 +353
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 6 617 117 +500

Acceptance criteria:

  • [P1] Review maintainer acceptance of the four queue selector options as SDK-visible API.
  • [P1] Confirm the 30-minute lease and drain-heartbeat fail-closed policy are the intended Telegram recovery contract.
  • [P1] Keep non-spooled webhook durability and global startup sweep work tracked separately.

What I checked:

  • Live PR state: gh pr view 97118 --json ... showed the PR open, MERGEABLE/CLEAN, head ea6687229204c7e947993388e626aa9b1e9c0086, maintainer/proof labels, and 6 changed files. (ea6687229204)
  • Repository clean: The checkout was clean on current main before read-only inspection. (f857e8d66eb3)
  • Queue selector surface: The PR adds orderBy, scanLimit, candidateIds, and deriveLaneKey to the ChannelIngressQueue.claimNext options shape. (src/channels/message/ingress-queue.ts:136, ea6687229204)
  • Queue-native claim implementation: claimNext now filters candidate IDs, blocks lanes already claimed in the candidate snapshot, optionally derives missing lane keys, and claims the selected row transactionally. (src/channels/message/ingress-queue.ts:459, ea6687229204)
  • Telegram drain path: The Telegram drain loop now recovers stale claims with the 30-minute lease, snapshots candidates, claims through claimNextTelegramSpooledUpdate, and starts handlers only after queue-native claims. (extensions/telegram/src/polling-session.ts:898, ea6687229204)
  • Drain heartbeat policy: Claim refresh now fails closed when the shared drain health timestamp is too old, so a wedged drain stops renewing claims. (extensions/telegram/src/polling-session.ts:1266, ea6687229204)

Likely related people:

  • vincentkoc: Authored the final stalled-drain lease/heartbeat commits and is the live PR author handling the maintainer tradeoff. (role: recent area contributor and current PR owner; confidence: high; commits: 25a99f2332e3, ea6687229204; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/telegram-ingress-spool.ts)
  • RomneyDa: Authored the queue-native claim and PID-reuse work carried forward from the closed superseded PR fix(telegram): drain ingress with native queue claims #96912. (role: source queue-native claim contributor; confidence: medium; commits: 85739990bf4a, 21f2bf451f44, a8af3b975e13; files: src/channels/message/ingress-queue.ts, extensions/telegram/src/polling-session.ts, extensions/telegram/src/telegram-ingress-spool.ts)
  • joshavant: Merged Fix Telegram spooled claim refresh #96962 added the active spooled-claim refresh behavior that this PR modifies with stalled-drain health gating. (role: adjacent Telegram claim-refresh contributor; confidence: high; commits: db255b1154c1; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/telegram-ingress-spool.ts, src/channels/message/ingress-queue.ts)
  • steipete: Prior shared SQLite ingress queue storage history is directly adjacent to the queue API and persistence path this PR changes. (role: shared queue storage contributor; confidence: medium; commits: b0679d1f13da; files: src/channels/message/ingress-queue.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. mantis: telegram-visible-proof Mantis should capture Telegram visible proof. labels Jun 26, 2026
@clawsweeper
clawsweeper Bot temporarily deployed to qa-live-shared June 26, 2026 23:03 Inactive
@vincentkoc
vincentkoc force-pushed the fix/telegram-spool-progress-lease branch from 2068f08 to bd5d6a0 Compare June 26, 2026 23:28
@vincentkoc vincentkoc mentioned this pull request Jun 26, 2026
Closed
@clawsweeper clawsweeper Bot added merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 26, 2026
@vincentkoc

vincentkoc commented Jun 26, 2026

Copy link
Copy Markdown
Member Author

@openclaw-mantis telegram desktop proof: verify stalled Telegram ingress spool claims stop renewing, recover, and queued messages drain in order.

@clawsweeper clawsweeper Bot mentioned this pull request Jun 27, 2026
Closed
@vincentkoc

vincentkoc commented Jun 27, 2026

Copy link
Copy Markdown
Member Author

@openclaw-mantis telegram desktop proof: rerun stalled Telegram ingress spool recovery for this PR, but avoid the previous synthetic-proof trap. Use real Telegram Desktop user messages as the seed source so queued rows carry real message_id/thread metadata from the leased chat. If you seed channel_ingress_events manually, copy those real message_id/thread/chat/from fields into the payload instead of inventing fake message ids. The mock model response is static, so do not require per-row echoed markers like 971181/971182; prove candidate success by DB rows draining plus new bot reply message ids appearing after the seeded rows, and prove baseline by the stale claimed row still blocking the later same-lane row. Capture native Telegram Desktop before/after GIFs only if the visible delta is honest.

@openclaw-mantis

openclaw-mantis Bot commented Jun 27, 2026

Copy link
Copy Markdown
Contributor

Mantis Telegram Desktop Proof

Summary: Mantis captured native Telegram Desktop before/after GIF evidence with Convex-leased Telegram credentials.

Main screenshot This PR screenshot
Baseline native Telegram Desktop screenshot Candidate native Telegram Desktop screenshot
Main This PR
Baseline native Telegram Desktop proof GIF Candidate native Telegram Desktop proof GIF

Motion-trimmed clips:

Raw QA files: https://artifacts.openclaw.ai/mantis/telegram-desktop/pr-97118/run-28291142592-1/index.json

@vincentkoc vincentkoc self-assigned this Jun 27, 2026
@vincentkoc
vincentkoc marked this pull request as ready for review June 27, 2026 14:32
@clawsweeper clawsweeper Bot added proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: 🎥 video Contributor real behavior proof includes video or recording evidence. and removed proof: 📸 screenshot Contributor real behavior proof includes screenshot evidence. labels Jun 27, 2026
@vincentkoc

vincentkoc commented Jun 27, 2026

Copy link
Copy Markdown
Member Author

Ready to land.

Proof checked for head bd5d6a072cffdd2072814f148e499f15fbb6523e:

  • Focused regression tests passed across polling-session, telegram-ingress-spool, and shared ingress-queue suites: 99 tests.
  • Local format and oxlint passed for all six changed files.
  • Testbox-through-Crabbox check:changed passed on tbx_01kw33f6bjngpjv0z5c1vtxzqv / Actions run 28270524234.
  • Autoreview clean: no accepted/actionable findings.
  • Mantis native Telegram Desktop proof passed: run 28291142592, job 83823345394, artifact 7925262302, evidence comment fix(telegram): recover stalled ingress spool claims #97118 (comment).
  • Fresh ready-for-review GitHub checks passed: 64 successful, 36 skipped, 0 failing, 0 pending.
  • OPENCLAW_TESTBOX=1 scripts/pr prepare-run 97118 passed using hosted exact-head gate proof.

Explicit maintainer tradeoff accepted here: the fix uses a 30-minute stale-claim lease and exposes the generic claimNext selector API so Telegram can recover stalled same-lane spool claims without duplicating queue logic.

@vincentkoc
vincentkoc merged commit b8e3de1 into main Jun 27, 2026
110 of 116 checks passed
@vincentkoc

vincentkoc commented Jun 27, 2026

Copy link
Copy Markdown
Member Author

Merged via squash.

@github-actions github-actions Bot mentioned this pull request Jun 27, 2026
Open
This was referenced Jun 28, 2026
Closed
Closed
@vincentkoc vincentkoc mentioned this pull request Jun 28, 2026
Merged
This was referenced Jun 29, 2026
Closed
Closed
Merged
Closed
Closed
Merged
Closed
Open
@amittell amittell mentioned this pull request Jul 5, 2026
Closed
@steipete steipete mentioned this pull request Jul 13, 2026
Closed
@charliemeyer2000 charliemeyer2000 mentioned this pull request Jul 13, 2026
Closed
@clawsweeper clawsweeper Bot mentioned this pull request Jul 13, 2026
Closed
@pkosaai pkosaai mentioned this pull request Jul 15, 2026
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

channel: telegram Channel integration: telegram maintainer Maintainer-authored PR mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: 🎥 video Contributor real behavior proof includes video or recording evidence. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@vincentkoc @RomneyDa