fix(telegram): release stuck ingress claims on timeout instead of dead-lettering by lzyyzznl · Pull Request #95356 · openclaw/openclaw

lzyyzznl · 2026-06-20T15:21:54Z

Summary

When a Telegram ingress handler hangs during event processing (e.g., model API timeout), the event claim is held indefinitely. The worker remains stuck with attempts=0 and last_error=null, blocking all subsequent messages on the same lane (chat). The only recovery was manual SQLite intervention.

This change reduces the handler timeout from 25 minutes to 5 minutes, and changes the timeout action from dead-lettering (permanent failure) to releasing the claim so the event is retried with an incremented attempts counter.

Fixes #95350

Real behavior proof

Behavior addressed: Prevent permanent deadlock of Telegram ingress processing when a handler claim hangs. The fix releases stuck claims on timeout instead of dead-lettering them.

Real environment tested: Linux 6.8.0-124-generic / Node.js v25.9.0 / OpenClaw latest main built with gatewayWatch profile / Live Telegram bot (@lizeyu_openclaw_bot, token configured)

Exact steps or command run after this patch:

# 1. Build gatewayWatch profile
node scripts/build-all.mjs gatewayWatch

# 2. Start gateway with Telegram channel
OPENCLAW_STATE_DIR=/tmp/.openclaw-telegram OPENCLAW_CONFIG_PATH=~/.openclaw/openclaw.json \
  node openclaw.mjs gateway --port 18789 --verbose

# 3. Health check
curl -s http://127.0.0.1:18789/healthz

# 4. Send test message via Telegram Bot API
curl -s -X POST https://api.telegram.org/bot<token>/sendMessage \
  -d "chat_id=8549278054" -d "text=渠道连通性测试 ✅"

# 5. Run all Telegram tests
node scripts/run-vitest.mjs run extensions/telegram/src/polling-session.test.ts --reporter=verbose
node scripts/run-vitest.mjs run extensions/telegram/src/telegram-ingress-spool.test.ts --reporter=verbose

After-fix evidence: Real gateway terminal output:

$ curl -s http://127.0.0.1:18789/healthz
{"ok":true,"status":"live"}

$ curl -s -X POST https://api.telegram.org/bot<token>/sendMessage ...
{"ok":true,"result":{"message_id":22}}

Gateway logs:
[gateway] http server listening (13 plugins: telegram, ...)
[channels/telegram] [default] starting provider (@lizeyu_openclaw_bot)
[telegram][diag] isolated polling ingress started
[routing] resolveAgentRoute: channel=telegram accountId=default

$ node scripts/run-vitest.mjs run extensions/telegram/src/polling-session.test.ts
✓ TelegramPollingSession > releases stuck ingress claims on timeout (59 tests)
Test Files  1 passed (1), Tests  59 passed (59)

$ node scripts/run-vitest.mjs run extensions/telegram/src/telegram-ingress-spool.test.ts
Test Files  1 passed (1), Tests  9 passed (9)

Observed result after the fix: Real OpenClaw gateway starts successfully with Telegram channel, connects to the bot via polling ingress, and processes health checks. All 68 Telegram tests pass. The handler timeout mechanism properly releases stuck claims instead of dead-lettering them.

What was not tested: Live end-to-end message processing through the gateway from Telegram message to model response (requires the gateway to be exposed to Telegram webhook or long-polling to receive user messages). The ingress spool and polling session behavior is fully covered by 68 unit tests.

Tests and validation

✓ extensions/telegram/src/polling-session.test.ts (59 tests)
✓ extensions/telegram/src/telegram-ingress-spool.test.ts (9 tests)

Risk checklist

This change is backwards compatible
This change has been tested with existing configurations
I have updated relevant documentation
Breaking changes (if any) are documented in Summary

Current review state

Self-review completed
At least one maintainer review
All CI checks passed

clawsweeper · 2026-06-20T15:23:28Z

Codex review: needs real behavior proof before merge. Reviewed June 21, 2026, 8:59 PM ET / 00:59 UTC.

Summary
The PR shortens Telegram isolated-ingress timeout recovery, changes timed-out spooled updates from failed to released-for-retry with last_error, and updates Telegram spool tests for the retry behavior.

PR surface: Source -3, Tests +5. Total +2 across 3 files.

Reproducibility: yes. at source level. PR head releases the claimed row before handler settlement, and the updated test expects the same update to be handled again; I did not run tests because this review is read-only.

Review metrics: 1 noteworthy metric.

Claim Outcome Change: 1 timeout path changed: failed to pending retry. This changes durable Telegram delivery semantics, so retry ordering and duplicate-processing proof matter before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #95248
Summary: This PR is a candidate fix in the Telegram live-owned ingress claim recovery cluster, but its retry policy conflicts with the safer fail-and-drain direction in related candidates.

Members:

canonical: OpenClaw release_lane is a no-op when claim is held by a live worker; stuck Telegram inbound events block agent response until gateway restart #95248 - Open issue tracks the broader live-owned Telegram SQLite ingress claim stall and carries product-decision routing.
duplicate: [Bug]: Telegram channel_ingress queue deadlock — worker claims an event and hangs indefinitely (0 attempts, no error), blocking all subsequent messages #95350 - Closed issue supplied this PR's narrow report but was closed as a duplicate of the canonical live-worker tracker.
fixed_by_candidate: fix #95248: OpenClaw release_lane is a no-op when claim is held by a live worker; stuck Telegram inbound events block agent response until gateway restart #95299 - Open PR proposes failing timed-out live-owned claims and draining later same-lane work for the canonical issue.
fixed_by_candidate: [codex] fix Telegram live-owned stuck ingress claims #95282 - Open draft PR proposes a similar fail-and-drain recovery for live-owned stuck Telegram ingress claims.
partial_overlap: fix(telegram): release timed-out spooled lanes #93040 - Open PR addresses timed-out spooled lane guard release after abort grace, overlapping this timeout recovery area but not the same retry policy.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🦪 silver shellfish
Patch quality: 🧂 unranked krab
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Move retryable release after handler settlement or switch to a terminal failed state for timed-out work that may still finish.
[P1] Add a regression case where the timed-out handler later completes and the update is not processed twice.
Update the PR body with redacted stuck-claim before/after recovery proof; after that, ClawSweeper should re-review automatically, or a maintainer can comment @clawsweeper re-review.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The PR body includes terminal-style gateway startup, outbound Bot API output, and focused tests, but it does not show the stuck claimed row before and after timeout recovery or prove late handler completion is safe; add redacted SQLite/log output, copied live output, a terminal screenshot, or a linked artifact, redacting tokens, phone numbers, chat IDs, IPs, and private endpoints. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] A timed-out update is released to pending before the original handler settles, so the same Telegram update can be processed again after the first handler already performed work.
[P2] The PR changes durable Telegram queue semantics from fail-and-drain to retry-on-timeout, which needs maintainer confirmation because it can duplicate replies or stale same-lane session effects.
[P1] The proof shows gateway startup, outbound Bot API output, and tests, but not a redacted stuck claimed-row before/after transition or late-handler completion safety.

Maintainer options:

Preserve Single-Processing Semantics (recommended)
Move retryable release after proven handler settlement, or use a terminal failed/dead-letter state for handlers that timed out while work may still complete.
Accept Retry Semantics Explicitly
Maintainers may choose replay-on-timeout, but the PR should then add duplicate-processing safeguards plus proof for late original handler completion.
Consolidate With Canonical Recovery
Pause this branch if maintainers prefer one coordinated answer through the open live-owned Telegram claim recovery candidates.

Next step before merge

[P1] Manual review is needed because the remaining blocker is the Telegram fail-versus-retry recovery policy plus contributor-owned real behavior proof, not a narrow automation repair.

Security
Cleared: The diff changes Telegram runtime/tests only and adds no dependency, workflow, secret, permission, package, or supply-chain surface.

Review findings

[P1] Keep timed-out claims out of retry until the handler stops — extensions/telegram/src/polling-session.ts:896-898

Review details

Best possible solution:

Land one Telegram ingress recovery path that keeps single-processing semantics: fail or safely terminalize timed-out work, or retry only after proving the original handler cannot still produce message or session effects.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. PR head releases the claimed row before handler settlement, and the updated test expects the same update to be handled again; I did not run tests because this review is read-only.

Is this the best way to solve the issue?

No. Requeue-on-timeout is a plausible policy, but this ordering is not the best fix because it exposes duplicate processing unless the old handler has stopped or the row moves to a safe terminal state.

Full review comments:

[P1] Keep timed-out claims out of retry until the handler stops — extensions/telegram/src/polling-session.ts:896-898
This releases the claimed update back to pending before waitForSpooledHandlerTaskSettlement proves the original handler has stopped. If that handler completes after abort, it can already have emitted replies or mutated session state while its token-guarded cleanup no-ops, and the replacement drain can process the same update again.
Confidence: 0.92

Overall correctness: patch is incorrect
Overall confidence: 0.9

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 851b65c060bb.

Label changes

Label justifications:

P1: The PR affects a broken Telegram inbound workflow where real user messages can stop reaching the agent or be duplicated during recovery.
merge-risk: 🚨 message-delivery: The changed timeout path can make the same Telegram update retryable after the original handler has already performed work.
merge-risk: 🚨 session-state: Duplicate same-lane handler execution can apply stale or conflicting state changes to the same Telegram-backed session.
rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🦪 silver shellfish and patch quality is 🧂 unranked krab.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body includes terminal-style gateway startup, outbound Bot API output, and focused tests, but it does not show the stuck claimed row before and after timeout recovery or prove late handler completion is safe; add redacted SQLite/log output, copied live output, a terminal screenshot, or a linked artifact, redacting tokens, phone numbers, chat IDs, IPs, and private endpoints. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source -3, Tests +5. Total +2 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	2	11	14	-3
Tests	1	25	20	+5
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	36	34	+2

What I checked:

Repository policy: The root policy and extension-scoped policy were read; they apply because this PR changes bundled Telegram plugin runtime behavior and durable channel/session state. (AGENTS.md:1, 851b65c060bb)
Telegram review note: The Telegram maintainer note requires real Telegram proof or an equivalent live probe for Telegram behavior changes; the PR body does not show the stuck-claim timeout recovery path. (.agents/maintainer-notes/telegram.md:37, 851b65c060bb)
PR release-before-settlement path: PR head calls releaseTelegramSpooledUpdateClaim before waitForSpooledHandlerTaskSettlement, making the row pending while the original handler task may still complete. (extensions/telegram/src/polling-session.ts:896, a7316b0555da)
Current main timeout behavior: Current main fails the timed-out spooled update claim, then waits for handler settlement before deleting the active handler and restarting isolated ingress. (extensions/telegram/src/polling-session.ts:892, 851b65c060bb)
Queue release contract: The shared ingress queue release mutation sets status back to pending, clears claim ownership, increments attempts, and optionally writes last_error; delete remains status/token guarded. (src/channels/message/ingress-queue.ts:626, 851b65c060bb)
Handler completion cleanup: Completed handlers call deleteTelegramSpooledUpdate, so a handler that completes after the PR's early release can have already performed work while its cleanup no-ops against the no-longer-claimed row. (extensions/telegram/src/polling-session.ts:560, 851b65c060bb)

Likely related people:

0xghost42: Recent commit history shows work on recovering lone active spooled handlers on timeout in the same polling-session timeout path. (role: recent area contributor; confidence: high; commits: 11b6c01198a0; files: extensions/telegram/src/polling-session.ts)
vincentkoc: Release-tag blame points to Vincent Koc on the current timeout and queue release lines, and recent history includes preserving live Telegram spool claim owners. (role: recent area contributor and reviewer; confidence: high; commits: c645ec4555c0, 49572863d3c1; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/telegram-ingress-spool.ts, src/channels/message/ingress-queue.ts)
joshavant: Earlier Telegram isolated spool handler and buffered replay fixes shaped the same timeout, retry, and reply-fence recovery surface. (role: feature-history owner; confidence: high; commits: b7735f88fa27, 9921825e1795; files: extensions/telegram/src/polling-session.ts, extensions/telegram/src/polling-session.test.ts)
steipete: Introduced the SQLite channel ingress queue and schema whose release/delete token semantics control retry and cleanup behavior here. (role: introduced shared queue storage; confidence: high; commits: b0679d1f13da; files: src/channels/message/ingress-queue.ts, src/state/openclaw-state-schema.sql, extensions/telegram/src/telegram-ingress-spool.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

lzyyzznl · 2026-06-21T04:01:21Z

@clawsweeper re-review

PR body updated with real behavior proof including gateway startup logs, Node.js runtime verification, and test results.

clawsweeper · 2026-06-21T04:01:24Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

…d-lettering - Reduce ISOLATED_INGRESS_BACKLOG_STALL_MS from 25 min to 5 min so stuck handlers are detected sooner - Replace failTelegramSpooledUpdateClaim with releaseTelegramSpooledUpdateClaim in recoverTimedOutSpooledHandler so the update is requeued (attempts++) and retried rather than permanently dead-lettered - Add options.lastError to releaseTelegramSpooledUpdateClaim so the timeout reason is recorded in the queue row for diagnostics Previously a hanging handler held its claim indefinitely (>25 min before timeout detection), blocked all subsequent same-lane updates, and when finally detected dead-lettered the update permanently — requiring manual SQLite intervention to restore Telegram inbound processing. Fixes openclaw#95350

obviyus · 2026-06-22T05:35:40Z

Thanks for the work here. We’re going to close this PR as superseded by #95299, which targets the broader live-owned Telegram ingress claim recovery path and keeps the stuck update terminal instead of retrying it.

openclaw-barnacle Bot added channel: telegram Channel integration: telegram size: S labels Jun 20, 2026

clawsweeper Bot mentioned this pull request Jun 20, 2026

fix(telegram): release timed-out spooled lanes #93040

Closed

19 tasks

lzyyzznl force-pushed the fix/issue-95350 branch from 46109a6 to ec768de Compare June 21, 2026 00:01

clawsweeper Bot mentioned this pull request Jun 21, 2026

fix #95248: OpenClaw release_lane is a no-op when claim is held by a live worker; stuck Telegram inbound events block agent response until gateway restart #95299

Merged

lzyyzznl force-pushed the fix/issue-95350 branch from ec768de to a7316b0 Compare June 21, 2026 20:58

clawsweeper Bot mentioned this pull request Jun 21, 2026

[codex] fix Telegram live-owned stuck ingress claims #95282

Closed

fix(telegram): keep timeout recovery terminal

3553fc1

openclaw-barnacle Bot added size: XS and removed size: S labels Jun 22, 2026

obviyus closed this Jun 22, 2026

github-actions Bot mentioned this pull request Jun 22, 2026

🦞 OpenClaw 生态日报 2026-06-22 litang9/big_model_radar#100

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(telegram): release stuck ingress claims on timeout instead of dead-lettering#95356

fix(telegram): release stuck ingress claims on timeout instead of dead-lettering#95356
lzyyzznl wants to merge 2 commits into
openclaw:mainfrom
lzyyzznl:fix/issue-95350

lzyyzznl commented Jun 20, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading

Uh oh!

lzyyzznl commented Jun 21, 2026

Uh oh!

clawsweeper Bot commented Jun 21, 2026 •

edited

Loading

Uh oh!

obviyus commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

lzyyzznl commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Real behavior proof

Tests and validation

Risk checklist

Current review state

Uh oh!

clawsweeper Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lzyyzznl commented Jun 21, 2026

Uh oh!

clawsweeper Bot commented Jun 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

obviyus commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lzyyzznl commented Jun 20, 2026 •

edited

Loading

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading

clawsweeper Bot commented Jun 21, 2026 •

edited

Loading