Skip to content

[Bug]: Telegram channel_ingress queue deadlock — worker claims an event and hangs indefinitely (0 attempts, no error), blocking all subsequent messages #95350

Description

@WangXuexin24

Bug type

Behavior bug (incorrect output/state without crash)

Beta release blocker

No

Summary

After some period of normal operation, Telegram's channel_ingress_events SQLite queue enters a deadlock: a worker claims one ingress event (status=claimed, attempts=0, last_error=null) and never completes or releases it, causing all subsequent events (32+ in this case) to remain stuck in status=pending forever. This silently kills all Telegram inbound processing—the bot's long poll continues receiving updates and enqueuing them, but no agent session is ever created. No crash, no error log, no restart required. The only fix is manually releasing the stuck claim via SQLite and restarting the gateway.

Steps to reproduce

We have not isolated a deterministic repro, but the following environment exhibited the bug after ~2 hours of normal operation:

  1. Run OpenClaw 2026.6.8 in Docker (ghcr.io/openclaw/openclaw:2026.6.8) on WSL2 (Ubuntu 24.04, kernel 6.6.87.2-microsoft-standard-WSL2).
  2. Configure Telegram bot with channels.telegram.accounts.oracle.botToken (account-level config, single account "oracle", enabled: true, dmPolicy: "allowlist", allowFrom: [8600735300]).
  3. Proxy env: HTTP_PROXY=http://host.docker.internal:7890 (Clash Meta-compatible proxy on Windows host), NO_PROXY=localhost,127.0.0.1,.local,.openclaw,.openclaw.ai.
  4. Send several DM messages to the bot over the course of ~2 hours.
  5. At some point, messages stop being processed. The bot's long poll continues (getUpdates returns empty → consumed), events arrive in channel_ingress_events with status=pending, but they are never dispatched to the agent.

Diagnosis (found post-mortem via SQLite):

-- Check stuck claims
SELECT status, COUNT(*) FROM channel_ingress_events
WHERE channel_id = 'telegram' GROUP BY status;
-- Result: claimed=1, pending=23 (oracle account)

-- The stuck event
SELECT claim_owner, claimed_at, attempts, last_error FROM channel_ingress_events
WHERE channel_id = 'telegram' AND status = 'claimed';

-- claimed_at: 2026-06-20 13:13:22 UTC
-- attempts: 0
-- last_error: NULL

After restarting the gateway (SIGUSR1), a new event was claimed but immediately got stuck again with the same symptoms (claimed_at~14:52 UTC, still 0 attempts, no error). This suggests the hang occurs during message processing (possibly session creation or model invocation), and the worker does not have a timeout/retry mechanism.

Expected behavior

A Telegram ingress worker should:

  • Time out a stuck claim after a reasonable duration (e.g., 2–5 minutes if no model API call in flight, or configurable claimTimeoutMs).
  • Increment attempts on failure instead of staying at 0.
  • Set last_error to describe the failure reason.
  • Requeue or fail the event so the next pending event can proceed.

Actual behavior

The worker claims one event and hangs permanently:

  • attempts stays at 0 (no retry).
  • last_error stays NULL (no diagnostic signal).
  • All subsequent events remain status=pending indefinitely.
  • The only recovery is manual SQLite intervention (UPDATE ... SET status='pending', claim_token=NULL, claim_owner=NULL, claimed_at=NULL) followed by gateway restart.

OpenClaw version

2026.6.8 (docker image ghcr.io/openclaw/openclaw:2026.6.8)

Operating system

Windows 11 (host) → WSL2 (Docker container host) → Docker container (linux/amd64, kernel 6.6.87.2-microsoft-standard-WSL2)

Install method

docker (ghcr.io/openclaw/openclaw:2026.6.8)

Model

deepseek/deepseek-v4-flash (primary), moonshot/kimi-k2.6 (fallback)

Provider / routing chain

openclaw → HTTP_PROXY=http://host.docker.internal:7890 → api.deepseek.com

Additional provider/model setup details

Default route goes through an HTTP proxy on the Windows host (Clash Meta-compatible, port 7890). The proxy env vars (HTTP_PROXY, HTTPS_PROXY) are set on the container. NO_PROXY excludes localhost,127.0.0.1,.local,.openclaw,.openclaw.ai but does NOT exclude api.telegram.org. The proxy is stable and all curl-based tests through it succeeded.

Logs, screenshots, and evidence

Status probe output (while stuck):

- Telegram oracle: enabled, configured, running, connected, transport:just now, mode:polling, bot:@oracle_Clio_bot, token:config, groups:unmentioned, works, audit failed

SQLite diagnostics (from /home/node/.openclaw/state/openclaw.sqlite):

The channel_ingress_events table had 1 claimed + 23 pending oracle events, plus separate pending/claimed for a "xin" account. Oldest pending was ~2h 24m old at time of discovery.

Event timeline (UTC):

  • 12:27:13 — oldest pending event received
  • 13:13:22 — the stuck claim was made (message Chat UI: strip thinking blocks from assistant text #109: "镜晞有问题。神谕没有问题。")
  • 13:25:23–14:49:22 — 23 subsequent messages all stuck in pending
  • 14:49:22 — most recent stuck message (updateId 514882262, msg#142: "测试")

Polling offset (from plugin_state_entries):
The telegram.update-offsets for oracle showed lastUpdateId: 514882240 while the ingress queue had events up to updateId 514882262. This gap suggests the polling progressed past the claimed event's update ID but the ingress processing never advanced.

Observation after workaround:
After manually releasing the claim and clearing the queue, plus resetting the update offset, a gateway restart restored normal operation—messages were processed and agent responses were delivered.

Impact and severity

  • Affected: Telegram bot users on this deployment (single-owner bot, direct messages)
  • Severity: High (complete loss of Telegram inbound processing until manual intervention)
  • Frequency: Occurred after ~2 hours of normal operation; exact trigger unknown
  • Consequence: User sent 23+ messages over 2+ hours with zero response, no error indication visible to the user or in gateway status (showed "running, connected, works"). The only way to detect this is via openclaw channels status --probe (which showed "audit failed" but no actionable detail) or direct SQLite inspection.

Additional information

  • The message tool also failed with "Telegram bot token missing" because the bot token is configured under channels.telegram.accounts.oracle.botToken but the message tool appears to only look for channels.telegram.botToken (top-level) or TELEGRAM_BOT_TOKEN env. This may be a separate issue but contributed to difficulty in diagnosing the problem (could not send outbound test messages to verify channel health).
  • The audit failed status in --probe output could be enhanced to surface queue depth or claim staleness as an actionable health signal.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions