You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Behavior bug (incorrect output/state without crash)
Beta release blocker
No
Summary
After some period of normal operation, Telegram's channel_ingress_events SQLite queue enters a deadlock: a worker claims one ingress event (status=claimed, attempts=0, last_error=null) and never completes or releases it, causing all subsequent events (32+ in this case) to remain stuck in status=pending forever. This silently kills all Telegram inbound processing—the bot's long poll continues receiving updates and enqueuing them, but no agent session is ever created. No crash, no error log, no restart required. The only fix is manually releasing the stuck claim via SQLite and restarting the gateway.
Steps to reproduce
We have not isolated a deterministic repro, but the following environment exhibited the bug after ~2 hours of normal operation:
Run OpenClaw 2026.6.8 in Docker (ghcr.io/openclaw/openclaw:2026.6.8) on WSL2 (Ubuntu 24.04, kernel 6.6.87.2-microsoft-standard-WSL2).
Configure Telegram bot with channels.telegram.accounts.oracle.botToken (account-level config, single account "oracle", enabled: true, dmPolicy: "allowlist", allowFrom: [8600735300]).
Proxy env: HTTP_PROXY=http://host.docker.internal:7890 (Clash Meta-compatible proxy on Windows host), NO_PROXY=localhost,127.0.0.1,.local,.openclaw,.openclaw.ai.
Send several DM messages to the bot over the course of ~2 hours.
At some point, messages stop being processed. The bot's long poll continues (getUpdates returns empty → consumed), events arrive in channel_ingress_events with status=pending, but they are never dispatched to the agent.
Diagnosis (found post-mortem via SQLite):
-- Check stuck claimsSELECT status, COUNT(*) FROM channel_ingress_events
WHERE channel_id ='telegram'GROUP BY status;
-- Result: claimed=1, pending=23 (oracle account)-- The stuck eventSELECT claim_owner, claimed_at, attempts, last_error FROM channel_ingress_events
WHERE channel_id ='telegram'AND status ='claimed';
-- claimed_at: 2026-06-20 13:13:22 UTC-- attempts: 0-- last_error: NULL
After restarting the gateway (SIGUSR1), a new event was claimed but immediately got stuck again with the same symptoms (claimed_at~14:52 UTC, still 0 attempts, no error). This suggests the hang occurs during message processing (possibly session creation or model invocation), and the worker does not have a timeout/retry mechanism.
Expected behavior
A Telegram ingress worker should:
Time out a stuck claim after a reasonable duration (e.g., 2–5 minutes if no model API call in flight, or configurable claimTimeoutMs).
Increment attempts on failure instead of staying at 0.
Set last_error to describe the failure reason.
Requeue or fail the event so the next pending event can proceed.
Actual behavior
The worker claims one event and hangs permanently:
attempts stays at 0 (no retry).
last_error stays NULL (no diagnostic signal).
All subsequent events remain status=pending indefinitely.
The only recovery is manual SQLite intervention (UPDATE ... SET status='pending', claim_token=NULL, claim_owner=NULL, claimed_at=NULL) followed by gateway restart.
Default route goes through an HTTP proxy on the Windows host (Clash Meta-compatible, port 7890). The proxy env vars (HTTP_PROXY, HTTPS_PROXY) are set on the container. NO_PROXY excludes localhost,127.0.0.1,.local,.openclaw,.openclaw.ai but does NOT exclude api.telegram.org. The proxy is stable and all curl-based tests through it succeeded.
The channel_ingress_events table had 1 claimed + 23 pending oracle events, plus separate pending/claimed for a "xin" account. Oldest pending was ~2h 24m old at time of discovery.
13:25:23–14:49:22 — 23 subsequent messages all stuck in pending
14:49:22 — most recent stuck message (updateId 514882262, msg#142: "测试")
Polling offset (from plugin_state_entries):
The telegram.update-offsets for oracle showed lastUpdateId: 514882240 while the ingress queue had events up to updateId 514882262. This gap suggests the polling progressed past the claimed event's update ID but the ingress processing never advanced.
Observation after workaround:
After manually releasing the claim and clearing the queue, plus resetting the update offset, a gateway restart restored normal operation—messages were processed and agent responses were delivered.
Impact and severity
Affected: Telegram bot users on this deployment (single-owner bot, direct messages)
Severity: High (complete loss of Telegram inbound processing until manual intervention)
Frequency: Occurred after ~2 hours of normal operation; exact trigger unknown
Consequence: User sent 23+ messages over 2+ hours with zero response, no error indication visible to the user or in gateway status (showed "running, connected, works"). The only way to detect this is via openclaw channels status --probe (which showed "audit failed" but no actionable detail) or direct SQLite inspection.
Additional information
The message tool also failed with "Telegram bot token missing" because the bot token is configured under channels.telegram.accounts.oracle.botToken but the message tool appears to only look for channels.telegram.botToken (top-level) or TELEGRAM_BOT_TOKEN env. This may be a separate issue but contributed to difficulty in diagnosing the problem (could not send outbound test messages to verify channel health).
The audit failed status in --probe output could be enhanced to surface queue depth or claim staleness as an actionable health signal.
Bug type
Behavior bug (incorrect output/state without crash)
Beta release blocker
No
Summary
After some period of normal operation, Telegram's
channel_ingress_eventsSQLite queue enters a deadlock: a worker claims one ingress event (status=claimed,attempts=0,last_error=null) and never completes or releases it, causing all subsequent events (32+ in this case) to remain stuck instatus=pendingforever. This silently kills all Telegram inbound processing—the bot's long poll continues receiving updates and enqueuing them, but no agent session is ever created. No crash, no error log, no restart required. The only fix is manually releasing the stuck claim via SQLite and restarting the gateway.Steps to reproduce
We have not isolated a deterministic repro, but the following environment exhibited the bug after ~2 hours of normal operation:
channels.telegram.accounts.oracle.botToken(account-level config, single account "oracle",enabled: true,dmPolicy: "allowlist",allowFrom: [8600735300]).HTTP_PROXY=http://host.docker.internal:7890(Clash Meta-compatible proxy on Windows host),NO_PROXY=localhost,127.0.0.1,.local,.openclaw,.openclaw.ai.getUpdatesreturns empty → consumed), events arrive inchannel_ingress_eventswithstatus=pending, but they are never dispatched to the agent.Diagnosis (found post-mortem via SQLite):
After restarting the gateway (SIGUSR1), a new event was claimed but immediately got stuck again with the same symptoms (claimed_at~14:52 UTC, still 0 attempts, no error). This suggests the hang occurs during message processing (possibly session creation or model invocation), and the worker does not have a timeout/retry mechanism.
Expected behavior
A Telegram ingress worker should:
claimTimeoutMs).attemptson failure instead of staying at 0.last_errorto describe the failure reason.Actual behavior
The worker claims one event and hangs permanently:
attemptsstays at 0 (no retry).last_errorstays NULL (no diagnostic signal).status=pendingindefinitely.UPDATE ... SET status='pending', claim_token=NULL, claim_owner=NULL, claimed_at=NULL) followed by gateway restart.OpenClaw version
2026.6.8 (docker image ghcr.io/openclaw/openclaw:2026.6.8)
Operating system
Windows 11 (host) → WSL2 (Docker container host) → Docker container (linux/amd64, kernel 6.6.87.2-microsoft-standard-WSL2)
Install method
docker (ghcr.io/openclaw/openclaw:2026.6.8)
Model
deepseek/deepseek-v4-flash (primary), moonshot/kimi-k2.6 (fallback)
Provider / routing chain
openclaw → HTTP_PROXY=http://host.docker.internal:7890 → api.deepseek.com
Additional provider/model setup details
Default route goes through an HTTP proxy on the Windows host (Clash Meta-compatible, port 7890). The proxy env vars (
HTTP_PROXY,HTTPS_PROXY) are set on the container.NO_PROXYexcludeslocalhost,127.0.0.1,.local,.openclaw,.openclaw.aibut does NOT excludeapi.telegram.org. The proxy is stable and all curl-based tests through it succeeded.Logs, screenshots, and evidence
Status probe output (while stuck):
SQLite diagnostics (from
/home/node/.openclaw/state/openclaw.sqlite):The
channel_ingress_eventstable had 1 claimed + 23 pending oracle events, plus separate pending/claimed for a "xin" account. Oldest pending was ~2h 24m old at time of discovery.Event timeline (UTC):
Polling offset (from
plugin_state_entries):The
telegram.update-offsetsfor oracle showedlastUpdateId: 514882240while the ingress queue had events up to updateId 514882262. This gap suggests the polling progressed past the claimed event's update ID but the ingress processing never advanced.Observation after workaround:
After manually releasing the claim and clearing the queue, plus resetting the update offset, a gateway restart restored normal operation—messages were processed and agent responses were delivered.
Impact and severity
openclaw channels status --probe(which showed "audit failed" but no actionable detail) or direct SQLite inspection.Additional information
messagetool also failed with "Telegram bot token missing" because the bot token is configured underchannels.telegram.accounts.oracle.botTokenbut themessagetool appears to only look forchannels.telegram.botToken(top-level) orTELEGRAM_BOT_TOKENenv. This may be a separate issue but contributed to difficulty in diagnosing the problem (could not send outbound test messages to verify channel health).audit failedstatus in--probeoutput could be enhanced to surface queue depth or claim staleness as an actionable health signal.