Skip to content

Stuck "claimed" rows in channel_ingress_events block all inbound delivery after restart (no stale-claim recovery) #96374

Description

@love1106

OpenClaw version: 2026.6.10
Channel: Telegram (long-polling / isolated ingress mode)
Storage: channel_ingress_events table in state/openclaw.sqlite

Summary

Inbound messages stop being delivered to the agent after the gateway is restarted while a worker has an in-flight claimed row in channel_ingress_events. The claim is never released, so the stuck rows block all subsequent pending events on the same queue indefinitely. The agent keeps polling and reports healthy, but never processes or replies to any incoming message.

This is a classic stale-claim / orphaned-lease bug: the queue has no visibility-timeout or reaper to recover claims owned by a dead worker.

Impact

  • Agent appears fully healthy: /ready and /health return OK, Telegram polling logs show isolated polling ingress started, no errors.
  • Telegram side shows pending_update_count: 0 (updates are fetched), but the agent never emits an Inbound message log and never replies.
  • Every message sent after the stuck claim is silently queued as pending and never dispatched.
  • Symptom is indistinguishable from "channel broken" at the surface; openclaw doctor --lint does not detect it (config is valid; the failure is purely at the runtime queue layer).

Root cause

channel_ingress_events lifecycle: pendingclaimed (worker takes it) → completed (purged) / failed.

When a worker claims an event (status='claimed', sets claim_token/claim_owner/claimed_at) and the gateway process is then killed mid-flight (SIGKILL, container restart, resource resize), the row stays claimed forever. On restart, a new worker does not reclaim these orphaned rows, and they sit ahead of newer pending rows in the queue ordering, blocking delivery.

The table schema already carries everything needed to detect this (claimed_at, attempts, last_attempt_at) — there is simply no logic that releases claims older than a timeout.

Reproduction (observed in production)

  1. Agent (Telegram, isolated polling ingress) is actively processing.
  2. Container is restarted abruptly (in our case: an orchestrator stop → resize → start sequence, plus a few manual restarts during debugging) while several events were claimed.
  3. After restart: polling resumes cleanly, no conflict, healthy — but agent never processes any inbound message.

Observed DB state (40 stuck rows for one telegram queue):

status='claimed'  count=31   (all claim_owner='7:239da3b5-...', claimed_at≈resize time, attempts=0, last_error=NULL)
status='pending'  count=9    (every message the user sent afterwards, never dispatched)

All 31 claimed rows were owned by a single dead worker id; none ever advanced to completed. Telegram offset on the API side had advanced past them (updates fetched), but the agent never saw them because they were stuck in the local DB queue.

Workaround used

DELETE FROM channel_ingress_events
WHERE channel_id = 'telegram'
  AND status IN ('claimed', 'pending');

After clearing the stuck rows and restarting, the new worker claimed → dispatched → completed normally (Inbound message logged, outbound send ok, queue back to empty).

Suggested fix

Add stale-claim recovery to the channel_ingress_events consumer, e.g. any of:

  1. Visibility timeout / claim TTL — on claim acquisition, reset rows where status='claimed' AND claimed_at < now() - CLAIM_TTL_MS back to pending (clear claim_token/claim_owner), bump attempts.
  2. Startup reaper — on gateway boot, release all claimed rows whose claim_owner doesn't match a live worker (workers are short-lived per-process, so any pre-restart claim is by definition orphaned).
  3. Worker heartbeat / lease renewal — require periodic claim renewal; expired leases auto-release.

Option (2) alone (release-on-boot) would have fully prevented this case, since a restart guarantees the previous claim owner is dead.

Optionally: have openclaw doctor surface a warning when channel_ingress_events has claimed rows with claimed_at older than N minutes — this would have made diagnosis immediate.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions