Skip to content

channel_ingress_events: claimed events never released when worker/session dies (no TTL or heartbeat) #90940

Description

@GDXbsv

Problem

When a session dies mid-processing (e.g. gateway crash, kill -HUP, unhandled rejection), the channel_ingress_events row stays in claimed state permanently. Workers that pick up new messages for the same session also fail and leave new events claimed. The queue deadlocks until manually unblocked.

Steps to reproduce

  1. Have a session actively running (e.g. Telegram direct session)
  2. Kill the gateway process mid-turn (e.g. via kill -HUP or unhandled rejection during parallel exec calls)
  3. Send any new message to that session
  4. Observe: new message gets claimed by a worker, worker fails, claim is never released
  5. All subsequent messages to that session are stuck in claimed → queue deadlocked

Root cause

channel_ingress_events has claim_token, claim_owner, claimed_at columns but no TTL enforcement or heartbeat mechanism:

  • No background job to expire stale claims
  • No liveness check on claim_owner before claiming
  • No heartbeat update from the session while it processes

Impact

  • Telegram (and likely other channels) sessions become silently unresponsive
  • Manual fix required: reset claimedpending via direct SQLite write + cron wake
  • Happens every time a session crashes mid-turn

Expected behavior

One of:

  1. Worker checks if claim_owner process is still alive before honouring a stale claim
  2. Sessions heartbeat claimed_at while processing; background reaper expires claims older than N×heartbeat_interval
  3. On gateway startup, all claimed events from the previous run are automatically released

Workaround

import sqlite3, time
conn = sqlite3.connect("/home/node/.openclaw/state/openclaw.sqlite")
cur = conn.cursor()
cutoff = int(time.time() * 1000) - 10 * 60 * 1000  # 10 min
cur.execute("""
  UPDATE channel_ingress_events
  SET status='pending', claim_token=NULL, claim_owner=NULL, claimed_at=NULL
  WHERE status='claimed' AND claimed_at < ?
""", (cutoff,))
conn.commit()
print(f"Reset {cur.rowcount} stuck claims")
conn.close()

Then wake the queue processor.

Environment

  • OpenClaw 2026.6.1
  • Channel: Telegram
  • Triggered by: parallel exec calls + kill -HUP on gateway PID

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions