Skip to content

Conversational agent lane wedges indefinitely: ingress events never reconcile (re-delivery storm on reconnect) + abort_embedded_run leaves owner lock unreleased (released=0) #97538

Description

@motteman

Summary

Two related defects in the gateway's inbound-processing and stalled-run recovery paths combine to wedge a conversational agent's reply lane indefinitely, with no organic self-heal. Observed on a live WhatsApp deployment. The stall/abort machinery predates v6.5 (it was active on v5.20), so this does not appear to be a 6.5 regression.

Environment: OpenClaw gateway v2026.6.5, macOS (launchd-managed), WhatsApp + Slack channels.


Bug 1 — abort_embedded_run does not release the session owner lock (released=0)

When a conversational embedded_run stalls (activeWorkKind=embedded_run, lastProgress=embedded_run:started, never advancing), the stuck-session recovery sweep fires:

[diagnostic] stalled session: state=idle age=<N>s queueDepth=<N>
  reason=active_work_without_progress classification=stalled_agent_run
  activeWorkKind=embedded_run lastProgress=embedded_run:started recovery=none
...
stuck session recovery -> action=abort_embedded_run aborted=true drained=true released=0

The abort reports aborted=true drained=true but released=0 — the in-memory session active-owner lock is not freed. As a result:

  • recovery_skipped_active_owner fires every ~30s (the sweep sees a still-"active" owner and skips recovery);
  • the next abort_embedded_run fires ~6 min later, again released=0;
  • a self-perpetuating loop results. The lane never recovers on its own (observed sustained >25 min).

Only a gateway restart clears the held lock.

Expected: aborting a stalled embedded_run should release its owner lock as part of the abort, so the lane recovers without a restart.


Bug 2 — Ingress events never reconcile to completed; re-deliver on every reconnect

Inbound events in channel_ingress_events are processed but never marked completed — they remain pending indefinitely. On every gateway reconnect/relink, the pending backlog re-delivers.

Observed: ~25 pending events accumulated over 7 days, the oldest already processed a week earlier (e.g. media attachments whose derived artifacts were built at the time). Only a handful of events ever reached completed. On each relink the gateway re-pushed the pending batch — confirmed via dedup (no new ingress rows created; the same events re-delivered).

When the re-delivered batch includes heavy multimodal payloads (PDFs/images), the resulting embedded_run stalls, triggering Bug 1.

Critical operational interaction — restart makes it worse: a restart forces a reconnect, so restarting the gateway re-triggers a full re-delivery storm of the pending backlog, which immediately re-wedges the lane via Bug 1. The intuitive remediation (restart) therefore perpetuates the failure unless the pending queue is cleared first.

Expected: ingress events should reconcile to completed after successful processing (ack), so they are not re-delivered on reconnect.


Combined user-visible impact

A conversational agent stops replying entirely on the affected channel. Inbound messages are received and broadcast but no reply is sent (run wedged at :started); subsequent legitimate messages queue behind the wedge and are starved. There is no supported recovery path:

  • No CLI/API verb exists to inspect / drain / ack the ingress queue (channels supports only add/remove/login/inspect).
  • The only remediation found was: (1) manually mark the pending channel_ingress_events rows completed via raw SQLite, then (2) restart — in that order, because restarting with a full queue re-triggers the storm.

Repro (observed)

  1. Send a heavy multimodal batch (several PDFs/images) to a conversational agent over WhatsApp; it processes.
  2. The ingress events remain pending (never reconciled).
  3. On any subsequent gateway reconnect/restart, the batch re-delivers; the embedded_run stalls at :started.
  4. The recovery sweep aborts the run with released=0; the lock never frees; the loop persists until the queue is cleared and the gateway restarted.

Suggested fixes / asks

  1. Release the session owner lock as part of abort_embedded_run (Bug 1).
  2. Reconcile ingress events to completed on successful processing so they do not re-deliver on reconnect (Bug 2).
  3. (Operability) A supported verb to inspect / drain / ack the ingress queue, so operators are not forced into raw SQLite writes.
  4. (Defensive) On reconnect, avoid re-delivering events already processed or older than a threshold.

The two bugs are separable but interact: Bug 2 supplies the trigger (re-delivery storm), Bug 1 makes the resulting stall permanent. Happy to split into two issues if preferred.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions