Skip to content

[Bug]: delivery-recovery 0 recovered / N failed after gateway restart — recovery starts before channel transport ready #91212

Description

@Enominera

Summary

After a gateway restart, delivery-recovery reports "0 recovered, N failed" for pending feishu delivery entries. Messages are silently lost because recovery attempts delivery before channel transports (WebSocket) have reconnected, and the artificially-incremented retryCount persists across restarts.

Reproduction

  1. Have pending delivery entries in the queue
  2. Trigger gateway restart (SIGUSR1 or gateway restart)
  3. Observe restart log: [delivery-recovery] Delivery recovery complete: 0 recovered, 6 failed, 0 skipped (max retries), 0 deferred (backoff)
  4. Expected: pending entries should be delivered once channels reconnect. Actual: all fail during recovery.

Root Cause Analysis

Three contributing factors in the code:

1. Recovery starts before transport is ready (server-runtime-services.js)
The recovery cycle runs immediately at startup, before channel adapters (e.g., feishu WebSocket) have connected. Delivery attempts during this window always fail.

2. deferRemainingEntriesForBudget inflates retryCount on unprocessed entries (delivery-queue.js)
When the recovery time budget (default 60s) expires, unprocessed entries are passed to failDelivery() which increments their retryCount — even though no actual delivery was attempted. This is a clear bug: entries that were never tried count as failed.

Pseudo-code:

if (Date.now() >= deadline) {
    entries.map(e => failDelivery(e.id, "recovery time budget exceeded", stateDir))
    // ↑ inflates retryCount on entries never actually attempted
}

3. retryCount persists across restarts with no reset (delivery-queue.js)
failDelivery() unconditionally increments retryCount for all failure types, without distinguishing "API rejected the message" from "gateway just started, network not ready". After 5 restarts, entries with retryCount ≥ 5 are permanently discarded.

Impact

  • Channel messages silently lost during gateway restart
  • Worse on consecutive restarts: retryCount accumulates across restarts → permanent data loss
  • "0 skipped (max retries)" in the log is misleading — entries may have hit max retries DURING recovery, not before

Suggested Fix Direction

  1. Recovery should wait for channel transports to be in ready state before draining
  2. deferRemainingEntriesForBudget should NOT call failDelivery (don't increment retryCount for un-attempted entries)
  3. Consider resetting or capping retryCount for entries recovered after restart (the restart boundary is a "new session" context)

Environment

  • OpenClaw 2026.6.1
  • macOS (arm64)
  • Feishu channel (WebSocket mode)

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivity

Type

No type

Fields

Priority

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions