-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[Bug]: delivery-recovery 0 recovered / N failed after gateway restart — recovery starts before channel transport ready #91212
Copy link
Copy link
Closed
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Summary
After a gateway restart,
delivery-recoveryreports "0 recovered, N failed" for pending feishu delivery entries. Messages are silently lost because recovery attempts delivery before channel transports (WebSocket) have reconnected, and the artificially-incremented retryCount persists across restarts.Reproduction
SIGUSR1orgateway restart)[delivery-recovery] Delivery recovery complete: 0 recovered, 6 failed, 0 skipped (max retries), 0 deferred (backoff)Root Cause Analysis
Three contributing factors in the code:
1. Recovery starts before transport is ready (
server-runtime-services.js)The recovery cycle runs immediately at startup, before channel adapters (e.g., feishu WebSocket) have connected. Delivery attempts during this window always fail.
2.
deferRemainingEntriesForBudgetinflates retryCount on unprocessed entries (delivery-queue.js)When the recovery time budget (default 60s) expires, unprocessed entries are passed to
failDelivery()which increments theirretryCount— even though no actual delivery was attempted. This is a clear bug: entries that were never tried count as failed.Pseudo-code:
3. retryCount persists across restarts with no reset (
delivery-queue.js)failDelivery()unconditionally incrementsretryCountfor all failure types, without distinguishing "API rejected the message" from "gateway just started, network not ready". After 5 restarts, entries with retryCount ≥ 5 are permanently discarded.Impact
Suggested Fix Direction
readystate before drainingdeferRemainingEntriesForBudgetshould NOT callfailDelivery(don't increment retryCount for un-attempted entries)Environment