fix(channels): recover stale ingress queue claims at gateway startup#90989
fix(channels): recover stale ingress queue claims at gateway startup#90989849261680 wants to merge 6 commits into
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 30, 2026, 3:55 PM ET / 19:55 UTC. Summary PR surface: Source +210, Tests +422. Total +632 across 12 files. Reproducibility: yes. at source level. Current main has only queue-local recovery and no gateway-wide startup sweep, and the linked issue plus PR proof show stale Review metrics: 3 noteworthy metrics.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Mantis proof suggestion Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Rebase onto current main, adapt the skip guards to the SQLite restart-handoff path, preserve the token-guarded startup sweep, and merge only after focused queue/startup/restart validation confirms no live inbound work is requeued. Do we have a high-confidence way to reproduce the issue? Yes, at source level. Current main has only queue-local recovery and no gateway-wide startup sweep, and the linked issue plus PR proof show stale Is this the best way to solve the issue? Yes for the direction, but not ready as-is. Startup recovery is the right layer for crash-left SQLite ingress claims before channel drains start, while the current branch must be adapted to main's SQLite restart-handoff path and revalidated. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 076da567f434. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +210, Tests +422. Total +632 across 12 files. View PR surface stats
Acceptance criteria:
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (1 earlier review cycle)
|
0d839c2 to
dc8f2c5
Compare
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
dc8f2c5 to
12cedda
Compare
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
Added live gateway recovery proof (addresses the remaining Booted a real This is the user-visible deadlock-clearing transition #90945 needs: the row left @clawsweeper re-review |
|
🦞👀 Command router queued. I will update this comment with the next step. |
|
This pull request has been automatically marked as stale due to inactivity. |
Summary
Problem: Issue channel_ingress_events (SQLite): stale claims never recovered after session crash — Telegram DM deadlocks #90945 — after a gateway crash,
channel_ingress_eventsrows left inclaimedstate are never recovered. Every new message into the affected session adds another stuck claim, silently deadlocking the channel (observed with Telegram DM). Per-queue recovery (recoverStaleClaims) only runs inside channel drain loops, which have not started yet right after a crash, so the claimed rows sit unrecovered until manual intervention.Fix: Add
recoverAllStaleChannelIngressClaims()iningress-queue.tsand run it fromprepareGatewayPluginBootstrap(beforestartChannels()) via a newrunStartupIngressClaimSweep, mirroring the existingrunStartupSessionMigrationpattern. The sweep selects allclaimedrows, keeps those older thanstaleMs(default 60s) whose owner process is gone, and batch-releases them topendingin a single write transaction. It is best-effort and non-blocking — failures are caught, logged, and startup continues.Claim-owner liveness:
claim_owneris either a bare PID (queue default) or a compound${pid}:${uuid}(Telegram spool —extensions/telegram/src/telegram-ingress-spool.ts). Liveness parses the PID prefix before:(matching the spool's ownprocessPidFromOwnerId), so a live sibling gateway's compound claim is never misread as dead.Race guard: Each release is guarded by the exact
claim_tokenobserved during selection. Between the read and the write a sibling gateway could release and re-claim the same event (new token); releasing onstatus = "claimed"alone would drop that fresh claim into duplicate processing. The sweep also returns the actual number of rows affected.What changed:
src/channels/message/ingress-queue.ts— new exportedrecoverAllStaleChannelIngressClaims()(+staleMsage guard,claim_tokenrelease guard) andisIngressClaimProcessAlive()(parses bare and compound PID owners).src/gateway/server-startup-ingress-sweep.ts— new startup sweep, follows theserver-startup-session-migration.tspattern (deps injection, try/catch with warn).src/gateway/server-startup-plugins.ts— wires the sweep intoprepareGatewayPluginBootstrapalongside existing startup tasks (+5 lines).src/channels/message/ingress-queue.test.ts— 8 new sweep tests (see Evidence).src/gateway/server-startup-ingress-sweep.test.ts— 3 new tests: success logging, error recovery, no-op path.What did NOT change (scope boundary): per-queue
recoverStaleClaims(),claimNext(), and Telegram spool recovery are unchanged — the startup sweep is additive. No config, plugin, or gateway-protocol surface changes.Reproduction
channel_ingress_events.kill -HUP <pid>).channel_ingress_eventsnow contains rows withstatus = "claimed"andclaim_owner= dead PID.pendingbefore channels start, messages resume.Real behavior proof
Behavior addressed: After a gateway crash, stale claimed rows in
channel_ingress_eventsare recovered at next startup; live local and sibling-gateway claims (bare PID, compoundpid:uuid, and rows re-claimed by a live worker) are preserved; channel message processing resumes. Fixes #90945.Real environment tested: macOS Darwin 24.6.0, Node 22.22. Two layers: (1) the SQLite claim lifecycle is exercised against real on-disk state databases (created via
openOpenClawStateDatabase, no mocks) by deterministic Vitest cases covering dead, live-bare-PID, live-compound-PID, re-claimed-by-live-worker, null, and recent-but-dead owners; (2) a realopenclaw gatewayprocess booted against an isolated state DB pre-seeded with a stuck claim, simulating recovery after a crashed session.Exact steps or command run after this patch:
Evidence after fix: Live gateway startup recovery (real
openclaw gatewayprocess, isolated state DB) —Unit layer:
Observed result after fix: On a real gateway boot, the stuck
claimedrow (dead PID, 2 min old) was released topendingbefore channels started — exactly the deadlock-clearing transition #90945 needs, so the event is deliverable again. Live owners (bare and compoundpid:uuid) and rows re-claimed by a live worker are preserved; recent claims within the staleMs window are age-protected; the sweep is best-effort and does not block startup on failure.What was not tested: An end-to-end Telegram transport round-trip (sending a real DM, crashing mid-turn, observing it resume in the Telegram client). The recovery mechanism — the gateway startup sweep transitioning the SQLite claim from
claimedtopending— is proven above with a real gateway process; a Telegram round-trip would only add transport-layer coverage on top of the proven claim-lifecycle fix.Risk / Mitigation
staleMswindow means only old claims are candidates, so a recycled PID from a recently-started process won't match. Same trade-off as the existing Telegram spool recovery.claim_tokenguard releases only the exact claim that was inspected, so a fresh claim is never dropped (regression test: "does not release a stale event that was re-claimed by a live worker").Change Type (select all)
Scope (select all touched areas)
Linked Issue/PR
Fixes #90945