fix: drain inbound debounce buffer and followup queues before SIGUSR1 reload#46303
fix: drain inbound debounce buffer and followup queues before SIGUSR1 reload#46303joeykrug wants to merge 37 commits into
Conversation
Greptile SummaryThis PR fixes a class of silent message loss on SIGUSR1 restart by draining two in-memory buffers — per-channel inbound debounce timers and followup queues — before the server tears down. The solution is well-architected: a global debouncer registry ( Key observations from the review:
Confidence Score: 4/5
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95d3b9495e
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c43e101bd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be0cc5ed07
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
4d629a9 to
4715f9f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2d6896bb44
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
Reconcile the stale branch onto current main while retaining only the core restart drain integration.
Count overflow-summary backlog and cover the private admission, debounce, channel, followup, and run-loop drain contracts.
Synchronize the reviewed restart-drain patch with cutoff 262baed.
|
@clawsweeper re-review Rebuilt on main cutoff |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Current exact head is |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Current exact head is |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Exact head
Focused regressions cover the pre-created command enqueue, sustained ingress, the exact final-pass boundary, and a slow callback at its deadline. The PR body now uses the required headings and records exact-head validation. |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@openclaw-mantis telegram live: reproduce an inbound message buffered immediately before SIGUSR1/config reload and capture redacted proof that it is processed exactly once after the drain. |
|
@clawsweeper re-review Exact head |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Exact head
Exact local validation: 4 focused files / 164 tests, type-aware lint and formatting, core production/test tsgo, and plugin SDK declaration tsgo all pass. The PR body and exact tree identity are updated. |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
@clawsweeper re-review Exact head
The timer in-flight tracking/settlement regression remains in place. Exact validation is now 4 files / 166 tests, formatting/type-aware lint, core production/test tsgo, and plugin SDK declaration tsgo. PR body/tree identity are updated. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@clawsweeper re-review Exact head
Exact local validation: 4 focused files / 167 tests; formatting and type-aware lint; core production/test tsgo; and plugin SDK declaration tsgo all pass. The PR body and exact tree identity are updated. |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
|
@joeykrug thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward. Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive. Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it. |
Related: #51620
What Problem This Solves
Fixes an issue where users whose inbound message was accepted shortly before a SIGUSR1 or config reload could have buffered message content silently dropped before it reached the reply pipeline.
Why This Change Was Made
The gateway now closes ordinary work admission synchronously, flushes accepted inbound debounce work inside one core-private restart continuation, drains channel, active-turn, and followup work in order, then rechecks work created by followup delivery. Pre-created debounce callbacks reserve tracked required-continuation ownership, including parentless accepted work. Already-reserved required continuations may finish during the signal-pending gap while ordinary roots remain fenced, and removed-but-unsettled timer flushes remain visible until settlement. Buffered callbacks can also be explicitly rebound to the private restart admission context, while the global sweep is bounded by both the remaining restart deadline and a 100-pass/30-second ceiling so continued ingress cannot stall reload indefinitely.
Every non-forced phase now shares one absolute restart-drain deadline. Initial active-task and embedded-run waits receive only the budget remaining after inbound debounce and channel queue draining, so earlier phases cannot reset the configured maximum.
Forced restarts retain the existing skip-drain behavior. This does not persist messages sent while the gateway is offline.
User Impact
Messages accepted immediately before an in-process reload can complete through the normal command/reply path instead of being rejected by the restart fence or disappearing from in-memory buffers. A sustained stream of new transport events also cannot hold the gateway forever in the debounce-flush phase, and the configured restart deferral maximum is honored across the full drain sequence.
Evidence
Exact-head validation for
bb8f1b842e85c6745e7f8a1bfb8303d73ca6f7f6:src/auto-reply/inbound.test.ts: 65 passed.src/process/command-queue.test.ts: 42 passed.src/process/gateway-work-admission.test.ts: 16 passed.src/cli/gateway-cli/run-loop.test.ts: 44 passed.oxfmt --checkand type-awareoxlintpassed on all touched files.tsgochecks passed.37822eb909f4bd797d9bc6eb7d8488286f39637ais the exact locally tested tree; both newly uploaded blob SHAs matched the local Git objects.A live Signal/Telegram account is not paired in this environment. I have requested the repository's Mantis Telegram proof flow below so maintainers can attach redacted channel-to-gateway reload evidence to this exact head.
Implementation and Safety Notes
createInboundDebouncer()retains its publicenqueue/flushKey/cancelKeyreturn shape.ChannelRunQueueretains its publicenqueue/deactivateshape.enqueueInternal,allowDuringGatewayDrain, or command-queue admission bypass is exposed.undefinedand wait indefinitely.Synchronization
bb8f1b842e85c6745e7f8a1bfb8303d73ca6f7f638befdcf347331814f3453093ff3ef0c31cc33c0262baedcf7fa3d66dc35cce7d0253e0a31f0cc0e5ae09ec76ee9b851cb24b07acad291724c185868has first parentda4b1b9c15b6a7b965657fc753e0c25d8dd0e4faand second parent262baedcf7fa3d66dc35cce7d0253e0a31f0cc0e.