Skip to content

fix: drain inbound debounce buffer and followup queues before SIGUSR1 reload#46303

Open
joeykrug wants to merge 37 commits into
openclaw:mainfrom
joeykrug:fix/drain-queue-before-sigusr1
Open

fix: drain inbound debounce buffer and followup queues before SIGUSR1 reload#46303
joeykrug wants to merge 37 commits into
openclaw:mainfrom
joeykrug:fix/drain-queue-before-sigusr1

Conversation

@joeykrug

@joeykrug joeykrug commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Related: #51620

What Problem This Solves

Fixes an issue where users whose inbound message was accepted shortly before a SIGUSR1 or config reload could have buffered message content silently dropped before it reached the reply pipeline.

Why This Change Was Made

The gateway now closes ordinary work admission synchronously, flushes accepted inbound debounce work inside one core-private restart continuation, drains channel, active-turn, and followup work in order, then rechecks work created by followup delivery. Pre-created debounce callbacks reserve tracked required-continuation ownership, including parentless accepted work. Already-reserved required continuations may finish during the signal-pending gap while ordinary roots remain fenced, and removed-but-unsettled timer flushes remain visible until settlement. Buffered callbacks can also be explicitly rebound to the private restart admission context, while the global sweep is bounded by both the remaining restart deadline and a 100-pass/30-second ceiling so continued ingress cannot stall reload indefinitely.

Every non-forced phase now shares one absolute restart-drain deadline. Initial active-task and embedded-run waits receive only the budget remaining after inbound debounce and channel queue draining, so earlier phases cannot reset the configured maximum.

Forced restarts retain the existing skip-drain behavior. This does not persist messages sent while the gateway is offline.

User Impact

Messages accepted immediately before an in-process reload can complete through the normal command/reply path instead of being rejected by the restart fence or disappearing from in-memory buffers. A sustained stream of new transport events also cannot hold the gateway forever in the debounce-flush phase, and the configured restart deferral maximum is honored across the full drain sequence.

Evidence

Exact-head validation for bb8f1b842e85c6745e7f8a1bfb8303d73ca6f7f6:

  • Focused behavior tests: 4 files, 167 tests passed.
    • src/auto-reply/inbound.test.ts: 65 passed.
    • src/process/command-queue.test.ts: 42 passed.
    • src/process/gateway-work-admission.test.ts: 16 passed.
    • src/cli/gateway-cli/run-loop.test.ts: 44 passed.
  • New P1 regressions: a buffer created inside an admitted inbound root survives that root's release; timer-started required work can finish through the restart-signal-pending fence while ordinary roots remain blocked; a timer-started callback remains visible and keeps the restart sweep pending until its real command enqueue settles; parentless accepted continuations retain required ownership; continuous ingress stops at the pass cap; the exact final-pass boundary reports drained correctly; a slow callback returns at the supplied deadline; and a deterministic 100 ms restart budget reduced to 40 ms by earlier debounce/channel work is passed as 40 ms to both initial active-work waits and the final close drain.
  • Formatting/lint: oxfmt --check and type-aware oxlint passed on all touched files.
  • Type safety: core production, core test, and plugin SDK declaration tsgo checks passed.
  • Tree identity: GitHub tree 37822eb909f4bd797d9bc6eb7d8488286f39637a is the exact locally tested tree; both newly uploaded blob SHAs matched the local Git objects.
  • CI: GitHub Actions is running on the exact head above.

A live Signal/Telegram account is not paired in this environment. I have requested the repository's Mantis Telegram proof flow below so maintainers can attach redacted channel-to-gateway reload evidence to this exact head.

Implementation and Safety Notes

  • createInboundDebouncer() retains its public enqueue / flushKey / cancelKey return shape.
  • ChannelRunQueue retains its public enqueue / deactivate shape.
  • No public enqueueInternal, allowDuringGatewayDrain, or command-queue admission bypass is exposed.
  • The debounce registry is weak, avoiding channel teardown requirements.
  • Overflow-summary backlog is included in followup pending counts.
  • Bounded restarts use one absolute deadline; unbounded restarts continue passing undefined and wait indefinitely.
  • The branch remains a focused 13-file change.

Synchronization

  • PR head: bb8f1b842e85c6745e7f8a1bfb8303d73ca6f7f6
  • Immediate parent: 38befdcf347331814f3453093ff3ef0c31cc33c0
  • Synchronized main cutoff: 262baedcf7fa3d66dc35cce7d0253e0a31f0cc0e
  • Cutoff merge topology: synchronization commit 5ae09ec76ee9b851cb24b07acad291724c185868 has first parent da4b1b9c15b6a7b965657fc753e0c25d8dd0e4fa and second parent 262baedcf7fa3d66dc35cce7d0253e0a31f0cc0e.
  • Diff: 13 files, +1167/-155.

@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes size: M labels Mar 14, 2026
@greptile-apps

greptile-apps Bot commented Mar 14, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes a class of silent message loss on SIGUSR1 restart by draining two in-memory buffers — per-channel inbound debounce timers and followup queues — before the server tears down. The solution is well-architected: a global debouncer registry (Symbol.for("openclaw.inboundDebouncers")) allows all debouncers to be collectively flushed, a new waitForFollowupQueueDrain poller gives newly-flushed items time to process, and the SIGUSR1 restart sequence is extended with a carefully-ordered set of drain steps.

Key observations from the review:

  • All issues raised in previous review rounds have been addressed: the flush count now reflects actual buffers delivered (not debouncers registered), getPendingCount correctly sums per-queue draining sentinels, the flushAllInternal while-loop has a deadline guard, individual flush errors are isolated with try/catch, flushBuffer correctly tracks delivered vs. error-swallowed, the force-exit timer is re-armed conditionally after the debounce flush, and unregister is exposed and wired up across Discord, Slack, Mattermost, WhatsApp, and BlueBubbles monitors.
  • Two test files not in the PR diffextensions/slack/src/monitor/message-handler.app-mention-race.test.ts and extensions/bluebubbles/src/monitor.test.ts — contain stale flushKey mocks returning Promise<void> and omit flushAll / unregister. These won't cause runtime failures (the missing methods are never called in those tests) but may surface as TypeScript type errors under strict checking.

Confidence Score: 4/5

  • Safe to merge; the fix is well-tested and all previous review concerns have been addressed, with only a minor stale-mock issue in two unmodified test files.
  • The core implementation is correct and the restart drain sequence is properly ordered and guarded. All previously-raised logic bugs are fixed. The only remaining concern is that two test files outside the PR diff have incomplete mocks for the new debouncer interface, which could cause TypeScript errors but will not affect production behavior or the 118 tests the author references.
  • extensions/slack/src/monitor/message-handler.app-mention-race.test.ts and extensions/bluebubbles/src/monitor.test.ts — stale debouncer mock types that were not updated alongside the interface changes.

Comments Outside Diff (1)

  1. extensions/slack/src/monitor/message-handler.app-mention-race.test.ts, line 29-30 (link)

    Stale mock missing new debouncer fields

    This test's mock of createChannelInboundDebouncer was not updated alongside inbound-debounce.ts. The return object is missing flushAll and unregister, and flushKey still returns Promise<void> instead of the new Promise<boolean>.

    While deactivate (which calls unregister) is never invoked in this test so it won't cause a runtime failure, the stale types may produce a TypeScript error if the mock is checked against the createChannelInboundDebouncer return type under strict mode. The same issue exists at extensions/bluebubbles/src/monitor.test.ts:1104 where flushKey is mocked with vi.fn(async (key: string) => { await flush(key); }) — also returning Promise<void> and omitting flushAll and unregister.

Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/slack/src/monitor/message-handler.app-mention-race.test.ts
Line: 29-30

Comment:
**Stale mock missing new debouncer fields**

This test's mock of `createChannelInboundDebouncer` was not updated alongside `inbound-debounce.ts`. The return object is missing `flushAll` and `unregister`, and `flushKey` still returns `Promise<void>` instead of the new `Promise<boolean>`.

While `deactivate` (which calls `unregister`) is never invoked in this test so it won't cause a runtime failure, the stale types may produce a TypeScript error if the mock is checked against the `createChannelInboundDebouncer` return type under strict mode. The same issue exists at `extensions/bluebubbles/src/monitor.test.ts:1104` where `flushKey` is mocked with `vi.fn(async (key: string) => { await flush(key); })` — also returning `Promise<void>` and omitting `flushAll` and `unregister`.

How can I resolve this? If you propose a fix, please make it concise.

Last reviewed commit: 5092f7d

Comment thread src/auto-reply/inbound-debounce.ts Outdated
Comment thread src/auto-reply/reply/queue/drain-all.ts Outdated
@openclaw-barnacle openclaw-barnacle Bot added channel: feishu Channel integration: feishu size: L and removed size: M labels Mar 14, 2026
@joeykrug
joeykrug marked this pull request as draft March 14, 2026 19:21
@joeykrug
joeykrug marked this pull request as ready for review March 14, 2026 19:21
Comment thread src/cli/gateway-cli/run-loop.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95d3b9495e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/auto-reply/inbound-debounce.ts Outdated
Comment thread src/cli/gateway-cli/run-loop.ts Outdated
@joeykrug
joeykrug marked this pull request as draft March 14, 2026 19:29
@joeykrug
joeykrug marked this pull request as ready for review March 14, 2026 20:29
Comment thread src/auto-reply/inbound-debounce.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9c43e101bd

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/auto-reply/inbound-debounce.ts Outdated
Comment thread src/cli/gateway-cli/run-loop.ts Outdated
@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime agents Agent runtime and tooling labels Mar 14, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: be0cc5ed07

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/cli/gateway-cli/run-loop.ts Outdated
@openclaw-barnacle openclaw-barnacle Bot added the channel: bluebubbles Channel integration: bluebubbles label Mar 14, 2026
@joeykrug
joeykrug marked this pull request as draft March 14, 2026 21:27
@joeykrug
joeykrug force-pushed the fix/drain-queue-before-sigusr1 branch from 4d629a9 to 4715f9f Compare March 14, 2026 21:29
@openclaw-barnacle openclaw-barnacle Bot removed gateway Gateway runtime agents Agent runtime and tooling labels Mar 14, 2026
@joeykrug
joeykrug marked this pull request as ready for review March 14, 2026 21:30
Comment thread src/auto-reply/inbound-debounce.ts Outdated
Comment thread src/cli/gateway-cli/run-loop.ts Outdated
@joeykrug
joeykrug marked this pull request as draft March 14, 2026 21:39
@joeykrug
joeykrug marked this pull request as ready for review March 14, 2026 23:38
Comment thread src/auto-reply/inbound-debounce.ts Outdated
Comment thread src/auto-reply/inbound-debounce.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2d6896bb44

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread src/auto-reply/inbound-debounce.ts Outdated
@joeykrug
joeykrug marked this pull request as draft March 15, 2026 01:25
joeykrug added 3 commits July 11, 2026 16:27
Reconcile the stale branch onto current main while retaining only the core restart drain integration.
Count overflow-summary backlog and cover the private admission, debounce, channel, followup, and run-loop drain contracts.
Synchronize the reviewed restart-drain patch with cutoff 262baed.

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Rebuilt on main cutoff 262baedcf7fa3d66dc35cce7d0253e0a31f0cc0e; current head is 5ae09ec76ee9b851cb24b07acad291724c185868. The diff is narrowed to 13 files, the public SDK drain bypass is removed, followup drain counts droppedCount, and the PR body now contains exact-head runtime proof plus focused test/type/lint results.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Current exact head is 5c06769703c80bf306311dac261e0e3cefce402f. The follow-up commit fixes the two type-aware oxlint findings from exact-head CI; the exact CI tsconfig, affected 53 tests, core typecheck, and runtime proof now pass on this head.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Current exact head is 1952ec24195d85e8f6ea032fe76b44806d952f69. The only new change isolates the followup drain test from shared FOLLOWUP_QUEUES state exposed by the full CI shard. The exact two-file shard (18 tests) and full six-file focused suite (166 tests) now pass.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Exact head c9067e7397281341627e8fdef266b7b865ff6101 addresses both blocking P1 findings:

  • pre-created debounce callbacks are rebound with a captured async context from the core-private restart continuation before their reserved tasks are released;
  • the global sweep receives the remaining restart deadline and is additionally capped at 100 passes / 30 seconds.

Focused regressions cover the pre-created command enqueue, sustained ingress, the exact final-pass boundary, and a slow callback at its deadline. The PR body now uses the required headings and records exact-head validation.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Copy link
Copy Markdown
Contributor Author

@openclaw-mantis telegram live: reproduce an inbound message buffered immediately before SIGUSR1/config reload and capture redacted proof that it is processed exactly once after the drain.

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Exact head 8c0a7a4e4cf7c35b97309c41b109e3134d94fb46 strengthens the admission regression to match the reported reproduction literally: the debounce buffer is created inside an admitted inbound root, that original root is released, restart admission closes, and the pre-created callback then successfully enqueues real command work only through the core-private drain context. The process suite remains 40/40 green; the PR body and exact tree identity are updated.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Exact head 9ec51a263dbfa5a325d3300070db1637f190d1c9 addresses the timer-boundary P1 across both ownership and accounting:

  • each reserved debounce key task is created inside a tracked required continuation before it waits on the buffer's ready gate;
  • parentless accepted buffers now receive the same required-continuation ownership through the normal admission fence;
  • flushBuffer atomically tracks a buffer as in-flight before removing it, and the global sweep counts/awaits removed-but-unsettled tasks even when it claims zero new buffers;
  • the exact regression fires the normal timer after restart admission closes, pauses inside onFlush, proves the restart sweep remains unsettled, then releases a real command enqueue and verifies successful exactly-once settlement.

Exact local validation: 4 focused files / 164 tests, type-aware lint and formatting, core production/test tsgo, and plugin SDK declaration tsgo all pass. The PR body and exact tree identity are updated.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Exact head 38befdcf347331814f3453093ff3ef0c31cc33c0 closes the restart-signal-pending gap while preserving the private fence:

  • ordinary/new roots remain rejected as soon as beginGatewayRestartSignalAdmission() runs;
  • only already-reserved requiredContinuation roots may finish subordinate work during that pending-signal window;
  • a real debounce regression creates the buffer under accepted admission, closes the pending-signal fence, fires its normal timer, and verifies its command enqueue completes; a unit regression simultaneously proves a new root is still refused.

The timer in-flight tracking/settlement regression remains in place. Exact validation is now 4 files / 166 tests, formatting/type-aware lint, core production/test tsgo, and plugin SDK declaration tsgo. PR body/tree identity are updated.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Exact head bb8f1b842e85c6745e7f8a1bfb8303d73ca6f7f6 addresses the remaining restart-deadline P1:

  • the initial active-task and embedded-run waits now receive one shared snapshot of resolveRemainingRestartDrainTimeoutMs(), after debounce and channel drains have consumed their time;
  • bounded restarts therefore preserve one absolute deadline, while unbounded restarts still pass undefined;
  • the focused deterministic regression starts with 100 ms, consumes 35 ms in debounce drain and 25 ms in channel drain, then proves both active-work waits and final close receive at most the 40 ms remainder.

Exact local validation: 4 focused files / 167 tests; formatting and type-aware lint; core production/test tsgo; and plugin SDK declaration tsgo all pass. The PR body and exact tree identity are updated.

@clawsweeper

clawsweeper Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper

clawsweeper Bot commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

@joeykrug thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward.

Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive.

Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: bluebubbles Channel integration: bluebubbles cli CLI command changes mantis: telegram-visible-proof Mantis should capture Telegram visible proof. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P1 High-priority user-facing bug, regression, or broken workflow. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: XL status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: dirty-candidate Candidate: broad unrelated surfaces; may need splitting or cleanup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant