fix(discord): sustained gateway bursts stop growing memory#110954
Conversation
|
Codex review: needs real behavior proof before merge. Reviewed July 18, 2026, 10:00 PM ET / July 19, 2026, 02:00 UTC. Summary PR surface: Source +43, Tests +40. Total +83 across 5 files. Reproducibility: unclear. whether we have a high-confidence live reproduction: the added owner-level test describes a deterministic saturation path, but no current final-head real Gateway run was provided for inspection. Review metrics: 1 noteworthy metric.
Stored data model Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Retain the queue-owner bound only after a redacted final-head Gateway saturation run proves bounded recovery and the existing owner review confirms that newest-state preservation is the intended non-critical delivery contract. Do we have a high-confidence way to reproduce the issue? Unclear whether we have a high-confidence live reproduction: the added owner-level test describes a deterministic saturation path, but no current final-head real Gateway run was provided for inspection. Is this the best way to solve the issue? Unclear: placing the limit in the serialized queue owner is the narrowest apparent implementation, but final confirmation needs real overload evidence and review of non-critical caller delivery expectations. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 047232bd16e4. Label changesLabel justifications:
Evidence reviewedPR surface: Source +43, Tests +40. Total +83 across 5 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (5 earlier review cycles)
|
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. Re-review progress:
|
2dac65d to
208e936
Compare
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(discord): sustained gateway bursts stop growing memory This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
Co-authored-by: 张贵萍0668001030 <[email protected]>
Co-authored-by: 张贵萍0668001030 <[email protected]>
208e936 to
cf9c716
Compare
47104d5 to
cf9c716
Compare
|
Merged via squash.
|
…110954) * fix(discord): sustained gateway bursts stop growing memory * fix(discord): contain gateway queue overflow * fix(discord): drop oldest saturated gateway sends Co-authored-by: 张贵萍0668001030 <[email protected]> * fix(discord): surface gateway overflow warnings Co-authored-by: 张贵萍0668001030 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]>
* fix: gate diagnostics command to owners (cherry picked from commit 170bf72) * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup (#96100) * fix(agent): wait for retained session write before releasing held lock on abort * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup * fix(test): reject fallback acquire with SessionWriteLockTimeoutError in active-scope cleanup test * fix(agent): trim retained-lock comments Signed-off-by: sallyom <[email protected]> --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: sallyom <[email protected]> (cherry picked from commit 0a042f6) * fix(gateway): resume channel after pending task recovery (cherry picked from commit 6039da3) * fix(gateway): resume channel after pending task recovery (cherry picked from commit ecd29fe) * fix(outbound): ignore empty delivery receipts (#79811) (cherry picked from commit 9a735be) * fix(agents): guard delivery-evidence attachment recursion against cycles (#97041) * fix(agents): guard delivery-evidence attachment recursion against cycles * fix(agents): guard delivery-evidence attachment recursion against cycles * fix(agents): guard delivery-evidence attachment recursion against cycles --------- Co-authored-by: Pick-cat <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 4985671) * fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort (#97128) * fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort When the opencode-go model finalizes a tool call and deliberates before the next one, the provider emits real block-boundary SSE events (text_end, thinking_end, toolcall_start, toolcall_end) that prove the socket is alive, but the watchdog's isProviderProgressEvent only returned true for token deltas (text_delta, thinking_delta, toolcall_delta). This caused the idle timer to fire and falsely abort a live stream, replacing a completed answer with a stalled error and dropping the provider's real done event. Fix: include block-boundary events in isProviderProgressEvent so the idle timer is re-armed on any forward-progress provider event. text_start and thinking_start are intentionally excluded because they are synthetic preamble events that should not shorten the first-event window. Closes #96518 Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(opencode-go): satisfy lint in stream regression * test(opencode-go): satisfy lint in stream regression * test(opencode-go): satisfy lint in stream regression --------- Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 552ec2b) * fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations (#90908) * fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations When the LLM API closes the connection mid-stream, the fetch layer surfaces AbortError("This operation was aborted") with no external abort signal triggered. The old guard `shouldRethrowAbort()` returned false for these errors (because isTimeoutError matched the message), so they fell through to the fallback loop but were never retried — the error propagated up and produced SILENT_REPLY_TOKEN in group sessions, permanently silencing the topic. Replace the guard with a direct check: only rethrow AbortError when the external abort signal is actually set (user/gateway cancellation). Provider-side AbortErrors without an external signal now fall through to the next fallback candidate, giving the system a chance to recover. * fix(cron): forward abort signal into runWithModelFallback Thread the cron executor's abort signal into the shared runWithModelFallback call so that cron timeouts and cancellations stop the fallback chain instead of retrying with the next candidate. Previously, the run callback checked params.abortSignal?.aborted and threw, but runWithModelFallback itself had no signal — so the new guard in model-fallback.ts could not distinguish a caller abort from a provider-side AbortError and would retry silently. Also adds a focused regression test verifying the signal is forwarded. --------- Co-authored-by: Shengting Xie <[email protected]> Co-authored-by: yayu <[email protected]> (cherry picked from commit 98ed83f) * fix(browser): block node routes when sandbox host control is disabled (#97958) (cherry picked from commit 2cf765f) * fix(exec): bind Windows allowlist execution path (#98260) * fix(exec): bind windows allowlist execution path * fix(exec): add windows shadow execution proof * fix(exec): preserve wildcard allowlist behavior * fix(exec): correct blocked plan test fixture (cherry picked from commit 3811001) * fix(mcp): suppress unhandled error on stderr pipe in stdio transport (#99803) * fix(mcp): suppress unhandled error on stderr pipe in stdio transport When child.stderr is piped to stderrStream without an error handler, a stream-level error (EPIPE, I/O failure) crashes the process. Add a noop error handler before the pipe, consistent with the error handlers already present on stdin and stdout. Co-Authored-By: Claude <[email protected]> * test(mcp): add regression test for stderr pipe error suppression Co-Authored-By: Claude <[email protected]> * fix(mcp): report stderr stream errors * fix(mcp): report stderr stream errors --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 1b84316) * Harden macOS SQLite WAL checkpoints (#99067) (cherry picked from commit f7f1be2) * fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver (#100521) * fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver * proof(secrets): add real behavior proof script for exec resolver stream error catch * proof(secrets): replace wrapper with real exec resolver stream error proof * style: apply oxfmt to changed files (cherry picked from commit c9a0783) * fix(agents): retry transient filesystem races when reading workspace bootstrap files (#100910) * fix(agents): retry transient filesystem races when reading workspace bootstrap files * fix(agents): retry transient boundary resolution --------- Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit f36d170) * fix(gateway): finish plugin HTTP responses after post-header failures (#102125) * fix(gateway): finish plugin HTTP responses after post-header failures * test(gateway): satisfy plugin HTTP regression lint * fix(gateway): skip ending destroyed plugin responses --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 240d350) * fix(gateway): validate exact custom browser origins (#38290) Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit fa0349a) * fix: block unspecified trusted DNS targets (#103075) (cherry picked from commit c70f3d0) * fix(channels): make nack callbacks idempotent (#104919) * fix(channels): make nack callbacks idempotent * fix(channels): coalesce overlapping nack callbacks --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 02d307e) * fix(channels): prevent base URL credentials in status output (#107754) * fix(channels): redact credentials in account URLs * fix(channels): sanitize final status summaries (cherry picked from commit 210340f) * fix(channels): prevent lifecycle listener buildup (#109108) (cherry picked from commit 0e1fad7) * fix(sandbox): use Buffer.byteLength for env var value size limit (#105017) * fix(sandbox): use Buffer.byteLength for env var value size limit validateEnvVarValue checked value.length (UTF-16 code units) against the 32768-byte limit, so multi-byte CJK values like "值".repeat(11000) passed the check despite exceeding 33 KB in UTF-8. Switch to Buffer.byteLength(value, "utf8") so the limit matches the actual byte count the OS and child processes see. * test(sandbox): simplify env byte-limit coverage Co-authored-by: 唐梓夷0668001293 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 84fb48c) * fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync (#109590) * fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync A verified gateway process can exit between the argv validation check and the process.kill call, causing an unhandled ESRCH error. Wrap the kill in try-catch and silently swallow ESRCH (process already gone = signal already delivered). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * docs(gateway): explain ESRCH signal race Co-authored-by: 丁宇婷0668001435 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 853b1a8) * fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS SSRF bypass (#110693) * fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS bypass The isAutoAllowedLitellmHostname helper auto-enables private-network access for loopback-style hosts. Before this fix, lowered.startsWith("127.") matched DNS hostnames like 127.evil.com, letting remote endpoints bypass the explicit allowPrivateNetwork opt-in — a SSRF risk. Add isIP(host)===4 guard so only literal IPv4 loopback addresses qualify. Same canonical pattern as extensions/slack/src/monitor/relay-source.ts:271 and the codex loopback fix. Co-Authored-By: Claude <[email protected]> * test(litellm): cover loopback endpoint policy --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 3d03b60) * fix(discord): sustained gateway bursts stop growing memory (#110954) * fix(discord): sustained gateway bursts stop growing memory * fix(discord): contain gateway queue overflow * fix(discord): drop oldest saturated gateway sends Co-authored-by: 张贵萍0668001030 <[email protected]> * fix(discord): surface gateway overflow warnings Co-authored-by: 张贵萍0668001030 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 69aeba9) * fix(gateway): bound busy channel health by real run age (#103793) * fix(gateway): bound busy channel health by real run age The channel health policy treats a channel as healthy-busy even while disconnected, bounded only by a 25 minute stale ceiling measured from lastRunActivityAt. The run-state heartbeat refreshes lastRunActivityAt every 60 seconds for as long as any run is active, so a run that hangs forever (for example a send blocking on a dead socket after the transport already reported connected:false) keeps that timestamp fresh and the stuck ceiling is never reached. The account is then reported healthy forever by the health monitor, readiness probe, and health CLI, and no restart ever fires. createRunStateMachine now tracks each in-flight run's start time keyed by an opaque run handle and publishes the oldest still-active run's start as activeRunStartedAt. The health policy busy override keys its ceiling off the real run age, so a run stuck longer than the threshold reports stuck and the monitor can restart it. Because the reported start is the oldest active run and advances to the next-oldest as runs complete, a channel churning through many short overlapping runs (activeRuns above 1 across concurrent queue keys) stays healthy; only a genuinely hung run breaches the ceiling. Short and active runs stay healthy and the existing lastRunActivityAt fallback is preserved for snapshots without a start time. * fix(channels): retain run-state callback compatibility Keep the released zero-argument onRunEnd callback source-compatible while allowing internal queue callers to pass a run handle for exact concurrent-run accounting. The compatibility path closes the oldest active run, preserving existing lifecycle behavior for consumers that do not use handles. * fix(channels): keep anonymous runs out of age tracking The zero-argument lifecycle callbacks cannot identify which concurrent run completed, so they must not update the identity-sensitive run start used by channel health. Keep their busy count separately and reserve exact start tracking for the shared queue's handle-aware lifecycle path. * fix(channels): keep tracked runs internal Keep the public run-state lifecycle callbacks unchanged. The channel queue now owns opaque run identity and augments its status updates with the oldest active queue run, so implementation details do not expand the SDK surface. * fix(channels): type queue run start status Keep activeRunStartedAt in the internal status patch type so the queue can publish its private tracked-run age through the existing status sink. * fix(channels): wrap isActive to satisfy unbound-method lint * fix(gateway): gate busy run-age ceiling on disconnected transport (cherry picked from commit 18b79d9) * fix(deps): update fast-uri past advisory (cherry picked from commit 1be9db0) * fix(release): adapt maintenance-line hardening Backport/adapt 18ec9ce, dea1fe1, 7f32b6c, 1da345e, 931ac3e, 89780d5, and c0d99ed for the 2026.6 extended-stable maintenance line. * fix(deps): bump protobufjs to 7.6.5 Backport-adapted from a230f74. * test(gateway): cover bounded macOS process probe * chore(release): prepare 2026.6.34 * test(dotenv): share path override environment assertions * fix(release): resolve 2026.6.34 CI blockers --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: joshavant <[email protected]> Co-authored-by: Peter Lee <[email protected]> Co-authored-by: sallyom <[email protected]> Co-authored-by: openclaw-clownfish[bot] <280122609+openclaw-clownfish[bot]@users.noreply.github.com> Co-authored-by: Liu Wenyu <[email protected]> Co-authored-by: pick-cat <[email protected]> Co-authored-by: Pick-cat <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: weiqinl <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: shengting <[email protected]> Co-authored-by: Shengting Xie <[email protected]> Co-authored-by: yayu <[email protected]> Co-authored-by: Agustin Rivera <[email protected]> Co-authored-by: cxbAsDev <[email protected]> Co-authored-by: ooiuuii <[email protected]> Co-authored-by: Masato Hoshino <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: mushuiyu886 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> Co-authored-by: Bruno Wowk (Volky) <[email protected]> Co-authored-by: Pavan Kumar Gondhi <[email protected]> Co-authored-by: Glucksberg <[email protected]> Co-authored-by: xingzhou <[email protected]> Co-authored-by: tzy-17 <[email protected]> Co-authored-by: krissding <[email protected]> Co-authored-by: lsr911 <[email protected]> Co-authored-by: Yuval Dinodia <[email protected]>
What Problem This Solves
Discord Gateway commands above the 120-events-per-60-seconds connection limit were appended to an unbounded in-memory queue. Sustained presence or member-request traffic could therefore grow memory indefinitely while retaining increasingly stale work.
Why This Change Was Made
The Gateway send limiter now retains at most one additional 120-event window. When that queue is saturated, it drops the oldest deferred command and keeps the newest command instead of throwing into presence/member callers. The limiter emits one structured warning per saturation episode and exposes the cumulative dropped-event count in its existing status snapshot.
This policy stays in the queue owner. Heartbeats, identify/resume, and voice-state updates remain critical sends and bypass the ordinary deferred queue. Reconnect cleanup remains unchanged.
The generic bounded-cache helpers were not reused because they own dedupe or insertion-ordered Map semantics rather than a serialized FIFO. The shared message queue helper is not a plugin SDK surface. LINE and Slack durable ingress spools retain events because their webhook/relay acknowledgement contracts require redelivery; Discord presence/member commands have no comparable durable-delivery contract, so keeping the newest bounded state is the better overload behavior.
Discord documents a limit of 120 Gateway events per connection every 60 seconds: https://docs.discord.com/developers/events/gateway#rate-limiting
User Impact
Long-running Discord connections no longer accumulate an unbounded send backlog. During sustained overload, stale deferred presence/member commands are discarded without surfacing synchronous queue-full failures to ordinary callers, and operators receive a bounded warning signal.
Evidence
Focused behavior proof:
The owner-level overflow regression exhausts the live send window, queues 122 indexed presence commands, verifies the two oldest deferred commands are dropped, verifies the newest 120 flush in order, checks the cumulative drop count, and proves one warning is emitted for the continuous saturation episode.
Changed-surface proof delegated to Blacksmith Testbox and passed formatting, extension production/test typechecks, extension lint, plugin boundaries, dependency guards, and runtime import-cycle checks:
https://github.com/openclaw/openclaw/actions/runs/29669128543
Structured autoreview completed cleanly on the final rewrite with the focused proof running in parallel.
No config, environment variable, UI, protocol, SDK, dependency, storage, or migration surface changes.