fix(channels): prevent lifecycle listener buildup across restarts#109108
Conversation
|
Codex review: needs maintainer review before merge. Reviewed July 16, 2026, 11:42 AM ET / 15:42 UTC. Summary PR surface: Source +2, Tests +32. Total +34 across 4 files. Reproducibility: yes. from source: create a run-state machine or Discord handler with a live AbortSignal, call manual deactivation, and current main leaves its registered abort listeners attached; the PR body also supplies exact before-and-after listener counts. Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Keep abort-listener ownership symmetric: every lifecycle component should detach the exact callback it registered during public deactivation, with listener-count regressions protecting manual teardown and parent-signal abort paths. Do we have a high-confidence way to reproduce the issue? Yes from source: create a run-state machine or Discord handler with a live AbortSignal, call manual deactivation, and current main leaves its registered abort listeners attached; the PR body also supplies exact before-and-after listener counts. Is this the best way to solve the issue? Yes; detaching callbacks in the same lifecycle owners that register them is narrower and safer than changing queue ordering, gateway ownership, or abort semantics. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against d1e2fbd6823e. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +2, Tests +32. Total +34 across 4 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
Merged via squash.
|
* fix: gate diagnostics command to owners (cherry picked from commit 170bf72) * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup (#96100) * fix(agent): wait for retained session write before releasing held lock on abort * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup * fix(test): reject fallback acquire with SessionWriteLockTimeoutError in active-scope cleanup test * fix(agent): trim retained-lock comments Signed-off-by: sallyom <[email protected]> --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: sallyom <[email protected]> (cherry picked from commit 0a042f6) * fix(gateway): resume channel after pending task recovery (cherry picked from commit 6039da3) * fix(gateway): resume channel after pending task recovery (cherry picked from commit ecd29fe) * fix(outbound): ignore empty delivery receipts (#79811) (cherry picked from commit 9a735be) * fix(agents): guard delivery-evidence attachment recursion against cycles (#97041) * fix(agents): guard delivery-evidence attachment recursion against cycles * fix(agents): guard delivery-evidence attachment recursion against cycles * fix(agents): guard delivery-evidence attachment recursion against cycles --------- Co-authored-by: Pick-cat <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 4985671) * fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort (#97128) * fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort When the opencode-go model finalizes a tool call and deliberates before the next one, the provider emits real block-boundary SSE events (text_end, thinking_end, toolcall_start, toolcall_end) that prove the socket is alive, but the watchdog's isProviderProgressEvent only returned true for token deltas (text_delta, thinking_delta, toolcall_delta). This caused the idle timer to fire and falsely abort a live stream, replacing a completed answer with a stalled error and dropping the provider's real done event. Fix: include block-boundary events in isProviderProgressEvent so the idle timer is re-armed on any forward-progress provider event. text_start and thinking_start are intentionally excluded because they are synthetic preamble events that should not shorten the first-event window. Closes #96518 Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(opencode-go): satisfy lint in stream regression * test(opencode-go): satisfy lint in stream regression * test(opencode-go): satisfy lint in stream regression --------- Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 552ec2b) * fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations (#90908) * fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations When the LLM API closes the connection mid-stream, the fetch layer surfaces AbortError("This operation was aborted") with no external abort signal triggered. The old guard `shouldRethrowAbort()` returned false for these errors (because isTimeoutError matched the message), so they fell through to the fallback loop but were never retried — the error propagated up and produced SILENT_REPLY_TOKEN in group sessions, permanently silencing the topic. Replace the guard with a direct check: only rethrow AbortError when the external abort signal is actually set (user/gateway cancellation). Provider-side AbortErrors without an external signal now fall through to the next fallback candidate, giving the system a chance to recover. * fix(cron): forward abort signal into runWithModelFallback Thread the cron executor's abort signal into the shared runWithModelFallback call so that cron timeouts and cancellations stop the fallback chain instead of retrying with the next candidate. Previously, the run callback checked params.abortSignal?.aborted and threw, but runWithModelFallback itself had no signal — so the new guard in model-fallback.ts could not distinguish a caller abort from a provider-side AbortError and would retry silently. Also adds a focused regression test verifying the signal is forwarded. --------- Co-authored-by: Shengting Xie <[email protected]> Co-authored-by: yayu <[email protected]> (cherry picked from commit 98ed83f) * fix(browser): block node routes when sandbox host control is disabled (#97958) (cherry picked from commit 2cf765f) * fix(exec): bind Windows allowlist execution path (#98260) * fix(exec): bind windows allowlist execution path * fix(exec): add windows shadow execution proof * fix(exec): preserve wildcard allowlist behavior * fix(exec): correct blocked plan test fixture (cherry picked from commit 3811001) * fix(mcp): suppress unhandled error on stderr pipe in stdio transport (#99803) * fix(mcp): suppress unhandled error on stderr pipe in stdio transport When child.stderr is piped to stderrStream without an error handler, a stream-level error (EPIPE, I/O failure) crashes the process. Add a noop error handler before the pipe, consistent with the error handlers already present on stdin and stdout. Co-Authored-By: Claude <[email protected]> * test(mcp): add regression test for stderr pipe error suppression Co-Authored-By: Claude <[email protected]> * fix(mcp): report stderr stream errors * fix(mcp): report stderr stream errors --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 1b84316) * Harden macOS SQLite WAL checkpoints (#99067) (cherry picked from commit f7f1be2) * fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver (#100521) * fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver * proof(secrets): add real behavior proof script for exec resolver stream error catch * proof(secrets): replace wrapper with real exec resolver stream error proof * style: apply oxfmt to changed files (cherry picked from commit c9a0783) * fix(agents): retry transient filesystem races when reading workspace bootstrap files (#100910) * fix(agents): retry transient filesystem races when reading workspace bootstrap files * fix(agents): retry transient boundary resolution --------- Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit f36d170) * fix(gateway): finish plugin HTTP responses after post-header failures (#102125) * fix(gateway): finish plugin HTTP responses after post-header failures * test(gateway): satisfy plugin HTTP regression lint * fix(gateway): skip ending destroyed plugin responses --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 240d350) * fix(gateway): validate exact custom browser origins (#38290) Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit fa0349a) * fix: block unspecified trusted DNS targets (#103075) (cherry picked from commit c70f3d0) * fix(channels): make nack callbacks idempotent (#104919) * fix(channels): make nack callbacks idempotent * fix(channels): coalesce overlapping nack callbacks --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 02d307e) * fix(channels): prevent base URL credentials in status output (#107754) * fix(channels): redact credentials in account URLs * fix(channels): sanitize final status summaries (cherry picked from commit 210340f) * fix(channels): prevent lifecycle listener buildup (#109108) (cherry picked from commit 0e1fad7) * fix(sandbox): use Buffer.byteLength for env var value size limit (#105017) * fix(sandbox): use Buffer.byteLength for env var value size limit validateEnvVarValue checked value.length (UTF-16 code units) against the 32768-byte limit, so multi-byte CJK values like "值".repeat(11000) passed the check despite exceeding 33 KB in UTF-8. Switch to Buffer.byteLength(value, "utf8") so the limit matches the actual byte count the OS and child processes see. * test(sandbox): simplify env byte-limit coverage Co-authored-by: 唐梓夷0668001293 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 84fb48c) * fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync (#109590) * fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync A verified gateway process can exit between the argv validation check and the process.kill call, causing an unhandled ESRCH error. Wrap the kill in try-catch and silently swallow ESRCH (process already gone = signal already delivered). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * docs(gateway): explain ESRCH signal race Co-authored-by: 丁宇婷0668001435 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 853b1a8) * fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS SSRF bypass (#110693) * fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS bypass The isAutoAllowedLitellmHostname helper auto-enables private-network access for loopback-style hosts. Before this fix, lowered.startsWith("127.") matched DNS hostnames like 127.evil.com, letting remote endpoints bypass the explicit allowPrivateNetwork opt-in — a SSRF risk. Add isIP(host)===4 guard so only literal IPv4 loopback addresses qualify. Same canonical pattern as extensions/slack/src/monitor/relay-source.ts:271 and the codex loopback fix. Co-Authored-By: Claude <[email protected]> * test(litellm): cover loopback endpoint policy --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 3d03b60) * fix(discord): sustained gateway bursts stop growing memory (#110954) * fix(discord): sustained gateway bursts stop growing memory * fix(discord): contain gateway queue overflow * fix(discord): drop oldest saturated gateway sends Co-authored-by: 张贵萍0668001030 <[email protected]> * fix(discord): surface gateway overflow warnings Co-authored-by: 张贵萍0668001030 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 69aeba9) * fix(gateway): bound busy channel health by real run age (#103793) * fix(gateway): bound busy channel health by real run age The channel health policy treats a channel as healthy-busy even while disconnected, bounded only by a 25 minute stale ceiling measured from lastRunActivityAt. The run-state heartbeat refreshes lastRunActivityAt every 60 seconds for as long as any run is active, so a run that hangs forever (for example a send blocking on a dead socket after the transport already reported connected:false) keeps that timestamp fresh and the stuck ceiling is never reached. The account is then reported healthy forever by the health monitor, readiness probe, and health CLI, and no restart ever fires. createRunStateMachine now tracks each in-flight run's start time keyed by an opaque run handle and publishes the oldest still-active run's start as activeRunStartedAt. The health policy busy override keys its ceiling off the real run age, so a run stuck longer than the threshold reports stuck and the monitor can restart it. Because the reported start is the oldest active run and advances to the next-oldest as runs complete, a channel churning through many short overlapping runs (activeRuns above 1 across concurrent queue keys) stays healthy; only a genuinely hung run breaches the ceiling. Short and active runs stay healthy and the existing lastRunActivityAt fallback is preserved for snapshots without a start time. * fix(channels): retain run-state callback compatibility Keep the released zero-argument onRunEnd callback source-compatible while allowing internal queue callers to pass a run handle for exact concurrent-run accounting. The compatibility path closes the oldest active run, preserving existing lifecycle behavior for consumers that do not use handles. * fix(channels): keep anonymous runs out of age tracking The zero-argument lifecycle callbacks cannot identify which concurrent run completed, so they must not update the identity-sensitive run start used by channel health. Keep their busy count separately and reserve exact start tracking for the shared queue's handle-aware lifecycle path. * fix(channels): keep tracked runs internal Keep the public run-state lifecycle callbacks unchanged. The channel queue now owns opaque run identity and augments its status updates with the oldest active queue run, so implementation details do not expand the SDK surface. * fix(channels): type queue run start status Keep activeRunStartedAt in the internal status patch type so the queue can publish its private tracked-run age through the existing status sink. * fix(channels): wrap isActive to satisfy unbound-method lint * fix(gateway): gate busy run-age ceiling on disconnected transport (cherry picked from commit 18b79d9) * fix(deps): update fast-uri past advisory (cherry picked from commit 1be9db0) * fix(release): adapt maintenance-line hardening Backport/adapt 18ec9ce, dea1fe1, 7f32b6c, 1da345e, 931ac3e, 89780d5, and c0d99ed for the 2026.6 extended-stable maintenance line. * fix(deps): bump protobufjs to 7.6.5 Backport-adapted from a230f74. * test(gateway): cover bounded macOS process probe * chore(release): prepare 2026.6.34 * test(dotenv): share path override environment assertions * fix(release): resolve 2026.6.34 CI blockers --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: joshavant <[email protected]> Co-authored-by: Peter Lee <[email protected]> Co-authored-by: sallyom <[email protected]> Co-authored-by: openclaw-clownfish[bot] <280122609+openclaw-clownfish[bot]@users.noreply.github.com> Co-authored-by: Liu Wenyu <[email protected]> Co-authored-by: pick-cat <[email protected]> Co-authored-by: Pick-cat <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: weiqinl <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: shengting <[email protected]> Co-authored-by: Shengting Xie <[email protected]> Co-authored-by: yayu <[email protected]> Co-authored-by: Agustin Rivera <[email protected]> Co-authored-by: cxbAsDev <[email protected]> Co-authored-by: ooiuuii <[email protected]> Co-authored-by: Masato Hoshino <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: mushuiyu886 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> Co-authored-by: Bruno Wowk (Volky) <[email protected]> Co-authored-by: Pavan Kumar Gondhi <[email protected]> Co-authored-by: Glucksberg <[email protected]> Co-authored-by: xingzhou <[email protected]> Co-authored-by: tzy-17 <[email protected]> Co-authored-by: krissding <[email protected]> Co-authored-by: lsr911 <[email protected]> Co-authored-by: Yuval Dinodia <[email protected]>
What Problem This Solves
Fixes an issue where repeated Discord startup or teardown attempts could accumulate stale queue lifecycle listeners when cleanup deactivated a handler before its longer-lived gateway abort signal fired.
Why This Change Was Made
The shared run-state owner and Discord skipped-job cleanup owner now remove the abort listeners they registered whenever public handler deactivation runs, alongside the existing heartbeat and queued-job cleanup. The change is limited to lifecycle resource ownership; status behavior, queue ordering, replay handling, configuration, and public API shapes are unchanged.
User Impact
Discord restarts and startup-failure cleanup no longer leave inactive queue closures attached to the parent lifecycle signal. This prevents listener buildup across retries without changing normal message processing.
Evidence
b8b064a9483e: the production Discord message queue reportedlistenerCountAfterCreate=2andlistenerCountAfterDeactivate=2, confirming the complete handler problem was still present immediately before submission review.listenerCountAfterCreate=2,listenerCountAfterDeactivate=0, andproblemPresent=false.node scripts/run-vitest.mjs src/channels/run-state-machine.test.ts— 4/4 passed.node scripts/run-vitest.mjs src/plugin-sdk/channel-lifecycle.queue.test.ts— 4/4 passed.node scripts/run-vitest.mjs extensions/discord/src/monitor/message-handler.queue.test.ts— 13/13 passed.git diff --checkpassed for the changed files.