fix(agent): replace self-wait with deferred release in retained-lock abort cleanup#96100
Conversation
|
@clawsweeper review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
|
Codex review: needs maintainer review before merge. Reviewed June 24, 2026, 6:26 PM ET / 22:26 UTC. Summary PR surface: Source +13, Tests +97. Total +110 across 2 files. Reproducibility: yes. at source level: current main can return early from retained-lock abort cleanup inside an active write scope, and the linked production issue shows the corresponding session lock timeout symptom. I did not run a live current-main multi-channel reproduction in this read-only review. Review metrics: 1 noteworthy metric.
Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the narrow fence-preserving deferred-release fix after session-lock owner acceptance, while tracking broader dispose, force-release, and memory-retention work separately. Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main can return early from retained-lock abort cleanup inside an active write scope, and the linked production issue shows the corresponding session lock timeout symptom. I did not run a live current-main multi-channel reproduction in this read-only review. Is this the best way to solve the issue? Likely yes: deferring release until after the active retained writer unwinds is narrower and safer than force-releasing while that writer still owns the scope. Broader force-release or bounded-dispose fixes may still be needed for adjacent failure modes, but they are not safer replacements for this active-scope abort fix. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against c6f5725906dc. Label changesLabel justifications:
Evidence reviewedPR surface: Source +13, Tests +97. Total +110 across 2 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
|
@clawsweeper review |
|
@clawsweeper re-review |
|
🦞🧹 I asked ClawSweeper to review this item again. |
…in active-scope cleanup test
Signed-off-by: sallyom <[email protected]>
|
Pushed maintainer edit 457aa1e to trim excessive code comments before merging. This is a comment-only cleanup; no behavior changes intended. |
|
Merge-ready at 457aa1e. CI is green, local review clean, the maintainer comment-trim commit is comment-only, and I do not see SDK/API/config/default/migration compatibility concerns in the final diff. Scope remains the retained session write-lock release fix; any broader heap-growth or dispose/force-release follow-up stays separate. Determined best narrow fix. |
…abort cleanup (openclaw#96100) * fix(agent): wait for retained session write before releasing held lock on abort * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup * fix(test): reject fallback acquire with SessionWriteLockTimeoutError in active-scope cleanup test * fix(agent): trim retained-lock comments Signed-off-by: sallyom <[email protected]> --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: sallyom <[email protected]>
…abort cleanup (openclaw#96100) * fix(agent): wait for retained session write before releasing held lock on abort * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup * fix(test): reject fallback acquire with SessionWriteLockTimeoutError in active-scope cleanup test * fix(agent): trim retained-lock comments Signed-off-by: sallyom <[email protected]> --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: sallyom <[email protected]>
* fix: gate diagnostics command to owners (cherry picked from commit 170bf72) * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup (#96100) * fix(agent): wait for retained session write before releasing held lock on abort * fix(agent): replace self-wait with deferred release in retained-lock abort cleanup * fix(test): reject fallback acquire with SessionWriteLockTimeoutError in active-scope cleanup test * fix(agent): trim retained-lock comments Signed-off-by: sallyom <[email protected]> --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: sallyom <[email protected]> (cherry picked from commit 0a042f6) * fix(gateway): resume channel after pending task recovery (cherry picked from commit 6039da3) * fix(gateway): resume channel after pending task recovery (cherry picked from commit ecd29fe) * fix(outbound): ignore empty delivery receipts (#79811) (cherry picked from commit 9a735be) * fix(agents): guard delivery-evidence attachment recursion against cycles (#97041) * fix(agents): guard delivery-evidence attachment recursion against cycles * fix(agents): guard delivery-evidence attachment recursion against cycles * fix(agents): guard delivery-evidence attachment recursion against cycles --------- Co-authored-by: Pick-cat <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 4985671) * fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort (#97128) * fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort When the opencode-go model finalizes a tool call and deliberates before the next one, the provider emits real block-boundary SSE events (text_end, thinking_end, toolcall_start, toolcall_end) that prove the socket is alive, but the watchdog's isProviderProgressEvent only returned true for token deltas (text_delta, thinking_delta, toolcall_delta). This caused the idle timer to fire and falsely abort a live stream, replacing a completed answer with a stalled error and dropping the provider's real done event. Fix: include block-boundary events in isProviderProgressEvent so the idle timer is re-armed on any forward-progress provider event. text_start and thinking_start are intentionally excluded because they are synthetic preamble events that should not shorten the first-event window. Closes #96518 Co-Authored-By: Claude Opus 4.8 <[email protected]> * test(opencode-go): satisfy lint in stream regression * test(opencode-go): satisfy lint in stream regression * test(opencode-go): satisfy lint in stream regression --------- Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 552ec2b) * fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations (#90908) * fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations When the LLM API closes the connection mid-stream, the fetch layer surfaces AbortError("This operation was aborted") with no external abort signal triggered. The old guard `shouldRethrowAbort()` returned false for these errors (because isTimeoutError matched the message), so they fell through to the fallback loop but were never retried — the error propagated up and produced SILENT_REPLY_TOKEN in group sessions, permanently silencing the topic. Replace the guard with a direct check: only rethrow AbortError when the external abort signal is actually set (user/gateway cancellation). Provider-side AbortErrors without an external signal now fall through to the next fallback candidate, giving the system a chance to recover. * fix(cron): forward abort signal into runWithModelFallback Thread the cron executor's abort signal into the shared runWithModelFallback call so that cron timeouts and cancellations stop the fallback chain instead of retrying with the next candidate. Previously, the run callback checked params.abortSignal?.aborted and threw, but runWithModelFallback itself had no signal — so the new guard in model-fallback.ts could not distinguish a caller abort from a provider-side AbortError and would retry silently. Also adds a focused regression test verifying the signal is forwarded. --------- Co-authored-by: Shengting Xie <[email protected]> Co-authored-by: yayu <[email protected]> (cherry picked from commit 98ed83f) * fix(browser): block node routes when sandbox host control is disabled (#97958) (cherry picked from commit 2cf765f) * fix(exec): bind Windows allowlist execution path (#98260) * fix(exec): bind windows allowlist execution path * fix(exec): add windows shadow execution proof * fix(exec): preserve wildcard allowlist behavior * fix(exec): correct blocked plan test fixture (cherry picked from commit 3811001) * fix(mcp): suppress unhandled error on stderr pipe in stdio transport (#99803) * fix(mcp): suppress unhandled error on stderr pipe in stdio transport When child.stderr is piped to stderrStream without an error handler, a stream-level error (EPIPE, I/O failure) crashes the process. Add a noop error handler before the pipe, consistent with the error handlers already present on stdin and stdout. Co-Authored-By: Claude <[email protected]> * test(mcp): add regression test for stderr pipe error suppression Co-Authored-By: Claude <[email protected]> * fix(mcp): report stderr stream errors * fix(mcp): report stderr stream errors --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit 1b84316) * Harden macOS SQLite WAL checkpoints (#99067) (cherry picked from commit f7f1be2) * fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver (#100521) * fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver * proof(secrets): add real behavior proof script for exec resolver stream error catch * proof(secrets): replace wrapper with real exec resolver stream error proof * style: apply oxfmt to changed files (cherry picked from commit c9a0783) * fix(agents): retry transient filesystem races when reading workspace bootstrap files (#100910) * fix(agents): retry transient filesystem races when reading workspace bootstrap files * fix(agents): retry transient boundary resolution --------- Co-authored-by: Vincent Koc <[email protected]> (cherry picked from commit f36d170) * fix(gateway): finish plugin HTTP responses after post-header failures (#102125) * fix(gateway): finish plugin HTTP responses after post-header failures * test(gateway): satisfy plugin HTTP regression lint * fix(gateway): skip ending destroyed plugin responses --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 240d350) * fix(gateway): validate exact custom browser origins (#38290) Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit fa0349a) * fix: block unspecified trusted DNS targets (#103075) (cherry picked from commit c70f3d0) * fix(channels): make nack callbacks idempotent (#104919) * fix(channels): make nack callbacks idempotent * fix(channels): coalesce overlapping nack callbacks --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 02d307e) * fix(channels): prevent base URL credentials in status output (#107754) * fix(channels): redact credentials in account URLs * fix(channels): sanitize final status summaries (cherry picked from commit 210340f) * fix(channels): prevent lifecycle listener buildup (#109108) (cherry picked from commit 0e1fad7) * fix(sandbox): use Buffer.byteLength for env var value size limit (#105017) * fix(sandbox): use Buffer.byteLength for env var value size limit validateEnvVarValue checked value.length (UTF-16 code units) against the 32768-byte limit, so multi-byte CJK values like "值".repeat(11000) passed the check despite exceeding 33 KB in UTF-8. Switch to Buffer.byteLength(value, "utf8") so the limit matches the actual byte count the OS and child processes see. * test(sandbox): simplify env byte-limit coverage Co-authored-by: 唐梓夷0668001293 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 84fb48c) * fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync (#109590) * fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync A verified gateway process can exit between the argv validation check and the process.kill call, causing an unhandled ESRCH error. Wrap the kill in try-catch and silently swallow ESRCH (process already gone = signal already delivered). Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * docs(gateway): explain ESRCH signal race Co-authored-by: 丁宇婷0668001435 <[email protected]> --------- Co-authored-by: Claude Sonnet 4.6 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 853b1a8) * fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS SSRF bypass (#110693) * fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS bypass The isAutoAllowedLitellmHostname helper auto-enables private-network access for loopback-style hosts. Before this fix, lowered.startsWith("127.") matched DNS hostnames like 127.evil.com, letting remote endpoints bypass the explicit allowPrivateNetwork opt-in — a SSRF risk. Add isIP(host)===4 guard so only literal IPv4 loopback addresses qualify. Same canonical pattern as extensions/slack/src/monitor/relay-source.ts:271 and the codex loopback fix. Co-Authored-By: Claude <[email protected]> * test(litellm): cover loopback endpoint policy --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 3d03b60) * fix(discord): sustained gateway bursts stop growing memory (#110954) * fix(discord): sustained gateway bursts stop growing memory * fix(discord): contain gateway queue overflow * fix(discord): drop oldest saturated gateway sends Co-authored-by: 张贵萍0668001030 <[email protected]> * fix(discord): surface gateway overflow warnings Co-authored-by: 张贵萍0668001030 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]> (cherry picked from commit 69aeba9) * fix(gateway): bound busy channel health by real run age (#103793) * fix(gateway): bound busy channel health by real run age The channel health policy treats a channel as healthy-busy even while disconnected, bounded only by a 25 minute stale ceiling measured from lastRunActivityAt. The run-state heartbeat refreshes lastRunActivityAt every 60 seconds for as long as any run is active, so a run that hangs forever (for example a send blocking on a dead socket after the transport already reported connected:false) keeps that timestamp fresh and the stuck ceiling is never reached. The account is then reported healthy forever by the health monitor, readiness probe, and health CLI, and no restart ever fires. createRunStateMachine now tracks each in-flight run's start time keyed by an opaque run handle and publishes the oldest still-active run's start as activeRunStartedAt. The health policy busy override keys its ceiling off the real run age, so a run stuck longer than the threshold reports stuck and the monitor can restart it. Because the reported start is the oldest active run and advances to the next-oldest as runs complete, a channel churning through many short overlapping runs (activeRuns above 1 across concurrent queue keys) stays healthy; only a genuinely hung run breaches the ceiling. Short and active runs stay healthy and the existing lastRunActivityAt fallback is preserved for snapshots without a start time. * fix(channels): retain run-state callback compatibility Keep the released zero-argument onRunEnd callback source-compatible while allowing internal queue callers to pass a run handle for exact concurrent-run accounting. The compatibility path closes the oldest active run, preserving existing lifecycle behavior for consumers that do not use handles. * fix(channels): keep anonymous runs out of age tracking The zero-argument lifecycle callbacks cannot identify which concurrent run completed, so they must not update the identity-sensitive run start used by channel health. Keep their busy count separately and reserve exact start tracking for the shared queue's handle-aware lifecycle path. * fix(channels): keep tracked runs internal Keep the public run-state lifecycle callbacks unchanged. The channel queue now owns opaque run identity and augments its status updates with the oldest active queue run, so implementation details do not expand the SDK surface. * fix(channels): type queue run start status Keep activeRunStartedAt in the internal status patch type so the queue can publish its private tracked-run age through the existing status sink. * fix(channels): wrap isActive to satisfy unbound-method lint * fix(gateway): gate busy run-age ceiling on disconnected transport (cherry picked from commit 18b79d9) * fix(deps): update fast-uri past advisory (cherry picked from commit 1be9db0) * fix(release): adapt maintenance-line hardening Backport/adapt 18ec9ce, dea1fe1, 7f32b6c, 1da345e, 931ac3e, 89780d5, and c0d99ed for the 2026.6 extended-stable maintenance line. * fix(deps): bump protobufjs to 7.6.5 Backport-adapted from a230f74. * test(gateway): cover bounded macOS process probe * chore(release): prepare 2026.6.34 * test(dotenv): share path override environment assertions * fix(release): resolve 2026.6.34 CI blockers --------- Signed-off-by: sallyom <[email protected]> Co-authored-by: joshavant <[email protected]> Co-authored-by: Peter Lee <[email protected]> Co-authored-by: sallyom <[email protected]> Co-authored-by: openclaw-clownfish[bot] <280122609+openclaw-clownfish[bot]@users.noreply.github.com> Co-authored-by: Liu Wenyu <[email protected]> Co-authored-by: pick-cat <[email protected]> Co-authored-by: Pick-cat <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: weiqinl <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: shengting <[email protected]> Co-authored-by: Shengting Xie <[email protected]> Co-authored-by: yayu <[email protected]> Co-authored-by: Agustin Rivera <[email protected]> Co-authored-by: cxbAsDev <[email protected]> Co-authored-by: ooiuuii <[email protected]> Co-authored-by: Masato Hoshino <[email protected]> Co-authored-by: Vincent Koc <[email protected]> Co-authored-by: mushuiyu886 <[email protected]> Co-authored-by: Peter Steinberger <[email protected]> Co-authored-by: Bruno Wowk (Volky) <[email protected]> Co-authored-by: Pavan Kumar Gondhi <[email protected]> Co-authored-by: Glucksberg <[email protected]> Co-authored-by: xingzhou <[email protected]> Co-authored-by: tzy-17 <[email protected]> Co-authored-by: krissding <[email protected]> Co-authored-by: lsr911 <[email protected]> Co-authored-by: Yuval Dinodia <[email protected]>
Root Cause
When embedded run abort fires during an active retained session write,
releaseHeldLockWithFence()bails out becausewaitForRetainedLockIdle()returnsfalsewhen the current active write-lock scope is still alive. The held lock is left unreleased, andacquireCleanupLock()then triesacquireLock()which fails with EEXIST (the file lock is still held), returning a noopLock that never releases the underlying file lock. The session write lock leaks until themaxHoldMswatchdog timer fires (90–350s).The initial fix (commit
3213604) added a wait onretainedLockIdleWaitersinsidetakeHeldLockAfterRetainedIdle()anddisposeHeldLockAfterRetainedIdle(), but this introduced a self-deadlock: the waiter resolves whenreleaseRetainedUse()releases the retained count to zero, which only happens when the active scope unwinds — but the await is inside that same active scope, so unwinding can never complete.Summary
retainedLockIdleWaitersfromtakeHeldLockAfterRetainedIdle()anddisposeHeldLockAfterRetainedIdle(). When called from inside the active write scope, these now return early (undefined/void) instead of deadlocking.releaseHeldLockDeferredflag thatreleaseHeldLockWithFence()sets when it bails out because the active write scope is still alive.runWithPhysicalWriteLockScope, after the scope is deactivated and the retained-use count is released, checkreleaseHeldLockDeferredand re-attemptreleaseHeldLockWithFence(). At this pointwaitForRetainedLockIdle()can safely wait (the current scope is no longer the active one).acquireForCleanupis called from inside the active scope.cc37eda): tightened thetakeHeldLockAfterRetainedIdleactive-scope cleanup test so the fallbackacquireLockrejects withSessionWriteLockTimeoutError(same-pid lock-timeout branch), and the test assertsEmbeddedAttemptSessionTakeoverErrorfrom scope completion.Verification
pnpm test src/agents/embedded-agent-runner/run/attempt.session-lock.test.ts— 108 passed (0 regressions)pnpm check:test-types— passedpnpm build— succeeded, binary matches HEAD (cc37eda)Real behavior proof
Behavior addressed: Session write-lock abort cleanup during an active retained session write —
releaseHeldLockWithFencebails out when scope is active, deferred release re-attempted after scope deactivation, preventing lock leaks until the 300s watchdog.Real environment tested: Linux x86_64 (kernel 4.19.112-2.el8), Node v24.13.1, branch
fix/issue-95915-session-lock-abort, commitcc37edaac6, binaryOpenClaw 2026.6.9 (cc37eda)Exact steps or command run after this patch:
Evidence after fix:
Runtime abort during active retained write — real embedded agent turn with forced timeout, exercising the abort cleanup path during an active model call + session transcript write:
Post-abort lock verification:
SessionWriteLockTimeoutError,lock leak,releasing lock held forafter abort run — zero hits/home/0668000452/.openclaw/agents/main_assistant/sessions/searched for stale.jsonl.lockfiles — zero stale lock filesreleaseHeldLockForAbort→releaseHeldLockWithFence→ deferred release) completed silently without leaving a leaked lockRuntime log evidence — no lock errors post-abort:
Comparison with the original bug (#95915):
released=0on abort.jsonl.lockpersists after abortSessionWriteLockTimeoutErrorafter abortObserved result after fix: The embedded agent run was aborted (
aborted=true) during an active model call (streaming in progress, status=200). The timeout fired at 3s while the write lock scope was active. The deferred release mechanism released the held lock at scope deactivation — no stale.jsonl.lockfiles remain, no lock timeout errors in runtime logs, and subsequent agent turns on the same session work withoutSessionWriteLockTimeoutError.What was not tested: Multi-channel production deployment with stuck-session recovery firing at
stuckSessionAbortMs(requires gateway with live channels and concurrent sessions that are beyond what this contributor environment has). The deferred-release mechanism is validated via: (1) this real timeout-abort proof showing clean lock teardown afteraborted=true, (2) 108 controller regression tests covering the 3 key abort-during-scope scenarios, (3) compiled dist code verification confirming 4releaseHeldLockDeferredreferences in the production build.