Skip to content

fix: force-release session lock on dispose to prevent orphan locks#90419

Closed
longkunbetter wants to merge 1 commit into
openclaw:mainfrom
longkunbetter:fix/session-lock-dispose-orphan
Closed

fix: force-release session lock on dispose to prevent orphan locks#90419
longkunbetter wants to merge 1 commit into
openclaw:mainfrom
longkunbetter:fix/session-lock-dispose-orphan

Conversation

@longkunbetter

Copy link
Copy Markdown

Problem

When an embedded attempt (sub-agent) is torn down via dispose(), the
disposeHeldLockAfterRetainedIdle() function silently returns without
releasing the session write lock if a write operation is still in progress
(retainedLockUseCount > 0 with activeWriteLock store active).

The same silent-return issue exists in releaseHeldLockWithFence(), used
by releaseHeldLockForAbort().

This leaves orphaned .jsonl.lock files on disk that block subsequent
sub-agents from writing to the same session file. Because the lock's PID
matches the still-alive gateway process and the in-memory state.held
map still tracks the entry, the contention handler does not reclaim it
(heldByThisProcess=trueshouldTreatAsOrphanSelfLock returns false).
The lock is neither reclaimed nor released until the watchdog force-releases
it after maxHoldMs (default 5 min), but the watchdog only checks in-memory
entries and the file lock can persist indefinitely in some scenarios.

Result: new sub-agents get a 60-second timeout with "session file locked"
errors.

Fix

In both disposeHeldLockAfterRetainedIdle() and
releaseHeldLockWithFence(), when waitForRetainedLockIdle() returns
false, force-release the lock (heldLock = undefined; lock.release())
instead of silently returning. Teardown and abort paths must never leak a
lock.

Files changed

src/agents/embedded-agent-runner/run/attempt.session-lock.ts — two
functions modified, 16 lines added.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 4, 2026
@clawsweeper

clawsweeper Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 21, 2026, 10:48 AM ET / 14:48 UTC.

Summary
The PR adds deferred retained-lock idle callbacks in two embedded attempt abort/dispose release paths so the captured session lock is released after active writes drain.

PR surface: Source +17. Total +17 across 1 file.

Reproducibility: yes. at source level: current main and v2026.6.9 return early when waitForRetainedLockIdle() is false in the release/dispose paths, matching the reported active-write lock leak. I did not run the timing-sensitive GUI rapid-switching reproduction in this read-only review.

Review metrics: 1 noteworthy metric.

  • Deferred Release Branches: 2 added, 0 tests added. Both added branches affect session-lock abort/dispose behavior while retained writes are active, so focused coverage matters before merge.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Preserve the normal fence handoff in the deferred release path instead of only calling the captured lock's release method.
  • [P1] Add focused regression coverage for active-write abort/dispose plus external takeover detection after deferred release.
  • [P1] Add redacted after-fix real behavior proof from rapid session switching or an equivalent real-component lock harness.

Proof guidance:

  • [P1] Needs real behavior proof before merge: No after-fix real behavior proof is present; the PR discussion provides reproduction analysis and before-fix logs only. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The deferred callback can release the held file lock without activating the session fence, so a later write can miss an external session-file takeover in the active-write interleaving this PR targets.
  • [P1] The PR has no after-fix real setup output showing rapid session switching, or an equivalent real-component lock harness, no longer leaves a lock behind.
  • [P1] Current main has newer session-lock fence and owned-write state than the PR base, so the final owner review should inspect the refreshed merge result.

Maintainer options:

  1. Preserve The Fence Before Merge (recommended)
    Update the deferred release path to run the same fingerprint, trust, and fence activation handoff as the normal release path, then cover the active-write abort/dispose interleaving.
  2. Pause Behind The Lock Cluster
    If owners want one canonical session-lock repair, pause this branch until the adjacent abort-bound and maxHoldMs reclaim PRs are either landed or explicitly split from this cleanup gap.
  3. Accept The Session-State Risk Explicitly
    Maintainers could intentionally take the availability improvement despite the fence gap, but that should be an explicit owner decision rather than an automated pass.

Next step before merge

  • [P1] Human review remains needed because the PR has a session-fence correctness blocker and the missing real behavior proof is contributor-gated.

Security
Cleared: No concrete security or supply-chain concern found; the diff is limited to internal session-lock runtime logic.

Review findings

  • [P1] Preserve the fence before deferred release — src/agents/embedded-agent-runner/run/attempt.session-lock.ts:805-808
Review details

Best possible solution:

Defer the same fence-preserving graceful release work until the retained write drains, add focused active-write abort/dispose regression coverage, and include redacted after-fix proof from the rapid session-switching flow or an equivalent real-component harness.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main and v2026.6.9 return early when waitForRetainedLockIdle() is false in the release/dispose paths, matching the reported active-write lock leak. I did not run the timing-sensitive GUI rapid-switching reproduction in this read-only review.

Is this the best way to solve the issue?

No, not as-is. Deferred release is the right shape for preserving write serialization, but this implementation needs to preserve the existing fence handoff rather than only calling release after idle.

Full review comments:

  • [P1] Preserve the fence before deferred release — src/agents/embedded-agent-runner/run/attempt.session-lock.ts:805-808
    This callback releases the retained held lock without running the same fingerprint/trust/fence activation block used by the normal release path. If abort or dispose is invoked from an active retained write, a later write can reacquire after an external append with fenceActive still false, so assertSessionFileFence() will not detect takeover. Defer the existing graceful release work until idle instead of only calling release().
    Confidence: 0.85

Overall correctness: patch is incorrect
Overall confidence: 0.86

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 0ad48dad2c47.

Label changes

Label justifications:

  • P2: The PR targets a real embedded-attempt session-lock reliability bug with limited but user-visible blast radius.
  • merge-risk: 🚨 session-state: The diff changes when a session write lock is released while retained transcript writes and session-fence state may still be pending.
  • merge-risk: 🚨 availability: A wrong release interleaving can keep later embedded attempts blocked on the session file lock or recover unsafely.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No after-fix real behavior proof is present; the PR discussion provides reproduction analysis and before-fix logs only. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +17. Total +17 across 1 file.

View PR surface stats
Area Files Added Removed Net
Source 1 17 0 +17
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 1 17 0 +17

What I checked:

Likely related people:

  • vincentkoc: Current blame covers the retained-lock wait and release/dispose functions, and recent GitHub history shows additional session-lock controller maintenance. (role: recent session-lock area contributor; confidence: high; commits: 66e5cfdd8617, d9c4f9a964ed; files: src/agents/embedded-agent-runner/run/attempt.session-lock.ts)
  • TurboTheTurtle: Authored recent merged fixes for retained-lock release after takeover and owned announcement session-write publication in the same controller/test surface. (role: adjacent session-lock contributor; confidence: medium; commits: 357462deef98, 5df0ed3b9f56; files: src/agents/embedded-agent-runner/run/attempt.session-lock.ts, src/agents/embedded-agent-runner/run/attempt.session-lock.test.ts)
  • steipete: Authored recent manual-abort release and stale session-lock policy work adjacent to this abort/dispose lock behavior. (role: recent abort/session-lock contributor; confidence: high; commits: 95890fe15063, 7ca77124fea0, 5f68291f4f54; files: src/agents/embedded-agent-runner/run/attempt-abort.ts, src/agents/embedded-agent-runner/run/attempt.session-lock.ts, src/agents/session-write-lock.ts)
  • openperf: Authored the merged timeout-abort session-lock release fix and an open adjacent maxHoldMs reclaim PR in the same lock family. (role: adjacent prior fix author; confidence: high; commits: 65fb56513fb2, 25db525072d7; files: src/agents/embedded-agent-runner/run/attempt.ts, src/agents/embedded-agent-runner/run/attempt.session-lock.ts, src/agents/session-write-lock.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@byungskers

Copy link
Copy Markdown

Nice catch on the teardown leak. One thing I wanted to sanity-check: in the waitForRetainedLockIdle() === false path we now force-release heldLock even though a retained write is still in flight, and the PR notes that "the in-flight write will refresh the fence after it completes." Is there any path where that writer can still assume exclusive ownership of the session file after another attempt has already reacquired the lock? If that invariant already holds elsewhere, it might be worth calling it out in a short code comment here, because this is exactly the kind of teardown fix that future readers may otherwise "simplify" back into a plain return.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 4, 2026
When disposeHeldLockAfterRetainedIdle or releaseHeldLockWithFence
are called from within the active retained-write context
(waitForRetainedLockIdle returns false), register a deferred
release callback on retainedLockIdleWaiters instead of silently
returning.

This preserves write serialization (no concurrent writes from
force-release) while ensuring the lock is eventually released
when the retained-use counter drains to zero.

Previously the early return leaked the lock in memory and on disk,
causing subsequent sub-agents to time out with "session file locked"
errors when the web GUI rapidly switches sessions.
@longkunbetter
longkunbetter force-pushed the fix/session-lock-dispose-orphan branch from 4a3de67 to 881a33b Compare June 4, 2026 17:17
@longkunbetter

Copy link
Copy Markdown
Author

Reproduction Analysis

Thanks for the thorough review. I've updated the fix to use deferred release via
retainedLockIdleWaiters instead of force-release, following the recommendation
to preserve write serialization.

Trigger scenario

The orphan-lock bug is triggered by rapid session switching in the web GUI:

  1. User opens Session A, starts a stock analysis → sub-agent spawns, acquires session
    write lock, begins writing results via withSessionWriteLock
  2. Inside the write callback, activeWriteLock AsyncLocalStorage is set to
    {active: true} and retainedLockUseCount is incremented
  3. User switches to Session B → Session A's active attempt is aborted
  4. The abort handler calls releaseHeldLockForAbort()releaseHeldLockWithFence()
  5. waitForRetainedLockIdle() checks:
    • retainedLockUseCount === 0false (write in flight)
    • activeWriteLock.getStore()?.active === truetrue (abort invoked within
      the write's AsyncLocalStorage context)
    • Returns false immediately (cannot wait, would deadlock)
  6. On current main: function silently returns; the lock stays in
    state.held (in-memory) and the .jsonl.lock file persists on disk
  7. User switches back to Session A → new sub-agent tries to write
  8. Lock contention handler checks shouldReclaim:
    • PID matches gateway (alive), heldByThisProcess is true (lock still in
      state.held) → shouldTreatAsOrphanSelfLock returns false
    • Lock is NOT reclaimed
  9. New sub-agent waits 60s (acquireTimeoutMs) → timeout → "session file locked" error

Evidence

The actual error from the reporter's environment:

Embedded agent failed before reply: session file locked (timeout 60000ms):
pid=39935 /root/.openclaw/agents/main/sessions/
09d3446a-5f05-42ba-8b46-116668fd232f.jsonl.lock
  • PID 39935 is the gateway process (still alive) — rules out dead-PID reclamation
  • The lock was 14 minutes old at failure time — rules out temporary contention
  • heldByThisProcess returns true → orphan-self-lock detection skipped
  • The maxHoldMs watchdog (runLockWatchdogCheck) only iterates
    SESSION_LOCKS.heldEntries() (in-memory), and the entry was present but the
    watchdog may not have force-released it before the next attempt raced in

Updated fix

Both disposeHeldLockAfterRetainedIdle and releaseHeldLockWithFence now
register a deferred release callback on retainedLockIdleWaiters when
waitForRetainedLockIdle() returns false. The callback fires when the
retained-use counter drains to zero (the in-flight write completes):

if (!(await waitForRetainedLockIdle())) {
    const pending = heldLock;
    retainedLockIdleWaiters.add(() => {
        if (heldLock === pending) heldLock = undefined;
        pending?.release().catch(() => {});
    });
    return;
}

This preserves the write serialization invariant (lock released AFTER write
completes, not during) while ensuring the lock is reliably released — avoiding
both the force-release race concern and the silent-leak bug.

@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 4, 2026
@clawsweeper clawsweeper Bot removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 20, 2026
@openclaw-barnacle openclaw-barnacle Bot added the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Jun 20, 2026
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jun 20, 2026

sallyom commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

Thanks for this contribution. This is superseded by the narrower retained-lock cleanup work that has now landed in #96100 (merge commit 0a042f6): releaseHeldLockWithFence() now defers an active-scope release and retries after the retained write unwinds, so the abort-side early-return leak this PR targeted is covered on main.

The remaining dispose/stuck-retained-user case is still a distinct follow-up and is tracked by #95833 / #95854, which should be reviewed against the post-#96100 code. Closing this PR to avoid landing an older overlapping implementation.

@sallyom sallyom closed this Jun 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: XS status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants