Skip to content

fix(agents): scope compaction lock reentry to logical writer#88919

Merged
steipete merged 6 commits into
openclaw:mainfrom
plexustech2006:fix/preflight-compaction-reentrant-lock
Jul 16, 2026
Merged

fix(agents): scope compaction lock reentry to logical writer#88919
steipete merged 6 commits into
openclaw:mainfrom
plexustech2006:fix/preflight-compaction-reentrant-lock

Conversation

@plexustech2006

@plexustech2006 plexustech2006 commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Summary

  • let direct compaction reuse only the matching async-local transcript writer lock
  • keep unrelated same-process compaction serialized through the normal physical session lock
  • replace the source-text guard with behavior-level coverage for both lock paths

What Problem This Solves

Compaction can run inside an existing logical session writer. Reacquiring the non-reentrant physical lock there self-deadlocks until the 60-second timeout. A process-wide allowReentrant opt-in avoids the timeout but is too broad: unrelated async work in the same gateway could then overlap transcript mutations.

The final implementation adapts the existing matching withOwnedSessionTranscriptWrites context to the release-shaped lock used by direct compaction. No matching owner means the normal non-reentrant session lock remains mandatory.

User impact

Context-pressure recovery no longer self-deadlocks when compaction is genuinely nested in the active logical writer. Independent turns and cleanup remain serialized, avoiding transcript races or lost messages.

Evidence

The contributor reproduced the failure on a live Linux gateway in a Telegram topic whose transcript exceeded its context window. Before the fix, preflight compaction repeatedly failed with a same-PID session file locked (timeout 60000ms) error. After applying the original narrow hotfix, the exact topic completed successfully as TOPIC8_HOTFIX_OK, token usage fell from 144084 to 39386, and a lock-file sweep found no remaining *.jsonl.lock files.

That live proof establishes the self-lock failure and recovery behavior. Maintainer review tightened the implementation from process-wide reentry to logical-owner-scoped reuse.

Validation

  • node scripts/run-vitest.mjs src/agents/embedded-agent-runner/compact.hooks.test.ts src/config/sessions/transcript.test.ts src/agents/embedded-agent-runner/run/attempt.session-lock.test.ts src/agents/session-write-lock.test.ts
    • 4 files passed; 325 tests passed
  • git diff --check
    • passed
  • mandatory full-branch autoreview
    • clean; no accepted/actionable findings
  • exact-head GitHub CI
    • pending

Notes

  • No config, environment variable, UI, protocol, or capability surface changes.
  • Contributor credit preserved in maintainer commits and squash metadata.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 1, 2026
@clawsweeper

clawsweeper Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 29, 2026, 8:00 AM ET / 12:00 UTC.

Summary
The PR adds allowReentrant: true to direct embedded compaction's session write-lock acquisition and adds a small guard test for that lock call.

PR surface: Source +5, Tests +16. Total +21 across 2 files.

Reproducibility: yes. at source level. Current main routes overflow recovery into direct compaction, direct compaction acquires the session lock without allowReentrant, and the lock helper only re-enters when the option is explicitly true; I did not run the long live overflow scenario in this read-only review.

Review metrics: 1 noteworthy metric.

  • Session lock opt-ins: 1 added, 0 removed. The runtime change deliberately adds one reentrant transcript lock acquisition, which is the semantic boundary maintainers need to review before merge.

Stored data model
Persistent data-model change detected: serialized state: src/agents/embedded-agent-runner/compact-lock.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P2] Have a maintainer explicitly accept the direct compaction same-logical-writer boundary before merge.
  • Consider replacing the source-string guard with behavior-level nested-lock coverage if maintainers want stronger regression proof.

Risk before merge

Maintainer options:

  1. Accept Direct Compaction Reentry (recommended)
    Land the narrow opt-in if maintainers agree direct preflight and overflow compaction are part of the same logical session writer.
  2. Strengthen Lock Behavior Coverage
    Before merge, add a focused behavior test proving nested same-process direct compaction lock acquisition succeeds only for the intended same-writer path.
  3. Pause For Broader Lock Policy
    Hold this PR if maintainers want one owner-led decision covering direct compaction plus the other inner transcript writer sites.

Next step before merge

  • Maintainer review should decide the same-logical-writer boundary and whether to land this narrow PR versus waiting for broader inner-writer work.

Security
Cleared: No security or supply-chain concern was found; the diff only changes a TypeScript session-lock option and adds a focused test.

Review details

Best possible solution:

Land the narrow direct-compaction reentrant lock opt-in once maintainers accept this path as the same logical transcript writer, and keep broader inner-writer reentry review separate.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main routes overflow recovery into direct compaction, direct compaction acquires the session lock without allowReentrant, and the lock helper only re-enters when the option is explicitly true; I did not run the long live overflow scenario in this read-only review.

Is this the best way to solve the issue?

Yes, conditionally. The runtime change is the narrowest immediate fix for direct compaction self-locking, but maintainers still need to accept that this direct compaction path is the same logical transcript writer before merge.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against db2488b6e3d6.

Label changes

Label justifications:

  • P1: The PR addresses a live broken agent/channel workflow where required compaction near the context ceiling can fail with session lock contention.
  • merge-risk: 🚨 session-state: The diff changes transcript session-lock reentry behavior for direct compaction, so a wrong same-writer assumption could affect session mutation ordering.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body supplies live before-fix lock failure output, after-fix exact-topic success output, and a lock sweep from the affected environment.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body supplies live before-fix lock failure output, after-fix exact-topic success output, and a lock sweep from the affected environment.
Evidence reviewed

PR surface:

Source +5, Tests +16. Total +21 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 5 0 +5
Tests 1 16 0 +16
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 21 0 +21

Acceptance criteria:

  • [P1] node scripts/run-vitest.mjs src/agents/embedded-agent-runner/compact-lock.test.ts src/agents/session-write-lock.test.ts.
  • [P1] node scripts/run-vitest.mjs src/agents/embedded-agent-runner/compact.hooks.test.ts.

What I checked:

Likely related people:

  • steipete: Recent history shows repeated session-lock hardening and the merged active compaction retry fix adjacent to this lock-contention path. (role: recent area contributor and adjacent fix author; confidence: high; commits: 210adf1d1189, 7ca77124fea0, 9a571399bb7c; files: src/agents/session-write-lock.ts, src/agents/embedded-agent-runner/run/compaction-retry-aggregate-timeout.ts)
  • amknight: Recent current-main history added compaction model-call tracing in the same direct compaction module. (role: recent compaction contributor; confidence: medium; commits: 5a7857dc18c5; files: src/agents/embedded-agent-runner/compact.ts)
  • jalehman: Recent history includes embedded run session-target work in the same agent/session area that feeds the lock and compaction path. (role: recent embedded runner contributor; confidence: medium; commits: 0dfa22c6e0fc; files: src/agents/embedded-agent-runner/compact.ts, src/agents/embedded-agent-runner/run.ts)
  • snowzlm: Recent prompt-released session state work touched nested owned writes and session coordination adjacent to this same-process lock boundary. (role: recent session-state contributor; confidence: medium; commits: 1a002c2d9db7; files: src/agents/embedded-agent-runner/compact.ts, src/agents/embedded-agent-runner/run/attempt.session-lock.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@plexustech2006
plexustech2006 force-pushed the fix/preflight-compaction-reentrant-lock branch 2 times, most recently from 491de48 to df6bc96 Compare June 1, 2026 04:59
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 1, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 1, 2026
@zheng960121-oss

Copy link
Copy Markdown

👍 Confirming + widening the scope (Feishu channel, 6.5.0)

Hi @plexustech2006 — I hit the same root cause on 2026-06-11 in the Feishu channel (not just Telegram), so I want to confirm this is a broader issue and offer a slightly wider surface for the fix.

Repro context

  • Version: OpenClaw 2026.6.5 (commit 5181e4f), installed via npm install -g openclaw
  • Channel: Feishu DM (default), one long-lived session agent:main:feishu:direct:<open_id>
  • Workload: long-running sub-agent orchestration — main agent frequently spawns child agents that write to the same parent session file
  • Symptom: ~33 occurrences of SessionWriteLockTimeoutError ... session file locked (timeout=60000ms): pid=<gateway> alive=true in 24h, correlated with eventLoopDelayP99Ms=14000+ and gateway CPU at 100% spinning on the lock wait

Wider surface than just preflight compaction

While digging, I found the bug is not isolated to preflight compaction — proper-lockfile is invoked in 11 places in the bundled dist, all without allowReentrant: true:

dist/selection-But6hGR0.js        (3 self-calls in createEmbeddedAttemptSessionLockController!)
dist/attempt-execution-CgTGShuY.js
dist/run-attempt-DXkrH0fF.js
dist/transcript-hciCrqol.js
dist/transcript-rewrite-BCPGHqBo.js
dist/docker-Dxn37PTc.js
dist/tool-result-truncation-CSLN3t54.js
dist/compact-Dn5LqRdR.js
+ 3 more

selection-But6hGR0.js:6091-6097 is the most interesting one — createEmbeddedAttemptSessionLockController calls acquireLock() 3 times in the same function (heldLock = await acquireLock() on L6097, then acquireLock() again on L6141/L6310/L6359) before releasing the first lock. This is the textbook non-reentrant self-deadlock.

Suggestion

The fix may be better expressed as one of:

  1. Default the wrapper to allowReentrant: true for in-process same-file acquisitions (matches the "logical writer" pattern), and only force false at the cross-process / IPC boundary
  2. Or add an opt-in allowReentrant on all 11 call sites that match the same-process-writer pattern, plus a regression test asserting no site silently drops it

I just restarted the gateway (launchctl kickstart -k) to clear the immediate symptoms and will watch 6.5.0 closely. Happy to test this PR on a Feishu DM session if it helps unblock maintainer review — I can repro the deadlock reliably in under 30 min by triggering a slow sub-agent dispatch.

— 🛠️ Self-hosted on macOS 26.5.1, gateway pid 32929, fresh after restart

@dexiosmb

Copy link
Copy Markdown

Behavioral proof from live production trajectory (embedded-agent-runner path, Claude Sonnet 4.6)

Adding observed evidence that supports the fix here, plus a priority note for maintainers.

What the trajectory shows:
During a session that drifted to overflow (June 17, trajectory 91f28c2c), the gateway log recorded 5 overflow compaction attempts. Cross-referencing the trajectory file against the gateway log confirms the alternating lock-contention pattern:

• Overflow #1 (15:20:09 EDT): no session.started in trajectory — lock held by the prior threshold compaction model call (still running). Gave up at 19:21:09.583Z, within 33ms of the threshold call's session.ended at 19:21:09.626Z.
• Overflow #2 (15:21:42): session.started + prompt.submitted → model.completed at 143s, aborted=False ✅ acquired lock, ran.
• Overflow #3 (15:23:05): no session.started — lock held by #2's still-running model call. Gave up within ~33ms of #2's session.ended.
• Overflow #4 (15:25:01): session.started + prompt.submitted → model.completed at 187s, aborted=False ✅ acquired lock, ran.
• Overflow #5 (15:27:08): no session.started — lock held by #4's still-running model call.

This is live behavioral confirmation of the same-process lock contention the PR addresses. The alternating pattern (every odd attempt hits contention, every even attempt gets through) directly results from the retry interval (~60s) being shorter than the model call duration (~143–187s). ClawSweeper noted "a behavior-level regression test would be stronger than the added source-string guard" — this is that behavioral data from production.

Priority note for maintainers:
The allowReentrant: true fix here is correct and worth landing. However, it's downstream of a separate primary bug: COMPACTION_RETRY_AGGREGATE_TIMEOUT_MS = 6e4 in selection-Cy8dzQeR.js — the embedded session's outer wait — gives up after 60s regardless of call status. When that fires, the abandoned-but-still-running model call holds (or re-acquires) the lock for another 80–130s, and the next retry starts into contention. Fix the aggregate wait and the retry storm stops; the contention becomes rare rather than structural. The two fixes are independent but this one is secondary.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 14, 2026
@steipete steipete self-assigned this Jul 16, 2026
@steipete
steipete force-pushed the fix/preflight-compaction-reentrant-lock branch from df6bc96 to 6343a47 Compare July 16, 2026 21:31
@steipete
steipete force-pushed the fix/preflight-compaction-reentrant-lock branch from 6343a47 to 6ca6da9 Compare July 16, 2026 21:40
@steipete steipete changed the title fix: allow preflight compaction to reenter session locks fix(agents): scope compaction lock reentry to logical writer Jul 16, 2026
@openclaw-barnacle openclaw-barnacle Bot added triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. and removed triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 16, 2026
@steipete
steipete force-pushed the fix/preflight-compaction-reentrant-lock branch from 5734d2a to b3ccc7a Compare July 16, 2026 22:12
Plexus Technology and others added 6 commits July 16, 2026 23:35
Replace the source-text guard with behavior-level coverage through the compaction harness.

Co-authored-by: Plexus Technology <[email protected]>
Reuse only the matching logical transcript writer lock and keep unrelated same-process compaction serialized.

Co-authored-by: Plexus Technology <[email protected]>
@steipete
steipete force-pushed the fix/preflight-compaction-reentrant-lock branch from b3ccc7a to 300d409 Compare July 16, 2026 22:39
@steipete
steipete merged commit a434ee7 into openclaw:main Jul 16, 2026
118 of 136 checks passed
@steipete

Copy link
Copy Markdown
Contributor

Merged via squash.

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 17, 2026
…w#88919)

* fix: allow preflight compaction to reenter session locks

* chore: refresh proof checks

* test(agents): cover compaction lock reentry

Replace the source-text guard with behavior-level coverage through the compaction harness.

Co-authored-by: Plexus Technology <[email protected]>

* fix(agents): scope compaction lock reentry

Reuse only the matching logical transcript writer lock and keep unrelated same-process compaction serialized.

Co-authored-by: Plexus Technology <[email protected]>

* test(agents): preserve lock runner generics

Co-authored-by: Plexus Technology <[email protected]>

* test(agents): preserve generic lock wrapper type

Co-authored-by: Plexus Technology <[email protected]>

---------

Co-authored-by: Plexus Technology <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S stale Marked as stale due to inactivity status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants