Skip to content

fix(session-lock): allow reentrant acquire from inner transcript writers#96000

Closed
t2wei wants to merge 1 commit into
openclaw:mainfrom
t2wei:fix/session-lock-reentrant-inner-writers
Closed

fix(session-lock): allow reentrant acquire from inner transcript writers#96000
t2wei wants to merge 1 commit into
openclaw:mainfrom
t2wei:fix/session-lock-reentrant-inner-writers

Conversation

@t2wei

@t2wei t2wei commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

What Problem This Solves

Three mid-attempt transcript writers — compact.ts, tool-result-truncation.ts, transcript-rewrite.ts — acquire the session write lock without allowReentrant: true. The outer embedded-attempt lock (acquired by createEmbeddedAttemptSessionLockController) is non-reentrant by design, so same-pid contention deadlocks for 60 seconds when one of the inner writers is invoked mid-turn.

Why This Change Was Made

Reliable repro: in a long-running session where the agent produces a large tool result that triggers context overflow precheck in run.ts, the precheck calls truncateOversizedToolResultsInSession or compactEmbeddedAgentSessionDirect while the attempt still holds the outer lock. The inner writer hits the file lock manager, sees the lock held by the same PID, and waits the full `DEFAULT_SESSION_WRITE_LOCK_ACQUIRE_TIMEOUT_MS` (60s) before throwing.

src/config/sessions/transcript-append.ts already uses `allowReentrant: true` for the same reason (writes session events during the attempt's turn). This PR extends the same fix to the other three inner writers that share the same locking pattern.

User Impact

Without this fix, agents running parallel sessions_spawn with substantial tool outputs hit `SessionWriteLockTimeoutError` mid-turn:

```text
[diagnostic] lane task error: lane=main durationMs=~290s
error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms):
pid= alive=true ageMs=~60s
/path/.jsonl.lock"
Embedded agent failed before reply: session file locked ...
```

The agent fails to deliver its reply (the followup reply path falls back to other channels), turn is wasted, and the user sees a stalled response after ~5 minutes.

Evidence

  • Production journal from internal deploy showing the lock contention before the fix (60s timeout, alive=true, same gateway PID).
  • After fix deployed to dev environment: same parallel-spawn scenario completes normally, no SessionWriteLockTimeoutError events in the inner-writer path.
  • pnpm tsgo passes locally.
  • Same fix pattern already exists in transcript-append.ts:325 (`allowReentrant: true` with a comment explaining the same reasoning).

Test plan

  • Local typecheck (pnpm tsgo)
  • Manual scenario: parallel sessions_spawn with large tool results — no longer deadlocks
  • Reviewer can verify by injecting an artificial delay in any inner writer + outer attempt and confirming the outer doesn't time out

Existing tests in src/agents/session-write-lock.test.ts already cover reentrant acquire semantics (lines around 192/197/221/226), so the lock primitive itself is exercised.

🤖 AI-assisted PR. Diagnosis + patch produced with Claude Code; locked behavior verified against production journals.

Three mid-attempt transcript writers (compact, tool-result-truncation,
transcript-rewrite) acquired the session write lock without
`allowReentrant: true`. The outer embedded-attempt lock (acquired by
`createEmbeddedAttemptSessionLockController`) is non-reentrant by design,
so same-pid contention deadlocks for 60s when one of the inner writers
is invoked mid-turn.

The deadlock fires reliably in scenarios where a large tool result
triggers context overflow precheck in run.ts, which calls
`truncateOversizedToolResultsInSession` or compaction while the attempt
still holds the outer lock. Symptom in the journal:

```
[diagnostic] lane task error: lane=main durationMs=~290s
error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms):
       pid=<same gateway pid> alive=true ageMs=~60s
       /path/<session>.jsonl.lock"
Embedded agent failed before reply: session file locked ...
```

`transcript-append` already uses `allowReentrant: true` for the same
reason — applying the same fix to the other three inner writers.
@clawsweeper

clawsweeper Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 30, 2026, 5:46 PM ET / 21:46 UTC.

Summary
Adds allowReentrant: true to the embedded compaction, oversized tool-result truncation, and runtime transcript rewrite session write-lock acquisitions.

PR surface: Source +13. Total +13 across 3 files.

Reproducibility: yes. at source level. Current main holds the embedded attempt session lock without reentry, the three inner writer paths still acquire the same session lock without allowReentrant, and the fs-safe lock manager only re-enters when that option is explicitly true.

Review metrics: 1 noteworthy metric.

  • Session lock opt-ins: 3 added, 0 removed. Each added reentrant acquire changes a session-state concurrency boundary that maintainers should review before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦐 gold shrimp
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Update the transcript rewrite lock-options assertion or add focused coverage for allowReentrant: true.
  • [P1] Add concrete redacted after-fix terminal/log output for the reported parallel-spawn or overflow path.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR needs concrete redacted after-fix terminal/log output or a linked artifact showing the repaired scenario completing; redact private data, update the PR body for re-review, or ask a maintainer to comment @clawsweeper re-review.

Risk before merge

  • [P1] checks-node-compact-large-whole-1 is currently failing because the transcript rewrite mock expectation was not updated for the new lock option.
  • [P1] The PR body has before-error logs and a qualitative after-fix claim, but no concrete redacted after-fix terminal output, log output, or linked artifact showing the fixed scenario completing.
  • [P1] Changing three transcript lock acquisitions to reenter a same-process held lock is session-state sensitive; maintainers need confidence that all three sites are truly the same logical writer boundary.
  • [P1] The open proof-positive compaction-only PR covers only one of the three call sites, so maintainers should avoid treating that narrower PR and this broader PR as interchangeable without an explicit split decision.

Maintainer options:

  1. Repair Tests And Proof Before Merge (recommended)
    Update the transcript rewrite lock expectation or add focused coverage, then attach redacted after-fix terminal/log proof showing the reported parallel-spawn or overflow path completes without SessionWriteLockTimeoutError.
  2. Land Narrow Compaction First
    If maintainers only want the proof-positive direct compaction fix now, land or adapt fix(agents): scope compaction lock reentry to logical writer #88919 and split the other two inner writers into a separately proven follow-up.
  3. Accept The Broader Reentry Boundary
    Maintainers can intentionally accept the broader three-site session-lock reentry risk after owner review confirms each path is a same-process logical writer.

Next step before merge

  • [P1] Needs contributor or maintainer action for the failing test expectation, proof gap, and broader same-logical-writer session-lock boundary review; no repair lane is selected because the key proof gap is contributor-owned.

Security
Cleared: No concrete security or supply-chain concern was found; the diff only changes TypeScript session-lock options and comments.

Review findings

  • [P2] Update the lock-call test expectation — src/agents/embedded-agent-runner/transcript-rewrite.ts:450
Review details

Best possible solution:

Keep the broad allowReentrant approach only if maintainers accept all three inner writers as same-logical-writer paths, update the focused test expectation or coverage, and add concrete redacted after-fix proof for the reported scenario before merge.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level. Current main holds the embedded attempt session lock without reentry, the three inner writer paths still acquire the same session lock without allowReentrant, and the fs-safe lock manager only re-enters when that option is explicitly true.

Is this the best way to solve the issue?

Mostly yes, but not merge-ready as submitted. Adding allowReentrant at the inner writer lock calls is the narrow runtime fix, but the PR still needs the failing test expectation updated and concrete after-fix proof for the broader three-site claim.

Full review comments:

  • [P2] Update the lock-call test expectation — src/agents/embedded-agent-runner/transcript-rewrite.ts:450
    This new field is intentional, but rewriteTranscriptEntriesInRuntimeTranscript still has an exact mock assertion for the lock options without allowReentrant: true. The PR's checks-node-compact-large-whole-1 check is failing, so update that assertion or add focused coverage for the new lock contract before merge.
    Confidence: 0.98

Overall correctness: patch is incorrect
Overall confidence: 0.91

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 3811001d2783.

Label changes

Label justifications:

  • P1: The PR addresses a reported real embedded-agent workflow failure where session lock contention can stall or lose a reply mid-turn.
  • merge-risk: 🚨 session-state: The diff changes transcript session-lock reentry behavior for three inner writer paths, so a wrong same-writer assumption could affect session mutation ordering.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR needs concrete redacted after-fix terminal/log output or a linked artifact showing the repaired scenario completing; redact private data, update the PR body for re-review, or ask a maintainer to comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +13. Total +13 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 3 13 0 +13
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 13 0 +13

What I checked:

Likely related people:

  • steipete: Recent merged session-lock hardening and active-compaction retry work is adjacent to the same session-lock contention path. (role: recent area contributor and adjacent fix author; confidence: high; commits: 210adf1d1189, 7ca77124fea0, 9a571399bb7c; files: src/agents/session-write-lock.ts, src/agents/embedded-agent-runner/run/compaction-retry-aggregate-timeout.ts)
  • jalehman: Authored and merged recent embedded run/session target work that touched compact.ts and run.ts, directly adjacent to this lock path. (role: recent embedded runner contributor; confidence: medium; commits: 6589f6386dff, 0dfa22c6e0fc; files: src/agents/embedded-agent-runner/compact.ts, src/agents/embedded-agent-runner/run.ts)
  • snowzlm: Recent prompt-released session state work touched the outer attempt session-lock coordination path used by this nested-lock scenario. (role: recent session-lock coordination contributor; confidence: medium; commits: 1a002c2d9db7; files: src/agents/embedded-agent-runner/run/attempt.session-lock.ts, src/agents/embedded-agent-runner/compact.ts)
  • amknight: Recent current-main history added compaction model-call tracing in the direct compaction module that this PR changes. (role: recent compaction contributor; confidence: medium; commits: 5a7857dc18c5; files: src/agents/embedded-agent-runner/compact.ts)
  • plexustech2006: Authored the open proof-positive compaction-only reentrant-lock PR, so they have directly relevant context for one overlapping part of this fix family. (role: adjacent candidate fix author; confidence: medium; commits: df6bc96f40c0; files: src/agents/embedded-agent-runner/compact.ts, src/agents/embedded-agent-runner/compact-lock.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. labels Jun 23, 2026
@itanyplus

Copy link
Copy Markdown

Additional live reproduction from a Feishu direct session on 2026-06-29.

Environment:

  • OpenClaw app/package: 2026.6.10
  • Gateway: systemd user service, local loopback, running pid 346578
  • OS: Linux 7.0.0-22-generic x64
  • Node: 24.18.0 as reported by openclaw status
  • Channel/session surface: Feishu direct session (agent:main:feishu:direct:...)
  • The gateway was otherwise reachable when checked via openclaw status.

Observed user-facing error at 2026-06-29 14:46 GMT+8:

session file locked (timeout 60000ms): pid=346578 alive=true ageMs=119780 /home/itany/.openclaw/agents/main/sessions/6e8686fa-6ce1-42ef-be95-3ec6d4ab0bd4.jsonl.lock

Important detail: pid=346578 is the live gateway process itself from openclaw status, not a dead PID or orphaned lock owner.

A few minutes later the same lock path still existed. Its metadata was:

{
  "pid": 346578,
  "createdAt": "2026-06-29T06:51:09.800Z",
  "maxHoldMs": 300000,
  "starttime": 131306136
}

Operational context from openclaw status around the same time:

  • Gateway service: active/running, pid 346578
  • Main Feishu direct session active about 1 minute ago
  • Main TUI session active about 1 minute ago
  • Multiple recent spawn-child/subagent sessions were also present
  • The failing path was a normal Feishu user message before reply, surfacing as a 60s session write-lock timeout.

This looks like same-process/live-gateway session lock contention or unreleased/reentrant lock behavior rather than stale dead-PID cleanup.

This may be relevant to this PR because the observed owner is the same live gateway PID and the symptom is the exact 60s SessionWriteLockTimeoutError on a session JSONL lock. If the PR is intended to cover same-PID inner transcript writer contention, this Feishu reproduction is another recent field case to validate against.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Jul 16, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: XS stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants