Skip to content

fix: preserve subagent delivery after lock stalls#86540

Open
galiniliev wants to merge 1 commit into
openclaw:mainfrom
galiniliev:bug-035-036-subagent-delivery-locks
Open

fix: preserve subagent delivery after lock stalls#86540
galiniliev wants to merge 1 commit into
openclaw:mainfrom
galiniliev:bug-035-036-subagent-delivery-locks

Conversation

@galiniliev

@galiniliev galiniliev commented May 25, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Preserves suspended subagent final-delivery text into durable completion fallback state before expiring the outbox payload, so expiry is not a hard loss of the child result.
  • Distinguishes expiry after durable fallback preservation from true payload discard.
  • Adds session write-lock timeout owner diagnostics including liveness, starttime, current starttime, age, and stale reasons.
  • Out of scope: redesigning subagent scheduling or session JSONL lock ownership.

Linked context

Which issue does this close?

Closes #86537
Closes #86538

Which issues, PRs, or discussions are related?

Related #86537
Related #86538

Was this requested by a maintainer or owner?

Local bug evidence under /mnt/c/OpenClaw/bugs/BUG-035-subagent-terminal-reconciliation-times-out-then-discards-delivery and /mnt/c/OpenClaw/bugs/BUG-036-session-write-lock-timeouts-block-main-and-subagent-lanes.

Real behavior proof (required for external PRs)

  • Behavior addressed: Expired suspended subagent final delivery now keeps final text durably instead of dropping the only payload, and write-lock timeouts expose actionable owner diagnostics.
  • Real environment tested: Linux WSL2 dev checkout plus Azure Crabbox remote validation. Before proof used Azure Crabbox silver-prawn / cbx_e06ef86ae274; after proof used Azure Crabbox violet-lobster / cbx_3eda7a42d715.
  • Exact steps or command run after this patch: node scripts/run-vitest.mjs src/agents/subagent-registry.test.ts src/agents/session-write-lock.test.ts; git diff --check; .agents/skills/autoreview/scripts/autoreview --mode local; Azure Crabbox before/after focused regression proof.
  • Evidence after fix (screenshot, recording, terminal capture, console output, redacted runtime log, linked artifact, or copied live output):
RUN  v4.1.7 /home/galini/GitHub/worktrees/bug-035-036-subagent-delivery-locks

Test Files  4 passed (4)
     Tests  144 passed (144)
  Start at  15:01:23
  Duration  16.11s

autoreview clean: no accepted/actionable findings reported
overall: patch is correct (0.84)

Azure Crabbox after-fix evidence log:

Crabbox OpenClaw evidence
timestamp=20260525T214730Z
name=pr-86540-after-fix
crabbox_id=violet-lobster
exit_status=0

tests=
build=0
types=0
check=none
proof_command_count=1

CRABBOX_PHASE:context
CRABBOX_PHASE:bootstrap
CRABBOX_PHASE:install
CRABBOX_PHASE:evidence_1
AFTER_FIX_HEAD_SHA=f9c6fc7de291905233e7d5ca65201f6d299e00ab
AFTER_FIX_TEST_EXIT=0
AFTER_FIX_REGRESSION_PASSED=1
CRABBOX_PHASE:diff
CRABBOX_PHASE:done

run summary sync=30.982s command=1m2.528s total=1m33.578s sync_skipped=false exit=0 sync_steps=ssh:611ms,manifest:635ms,preflight:0s,fingerprint:2ms,fingerprint_remote:70ms,git_seed:19.231s,manifest_write:223ms,prune:67ms,rsync:8.012s,finalize:2.039s command_phases=user-command:2.86s,context:32ms,bootstrap:16.583s,install:10.482s,evidence_1:32.558s,diff:6ms,done:2ms
  • Observed result after fix: The subagent registry regression expires a suspended delivery and observes discardReason: "expired-after-durable-fallback" plus completion.fallbackResultText; the session lock regression observes timeout owner text with pid=, alive=true, starttime=, and currentStarttime=. Azure Crabbox after-fix proof passed the focused regression set at head f9c6fc7de291905233e7d5ca65201f6d299e00ab.
  • What was not tested: Live gateway retry with the original private sessions was not rerun.
  • Proof limitations or environment constraints: Azure Crabbox and local proof cover the durable state and timeout diagnostic seams; the original session keys and lock paths are private and redacted.
  • Before evidence (optional but encouraged):

Redacted original symptom evidence:

subagent wait timed out; deferring terminal state until session reconciliation
subagent suspended delivery discarded {"reason":"expired","runId":"[redacted run id]","childSessionKey":"[redacted child session key]","requesterSessionKey":"[redacted requester session key]"}
SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=[redacted pid] [redacted session lock path]

Azure Crabbox before-fix evidence log:

Crabbox OpenClaw evidence
timestamp=20260525T214615Z
name=pr-86540-before-regression
crabbox_id=silver-prawn
exit_status=0

tests=
build=0
types=0
check=none
proof_command_count=1

CRABBOX_PHASE:context
CRABBOX_PHASE:bootstrap
CRABBOX_PHASE:install
CRABBOX_PHASE:evidence_1
BEFORE_FIX_BASE_SHA=c4bce00727ec5f6a5b9c78e0ad34524f5dc1db54
BEFORE_FIX_TEST_EXIT=1
BEFORE_FIX_EXPECTED_REGRESSION=1
CRABBOX_PHASE:diff
CRABBOX_PHASE:done

run summary sync=1.005s command=16.214s total=18.025s sync_skipped=true exit=0 sync_steps=ssh:238ms,manifest:698ms,preflight:0s,fingerprint:2ms,fingerprint_remote:68ms command_phases=user-command:2.843s,context:41ms,bootstrap:646ms,install:390ms,evidence_1:12.284s,diff:4ms,done:3ms

Tests and validation

Which commands did you run?

  • node scripts/run-vitest.mjs src/agents/subagent-registry.test.ts src/agents/session-write-lock.test.ts
  • git diff --check
  • .agents/skills/autoreview/scripts/autoreview --mode local
  • Azure Crabbox before proof: silver-prawn / cbx_e06ef86ae274, base c4bce00727ec5f6a5b9c78e0ad34524f5dc1db54, observed BEFORE_FIX_TEST_EXIT=1 and BEFORE_FIX_EXPECTED_REGRESSION=1.
  • Azure Crabbox after proof: violet-lobster / cbx_3eda7a42d715, head f9c6fc7de291905233e7d5ca65201f6d299e00ab, observed AFTER_FIX_TEST_EXIT=0 and AFTER_FIX_REGRESSION_PASSED=1.

What regression coverage was added or updated?

  • src/agents/subagent-registry.test.ts now verifies durable fallback preservation on suspended final-delivery expiry.
  • src/agents/session-write-lock.test.ts now verifies timeout owner diagnostics include liveness/starttime details.

What failed before this fix, if known?

  • Redacted logs showed 25 suspended final deliveries discarded after repeated terminal reconciliation timeouts and 161 session write-lock timeout lines with only pid/path owner context.
  • Azure Crabbox before proof applied the focused regression tests to base c4bce00727ec5f6a5b9c78e0ad34524f5dc1db54 and observed the expected regression failure.

If no test was added, why not?

N/A

Risk checklist

Did user-visible behavior change? (Yes/No)

Yes

Did config, environment, or migration behavior change? (Yes/No)

No

Did security, auth, secrets, network, or tool execution behavior change? (Yes/No)

No

What is the highest-risk area?

Subagent final-delivery cleanup state.

How is that risk mitigated?

The delivery payload is still compacted on expiry, but final text is copied into the existing completion fallback field before the payload is removed; pressure-pruned deliveries remain unchanged.

Current review state

What is the next action?

Maintainer review and CI.

What is still waiting on author, maintainer, CI, or external proof?

Live replay of the original private sessions was not run. Required CI is currently blocked by an unrelated type error in extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.ts:872.

Which bot or reviewer comments were addressed?

Autoreview reported no accepted/actionable findings. Azure Crabbox before/after focused evidence was added to the PR description.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S maintainer Maintainer-authored PR labels May 25, 2026
@clawsweeper

clawsweeper Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed June 29, 2026, 8:36 AM ET / 12:36 UTC.

Summary
The PR preserves expired suspended subagent final-delivery text in completion fallback state, adds a distinct discard reason, expands session write-lock timeout diagnostics, and updates focused agent tests.

PR surface: Source +46, Tests +15. Total +61 across 5 files.

Reproducibility: yes. at source level: current main still clears completion fallback during suspended-delivery expiry while the recovery path reads that fallback text, and linked reports include repeated live lock/lost-delivery logs. I did not run the timing-sensitive live gateway race in this read-only review.

Review metrics: 1 noteworthy metric.

  • Persisted delivery state: 1 discard reason added; fallback text retained on expiry. The PR changes durable subagent recovery state, so maintainers need the rebased merge result and upgrade behavior in view before merge.

Stored data model
Persistent data-model change detected: serialized state: src/agents/session-write-lock.test.ts, serialized state: src/agents/session-write-lock.ts, unknown-data-model-change: src/agents/session-write-lock.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #86538
Summary: This PR is the open candidate fix for the canonical session write-lock timeout plus subagent completion delivery-loss cluster; related items overlap but do not safely replace it.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Resolve the dirty merge state against current main and rerun focused agent tests plus required CI.
  • [P1] Add redacted after-fix live gateway/session proof, runtime logs, or get an explicit maintainer proof waiver.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR has redacted before logs and focused local/Azure regression-test output, but the after-fix evidence is still test output rather than a live gateway/session replay or runtime log from the real contended requester path. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Mantis proof suggestion
A live transport transcript would materially help prove that stalled requester sessions still receive or can recover subagent final text after the core fix. A maintainer can ask Mantis to capture proof by posting this exact PR comment:

@openclaw-mantis telegram live proof: verify that subagent final text remains visible or recoverable after a requester session write-lock stall and suspended delivery expiry.

Risk before merge

  • [P1] The branch is currently dirty/conflicting against main, and current main has newer session-lock retry/stale-handling and subagent lifecycle work that the merge result must reconcile.
  • [P1] After-fix proof is focused local/Azure regression-test output rather than a live gateway/session replay or redacted runtime log from the contended requester path.
  • [P1] The PR changes durable subagent completion/discard state, so a careless rebase could preserve stale or wrong child output in requester recovery state.
  • [P1] Required checks currently have failures; the PR discussion says some may be unrelated, but landing still needs green checks or an explicit maintainer waiver.

Maintainer options:

  1. Rebase and prove the requester path (recommended)
    Resolve the dirty merge state, rerun focused agent tests and required CI, then add redacted after-fix live gateway/session proof or an explicit maintainer waiver before merge.
  2. Accept focused proof by waiver
    Maintainers can intentionally accept the Azure before/after regression proof for this private-session failure mode, but that waiver should be explicit because it substitutes for live behavior proof.
  3. Split diagnostics if current main changed too much
    If current session-lock changes make the diagnostic half stale, keep a narrower durable-delivery repair and route lock diagnostics to the active lock cluster.

Next step before merge

  • [P1] The protected maintainer label, dirty merge state, live-proof waiver choice, and overlapping session-lock work require maintainer handling rather than an automated repair lane.

Security
Cleared: No concrete security or supply-chain concern found; the diff is limited to core agent session-state logic and focused tests.

Review details

Best possible solution:

Rebase onto current main, keep the durable fallback preservation if it still fits the current lifecycle code, reconcile the lock-diagnostic overlap, and land only after maintainers accept live proof or explicitly waive it.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main still clears completion fallback during suspended-delivery expiry while the recovery path reads that fallback text, and linked reports include repeated live lock/lost-delivery logs. I did not run the timing-sensitive live gateway race in this read-only review.

Is this the best way to solve the issue?

Mostly yes: preserving final text in the existing completion fallback is the narrow repair for the durable-loss path. The exact PR is not merge-ready until it is rebased across current lock changes and maintainers accept live proof or waive it.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 355c43fe0c9e.

Label changes

Label justifications:

  • P1: The PR targets active reports of subagent completion/message loss and session write-lock timeouts in real agent/channel workflows.
  • merge-risk: 🚨 session-state: The diff changes persisted subagent completion fallback and discard state, which can affect what requester sessions later recover or display.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR has redacted before logs and focused local/Azure regression-test output, but the after-fix evidence is still test output rather than a live gateway/session replay or runtime log from the real contended requester path. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +46, Tests +15. Total +61 across 5 files.

View PR surface stats
Area Files Added Removed Net
Source 3 52 6 +46
Tests 2 19 4 +15
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 5 71 10 +61

What I checked:

  • Repository policy read: Root and scoped agent review guidance were read and applied; the PR touches core agent/session-state behavior where whole-surface review, proof, and merge-risk handling matter. (AGENTS.md:1, 355c43fe0c9e)
  • Scoped agent policy read: Scoped src/agents guidance was read; it is relevant because the touched tests and runtime paths live under src/agents. (src/agents/AGENTS.md:1, 355c43fe0c9e)
  • Current main still clears fallback on expiry: The suspended-delivery expiry path removes the payload and clears completion.fallbackResultText/fallbackCapturedAt, so current main has not implemented the PR's central durable-preservation behavior. (src/agents/subagent-registry.ts:825, 355c43fe0c9e)
  • Fallback text is a real recovery source: Subagent completion rendering reads completion.fallbackResultText after the primary payload result, so preserving the field affects requester-visible recovery rather than unused state. (src/agents/subagent-announce-output.ts:363, 355c43fe0c9e)
  • Current main diagnostics are narrower: Current timeout-owner formatting emits pid/alive/ageMs but not the PR's starttime/currentStarttime/stale-reason detail. (src/agents/session-write-lock.ts:729, 355c43fe0c9e)
  • PR implements the narrow fallback change: The PR diff copies suspended delivery payload text into completion fallback state before expiry compaction and adds expired-after-durable-fallback as the distinguishable discard reason. (src/agents/subagent-registry.ts:704, f9c6fc7de291)

Likely related people:

  • steipete: GitHub history shows repeated recent work on subagent delivery and session-lock paths, including split delivery state, steering queue handoffs, and transient stale-lock retry behavior. (role: recent area contributor and feature-history owner; confidence: high; commits: d73f3ac85d5f, f24a13879095, 210adf1d1189; files: src/agents/subagent-registry.ts, src/agents/session-write-lock.ts, src/agents/subagent-registry-run-manager.ts)
  • vincentkoc: Recent GitHub history shows work in the same agent/subagent area, including subagent registry caching and runtime guard exports that affect nearby code boundaries. (role: recent adjacent contributor; confidence: medium; commits: c4a0ca0b7a41, d1ea170c9b55; files: src/agents/subagent-registry.ts, src/agents/session-write-lock.ts, src/agents/subagent-announce-output.ts)
  • galiniliev: Beyond authoring this PR, recent main history includes adjacent subagent wake work by the same contributor, so they are a practical routing candidate for this repair branch. (role: active linked repair and recent adjacent contributor; confidence: high; commits: f9c6fc7de291, 6883c6c070f6; files: src/agents/subagent-registry.ts, src/agents/subagent-registry.test.ts, src/agents/session-write-lock.ts)
  • giodl73-repo: Recent session-lock doctor work touches the same owner-diagnostic area, though less directly than the subagent delivery path. (role: recent adjacent session-lock contributor; confidence: medium; commits: 704fc3504392; files: src/agents/session-write-lock.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (1 earlier review cycle)
  • reviewed 2026-06-24T18:43:49.036Z sha f9c6fc7 :: needs real behavior proof before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels May 25, 2026
@clawsweeper

clawsweeper Bot commented May 25, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@omarshahine

Copy link
Copy Markdown
Contributor

Thanks for the PR. Before this can move forward, please add live proof from the affected surface, not just unit tests, mocked tests, or source inspection.

A useful proof update should include:

  • the exact build/SHA tested
  • the real environment used
  • the command, UI flow, channel flow, provider request, or other live path exercised
  • before/after symptom evidence where applicable
  • the observed result after the patch
  • any remaining proof gaps

Please redact secrets, tokens, phone numbers, and private message content from logs or screenshots.

@galiniliev galiniliev self-assigned this May 25, 2026

@martingarramon martingarramon left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at f9c6fc7d.

Two distinct changes in this PR:

Lock timeout diagnostics (session-write-lock.ts): formatSessionLockTimeoutOwner assembles pid + liveness + starttime + age into the timeout error. The pinCurrentProcessStartTimeForTest addition in the test is needed to make starttime assertions deterministic. ✓

Durable fallback preservation (subagent-registry.ts): before expiring a suspended delivery, checks frozenResultText ?? fallbackFrozenResultText and persists it into completion.fallbackResultText if present. discardReason distinguishes "expired-after-durable-fallback" from "expired". The type in subagent-registry.types.ts is updated to match. ✓

Three CI failures need resolution before merge:

  • check-prod-types and check-additional-extension-package-boundary — likely the new fallbackResultText / fallbackCapturedAt fields on the completion type are not exported through the package boundary; verify src/agents/subagent-registry.types.ts re-exports reach the extension bundle entry.
  • check-test-types — if a new field is referenced in a test that the type declaration doesn't yet include.

LGTM pending CI.

@galiniliev

Copy link
Copy Markdown
Contributor Author

Maintainer prep update for f9c6fc7de291905233e7d5ca65201f6d299e00ab.

Evidence collected

  • Azure Crabbox before proof: silver-prawn / cbx_e06ef86ae274, base c4bce00727ec5f6a5b9c78e0ad34524f5dc1db54, applied the PR regression tests to the base, observed BEFORE_FIX_TEST_EXIT=1 and BEFORE_FIX_EXPECTED_REGRESSION=1.
  • Azure Crabbox after proof: violet-lobster / cbx_3eda7a42d715, head f9c6fc7de291905233e7d5ca65201f6d299e00ab, ran pnpm test src/agents/subagent-registry.test.ts src/agents/session-write-lock.test.ts, observed AFTER_FIX_TEST_EXIT=0 and AFTER_FIX_REGRESSION_PASSED=1.
  • Local focused proof also passed via node scripts/run-vitest.mjs src/agents/subagent-registry.test.ts src/agents/session-write-lock.test.ts.
  • Autoreview was clean: no accepted/actionable findings, overall patch-correct confidence 0.84.

Review result
No blocking code findings from maintainer review. I do not see a feature drop or regression in the PR itself: the change preserves durable suspended subagent fallback text before compacting expired payloads, and the lock timeout diagnostic expansion stays scoped to owner/liveness/starttime/staleness metadata.

Current blocker
CI was rerun at https://github.com/openclaw/openclaw/actions/runs/26407542199 and still has three failed jobs: check-prod-types, check-test-types, and check-additional-extension-package-boundary. The shared failure is an unrelated type error in extensions/qa-lab/src/live-transports/whatsapp/whatsapp-live.runtime.ts:872 where rttMeasurement.source is inferred as string instead of the literal "request-to-observed-message". This PR only touches src/agents/session-write-lock* and src/agents/subagent-registry*.

I would not call this fully land-ready while required CI is red. The PR-specific code/test evidence is clean; remaining decision is whether to fix/override the unrelated qa-lab CI blocker and whether the targeted Azure before/after proof is enough versus requiring a live gateway/session replay.

@BingqingLyu

This comment was marked as spam.

@itanyplus

Copy link
Copy Markdown

Cross-posting fresh Feishu production evidence because this thread is related. Context for this thread: Open PR touching subagent delivery after lock stalls; adding production evidence for validation.

Additional production evidence from Feishu DM lane (2026-06-26)

I'm adding fresh production evidence from a Feishu direct-message session on OpenClaw 2026.6.10.

Environment

  • OpenClaw app: 2026.6.10
  • OS: Linux 7.0.0-22-generic x64
  • Node: 24.17.0
  • Channel: Feishu DM / p2p
  • Session model override: kuaipao-gpt/gpt-5.5
  • Workload pattern: serial sessions_spawn child runs for paper-reading tasks, with completion announcements back to the Feishu requester session
  • Session state at the time: ~540 active sessions; Feishu requester session around 70k/268k context; multiple recent spawn-child sessions

Summary

The Feishu lane repeatedly became unable to reliably receive normal user messages or subagent completion announcements while a long-running/compacting requester session was active. The failures were not isolated to one child run; they formed a cascade:

  1. Feishu DM dispatch enters an active/queued state.
  2. Child/subagent completion tries to announce back to the requester session.
  3. The requester session is busy or its prompt lock is released/reacquired during compaction/model work.
  4. Delivery fails via either SessionWriteLockTimeoutError, EmbeddedAttemptSessionTakeoverError, or gateway request timeout.
  5. Subagent completion retries eventually give up, so the user sees the workflow as "stalled" even though child output may already be written to disk.
  6. Feishu per-chat queue eventually hits the 300000ms cap and evicts work so later same-key messages can proceed.

Counts from today's Feishu-related gateway logs

From journalctl --user -u openclaw-gateway --since '2026-06-26 00:00:00', filtered for Feishu/session-lock/subagent/compaction terms:

  • SessionWriteLockTimeoutError: 12
  • Embedded agent failed before reply: 12
  • EmbeddedAttemptSessionTakeoverError: 13
  • Subagent completion direct announce failed: 38
  • Subagent announce give up: 8
  • Active requester session could not be woken: 24
  • Compaction timed out: 12
  • Full summarization failed: 9
  • per-chat task exceeded 300000ms cap: 5
  • lane wait exceeded: 45
  • long-running session: 2

Some early entries are provider/rate-limit failures, but the afternoon cluster below is specifically session lock / takeover / completion delivery related.

Representative timeline excerpt, redacted

2026-06-26T13:48:30+08:00 [diagnostic] lane task error: lane=main durationMs=60167 error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=124708 <session>.jsonl.lock"
2026-06-26T13:48:30+08:00 [diagnostic] lane task error: lane=session:agent:main:feishu:direct:<open-id> durationMs=60169 error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=124708 <session>.jsonl.lock"
2026-06-26T13:48:30+08:00 Embedded agent failed before reply: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=124708 <session>.jsonl.lock
2026-06-26T13:49:22+08:00 [agents/cli-compaction] CLI transcript compaction failed for kuaipao-gpt/gpt-5.5: Compaction timed out
2026-06-26T13:51:18+08:00 [diagnostic] lane task error: lane=main durationMs=168013 error="EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released: <session>.jsonl"
2026-06-26T13:52:04+08:00 [warn] Subagent completion direct announce failed for run <run-id>: gateway request timeout for agent
2026-06-26T13:52:06+08:00 [warn] Subagent completion direct announce failed for run <run-id>: EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released: <session>.jsonl
2026-06-26T13:52:08+08:00 [warn] Subagent announce give up (retry-limit) run=<run-id> child=agent:main:subagent:<child-id> requester=agent:main:feishu:direct:<open-id> retries=3 endedAgo=245s deliveryError="EmbeddedAttemptSessionTakeoverError..."

2026-06-26T14:15:04+08:00 [diagnostic] lane task error: lane=main durationMs=60138 error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=86720 <session>.jsonl.lock"
2026-06-26T14:16:19+08:00 [diagnostic] lane task error: lane=main durationMs=60139 error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=161689 <session>.jsonl.lock"
2026-06-26T14:16:34+08:00 [agents/cli-compaction] CLI transcript compaction failed for kuaipao-gpt/gpt-5.5: Compaction timed out
2026-06-26T14:21:02+08:00 [agents/cli-compaction] CLI transcript compaction failed for kuaipao-gpt/gpt-5.5: Compaction timed out

2026-06-26T14:32:44+08:00 [diagnostic] lane task error: lane=main durationMs=60146 error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=122137 <session>.jsonl.lock"
2026-06-26T14:33:28+08:00 [agents/cli-compaction] CLI transcript compaction failed for kuaipao-gpt/gpt-5.5: Compaction timed out
2026-06-26T14:35:16+08:00 [diagnostic] lane task error: lane=main durationMs=127727 error="EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released: <session>.jsonl"
2026-06-26T14:37:08+08:00 [warn] Subagent completion direct announce failed for run <run-id>: gateway request timeout for agent
2026-06-26T14:37:12+08:00 [warn] Subagent announce give up (retry-limit) run=<run-id> child=agent:main:subagent:<child-id> requester=agent:main:feishu:direct:<open-id> retries=3 endedAgo=246s deliveryError="EmbeddedAttemptSessionTakeoverError..."

2026-06-26T14:43:34+08:00 [diagnostic] lane task error: lane=main durationMs=60197 error="SessionWriteLockTimeoutError: session file locked (timeout 60000ms): pid=<gateway-pid> alive=true ageMs=125037 <session>.jsonl.lock"
2026-06-26T14:44:26+08:00 [agents/cli-compaction] CLI transcript compaction failed for kuaipao-gpt/gpt-5.5: Compaction timed out
2026-06-26T14:46:19+08:00 [diagnostic] lane task error: lane=main durationMs=111864 error="EmbeddedAttemptSessionTakeoverError: session file changed while embedded prompt lock was released: <session>.jsonl"
2026-06-26T14:46:21+08:00 [warn] Active requester session could not be woken for subagent completion; falling back to requester-agent handoff: active requester session could not be woken: queue_message_failed reason=source_reply_delivery_mode_mismatch sessionId=<session-id> gatewayHealth=live
2026-06-26T14:51:55+08:00 [diagnostic] long-running session: sessionId=<session-id> sessionKey=agent:main:feishu:direct:<open-id> state=processing age=130s queueDepth=1 reason=queued_behind_active_work classification=long_running activeWorkKind=model_call lastProgress=model_call:started lastProgressAge=67s recovery=none
2026-06-26T14:54:43+08:00 [feishu] feishu[default]: per-chat task exceeded 300000ms cap (key=feishu:default:<chat-id>); evicting from queue so later same-key messages can proceed (#70133)

Lock file state observed during the incident

The active Feishu requester session lock was owned by the live Gateway process, not an orphaned PID:

{
  "pid": <gateway-pid>,
  "createdAt": "2026-06-26T06:52:28.443Z",
  "maxHoldMs": 300000,
  "starttime": 131306136
}

The important mismatch remained: callers were failing after timeout 60000ms while the live lock could remain valid much longer (maxHoldMs: 300000 in this release; previously observed as 1020000 in older releases).

User-visible impact

The workflow was a serial paper-reading chain (P01..P20) where each child writes a note file and the parent should announce completion and spawn the next child. During the incident:

  • note files were written successfully through at least read_p11.md, so some child work completed;
  • the Feishu requester session still missed or delayed completion events;
  • several child completion announcements retried and gave up;
  • the human saw the workflow as stalled and had to repeatedly send manual nudges such as "child agent stopped again" / "why can't it continue automatically";
  • progress tracking lagged behind actual child output because the completion-to-parent handoff was unreliable.

Why this seems relevant here

This looks like the same family as the session write-lock / long-turn / subagent delivery lane issues, but today's evidence adds two details:

  1. On 2026.6.10, the lock maxHoldMs is now 300000, yet callers still fail at 60000ms, so the timeout mismatch still creates user-visible failures.
  2. Compaction timeouts and EmbeddedAttemptSessionTakeoverError appear to amplify the problem: once the requester session is busy/compacting, subagent completion delivery can fail repeatedly and eventually give up even though the child output exists.

If useful, I can provide a fuller redacted log bundle, but I avoided pasting full paths, Feishu IDs, or local project contents here.

@itanyplus

Copy link
Copy Markdown

Linking #96975 as related design context for this PR: preserving subagent delivery after lock stalls would be more robust if completion delivery is metadata-only by default rather than carrying large child payloads.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling maintainer Maintainer-authored PR merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Subagent terminal reconciliation can expire suspended final delivery [Bug]: Session write-lock timeouts block subagent delivery lanes

6 participants