Skip to content

fix(agents): retry transient filesystem races when reading workspace bootstrap files#100910

Merged
openclaw-clownfish[bot] merged 2 commits into
openclaw:mainfrom
masatohoshino:workspace-bootstrap-read-retry
Jul 6, 2026
Merged

fix(agents): retry transient filesystem races when reading workspace bootstrap files#100910
openclaw-clownfish[bot] merged 2 commits into
openclaw:mainfrom
masatohoshino:workspace-bootstrap-read-retry

Conversation

@masatohoshino

Copy link
Copy Markdown
Contributor

What Problem This Solves

readWorkspaceFileWithGuards in src/agents/workspace.ts reads each workspace
bootstrap file (AGENTS.md, SOUL.md, TOOLS.md, IDENTITY.md, USER.md, HEARTBEAT.md,
BOOTSTRAP.md, MEMORY.md) with a single openRootFile + readFileSync(fd) and no
retry. On a transient filesystem race (EAGAIN/EWOULDBLOCK/EINTR under load)
the read throws — or openRootFile returns { ok: false, reason: "io" } — and the
file is reported missing. Because loadWorkspaceBootstrapFiles runs every agent
turn
(bootstrap-cache.ts, "Refresh per turn so long-lived sessions pick up
edits"), a transient race silently drops the agent's bootstrap content for that
turn.

This is the unmigrated sibling of the transient-race fix in
fileContentDiffersFromTemplate: that function already retries reads via the
retryAsync + isTransientWorkspaceReadError helpers defined in this same file,
but the guard was only wired into the onboarding template check, not the per-turn
bootstrap reader.

Why This Change Was Made

Wrap the reader's open+read in the existing in-file retryAsync
(isTransientWorkspaceReadError, 3 attempts, 50 ms) so a transient race is retried
instead of dropping the file. The retry re-opens on each attempt rather than
re-reading the same fd: retrying readFileSync on an already-partially-read fd
could return truncated content, whereas a fresh open gives a clean read and
openRootFile's inode-identity guard still rejects a file swapped between attempts.
Transient failures on the open itself (surfaced by openRootFile as
reason: "io") are retried too, so the fix is not one-sided within the function.

Behavior is otherwise unchanged: deterministic path/validation failures are not
retried, and exhausted transient retries still return { ok: false, reason: "io" }
(surfaced as missing) rather than throwing — preserving this reader's existing
contract (unlike the template check, which propagates).

User Impact

Long-lived agent sessions no longer intermittently lose their AGENTS/SOUL/TOOLS/
HEARTBEAT bootstrap content for a turn under filesystem load; the agent keeps its
identity and instructions across transient IO races.

Evidence

  • Three regression tests in src/agents/workspace.test.ts drive the real
    loadWorkspaceBootstrapFiles production path against a real temp workspace,
    injecting a transient EAGAIN at the readFileSync/openSync boundary:
    • a transient read on the first attempt is retried and the real content loads
      (not missing);
    • a transient open failure is retried and the content loads (not missing);
    • exhausted transient retries surface the file as missing without an uncaught
      throw.
  • Focused suite src/agents/workspace.test.ts green — vitest reports Tests 54 passed (54), including the three new cases above.
  • Core typecheck (tsgo:core, tsgo:core:test) clean; formatter and linter clean.

@clawsweeper

This comment has been minimized.

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P2 Normal backlog priority with limited blast radius. labels Jul 6, 2026
@vincentkoc

Copy link
Copy Markdown
Member

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@vincentkoc vincentkoc left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validated exact head 460d8714cf60241d5279e21c58e6a19105d602e7 against production loadWorkspaceBootstrapFiles on a real temporary filesystem with no JS mocks.

  • Native DYLD_INTERPOSE injected one real EAGAIN into read: this head retried and preserved the sentinel; pre-PR parent 2a741bebb54fcebeb92eb36ae237233451559aec returned missing.
  • The same proof for open passed on this head and failed on the parent.
  • Symlink and hardlink boundary probes remained rejected with no outside content exposed.
  • Atomic replacement stress: 2,000 writer iterations, 500 reads, 212 complete A, 277 complete B, 11 fail-closed missing snapshots, and 0 truncated or mixed snapshots.

The remaining fail-closed missing snapshots are an existing inode-swap validation race, separate from this PR’s transient EAGAIN retry fix. The requested filesystem-stress/setup proof is satisfied.

@vincentkoc

Copy link
Copy Markdown
Member

@clawsweeper re-review

@vincentkoc vincentkoc added proof: sufficient ClawSweeper judged the real behavior proof convincing. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Jul 6, 2026
@openclaw-clownfish
openclaw-clownfish Bot merged commit f36d170 into openclaw:main Jul 6, 2026
144 of 153 checks passed
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 7, 2026
…bootstrap files (openclaw#100910)

* fix(agents): retry transient filesystem races when reading workspace bootstrap files

* fix(agents): retry transient boundary resolution

---------

Co-authored-by: Vincent Koc <[email protected]>
giodl73-repo pushed a commit to giodl73-repo/openclaw that referenced this pull request Jul 8, 2026
…bootstrap files (openclaw#100910)

* fix(agents): retry transient filesystem races when reading workspace bootstrap files

* fix(agents): retry transient boundary resolution

---------

Co-authored-by: Vincent Koc <[email protected]>
RomneyDa added a commit that referenced this pull request Jul 22, 2026
* fix: gate diagnostics command to owners

(cherry picked from commit 170bf72)

* fix(agent): replace self-wait with deferred release in retained-lock abort cleanup (#96100)

* fix(agent): wait for retained session write before releasing held lock on abort

* fix(agent): replace self-wait with deferred release in retained-lock abort cleanup

* fix(test): reject fallback acquire with SessionWriteLockTimeoutError in active-scope cleanup test

* fix(agent): trim retained-lock comments

Signed-off-by: sallyom <[email protected]>

---------

Signed-off-by: sallyom <[email protected]>
Co-authored-by: sallyom <[email protected]>
(cherry picked from commit 0a042f6)

* fix(gateway): resume channel after pending task recovery

(cherry picked from commit 6039da3)

* fix(gateway): resume channel after pending task recovery

(cherry picked from commit ecd29fe)

* fix(outbound): ignore empty delivery receipts (#79811)

(cherry picked from commit 9a735be)

* fix(agents): guard delivery-evidence attachment recursion against cycles (#97041)

* fix(agents): guard delivery-evidence attachment recursion against cycles

* fix(agents): guard delivery-evidence attachment recursion against cycles

* fix(agents): guard delivery-evidence attachment recursion against cycles

---------

Co-authored-by: Pick-cat <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
(cherry picked from commit 4985671)

* fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort (#97128)

* fix(opencode-go): re-arm idle timer on block-boundary events to prevent false stalled-stream abort

When the opencode-go model finalizes a tool call and deliberates before
the next one, the provider emits real block-boundary SSE events
(text_end, thinking_end, toolcall_start, toolcall_end) that prove the
socket is alive, but the watchdog's isProviderProgressEvent only
returned true for token deltas (text_delta, thinking_delta,
toolcall_delta). This caused the idle timer to fire and falsely abort a
live stream, replacing a completed answer with a stalled error and
dropping the provider's real done event.

Fix: include block-boundary events in isProviderProgressEvent so the
idle timer is re-armed on any forward-progress provider event.
text_start and thinking_start are intentionally excluded because they
are synthetic preamble events that should not shorten the first-event
window.

Closes #96518

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test(opencode-go): satisfy lint in stream regression

* test(opencode-go): satisfy lint in stream regression

* test(opencode-go): satisfy lint in stream regression

---------

Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
(cherry picked from commit 552ec2b)

* fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations (#90908)

* fix(model-fallback): don't rethrow provider-side AbortErrors as user cancellations

When the LLM API closes the connection mid-stream, the fetch layer
surfaces AbortError("This operation was aborted") with no external
abort signal triggered. The old guard `shouldRethrowAbort()` returned
false for these errors (because isTimeoutError matched the message),
so they fell through to the fallback loop but were never retried —
the error propagated up and produced SILENT_REPLY_TOKEN in group
sessions, permanently silencing the topic.

Replace the guard with a direct check: only rethrow AbortError when
the external abort signal is actually set (user/gateway cancellation).
Provider-side AbortErrors without an external signal now fall through
to the next fallback candidate, giving the system a chance to recover.

* fix(cron): forward abort signal into runWithModelFallback

Thread the cron executor's abort signal into the shared
runWithModelFallback call so that cron timeouts and cancellations
stop the fallback chain instead of retrying with the next candidate.

Previously, the run callback checked params.abortSignal?.aborted and
threw, but runWithModelFallback itself had no signal — so the new
guard in model-fallback.ts could not distinguish a caller abort from
a provider-side AbortError and would retry silently.

Also adds a focused regression test verifying the signal is forwarded.

---------

Co-authored-by: Shengting Xie <[email protected]>
Co-authored-by: yayu <[email protected]>
(cherry picked from commit 98ed83f)

* fix(browser): block node routes when sandbox host control is disabled (#97958)

(cherry picked from commit 2cf765f)

* fix(exec): bind Windows allowlist execution path (#98260)

* fix(exec): bind windows allowlist execution path

* fix(exec): add windows shadow execution proof

* fix(exec): preserve wildcard allowlist behavior

* fix(exec): correct blocked plan test fixture

(cherry picked from commit 3811001)

* fix(mcp): suppress unhandled error on stderr pipe in stdio transport (#99803)

* fix(mcp): suppress unhandled error on stderr pipe in stdio transport

When child.stderr is piped to stderrStream without an error
handler, a stream-level error (EPIPE, I/O failure) crashes the
process. Add a noop error handler before the pipe, consistent
with the error handlers already present on stdin and stdout.

Co-Authored-By: Claude <[email protected]>

* test(mcp): add regression test for stderr pipe error suppression

Co-Authored-By: Claude <[email protected]>

* fix(mcp): report stderr stream errors

* fix(mcp): report stderr stream errors

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
(cherry picked from commit 1b84316)

* Harden macOS SQLite WAL checkpoints (#99067)

(cherry picked from commit f7f1be2)

* fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver (#100521)

* fix(secrets): suppress unhandled stdout/stderr stream errors in exec resolver

* proof(secrets): add real behavior proof script for exec resolver stream error catch

* proof(secrets): replace wrapper with real exec resolver stream error proof

* style: apply oxfmt to changed files

(cherry picked from commit c9a0783)

* fix(agents): retry transient filesystem races when reading workspace bootstrap files (#100910)

* fix(agents): retry transient filesystem races when reading workspace bootstrap files

* fix(agents): retry transient boundary resolution

---------

Co-authored-by: Vincent Koc <[email protected]>
(cherry picked from commit f36d170)

* fix(gateway): finish plugin HTTP responses after post-header failures (#102125)

* fix(gateway): finish plugin HTTP responses after post-header failures

* test(gateway): satisfy plugin HTTP regression lint

* fix(gateway): skip ending destroyed plugin responses

---------

Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit 240d350)

* fix(gateway): validate exact custom browser origins (#38290)

Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit fa0349a)

* fix: block unspecified trusted DNS targets (#103075)

(cherry picked from commit c70f3d0)

* fix(channels): make nack callbacks idempotent (#104919)

* fix(channels): make nack callbacks idempotent

* fix(channels): coalesce overlapping nack callbacks

---------

Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit 02d307e)

* fix(channels): prevent base URL credentials in status output (#107754)

* fix(channels): redact credentials in account URLs

* fix(channels): sanitize final status summaries

(cherry picked from commit 210340f)

* fix(channels): prevent lifecycle listener buildup (#109108)

(cherry picked from commit 0e1fad7)

* fix(sandbox): use Buffer.byteLength for env var value size limit (#105017)

* fix(sandbox): use Buffer.byteLength for env var value size limit

validateEnvVarValue checked value.length (UTF-16 code units) against
the 32768-byte limit, so multi-byte CJK values like "值".repeat(11000)
passed the check despite exceeding 33 KB in UTF-8. Switch to
Buffer.byteLength(value, "utf8") so the limit matches the actual byte
count the OS and child processes see.

* test(sandbox): simplify env byte-limit coverage

Co-authored-by: 唐梓夷0668001293 <[email protected]>

---------

Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit 84fb48c)

* fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync (#109590)

* fix(gateway): guard process.kill ESRCH race in signalVerifiedGatewayPidSync

A verified gateway process can exit between the argv validation check and
the process.kill call, causing an unhandled ESRCH error. Wrap the kill in
try-catch and silently swallow ESRCH (process already gone = signal
already delivered).

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* docs(gateway): explain ESRCH signal race

Co-authored-by: 丁宇婷0668001435 <[email protected]>

---------

Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit 853b1a8)

* fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS SSRF bypass (#110693)

* fix(litellm): guard loopback hostname auto-allow with isIP to prevent DNS bypass

The isAutoAllowedLitellmHostname helper auto-enables private-network access
for loopback-style hosts. Before this fix, lowered.startsWith("127.")
matched DNS hostnames like 127.evil.com, letting remote endpoints bypass
the explicit allowPrivateNetwork opt-in — a SSRF risk.

Add isIP(host)===4 guard so only literal IPv4 loopback addresses qualify.
Same canonical pattern as extensions/slack/src/monitor/relay-source.ts:271
and the codex loopback fix.

Co-Authored-By: Claude <[email protected]>

* test(litellm): cover loopback endpoint policy

---------

Co-authored-by: Claude <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit 3d03b60)

* fix(discord): sustained gateway bursts stop growing memory (#110954)

* fix(discord): sustained gateway bursts stop growing memory

* fix(discord): contain gateway queue overflow

* fix(discord): drop oldest saturated gateway sends

Co-authored-by: 张贵萍0668001030 <[email protected]>

* fix(discord): surface gateway overflow warnings

Co-authored-by: 张贵萍0668001030 <[email protected]>

---------

Co-authored-by: Peter Steinberger <[email protected]>
(cherry picked from commit 69aeba9)

* fix(gateway): bound busy channel health by real run age (#103793)

* fix(gateway): bound busy channel health by real run age

The channel health policy treats a channel as healthy-busy even while
disconnected, bounded only by a 25 minute stale ceiling measured from
lastRunActivityAt. The run-state heartbeat refreshes lastRunActivityAt
every 60 seconds for as long as any run is active, so a run that hangs
forever (for example a send blocking on a dead socket after the
transport already reported connected:false) keeps that timestamp fresh
and the stuck ceiling is never reached. The account is then reported
healthy forever by the health monitor, readiness probe, and health CLI,
and no restart ever fires.

createRunStateMachine now tracks each in-flight run's start time keyed by
an opaque run handle and publishes the oldest still-active run's start as
activeRunStartedAt. The health policy busy override keys its ceiling off
the real run age, so a run stuck longer than the threshold reports stuck
and the monitor can restart it. Because the reported start is the oldest
active run and advances to the next-oldest as runs complete, a channel
churning through many short overlapping runs (activeRuns above 1 across
concurrent queue keys) stays healthy; only a genuinely hung run breaches
the ceiling. Short and active runs stay healthy and the existing
lastRunActivityAt fallback is preserved for snapshots without a start
time.

* fix(channels): retain run-state callback compatibility

Keep the released zero-argument onRunEnd callback source-compatible while allowing internal queue callers to pass a run handle for exact concurrent-run accounting. The compatibility path closes the oldest active run, preserving existing lifecycle behavior for consumers that do not use handles.

* fix(channels): keep anonymous runs out of age tracking

The zero-argument lifecycle callbacks cannot identify which concurrent run completed, so they must not update the identity-sensitive run start used by channel health. Keep their busy count separately and reserve exact start tracking for the shared queue's handle-aware lifecycle path.

* fix(channels): keep tracked runs internal

Keep the public run-state lifecycle callbacks unchanged. The channel queue now owns opaque run identity and augments its status updates with the oldest active queue run, so implementation details do not expand the SDK surface.

* fix(channels): type queue run start status

Keep activeRunStartedAt in the internal status patch type so the queue can publish its private tracked-run age through the existing status sink.

* fix(channels): wrap isActive to satisfy unbound-method lint

* fix(gateway): gate busy run-age ceiling on disconnected transport

(cherry picked from commit 18b79d9)

* fix(deps): update fast-uri past advisory

(cherry picked from commit 1be9db0)

* fix(release): adapt maintenance-line hardening

Backport/adapt 18ec9ce, dea1fe1, 7f32b6c, 1da345e, 931ac3e, 89780d5, and c0d99ed for the 2026.6 extended-stable maintenance line.

* fix(deps): bump protobufjs to 7.6.5

Backport-adapted from a230f74.

* test(gateway): cover bounded macOS process probe

* chore(release): prepare 2026.6.34

* test(dotenv): share path override environment assertions

* fix(release): resolve 2026.6.34 CI blockers

---------

Signed-off-by: sallyom <[email protected]>
Co-authored-by: joshavant <[email protected]>
Co-authored-by: Peter Lee <[email protected]>
Co-authored-by: sallyom <[email protected]>
Co-authored-by: openclaw-clownfish[bot] <280122609+openclaw-clownfish[bot]@users.noreply.github.com>
Co-authored-by: Liu Wenyu <[email protected]>
Co-authored-by: pick-cat <[email protected]>
Co-authored-by: Pick-cat <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
Co-authored-by: weiqinl <[email protected]>
Co-authored-by: Claude Opus 4.8 <[email protected]>
Co-authored-by: shengting <[email protected]>
Co-authored-by: Shengting Xie <[email protected]>
Co-authored-by: yayu <[email protected]>
Co-authored-by: Agustin Rivera <[email protected]>
Co-authored-by: cxbAsDev <[email protected]>
Co-authored-by: ooiuuii <[email protected]>
Co-authored-by: Masato Hoshino <[email protected]>
Co-authored-by: Vincent Koc <[email protected]>
Co-authored-by: mushuiyu886 <[email protected]>
Co-authored-by: Peter Steinberger <[email protected]>
Co-authored-by: Bruno Wowk (Volky) <[email protected]>
Co-authored-by: Pavan Kumar Gondhi <[email protected]>
Co-authored-by: Glucksberg <[email protected]>
Co-authored-by: xingzhou <[email protected]>
Co-authored-by: tzy-17 <[email protected]>
Co-authored-by: krissding <[email protected]>
Co-authored-by: lsr911 <[email protected]>
Co-authored-by: Yuval Dinodia <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants