Retry node shell fallback on ENOENT by clawSean · Pull Request #85543 · openclaw/openclaw

clawSean · 2026-05-22T23:29:44Z

Summary

Retry node-host system.run shell execution with trusted absolute /usr/bin/sh when the default /bin/sh spawn fails with ENOENT.
Keep PATH-resolved shell fallback out of the retry chain so execution does not switch to an inherited-PATH binary after approval analysis.
Keep non-shell and Windows invocations unchanged.
Add regression coverage for the runtime retry chain.

Why

Linux headless node hosts can report spawn /bin/sh ENOENT even when the node is connected and system.which/manual host checks show a usable shell at /usr/bin/sh. The previous node-host path treated the first spawn error as terminal, so exec host=node failed before trying another trusted POSIX shell path.

Verification

node scripts/test-projects.mjs src/node-host/invoke.node-shell-fallback.test.ts src/infra/node-shell.test.ts src/node-host/invoke.test.ts src/node-host/invoke.sanitize-env.test.ts src/agents/bash-tools.exec-host-node.test.ts
- 3 Vitest shards passed; 24 tests
pnpm tsgo:core:test
git diff --check

Real behavior proof

Behavior addressed:
Node-host system.run no longer treats a missing default /bin/sh spawn as the only possible POSIX shell path; the patched node host retries trusted absolute /usr/bin/sh on ENOENT.

Real environment tested:
Linux VPS source checkout at /root/projects/openclaw-node-shell-fallback, Node.js v22.22.2, patched OpenClaw node-host runtime code at PR head fd0fdfb445. I also ran the actual patched src/node-host/invoke.ts runCommand path inside a minimal Linux chroot proof root where /bin/sh is absent and /usr/bin/sh -> dash exists.

Exact steps or command run after this patch:
First ran the focused runtime/test/type gates from the patched checkout. Then ran the actual patched node-host runCommand implementation in a Linux chroot with an empty /bin directory and /usr/bin/sh available:

/usr/sbin/chroot /tmp/openclaw-node-shell-proof-root \
  /usr/bin/node --import /repo/node_modules/tsx/dist/esm/index.mjs \
  /repo/.tmp-node-shell-proof.mjs

The proof script called:

await runCommand(
  ["/bin/sh", "-lc", "printf NODE_SHELL_FALLBACK_OK"],
  undefined,
  undefined,
  5000,
);

Evidence after fix:
Terminal output from the live chroot runtime proof:

proof root /bin/sh exists: false
proof root /usr/bin/sh exists: true
proof root /usr/bin/sh target: dash
runCommand result: {
  "exitCode": 0,
  "timedOut": false,
  "success": true,
  "stdout": "NODE_SHELL_FALLBACK_OK",
  "stderr": "",
  "error": null,
  "truncated": false
}

Focused test output from the patched checkout:

[test] starting test/vitest/vitest.unit-fast.config.ts

 RUN  v4.1.7 /root/projects/openclaw-node-shell-fallback

 Test Files  2 passed (2)
      Tests  13 passed (13)

[test] starting test/vitest/vitest.unit.config.ts

 RUN  v4.1.7 /root/projects/openclaw-node-shell-fallback

 Test Files  2 passed (2)
      Tests  2 passed (2)

[test] starting test/vitest/vitest.agents.config.ts

 RUN  v4.1.7 /root/projects/openclaw-node-shell-fallback

 Test Files  1 passed (1)
      Tests  8 passed (8)

[test] passed 3 Vitest shards in 19.64s

Observed result after fix:
The actual patched node-host runtime path returned success from /usr/bin/sh with stdout NODE_SHELL_FALLBACK_OK when invoked in a Linux root where /bin/sh was absent and /usr/bin/sh existed. The focused regression separately asserts the retry sequence:

/bin/sh -lc "printf NODE_SHELL_FALLBACK_OK"
/usr/bin/sh -lc "printf NODE_SHELL_FALLBACK_OK"

What was not tested:
I still do not have the reporter's exact paired Linux headless node/systemd setup connected to this gateway, so this is runtime proof of the patched node-host command path under the same missing-/bin/sh failure condition, not a full paired-node exec host=node end-to-end recording from the original environment.

clawsweeper · 2026-05-22T23:31:46Z

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Current main and the latest inspected release already solve the central linked Linux node-exec failure at the safer gateway boundary by not forwarding gateway/default cwd to remote node execution; this PR is no longer needed as a landing path.

Root-cause cluster
Relationship: superseded
Canonical: #85202
Summary: This PR was a candidate for the canonical Linux node exec ENOENT issue, but the central failure is already addressed by the merged gateway-side cwd fix.

Members:

canonical: [Bug]: Linux headless node exec fails with spawn /bin/sh ENOENT despite connected nodes and existing /bin/sh #85202 - The PR body uses closing syntax for the Linux headless node exec failure.
fixed_by_candidate: fix(exec): skip gateway cwd injection for remote node host #58977 - The merged PR implements the gateway-side fix for not forwarding default cwd to remote node exec.
adjacent_distinct: Fix Android node system.run shell wrapper #73897 - This open PR shares node shell selection surface but targets Android/Termux /bin/sh absence, not the Linux forwarded-cwd failure.
superseded: [Fix] #85202 [Bug]: Linux headless node exec fails with spawn /bin/sh ENOENT despite connecte #86269 - This closed unmerged earlier attempt targeted the same linked issue but is not a live fix path.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Keep the shipped gateway-side cwd behavior and close this branch rather than adding a node-side fallback that can silently ignore an explicitly supplied missing workdir.

Do we have a high-confidence way to reproduce the issue?

Yes. The linked issue evidence shows /bin/sh exists and the reported setup has a Docker gateway plus user-systemd node services; current main/source history show the gateway no longer forwards its own default cwd to node exec when workdir is omitted.

Is this the best way to solve the issue?

No for this PR as a landing path. The best fix is already on main at the gateway boundary: omit gateway-derived cwd for node exec while preserving explicit node cwd values.

Security review:

Security review cleared: No supply-chain or security-sensitive dependency/workflow changes were introduced; the concern is redundancy and behavior fit, not a concrete security regression.

AGENTS.md: found and applied where relevant.

What I checked:

Current main implementation: For host=node, current main forwards only explicitWorkdir and no longer falls back to defaults.cwd or process.cwd(), so omitted node cwd uses the node host default instead of a gateway/container path. (src/agents/bash-tools.exec.ts:1583, 8a5cb85c31e5)
Current main regression coverage: Current main has coverage that explicit node workdirs are preserved and gateway default cwd is not forwarded when workdir is omitted. (src/agents/bash-tools.exec.approval-id.test.ts:588, 8a5cb85c31e5)
Git-history provenance: Commit 247a068 introduced the gateway-side fix to skip gateway cwd injection for remote node host execution. (src/agents/bash-tools.exec.ts:1583, 247a06813e01)
Fix commit metadata: The fix commit is fix: avoid gateway cwd for node exec (#58977) (thanks @Starhappysh), committed on 2026-04-03T02:04:26+09:00. (src/agents/bash-tools.exec.ts:1583, 247a06813e01)
Release provenance: Release tag v2026.6.10 contains the same host === "node" behavior that forwards only explicit workdir. (src/agents/bash-tools.exec.ts:1555, aa69b12d0086)

Likely related people:

Starhappysh: Authored the merged PR that stopped forwarding gateway default cwd to remote node exec when cwd is omitted. (role: introduced current fix in merged PR; confidence: high; commits: 247a06813e01; files: src/agents/bash-tools.exec.ts)
steipete: Merged the current gateway-side node cwd fix and appears in history around the node exec feature surface. (role: merger and feature-history owner; confidence: high; commits: 247a06813e01, ae0b4c49903d; files: src/agents/bash-tools.exec.ts, src/agents/bash-tools.exec-host-node-phases.ts)
vincentkoc: The v2026.6.10 release tag containing the current implementation is tagged by this person in local release metadata. (role: release/tag provenance; confidence: medium; commits: aa69b12d0086; files: src/agents/bash-tools.exec.ts)

Codex review notes: model internal, reasoning high; reviewed against 8a5cb85c31e5; fix evidence: release v2026.6.10, commit 247a06813e01.

clawsweeper · 2026-05-22T23:40:05Z

ClawSweeper PR egg

🔥 Warming up: real-behavior proof passed; findings, security review, or rank-up moves are still in progress.

Hatch command

Comment @clawsweeper hatch when this PR is hatchable.

Hatchability rules:

Merged PRs are hatchable.
Open PRs are hatchable when they are status: 👀 ready for maintainer look, status: 🚀 automerge armed, or labeled clawsweeper:automerge.
Closed unmerged PRs are hatchable only when one of those hatchable labels is still present in the durable record.

What is this egg doing here?

Eggs appear after the PR passes real-behavior proof. It is here for vibes, not verdicts: it does not change labels, ratings, merge decisions, or automation.
The shell reacts to review momentum: open follow-up work warms it up, re-review makes it wobble, and a clean final review lets it hatch.
Hatchability usually comes from sufficient real-behavior proof, no blocking P0/P1/P2 findings, no security attention needed, and clean correctness. A merged PR is already final, so merge makes the egg hatchable independently.
The hatch is seeded from this repository and PR number, so the same PR keeps the same creature; the reviewed head SHA can only change safe visual details.
Rarity is just collectible sparkle: 🥚 common, 🌱 uncommon, 💎 rare, ✨ glimmer, and 🌈 legendary.

clawSean · 2026-05-23T22:41:12Z

@clawsweeper re-review

Removed the PATH-resolved sh fallback and updated the PR body/proof; retry chain is now /bin/sh -> /usr/bin/sh only.

clawsweeper · 2026-05-23T22:41:15Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26446457068
Updated: 2026-05-26T10:27:13.178Z

clawSean · 2026-05-23T22:43:44Z

@clawsweeper re-review

Updated the Real behavior proof fields with the required plain field labels after removing the PATH fallback.

clawsweeper · 2026-05-23T22:43:47Z

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26345542481
Updated: 2026-05-23T22:55:10.377Z

clawSean · 2026-05-26T09:50:00Z

@clawsweeper re-review

Added live Linux chroot runtime proof for the actual patched node-host runCommand path: /bin/sh absent, /usr/bin/sh present, runCommand returns stdout NODE_SHELL_FALLBACK_OK via the fallback.

clawsweeper · 2026-05-26T09:50:04Z

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/26445149767
Updated: 2026-05-26T10:00:17.962Z

clawSean · 2026-06-14T21:32:47Z

@clawsweeper re-review

clawsweeper · 2026-06-14T21:32:50Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review finished, the durable review comment was updated, and the synced verdict was routed.
Run: https://github.com/openclaw/clawsweeper/actions/runs/27512723695
Updated: 2026-06-14T21:40:00.281Z

clawSean · 2026-06-20T09:30:33Z

@clawsweeper re-review

Updated this to address the patch-quality/security-boundary concern. The /bin/sh -> /usr/bin/sh fallback is no longer a retry inside the low-level spawn helper after policy/approval/audit have already bound a different argv.

What changed:

resolveNodeShellCommand() now canonicalizes node shell argv before execution planning when /bin/sh is absent and /usr/bin/sh exists.
system.run.prepare binds the fallback argv into the approval plan before returning it.
system.run parse normalizes legacy/local prepared plans before policy evaluation, durable approval persistence, allowlist-use recording, Mac exec-host forwarding, and exec.finished.
runCommand() now executes only the argv supplied by the validated system.run path; no below-policy retry remains.
Added focused tests proving prepared plan binding, runCommand/audit commandText parity, and no low-level retry.

Local proof on head 68d262145c:

node scripts/test-projects.mjs src/node-host/invoke.node-shell-fallback.test.ts src/infra/node-shell.test.ts src/node-host/invoke-system-run.test.ts src/agents/bash-tools.exec-host-node.test.ts src/agents/bash-tools.exec.approval-id.test.ts ✅
pnpm tsgo:core:test ✅
pnpm exec oxfmt --check src/infra/node-shell.ts src/infra/node-shell.test.ts src/node-host/invoke.ts src/node-host/invoke-system-run.ts src/node-host/invoke-system-run-plan.ts src/node-host/invoke-system-run.test.ts src/node-host/invoke.node-shell-fallback.test.ts ✅
pnpm exec oxlint src/infra/node-shell.ts src/infra/node-shell.test.ts src/node-host/invoke.ts src/node-host/invoke-system-run.ts src/node-host/invoke-system-run-plan.ts src/node-host/invoke-system-run.test.ts src/node-host/invoke.node-shell-fallback.test.ts ✅
git diff --check ✅

clawsweeper · 2026-06-20T09:30:36Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

…ENT) Linux headless node exec failed with `spawn /bin/sh ENOENT` even when /bin/sh exists (openclaw#85202). The real cause is a forwarded working directory that does not exist on the node host: Node fails the pre-exec chdir and surfaces it as `spawn <cmd> ENOENT` against the command rather than the directory. A gateway running in Docker forwards its own process.cwd() as the node workdir, which is a container path absent on the node; both /bin/sh and /usr/bin/sh point to the same dash, so swapping shells cannot help. Non-approval node exec now drops a cwd that does not resolve to a directory on the node host (falls back to the node default dir). The approval path is unchanged and still fails closed via resolveCanonicalApprovalCwdSync. The /bin/sh -> /usr/bin/sh fallback is retained for genuinely-missing shells. Adds resolveNodeExecCwd unit coverage and a /bin/sh-present + missing-cwd regression test.

clawSean · 2026-06-25T00:42:34Z

@clawsweeper re-review

Reworked this around the actual root cause of the canonical issue (#85202), which the previous shell-only fallback could not address.

Root cause (reproduced locally)

The linked reports all show /bin/sh present yet spawn /bin/sh ENOENT. That error is Node misattributing a failed pre-exec chdir: when spawn() is given a cwd that does not exist, libuv fails the chdir and surfaces it as spawn <command> ENOENT — pointing at the command, not the directory.

#85202 runs the gateway in Docker and the nodes as user systemd services. An unspecified node workdir defaults to the gateway's own process.cwd() (src/agents/bash-tools.exec.ts → explicitWorkdir ?? defaultWorkdir ?? process.cwd()), a container path that does not exist on the node. In #85202 both /bin/sh and /usr/bin/sh point to the same dash, so swapping shells cannot help — the chdir fails regardless of which shell is selected.

Local repro of the exact symptom:

$ node -e 'require("child_process").spawn("/bin/sh",["-lc","echo hi"],{cwd:"/does/not/exist"}).on("error",e=>console.log(e.message))'
spawn /bin/sh ENOENT   # /bin/sh exists; the missing cwd is the real cause

Fix

In the non-approval execution path (security=full / ask=off, the headless-node default), hardenApprovedExecutionPaths now drops a forwarded cwd that does not resolve to a directory on the node host, so execution falls back to the node's default working directory instead of failing. The approval path is unchanged — it still fails closed on a missing cwd via resolveCanonicalApprovalCwdSync. The existing /bin/sh → /usr/bin/sh fallback is retained for genuinely-missing shells.

The new helper resolveNodeExecCwd is the single chokepoint, and the cwd normalization stays above the policy/approval boundary (no below-policy retry).

Tests (the `/bin/sh`-present case the review asked for)

invoke-system-run.test.ts: full/off node exec with /bin/sh present + a non-existent forwarded cwd now drops the phantom cwd (runCommand receives undefined for cwd) and exec.finished/audit stay consistent.
node-shell.test.ts: unit coverage for resolveNodeExecCwd (missing dir, non-directory, existing dir, unset).

Proof on head `120991621d`

node scripts/test-projects.mjs src/infra/node-shell.test.ts src/node-host/invoke.node-shell-fallback.test.ts src/node-host/invoke-system-run.test.ts src/node-host/invoke-system-run-plan.test.ts src/agents/bash-tools.exec-host-node.test.ts ✅ (102 tests)
pnpm tsgo:core:test ✅
pnpm exec oxfmt --check on touched files ✅
pnpm exec oxlint on touched files ✅
git diff --check ✅

Follow-up worth flagging: the upstream contributor is the gateway defaulting a node workdir to its own process.cwd(); a separate change could stop forwarding a gateway-local cwd to remote nodes. The node-side fix here makes node exec robust regardless of what cwd is forwarded.

clawsweeper · 2026-06-25T00:42:37Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

clawsweeper · 2026-06-25T00:50:07Z

ClawSweeper applied the proposed close for this PR.

Action: closed this PR.
Close reason: already implemented on main.
Evidence: durable ClawSweeper review.

openclaw-barnacle Bot added size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 22, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 22, 2026

clawSean force-pushed the fix/node-shell-fallback branch from 1d5c804 to e100682 Compare May 23, 2026 00:26

clawSean force-pushed the fix/node-shell-fallback branch 2 times, most recently from 70203af to fd0fdfb Compare May 23, 2026 22:38

openclaw-barnacle Bot added triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. and removed proof: supplied External PR includes structured after-fix real behavior proof. labels May 23, 2026

openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 23, 2026

clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. labels May 23, 2026

clawsweeper Bot mentioned this pull request May 25, 2026

[Fix] #85202 [Bug]: Linux headless node exec fails with spawn /bin/sh ENOENT despite connecte #86269

Closed

clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels May 26, 2026

clawSean force-pushed the fix/node-shell-fallback branch from 494456e to 288e76c Compare June 13, 2026 08:36

openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 13, 2026

clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 13, 2026

clawsweeper Bot mentioned this pull request Jun 15, 2026

[Bug]: Linux headless node exec fails with spawn /bin/sh ENOENT despite connected nodes and existing /bin/sh #85202

Closed

clawSean force-pushed the fix/node-shell-fallback branch from 288e76c to 68d2621 Compare June 20, 2026 09:30

openclaw-barnacle Bot added size: M and removed size: S labels Jun 20, 2026

clawSean force-pushed the fix/node-shell-fallback branch from 68d2621 to 1209916 Compare June 25, 2026 00:41

clawsweeper Bot closed this Jun 25, 2026

clawSean mentioned this pull request Jun 26, 2026

fix(node-host): clarify unusable node exec cwd failures #97105

Merged

Uh oh!

Uh oh!

Conversation

clawSean commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why

Verification

Real behavior proof

Uh oh!

clawsweeper Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented May 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Hatch command

Uh oh!

clawSean commented May 23, 2026

Uh oh!

clawsweeper Bot commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawSean commented May 23, 2026

Uh oh!

clawsweeper Bot commented May 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawSean commented May 26, 2026

Uh oh!

clawsweeper Bot commented May 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawSean commented Jun 14, 2026

Uh oh!

clawsweeper Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawSean commented Jun 20, 2026

Uh oh!

clawsweeper Bot commented Jun 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawSean commented Jun 25, 2026

Root cause (reproduced locally)

Fix

Tests (the /bin/sh-present case the review asked for)

Proof on head 120991621d

Uh oh!

clawsweeper Bot commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

clawsweeper Bot commented Jun 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clawSean commented May 22, 2026 •

edited

Loading

clawsweeper Bot commented May 22, 2026 •

edited

Loading

clawsweeper Bot commented May 22, 2026 •

edited

Loading

clawsweeper Bot commented May 23, 2026 •

edited

Loading

clawsweeper Bot commented May 23, 2026 •

edited

Loading

clawsweeper Bot commented May 26, 2026 •

edited

Loading

clawsweeper Bot commented Jun 14, 2026 •

edited

Loading

clawsweeper Bot commented Jun 20, 2026 •

edited

Loading

Tests (the `/bin/sh`-present case the review asked for)

Proof on head `120991621d`

clawsweeper Bot commented Jun 25, 2026 •

edited

Loading