Skip to content

fix(codex): keep runCodexExecResume alive on stdout/stderr pipe errors#102127

Closed
wangmiao0668000666 wants to merge 5 commits into
openclaw:mainfrom
wangmiao0668000666:fix/codex-node-cli-sessions-stdio-stream-errors
Closed

fix(codex): keep runCodexExecResume alive on stdout/stderr pipe errors#102127
wangmiao0668000666 wants to merge 5 commits into
openclaw:mainfrom
wangmiao0668000666:fix/codex-node-cli-sessions-stdio-stream-errors

Conversation

@wangmiao0668000666

Copy link
Copy Markdown
Contributor

Summary

  • add child.stdout.on("error", ...) and child.stderr.on("error", ...) to the runCodexExecResume child-spawn path so a parent-side pipe break on either Readable is routed into the same rejection the existing child.on("error", reject) path already handles, instead of escaping as an unhandled error event
  • add two regression tests that drive the real production code path against a real Node child spawned via process.execPath --eval, with the spawn wrapper capturing the real ChildProcess reference and emitting a synthetic error on the real parent-side stdout / stderr readable

Why this change is needed

runCodexExecResume (extensions/codex/src/node-cli-sessions.ts:252) spawns the Codex CLI with stdio: ["pipe", "pipe", "pipe"] and registers a data handler on stdout and stderr, but neither stream has its own error handler. Node's EventEmitter contract is that an error event on a stream without a registered listener becomes an uncaughtException that crashes the gateway. On a real production teardown — the parent process exiting, the kernel resetting the pipe, the child being SIGKILLed mid-write — that contract fires and takes the gateway down.

This is the same pattern that Alix-007 fixed in #101596 (google-meet realtime) and #101597 (matrix deps) for sibling surfaces, and the same pattern that sunlit-deng fixed in #101777 for the Codex sandbox-exec-server subprocesses. This PR covers the parallel Codex node-cli-sessions surface that runs the codex exec resume CLI invocation.

The Codex app-server transport at extensions/codex/src/app-server/transport-stdio.ts:116 already routes stream errors through the CodexAppServerClient constructor (stdout at client.ts:145, stderr at client.ts:158, stdin EPIPE guard at client.ts:171), so that surface is covered and out of scope for this PR.

Verification

  • node scripts/run-vitest.mjs extensions/codex/src/node-cli-sessions.test.ts — passed; 1 file, 9 tests (7 pre-existing + 2 new real-spawn stream-error tests)
  • ./node_modules/.bin/oxfmt --check --threads=1 extensions/codex/src/node-cli-sessions.ts extensions/codex/src/node-cli-sessions.test.ts — all files correctly formatted
  • ./node_modules/.bin/oxlint --tsconfig config/tsconfig/oxlint.extensions.json extensions/codex/src/node-cli-sessions.ts extensions/codex/src/node-cli-sessions.test.ts — no findings
  • pnpm tsgo:extensions — passed
  • pnpm tsgo:extensions:test — passed (test-typecheck gap is not bypassed)
  • mutation check: renaming the new child.stdout?.on?.("error", ...) block to listen for "Xerror" instead of "error" makes the new rejects with the synthetic stream error when the child stdout pipe fails mid-run test fail with synthetic parent stdout read failure not being thrown; restoring the block makes the test pass. Renaming the stderr listener has the symmetric effect on the stderr test.

Evidence

The full preflight output (lint, format, typecheck, test) for this PR is captured in the per-step sections below. The decisive evidence is in ## Real behavior proof — the real child process spawned via process.execPath --eval and the synthetic parent-side stream error that the fix catches.

Real behavior proof

Behavior addressed: codex node-cli-sessions runCodexExecResume child stdout/stderr pipe errors no longer leak as unhandled error events. The new child.stdout.on("error", ...) and child.stderr.on("error", ...) listeners call reject(...) on the surrounding new Promise<number | null> that already hosts child.on("error", reject), so the existing runCodexExecResume failure path (and the upstream codex.cli.session.resume command handler) propagates a clear Error("synthetic parent stdout read failure") (or the stderr variant) to the caller instead of crashing the gateway with an unhandled error event.

Real environment tested: local Linux Node 22.22 source checkout. The new regression tests wrap the real node:child_process spawn (via vi.doMock + vi.importActual) and substitute the spawned command with process.execPath --eval running a long-lived script that calls process.stdin.resume(), writes a ready\n marker to stderr, and then idles. This produces a real Node child with the production stdio: ["pipe", "pipe", "pipe"] shape and real parent-side Readable streams. The wrapper records the real ChildProcess in a captured array; the test then emit("error", new Error("synthetic parent stdout read failure")) (or the stderr variant) on the real parent-side readable and asserts the surrounding handlePromise rejects with that exact error message.

Exact steps or command run after this patch:

  • node scripts/run-vitest.mjs extensions/codex/src/node-cli-sessions.test.ts --reporter=verbose — full file passes, 9 tests:
    • runCodexExecResume stream error handling > rejects with the synthetic stream error when the child stdout pipe fails mid-run (~170 ms)
    • runCodexExecResume stream error handling > rejects with the synthetic stream error when the child stderr pipe fails mid-run (~180 ms)

Evidence after fix (output of the verbose test run):

✓ |extension-codex| extensions/codex/src/node-cli-sessions.test.ts > runCodexExecResume stream error handling > rejects with the synthetic stream error when the child stdout pipe fails mid-run 168ms
✓ |extension-codex| extensions/codex/src/node-cli-sessions.test.ts > runCodexExecResume stream error handling > rejects with the synthetic stream error when the child stderr pipe fails mid-run 178ms

Test Files  1 passed (1)
     Tests  9 passed (9)

The stdout-pipe test asserts:

  • await expect(handlePromise).rejects.toThrow("synthetic parent stdout read failure")
  • The promise comes from the production runCodexExecResume after it has spawned a real Node child via process.execPath --eval and attached the stream-error handler in this PR
  • The capture wrapper around spawn returns the real ChildProcess and the test emits a synthetic error on the real child.stdout readable

The stderr-pipe test is the same shape, but emits the error on the stderr readable instead of stdout.

Observed result after fix: the previously unhandled child.stdout.on("error", ...) and child.stderr.on("error", ...) paths are now caught inside the existing runCodexExecResume promise contract. The error reaches the upstream codex.cli.session.resume command handler through the existing child.on("error", reject) rejection, which the calling command then surfaces to the gateway as a normal rejected command outcome — no unhandled error event, no gateway crash.

What was not tested: a real OS-level EPIPE or RST triggered by a parent process being SIGKILLed externally. Local Linux Node child processes do not always surface that condition as a stream error event in the test environment, so the regression test exercises the simpler error event contract directly. The pattern is identical to what Alix-007 used in #101597 (matrix deps MERGED 2026-07-08) and sunlit-deng used in #101777 (codex sandbox-exec-server subprocesses, OPEN, 🐚 ready for maintainer look) — those PRs also rely on a synthetic error event for their regression tests, not a kernel-level RST. The Codex app-server transport at extensions/codex/src/app-server/transport-stdio.ts:116 is a different long-lived spawn that already has full stream-error coverage in the CodexAppServerClient constructor; it is intentionally out of scope for this PR.

Campaign coordination

This PR is part of the active stream-error campaign wave in 2026-07:

Collision check: no open PR found for extensions/codex/src/node-cli-sessions.ts or for the runCodexExecResume stream-error issue. Verified via gh pr list --state all --search "node-cli-sessions" --json number,title,state and file-level gh pr list --state all --search "runCodexExecResume" --json number,title,state; the only fuzzy hits are unrelated test-infra PRs.

The Codex node-cli-sessions runCodexExecResume helper spawns the CLI with
stdio: ["pipe", "pipe", "pipe"] and registers data handlers on stdout and
stderr, but neither stream had its own error listener. A parent-side
pipe break (gateway teardown, kernel RST, child SIGKILL mid-write) would
fire an unhandled "error" event on the stream and crash the gateway.

Route both streams into the existing child.on("error", reject) promise so
the upstream codex.cli.session.resume command handler surfaces a clear
rejection. Cover the new behavior with two real-spawn tests that emit a
synthetic "error" on the real parent-side stdout/stderr readable.
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 10:36 PM ET / 02:36 UTC.

Summary
The PR adds stdout/stderr stream-error handling to runCodexExecResume and adds real-spawn regression tests for stdout, stderr, and burst stream-error handling.

PR surface: Source +37, Tests +147. Total +184 across 2 files.

Reproducibility: yes. source-reproducible: current main and v2026.6.11 attach stdout/stderr data listeners without stream error listeners, and Node's EventEmitter source throws on unhandled error events. I did not run a failing current-main crash repro in this read-only review.

Review metrics: 1 noteworthy metric.

  • Session-Guarded Command Path: 1 dangerous node-host command path changed. codex.cli.session.resume holds an active per-session guard around a live Codex CLI child, so child settlement ordering matters before merge.

Root-cause cluster
Relationship: canonical
Canonical: #102127
Summary: This PR is the direct runCodexExecResume stream-error fix; related campaign PRs cover adjacent subprocess surfaces or command policy rather than the same remaining work.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • none.

Risk before merge

  • [P1] This changes failure settlement for a session-guarded dangerous Codex CLI resume command; a lifecycle regression could release the active-session guard too early or leave a child process running.
  • [P1] The proof is strong for Node's stream-error contract with a real spawned child, but it does not reproduce a kernel-originated EPIPE/RST from a live Codex resume run.

Maintainer options:

  1. Land With Focused Proof (recommended)
    Accept the existing real-spawn stream-error proof and clean checks as enough for this rare subprocess lifecycle repair.
  2. Ask For Broader Runtime Proof
    Require a live Codex resume or OS-level pipe-failure artifact before merge if synthetic parent-side Readable errors are not enough for the session-guarded path.
  3. Pause For Lifecycle Consolidation
    Defer this PR if maintainers want a shared subprocess stream-error helper before landing more local handlers.

Next step before merge

  • [P2] No automated code repair remains; the next action is maintainer proof-sufficiency and merge-risk review for the current head.

Maintainer decision needed

  • Question: Is the real-spawn synthetic stdout/stderr stream-error proof sufficient for this rare Codex CLI resume child-process failure, or should merge wait for broader live Codex or OS-level pipe-failure proof?
  • Rationale: The source path, Node contract, Codex CLI contract, tests, and exact-head checks support the fix, but OS-originated parent-side Readable errors are timing-dependent and hard to force, so the final proof bar for this session-guarded command is maintainer judgment.
  • Likely owner: mbelinky — mbelinky has the strongest history signal for the Codex CLI node-session binding and can judge the proof bar for this command path.
  • Options:
    • Accept Focused Lifecycle Proof (recommended): Merge after ordinary maintainer review if the real spawned child tests, mutation checks, and clean exact-head checks are sufficient for the rare stream-error condition.
    • Request Broader Live Proof: Hold the PR until a contributor or maintainer provides a live Codex resume run or OS-level pipe-failure artifact showing the same containment behavior.
    • Pause For Shared Helper: Pause this patch if maintainers want the Codex and sibling subprocess stream-error campaign consolidated behind a common owner-approved helper first.

Security
Cleared: No dependency, workflow, secret, package-resolution, or command-source security regression was found in this focused child-process lifecycle patch.

Review details

Best possible solution:

Land the focused runCodexExecResume lifecycle handler with its real-spawn regressions after maintainers accept the proof bar, rather than reviving the broader superseded combined Codex stream-error PR.

Do we have a high-confidence way to reproduce the issue?

Yes, source-reproducible: current main and v2026.6.11 attach stdout/stderr data listeners without stream error listeners, and Node's EventEmitter source throws on unhandled error events. I did not run a failing current-main crash repro in this read-only review.

Is this the best way to solve the issue?

Yes. The patch is the narrow owner-boundary fix for this direct codex exec resume spawn; the app-server transport and sandbox subprocesses are distinct sibling surfaces with their own handling or open PRs.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 30b3a7cdeda9.

Label changes

Label justifications:

  • P2: This is a focused Codex subprocess lifecycle bugfix with limited surface and plausible gateway availability impact.
  • merge-risk: 🚨 session-state: The diff changes failure settlement for a session-guarded Codex resume command, where early guard release could allow overlapping turns if lifecycle handling regresses.
  • merge-risk: 🚨 availability: The diff changes child-process termination on stdout/stderr stream errors, where incorrect handling could leave a live Codex child or allow an unhandled stream error to crash the gateway.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body includes copied terminal output from real-spawn stdout/stderr stream-error tests plus mutation checks, while honestly noting that an OS-originated EPIPE/RST was not reproduced.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes copied terminal output from real-spawn stdout/stderr stream-error tests plus mutation checks, while honestly noting that an OS-originated EPIPE/RST was not reproduced.
Evidence reviewed

PR surface:

Source +37, Tests +147. Total +184 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 37 0 +37
Tests 1 147 0 +147
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 184 0 +184

What I checked:

  • Current main still lacks stream-error handlers: Current main spawns codex exec resume with piped stdout/stderr and attaches only data listeners before waiting on child error and exit; stdout/stderr stream error events remain unhandled on main. (extensions/codex/src/node-cli-sessions.ts:294, 30b3a7cdeda9)
  • Latest release is still affected: The latest release tag v2026.6.11 has the same data-only stdout/stderr handling at the resume spawn site, so this PR is not obsolete or already shipped. (extensions/codex/src/node-cli-sessions.ts:293, e085fa1a3ffd)
  • PR head routes stream errors through child lifecycle: The PR records stdout/stderr stream errors, arms a single SIGTERM/SIGKILL fallback, waits for the child exit settlement, clears the fallback timer, and throws the recorded stream error afterward. (extensions/codex/src/node-cli-sessions.ts:297, 5bc1256617de)
  • Regression tests use a real child process shape: The added tests wrap spawn, substitute a long-running real Node child with production stdio: ["pipe", "pipe", "pipe"], emit errors on real parent-side stdout/stderr Readables, and assert rejection plus child termination. (extensions/codex/src/node-cli-sessions.test.ts:274, 5bc1256617de)
  • Burst-dedupe test covers sibling lifecycle risk: The new burst test emits stdout and stderr errors back-to-back and asserts the handler sends exactly one SIGTERM, matching the lifecycle risk raised in earlier ClawSweeper cycles. (extensions/codex/src/node-cli-sessions.test.ts:370, 5bc1256617de)
  • Caller and owner boundary checked: The Codex plugin registers these node-host commands and node-invoke policies from the bundled Codex plugin entry; the fix stays inside the owning plugin command implementation. (extensions/codex/index.ts:97, 30b3a7cdeda9)

Likely related people:

  • mbelinky: Mariano Belinky introduced the Codex CLI node-session binding and the node-cli-sessions source/test files touched by this PR. (role: feature introducer; confidence: high; commits: 27c9564081f5, a5c1956ca17a; files: extensions/codex/src/node-cli-sessions.ts, extensions/codex/src/node-cli-sessions.test.ts, extensions/codex/src/conversation-binding.ts)
  • llagy009: llagy009 recently changed the same Codex CLI session source and tests for preview text boundary handling. (role: recent area contributor; confidence: medium; commits: 52e2b003e63d, b5c662f4f54c; files: extensions/codex/src/node-cli-sessions.ts, extensions/codex/src/node-cli-sessions.test.ts)
  • vincentkoc: Vincent Koc's recent refactor touched the current line blame for the command registration and resume helper in this shallow main checkout. (role: recent area contributor and merger; confidence: medium; commits: 6f6c3e260222; files: extensions/codex/src/node-cli-sessions.ts)
  • steipete: Peter Steinberger recently fixed invalid Codex history timestamp handling in the same source and test files. (role: recent area contributor; confidence: medium; commits: 28eb4cfa12e5; files: extensions/codex/src/node-cli-sessions.ts, extensions/codex/src/node-cli-sessions.test.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (8 earlier review cycles)
  • reviewed 2026-07-08T09:10:03.434Z sha 5f3efb4 :: needs changes before merge. :: [P2] Terminate the resume child on stream errors
  • reviewed 2026-07-08T09:20:56.310Z sha 5f3efb4 :: needs changes before merge. :: [P2] Terminate the resume child on stream errors
  • reviewed 2026-07-08T09:32:44.675Z sha 5f3efb4 :: needs changes before merge. :: [P2] Terminate the resume child on stream errors
  • reviewed 2026-07-08T12:04:10.668Z sha 7c02ef1 :: needs changes before merge. :: [P2] Wait for the resume child before rejecting
  • reviewed 2026-07-08T13:25:28.246Z sha bda9070 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T13:36:23.229Z sha bda9070 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T01:53:47.339Z sha 9bc565b :: needs maintainer review before merge. :: none
  • reviewed 2026-07-09T02:28:48.694Z sha 5bc1256 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jul 8, 2026
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 8, 2026
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jul 8, 2026
…ExecResume

When a broken pipe fires the error event on both stdout and stderr back-to-back, routeStreamError was calling child.kill(SIGTERM) twice and arming a redundant SIGKILL grace timer. Add a streamKillArmed flag so the kill path runs once per child, and clear the SIGKILL timer in the existing finally block so it does not fire after the child has already exited.

Adds a burst test that wraps child.kill, fires both streams, and asserts exactly one SIGTERM landed.
…utor-return

The burst-dedup test awaited `new Promise<void>((resolve) => setTimeout(() => resolve(), 100))`.
The arrow-body `() => resolve()` is flagged by eslint(no-promise-executor-return)
because the implicit `return resolve()` is read as "return in Promise executor".
Bracketing the body so resolve() is a statement (not an expression) clears the
rule while keeping the test behavior identical.

Caught by CI check-lint on the round-1 push.
@wangmiao0668000666

Copy link
Copy Markdown
Contributor Author

Closing this myself to trim the queue. Keeping runCodexExecResume alive on stdout/stderr pipe errors touches session-state and availability semantics (double merge-risk flag) and needs owner-level judgment on the Codex harness contract; after 10 days without maintainer pickup I'll defer to however the owners want to handle pipe failures. Happy to reopen or rework with owner guidance — branch stays on my fork.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

extensions: codex merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P2 Normal backlog priority with limited blast radius. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: S status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant