fix(process): UTF-8 command output corrupts at Windows byte limits by mushuiyu886 · Pull Request #105274 · openclaw/openclaw

mushuiyu886 · 2026-07-12T10:00:02Z

What Problem This Solves

Fixes an issue where byte-capped UTF-8 command output on Windows could be corrupted when the cap split a multibyte character. The retained incomplete UTF-8 prefix was passed through Windows console-code-page auto-detection, so output such as a followed by a partial emoji could decode with an invalid trailing character instead of the longest valid UTF-8 prefix, a.

This affects the SSH node-pairing identity probe because OpenSSH returns the remote JSON stream as UTF-8 while the probe keeps only the first 64 KiB.

Why This Change Was Made

The process runner now has a narrow UTF-8-specific entry point for commands whose output encoding is defined. It applies the existing byte-boundary cleanup on every platform and decodes the retained bytes as UTF-8. The default command runner keeps its existing Windows console-code-page auto-detection, including GBK and other legacy encodings.

The SSH node-pairing probe opts into the UTF-8 path. No SSH trust, pairing eligibility, timeout, command construction, public configuration, or protocol behavior changes.

User Impact

Windows gateways no longer corrupt a byte-capped SSH identity stream when truncation lands inside a UTF-8 character. Complete legacy-code-page command output remains unchanged.

Evidence

RED: node scripts/run-vitest.mjs src/process/exec.windows.test.ts returned a corrupted partial character with 3 truncated bytes instead of a with 5 truncated bytes.
GREEN: focused process, Windows mock, Windows integration routing, encoding, and gateway tests passed: 87 tests passed and 2 platform-specific tests skipped on Linux.
Legacy control: a truncated CP936 buffer containing the complete a测 prefix remains a测 with the original auto-detecting runner.
Tooling coverage: test/package-scripts.test.ts and test/scripts/openclaw-cross-os-release-workflow.test.ts passed (20 tests).
node scripts/run-tsgo.mjs -p tsconfig.core.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core.tsbuildinfo passed.
Targeted oxfmt --check, run-oxlint.mjs, and git diff --check passed.
Fresh codex review --base upstream/main found the initial legacy-code-page regression; the implementation was revised to isolate UTF-8 decoding, and the final review reported no defect.
Exact-head Windows 2025 proof and pnpm test:windows:ci passed on e22cfd34043e253010e2e5f83e4afb2077dda1ad: https://github.com/mushuiyu886/openclaw/actions/runs/29510723363/job/87663176141

Real behavior proof

Behavior addressed: Byte-capped UTF-8 stdout and stderr on Windows retain the longest complete UTF-8 prefix instead of passing an incomplete character to console-code-page decoding.
Real environment tested: GitHub-hosted Windows 2025 (Windows_NT 10.0.26100 x64) on exact PR head e22cfd34043e253010e2e5f83e4afb2077dda1ad. The run was manually dispatched for proof and did not use a test runner or mock subprocess for the production-runner scenario.
Exact steps or command run after this patch: Installed the exact-head checkout, then used node --import tsx to import the production src/process/exec.ts facade. The production runners spawned the real Windows Node executable for a three-byte head cap, separately exercised stdout and stderr, and ran a normal identity-JSON control. The same run then executed pnpm test:windows:ci.
Evidence after fix:

head=e22cfd34043e253010e2e5f83e4afb2077dda1ad
platform=win32
os=Windows_NT 10.0.26100 x64
default_head.code_points=U+0061,U+FFFD
default_head.truncated_bytes=3
utf8_head.code_points=U+0061
utf8_head.replacement_character=false
utf8_head.truncated_bytes=5
utf8_stderr.code=7
utf8_stderr.code_points=U+0061
utf8_stderr.replacement_character=false
utf8_stderr.truncated_bytes=5
identity_json_control.code=0
identity_json_control.unchanged=true
Observed result after fix: The unchanged default path reproduced the invalid partial-character decode. The UTF-8-specific production path returned a for both stdout and stderr, contained no replacement character, and accounted for all five omitted bytes. A normal identity JSON payload was returned unchanged with exit code 0.
What was not tested: A physical Windows gateway connected to a separate SSH server or a live device-pairing approval. The changed real Windows child-process capture, UTF-8 boundary cleanup, stdout/stderr decoding, normal JSON control, and Windows CI test path were exercised directly.

Risk / Compatibility

Low and localized. Existing callers keep the same automatic Windows decoding behavior. Only callers using the new internal UTF-8 runner receive forced UTF-8 decoding and boundary cleanup; this PR changes the SSH identity probe only.

clawsweeper · 2026-07-12T10:24:18Z

Codex review: needs real behavior proof before merge. Reviewed July 18, 2026, 5:34 PM ET / 21:34 UTC.

Summary
The PR adds a UTF-8-only bounded command runner, routes the SSH node-pairing identity probe through it, and adds Windows output-capture coverage.

PR surface: Source +22, Tests +68, Config 0. Total +90 across 7 files.

Reproducibility: no. high-confidence current-head reproduction is available in the evidence. The source and Windows-focused tests clearly identify the bounded UTF-8 decode path, but the live terminal run predates the relevant later branch change.

Review metrics: none identified.

Merge readiness
Overall: 🦐 gold shrimp
Proof: 🦐 gold shrimp
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Post a redacted Windows 2025 terminal transcript from 28c4cea9ceb528264e4ded3fff80a8d1da744db4 showing truncated UTF-8 stdout and stderr plus the unchanged identity-JSON control.
After updating the PR body with that proof, request a fresh review if one is not triggered automatically.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The PR contains strong redacted Windows terminal output for an earlier head, but the relevant decode branch changed afterward; add equivalent current-head Windows stdout/stderr and identity-JSON proof, with private paths, endpoints, and credentials redacted. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

[P1] The current head lacks equivalent real Windows stdout/stderr proof after the later UTF-8/code-page branch change, so a regression in the actual Windows decode path remains unverified before merge.

Maintainer options:

Decide the mitigation before merge
Retain the isolated UTF-8 runner and add a redacted Windows 2025 current-head terminal run that exercises truncated stdout and stderr plus the normal SSH identity-JSON control, then review the exact head with required checks.
Pause or close
Do not merge this PR until maintainers decide whether the risk is worth taking.

Next step before merge

[P1] No mechanical repair is indicated; the remaining merge gate is contributor-supplied current-head Windows behavior proof.

Security
Cleared: The patch adds no dependency, privilege, secret, workflow, or command-construction expansion; it preserves argv-based invocation and narrows decoding for a known UTF-8 caller.

Review details

Best possible solution:

Retain the isolated UTF-8 runner and add a redacted Windows 2025 current-head terminal run that exercises truncated stdout and stderr plus the normal SSH identity-JSON control, then review the exact head with required checks.

Do we have a high-confidence way to reproduce the issue?

No high-confidence current-head reproduction is available in the evidence. The source and Windows-focused tests clearly identify the bounded UTF-8 decode path, but the live terminal run predates the relevant later branch change.

Is this the best way to solve the issue?

Yes, conditionally: a separate UTF-8-only runner is narrower and safer than changing the default Windows code-page-detecting path, provided current-head Windows proof confirms stdout, stderr, and the SSH JSON control.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 77a42adce4d5.

Label changes

Label justifications:

P2: This is a focused Windows process-output correctness fix with bounded impact on SSH node pairing rather than an urgent broad runtime outage.
rating: 🦐 gold shrimp: Overall readiness is 🦐 gold shrimp; proof is 🦐 gold shrimp and patch quality is 🐚 platinum hermit.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR contains strong redacted Windows terminal output for an earlier head, but the relevant decode branch changed afterward; add equivalent current-head Windows stdout/stderr and identity-JSON proof, with private paths, endpoints, and credentials redacted. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Evidence reviewed

PR surface:

Source +22, Tests +68, Config 0. Total +90 across 7 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	4	36	14	+22
Tests	2	69	1	+68
Docs	0	0	0	0
Config	1	1	1	0
Generated	0	0	0	0
Other	0	0	0	0
Total	7	106	16	+90

What I checked:

Focused implementation boundary: The branch keeps the default runner intact while introducing a UTF-8-specific runner that bypasses Windows console-code-page decoding only for callers with an explicit UTF-8 contract. (src/process/exec-runner.ts:68, 28c4cea9ceb5)
Gateway caller: The SSH node-pairing identity probe is the only production caller changed to opt into the UTF-8-specific path, matching the stated OpenSSH JSON-output contract. (src/gateway/node-pairing-ssh-verify.runtime.ts:70, 28c4cea9ceb5)
Current-head proof gap: The PR body’s real Windows run is explicitly tied to e22cfd34043e253010e2e5f83e4afb2077dda1ad, while the later commit 4389f844e3d5f4d409746de9a77e178caef9a54f changes the relevant UTF-8/code-page branch and current head is later still. (src/process/exec-runner.ts:139, 4389f844e3d5)
Feature provenance: The related merged process refactor established the current Execa-backed process boundary; this PR is a narrow follow-up rather than a competing execution architecture. (src/process/exec.ts:10, db02a96c4c7a)

Likely related people:

steipete: Authored the later UTF-8/code-page adjustment on this branch, is assigned to the PR, and authored the merged process-runner refactor that introduced the current execution boundary. (role: recent area contributor; confidence: high; commits: 4389f844e3d5, db02a96c4c7a; files: src/process/exec-runner.ts, src/process/exec.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (13 earlier review cycles; latest 8 shown)

reviewed 2026-07-16T11:22:35.678Z sha f7701a4 :: needs maintainer review before merge. :: none
reviewed 2026-07-16T13:27:18.311Z sha e22cfd3 :: needs real behavior proof before merge. :: none
reviewed 2026-07-16T15:36:01.421Z sha e22cfd3 :: needs maintainer review before merge. :: none
reviewed 2026-07-16T16:22:35.933Z sha e22cfd3 :: needs maintainer review before merge. :: none
reviewed 2026-07-17T00:02:16.320Z sha 75a323c :: needs maintainer review before merge. :: none
reviewed 2026-07-17T17:45:31.706Z sha 5e96209 :: needs maintainer review before merge. :: none
reviewed 2026-07-18T01:12:56.965Z sha 1ae9cd4 :: needs maintainer review before merge. :: none
reviewed 2026-07-18T21:09:18.857Z sha 4389f84 :: needs real behavior proof before merge. :: none

mushuiyu886 · 2026-07-12T12:24:15Z

@clawsweeper re-review

clawsweeper · 2026-07-12T12:24:18Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29192783249
Updated: 2026-07-12T12:39:44.395Z

mushuiyu886 · 2026-07-16T11:50:53Z

@clawsweeper re-review

clawsweeper · 2026-07-16T11:50:56Z

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

mushuiyu886 · 2026-07-16T15:28:02Z

@clawsweeper re-review

clawsweeper · 2026-07-16T15:28:06Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The durable review comment was published and its verdict was queued for target-serialized routing.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29522033593
Updated: 2026-07-16T18:02:20.002Z

Co-authored-by: 杨浩宇0668001029 <[email protected]>

steipete · 2026-07-18T21:44:29Z

Merged via squash.

Prepared head SHA: 28c4cea9ceb528264e4ded3fff80a8d1da744db4
Landed commit: 4e11da2872eccadeaf4fbd9b7c8a6bf4b229adbe

…penclaw#105274) * fix(process): preserve byte-capped UTF-8 output on Windows * fix(process): skip codepage probe for UTF-8 output Co-authored-by: 杨浩宇0668001029 <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]>

openclaw-barnacle Bot added gateway Gateway runtime size: S labels Jul 12, 2026

fix(process): preserve byte-capped UTF-8 output on Windows

983f957

mushuiyu886 force-pushed the fix/followup-104180 branch from 294cd8d to 983f957 Compare July 16, 2026 06:58

mushuiyu886 changed the title ~~fix(gateway): cap SSH pairing output by UTF-8 bytes~~ fix(process): UTF-8 command output corrupts at Windows byte limits Jul 16, 2026

Merge branch 'main' into fix/followup-104180

0cdea64

Merge remote-tracking branch 'upstream/main' into pr-105274-windows-utf8

f7701a4

BradGroux mentioned this pull request Jul 16, 2026

WORKING: All Microsoft Issues and PRs (refresh) #74163

Draft

clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. labels Jul 16, 2026

mushuiyu886 added 3 commits July 17, 2026 07:55

Merge branch 'main' into fix/followup-104180

75a323c

Merge upstream/main into fix/followup-104180

5e96209

Merge upstream/main into fix/followup-104180

1ae9cd4

clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels Jul 18, 2026

steipete self-assigned this Jul 18, 2026

fix(process): skip codepage probe for UTF-8 output

4389f84

Co-authored-by: 杨浩宇0668001029 <[email protected]>

Merge branch 'main' into fix/followup-104180

28c4cea

steipete merged commit 4e11da2 into openclaw:main Jul 18, 2026
117 checks passed

steipete mentioned this pull request Jul 18, 2026

fix(process): preserve Windows output encoding through truncation #110764

Closed

github-actions Bot mentioned this pull request Jul 18, 2026

📡 Upstream Digest — 2026-07-18 22:39 UTC curtismercier/openclaw-mods#1339

Open

masatohoshino mentioned this pull request Jul 25, 2026

[Bug]: exec / read tools mangle GBK-encoded output on Windows (cp936), corrupting transcript and causing empty-response loop #113219

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(process): UTF-8 command output corrupts at Windows byte limits#105274

fix(process): UTF-8 command output corrupts at Windows byte limits#105274
steipete merged 9 commits into
openclaw:mainfrom
mushuiyu886:fix/followup-104180

mushuiyu886 commented Jul 12, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 12, 2026 •

edited

Loading

Uh oh!

mushuiyu886 commented Jul 12, 2026

Uh oh!

clawsweeper Bot commented Jul 12, 2026 •

edited

Loading

Uh oh!

mushuiyu886 commented Jul 16, 2026

Uh oh!

clawsweeper Bot commented Jul 16, 2026

Uh oh!

mushuiyu886 commented Jul 16, 2026

Uh oh!

clawsweeper Bot commented Jul 16, 2026 •

edited

Loading

Uh oh!

Uh oh!

steipete commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

mushuiyu886 commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Real behavior proof

Risk / Compatibility

Uh oh!

clawsweeper Bot commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mushuiyu886 commented Jul 12, 2026

Uh oh!

clawsweeper Bot commented Jul 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mushuiyu886 commented Jul 16, 2026

Uh oh!

clawsweeper Bot commented Jul 16, 2026

Uh oh!

mushuiyu886 commented Jul 16, 2026

Uh oh!

clawsweeper Bot commented Jul 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

steipete commented Jul 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mushuiyu886 commented Jul 12, 2026 •

edited

Loading

clawsweeper Bot commented Jul 12, 2026 •

edited

Loading

clawsweeper Bot commented Jul 12, 2026 •

edited

Loading

clawsweeper Bot commented Jul 16, 2026 •

edited

Loading