fix(file-transfer): keep bounded stderr tail UTF-16 safe by hugenshen · Pull Request #104151 · openclaw/openclaw

hugenshen · 2026-07-11T04:07:14Z

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers
can help update the branch when needed.

What Problem This Solves

Fixes an issue where file-transfer tar/dir-fetch stderr tail truncation can produce invalid Unicode when paths or archive listings contain emoji or other supplementary-plane characters.

appendBoundedTextTail() capped raw child.stderr chunks with next.slice(-maxChars) and failure reasons still used raw stderr.slice(-200/-300) at projection time. Tar stderr is unfiltered, so filenames like 📊report.csv can land on a UTF-16 surrogate boundary and leave a lone high/low surrogate in error text surfaced to agents/operators.

Concrete example: with production ring cap 4096 and input "x"×4095 + "🤖" + "y"×4095, raw .slice(-4096) keeps U+DD16 (lone low surrogate) without its high half; with reason cap 200 and input "🤖" + "f"×199, raw .slice(-200) has the same failure.

Why This Change Was Made

Extract shared helpers in extensions/file-transfer/src/shared/append-bounded-text-tail.ts:

appendBoundedTextTail — UTF-16-safe ring buffer for streaming stderr
projectBoundedTextTail — UTF-16-safe final projection for error reasons

Both use sliceUtf16Safe from openclaw/plugin-sdk/text-utility-runtime, matching the SCP stderr fix (#103757) pattern. Ring-buffer sizes and tar/dir-fetch behavior are unchanged; only truncation boundaries are surrogate-safe end-to-end.

User Impact

Before: dir_fetch / archive-list failures can surface replacement glyphs or invalid Unicode in agent/operator error text when stderr mentions emoji paths near the 4,096-char ring-buffer cap or 200/300-char error projection.

After: File-transfer stderr tails and surfaced failure reasons remain well-formed Unicode; boundary emoji are dropped whole instead of split.

Evidence

extensions/file-transfer/src/shared/append-bounded-text-tail.ts — shared UTF-16-safe ring buffer + projection helpers
extensions/file-transfer/src/tools/dir-fetch-tool.ts — tar list/budget/unpack stderr paths use both helpers
extensions/file-transfer/src/shared/node-invoke-policy.ts — archive list stderr path uses both helpers
Regression: append-bounded-text-tail.test.ts, caller-path tests in dir-fetch-tool.test.ts and node-invoke-policy.stream-errors.test.ts (fixtures place 🤖 so raw slice(-200) starts on the low surrogate)

Real behavior proof

Behavior or issue addressed: File-transfer bounded stderr tails and final tar failure reasons no longer leave unpaired UTF-16 surrogates when emoji cross the ring-buffer cap or 200/300-char error projection.
Canonical reachability path: dir_fetch tool preValidateTarball / node listDirFetchArchiveEntries → spawn(tar, …) → child.stderr chunks → appendBoundedTextTail ring buffer → projectBoundedTextTail in failure reason returned to agent/operator.
Boundary crossed: local-runtime; real Node child using the same stderr EventEmitter wiring as consumeChildOutput (production 4096 / 200 caps engineered to bisect 🤖), plus real /usr/bin/tar with an emoji missing path.
Shared helper / provider constraint check: Reused sliceUtf16Safe from openclaw/plugin-sdk/text-utility-runtime; provider/channel constraints N/A because this is plugin-local tar stderr handling.
Real environment tested: macOS (darwin), Node v22.22.0, branch fix/file-transfer-bounded-tail-utf16, /usr/bin/tar.
Exact steps or command run after this patch:

$ pnpm exec tsx .file-transfer-utf16-proof.mjs
$ node scripts/run-vitest.mjs extensions/file-transfer/src/shared/append-bounded-text-tail.test.ts extensions/file-transfer/src/tools/dir-fetch-tool.test.ts extensions/file-transfer/src/shared/node-invoke-policy.stream-errors.test.ts --reporter=verbose

Evidence after fix:

=== Real subprocess proof: file-transfer UTF-16 stderr tails ===
platform: darwin, node: v22.22.0

--- Case A: production 4096 ring bisects 🤖 (before/after) ---
payload code units: 8192
production ring cap: 4096, reason cap: 200
raw ring first code unit: U+DD16
raw ring unpaired_surrogates: true
raw reason unpaired_surrogates: false
fixed ring length: 4095
fixed ring first code unit: U+0079
fixed ring unpaired_surrogates: false
fixed reason unpaired_surrogates: false
fixed reason prefix: "tar -tzf exited 1: yyyyyyyyyyyyyyyyyyyyyyyyyyyyy"…
boundary before/after pass: true

--- Case B: production 200 reason projection bisects 🤖 (before/after) ---
payload code units: 201
production ring cap: 4096, reason cap: 200
raw ring first code unit: U+D83E
raw ring unpaired_surrogates: false
raw reason unpaired_surrogates: true
fixed ring length: 201
fixed ring first code unit: U+D83E
fixed ring unpaired_surrogates: false
fixed reason unpaired_surrogates: false
fixed reason prefix: "tar -tzf exited 1: fffffffffffffffffffffffffffff"…
boundary before/after pass: true

--- Case C: real /usr/bin/tar (emoji filename in stderr) ---
tar: /usr/bin/tar
stderr length: 4096
stderr unpaired_surrogates: false
reason unpaired_surrogates: false
reason tail: "e or directory\ntar: 📊missing.csv: Cannot stat: No such file or directory\ntar: Error exit delayed from previous errors.\n"

=== Summary ===
ring boundary before/after: true
reason boundary before/after: true
real tar surrogate-safe: true
all boundary proofs pass: true

$ node scripts/run-vitest.mjs … --reporter=verbose
 ✓ surfaces UTF-16 safe tar stderr tail when listing fails with emoji at projection boundary
 ✓ surfaces UTF-16 safe tar stderr tail when archive listing fails with emoji at projection boundary
 Test Files  3 passed (3)
      Tests  14 passed (14)

Observed result after fix: Case A shows raw production 4096 ring starts on lone low surrogate U+DD16 while patched appendBoundedTextTail drops that half and retains a well-formed y×4095 tail. Case B shows raw slice(-200) reason projection unpaired while projectBoundedTextTail drops 🤖 and keeps f×199. Case C runs real /usr/bin/tar with 📊missing.csv through the same helpers at the 4,096 ring cap with unpaired_surrogates=false.
What was not tested: Live dir_fetch / node archive-list on a remote host; Linux GNU tar stderr encoding differences.
Fix classification: Root cause fix.
AI-assisted (Cursor)
I understand what the code does
Change is focused and does not mix unrelated concerns
Single commit on upstream/main; no bundled scripts/ or release-script changes

clawsweeper · 2026-07-11T04:20:15Z

Codex review: needs maintainer review before merge. Reviewed July 11, 2026, 11:14 AM ET / 15:14 UTC.

Summary
Adds shared UTF-16-safe bounded stderr accumulation and projection helpers for file-transfer tar operations, applies them to list, budget, unpack, and node archive-list failures, and adds boundary regression tests.

PR surface: Source +20, Tests +91. Total +111 across 6 files.

Reproducibility: yes. Current main's raw bounded slices deterministically leave a lone surrogate at the stated code-unit boundaries, and the affected production tar stderr paths are directly traceable.

Review metrics: none identified.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Next step before merge

No automated repair is needed; routine maintainer review and exact-head merge gating are the remaining actions.

Security
Cleared: The patch changes only plugin-local string handling and tests, with no dependency, workflow, permission, secret, install, artifact-download, or publishing changes.

Review details

Best possible solution:

Merge the focused exact-head fix after routine maintainer review; it preserves existing caps and error semantics while making every affected file-transfer truncation boundary surrogate-safe.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main's raw bounded slices deterministically leave a lone surrogate at the stated code-unit boundaries, and the affected production tar stderr paths are directly traceable.

Is this the best way to solve the issue?

Yes. The PR applies the repository's existing exported safe-slice contract at both streaming and final projection boundaries, covers every matching file-transfer path, and does not alter limits or unrelated behavior.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against f5060701881b.

Label changes

Label justifications:

P2: This fixes rare malformed diagnostic output in one plugin without affecting successful transfers, configuration, protocols, or persisted state.
rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR provides observed before-and-after terminal output for both production truncation boundaries and a real /usr/bin/tar subprocess, supplemented by caller-path regression tests.
proof: sufficient: Contributor real behavior proof is sufficient. The PR provides observed before-and-after terminal output for both production truncation boundaries and a real /usr/bin/tar subprocess, supplemented by caller-path regression tests.

Evidence reviewed

PR surface:

Source +20, Tests +91. Total +111 across 6 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	3	34	14	+20
Tests	3	92	1	+91
Docs	0	0	0	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	6	126	15	+111

What I checked:

Current-main bug path: Current main still uses raw .slice(-200) and .slice(-300) projections that can begin on the low surrogate of a supplementary-plane character. (extensions/file-transfer/src/tools/dir-fetch-tool.ts:140, f5060701881b)
Complete affected-surface repair: The PR head routes all three file-transfer tar stderr accumulators and all four bounded failure projections through the shared safe helpers. (extensions/file-transfer/src/tools/dir-fetch-tool.ts:128, 7b3d27f8bbca)
UTF-16 contract: The reused SDK helper explicitly adjusts slice boundaries to avoid returning dangling high or low surrogate halves. (packages/normalization-core/src/utf16-slice.ts:15, f5060701881b)
Plugin SDK boundary: openclaw/plugin-sdk/text-utility-runtime is a declared exported plugin SDK subpath, so the new import follows the extensions boundary. (package.json:1428, f5060701881b)
Established repository pattern: Merged commit 6f6c802 repaired the same UTF-16 tail-truncation invariant in restart diagnostics using sliceUtf16Safe. (src/infra/restart-sentinel.ts:373, 6f6c802ea513)
Exact-head validation: Real-behavior proof, security scans, lint, types, builds, plugin boundaries, and quality checks pass on the reviewed head. (7b3d27f8bbca)

Likely related people:

steipete: Current-main blame and file history attribute the file-transfer tar stderr implementation and exported UTF-16 helper surface to commit d5fe0c5; the same person also merged the adjacent safe-tail precedent. (role: introduced current behavior and adjacent merger; confidence: high; commits: d5fe0c5e73d9, 6f6c802ea513; files: extensions/file-transfer/src/tools/dir-fetch-tool.ts, extensions/file-transfer/src/shared/node-invoke-policy.ts, packages/normalization-core/src/utf16-slice.ts)
zhangguiping-xydt: Authored the merged restart-diagnostic fix that established the repository pattern of using sliceUtf16Safe for bounded diagnostic tails. (role: introduced adjacent safe-tail pattern; confidence: medium; commits: 6f6c802ea513; files: src/infra/restart-sentinel.ts, src/infra/restart-sentinel.test.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (6 earlier review cycles)

reviewed 2026-07-11T04:18:41.726Z sha cb75851 :: needs real behavior proof before merge. :: [P2] Make the surfaced 200-character tail surrogate-safe
reviewed 2026-07-11T04:41:05.600Z sha 4474952 :: needs real behavior proof before merge. :: [P2] Protect the final surfaced stderr tails
reviewed 2026-07-11T05:13:39.760Z sha 1f6ff7d :: needs real behavior proof before merge. :: none
reviewed 2026-07-11T05:20:46.073Z sha 1f6ff7d :: needs real behavior proof before merge. :: none
reviewed 2026-07-11T05:46:59.064Z sha 5d3801b :: needs real behavior proof before merge. :: none
reviewed 2026-07-11T15:07:16.213Z sha 7b3d27f :: needs maintainer review before merge. :: none

hugenshen · 2026-07-11T05:04:32Z

@clawsweeper re-review

Branch cleanup for this round:

Rebased on latest upstream/main; merge conflicts cleared
Single commit (1f6ff7d0f00) — squashed 3 commits into one
Removed unrelated scripts/package-changelog.mjs changes (separate concern)
Complete UTF-16 fix: appendBoundedTextTail ring buffer + projectBoundedTextTail on all slice(-200/-300) error projections in dir-fetch-tool and node-invoke-policy
Caller-path proof (14/14 pass):

$ node scripts/run-vitest.mjs extensions/file-transfer/src/shared/append-bounded-text-tail.test.ts extensions/file-transfer/src/tools/dir-fetch-tool.test.ts extensions/file-transfer/src/shared/node-invoke-policy.stream-errors.test.ts --reporter=verbose

 ✓ surfaces UTF-16 safe tar stderr tail when listing fails with emoji at projection boundary
 ✓ surfaces UTF-16 safe tar stderr tail when archive listing fails with emoji at projection boundary

Test Files  3 passed (3)
     Tests  14 passed (14)

PR body updated with caller-path evidence.

clawsweeper · 2026-07-11T05:04:34Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29140925363
Updated: 2026-07-11T05:22:11.935Z

hugenshen · 2026-07-11T05:33:01Z

@clawsweeper re-review — added real /usr/bin/tar subprocess proof (Cases A–C) to Real behavior proof section.

clawsweeper · 2026-07-11T05:33:03Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29141567239
Updated: 2026-07-11T05:48:28.883Z

hugenshen · 2026-07-11T14:57:10Z

@clawsweeper re-review

Boundary-specific real behavior proof added on head 7b3d27f8bbc (rebased onto current upstream/main, single commit):

Case A: production 4096 ring engineered so raw .slice(-4096) starts on lone low surrogate U+DD16 of 🤖; patched appendBoundedTextTail drops the half (unpaired_surrogates=false).
Case B: production 200 reason projection engineered so raw .slice(-200) is unpaired; projectBoundedTextTail keeps f×199 only.
Case C: real /usr/bin/tar with 📊missing.csv through the same helpers at the 4096 ring cap.

Caller-path fixtures updated so the 200-char projection actually bisects 🤖.

clawsweeper · 2026-07-11T14:57:12Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

State: Complete
Detail: The targeted re-review published an exact durable review comment and queued an executing target-wide serialized router scan.
Run: https://github.com/openclaw/clawsweeper/actions/runs/29157390249
Updated: 2026-07-11T15:16:30.128Z

sallyom · 2026-07-11T17:05:22Z

Merge-ready at 7b3d27f8bbca962eee4ddf6b1ce810c258b2e067.

Clean review with no actionable findings. No compatibility, configuration, protocol, persisted-state, or public API changes identified. This is the best narrow fix: it repairs both file-transfer stderr accumulation and final error projection at the plugin owner boundary.

Exact-head CI and real-behavior proof are green. This complete fix supersedes #104098, which addresses accumulation but not the final projection boundaries. Thanks @hugenshen!

…penclaw#104151) Co-authored-by: NIO <[email protected]>

openclaw-barnacle Bot added plugin: file-transfer size: XS labels Jul 11, 2026

openclaw-barnacle Bot added scripts Repository scripts size: S and removed size: XS labels Jul 11, 2026

clawsweeper Bot added the merge-risk: 🚨 automation 🚨 May affect CI, automerge, proof capture, label sync, or maintainer automation. label Jul 11, 2026

openclaw-barnacle Bot added size: M and removed size: S labels Jul 11, 2026

hugenshen force-pushed the fix/file-transfer-bounded-tail-utf16 branch from 1e2d2f0 to 1f6ff7d Compare July 11, 2026 05:04

openclaw-barnacle Bot added size: S and removed scripts Repository scripts size: M labels Jul 11, 2026

hugenshen force-pushed the fix/file-transfer-bounded-tail-utf16 branch from 1f6ff7d to 5d3801b Compare July 11, 2026 05:32

fix(file-transfer): keep bounded stderr tail UTF-16 safe end-to-end

7b3d27f

hugenshen force-pushed the fix/file-transfer-bounded-tail-utf16 branch from 5d3801b to 7b3d27f Compare July 11, 2026 14:48

sallyom self-assigned this Jul 11, 2026

sallyom merged commit a498fc5 into openclaw:main Jul 11, 2026
141 of 151 checks passed

github-actions Bot mentioned this pull request Jul 11, 2026

📡 Upstream Digest — 2026-07-11 18:48 UTC curtismercier/openclaw-mods#1264

Open

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 12, 2026

fix(file-transfer): keep bounded stderr tail UTF-16 safe end-to-end (o…

90033f6

…penclaw#104151) Co-authored-by: NIO <[email protected]>

sunlit-deng mentioned this pull request Jul 15, 2026

fix(agents): keep session tool stderr tails UTF-8 safe #108012

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(file-transfer): keep bounded stderr tail UTF-16 safe#104151

fix(file-transfer): keep bounded stderr tail UTF-16 safe#104151
sallyom merged 1 commit into
openclaw:mainfrom
hugenshen:fix/file-transfer-bounded-tail-utf16

hugenshen commented Jul 11, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

hugenshen commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

hugenshen commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

hugenshen commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

Uh oh!

sallyom commented Jul 11, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

hugenshen commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Real behavior proof

Uh oh!

clawsweeper Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hugenshen commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hugenshen commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hugenshen commented Jul 11, 2026

Uh oh!

clawsweeper Bot commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sallyom commented Jul 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hugenshen commented Jul 11, 2026 •

edited

Loading

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

clawsweeper Bot commented Jul 11, 2026 •

edited

Loading

sallyom commented Jul 11, 2026 •

edited

Loading