Skip to content

fix(agents): keep extension exec truncation UTF-16 safe#102007

Closed
mushuiyu886 wants to merge 2 commits into
openclaw:mainfrom
mushuiyu886:fix/followup-101513
Closed

fix(agents): keep extension exec truncation UTF-16 safe#102007
mushuiyu886 wants to merge 2 commits into
openclaw:mainfrom
mushuiyu886:fix/followup-101513

Conversation

@mushuiyu886

Copy link
Copy Markdown
Contributor

Summary

  • Keep extension/custom-tool exec retained stdout and stderr UTF-16 safe when output is caller-capped.
  • Reuse the shared sliceUtf16Safe helper for both tail-retention and default output-limit truncation.
  • Add regression coverage for caller-supplied caps and the default output-limit path.

What Problem This Solves

Extension runtime commands can return stdout/stderr containing non-BMP characters such as emoji. execCommand retained output with String.prototype.slice, so a cap that landed between UTF-16 surrogate halves could return a dangling surrogate in the retained stream.

The anchor PR fixed the same class of truncation bug for exec auto-reviewer rationale text. This follow-up applies the same shared UTF-16-safe truncation invariant to extension/custom-tool command output retention.

User Impact

Users running extensions or custom tools that emit emoji or other non-BMP text could receive broken retained stdout/stderr after truncation. That broken text can be surfaced as command output, diagnostics, or model-visible tool output. After this patch, truncation may retain one fewer UTF-16 code unit at a split boundary, but it will not return an invalid surrogate half.

Origin / follow-up

Follows and completes #101513.

  • Anchor: fix(agents): keep exec auto-reviewer rationale truncation UTF-16 safe #101513 fixed exec auto-reviewer rationale truncation by using UTF-16-safe truncation.
  • Gap this fills: src/agents/sessions/exec.ts still used raw slice() for retained stdout/stderr in the extension/custom-tool command execution path.
  • Consistency: This patch reuses the existing @openclaw/normalization-core/utf16-slice helper instead of introducing another truncation implementation.

Competition / linked PR analysis

I checked current open PRs and issues for the same execCommand / extension exec UTF-16 truncation gap.

  • No open issue matched extension exec output UTF-16 surrogate or execCommand truncation surrogate.
  • No open PR matched execCommand truncation UTF-16 safe.
  • One open PR mentions surrogate-safe truncation in a different memory-core dreaming snippet path; it does not touch src/agents/sessions/exec.ts or the extension exec stdout/stderr retention path.
  • Related open PR scan: no same-file or same-runtime duplicate was found for this patch.

Real behavior proof

  • Behavior or issue addressed: extension/custom-tool exec retained stdout/stderr must not contain dangling UTF-16 surrogate halves after truncation.
  • Why it matters / User impact: extension command output is user-triggered text that can be returned as stdout/stderr; a split surrogate can show replacement characters or invalid text to the next consumer.
  • Canonical reachability path: user-triggered extension command output input → child-process pipe ingestion and Buffer-to-string type conversion in execCommandExecResult runtime object → retained stdout/stderr returned as the command effect.
  • Boundary crossed: child-process stdout/stderr pipe payloads are captured through the same execCommand retention logic used by extension command execution.
  • Shared helper / provider constraint check: existing repository helper sliceUtf16Safe is reused; provider-specific constraints are not applicable to this local command-output boundary.
  • Architecture / source-of-truth check: appendCapturedOutput is the source-of-truth retention boundary for ExecResult stdout/stderr caps. The patch fixes the invariant there instead of adding a downstream cleanup step, so stored/returned command output share the same safe retained text.
  • Real environment tested: local OpenClaw repo on Node/Vitest using the shipped child-process output capture boundary.
  • Exact steps or command run after this patch:
node scripts/test-projects.mjs src/agents/sessions/exec.test.ts
node --import tsx --input-type=module
# Imported src/agents/sessions/exec.ts, executed a child process that wrote
# A😀B to stdout and C😀D to stderr, and called execCommand(..., { maxOutputChars: 2 }).
  • Evidence after fix:
$ node scripts/test-projects.mjs src/agents/sessions/exec.test.ts
Test Files  1 passed (1)
Tests  7 passed (7)
[test] passed 1 Vitest shard in 11.57s
{
  "stdout": "B",
  "stderr": "D",
  "stdoutLength": 1,
  "stderrLength": 1,
  "stdoutCodes": ["42"],
  "stderrCodes": ["44"],
  "stdoutDangling": [],
  "stderrDangling": [],
  "stdoutTruncatedChars": 3,
  "stderrTruncatedChars": 3,
  "code": 0
}
  • Observed result after fix: stdout and stderr retain valid text with no dangling surrogate halves; truncation counts reflect the actual removed UTF-16 code units.
  • What was not tested: full dev gateway startup was not run because the changed boundary is the local execCommand child-process output capture path.
  • What did NOT change: no extension loader API, process spawning, timeout, abort, process-tree cleanup, model routing, or network behavior changed.
  • Target test file: src/agents/sessions/exec.test.ts.
  • Scenario locked in: caller-capped stdout A😀B and stderr C😀D with maxOutputChars: 2 now retain B and D without dangling surrogate halves; the default output-limit path also avoids retaining a half emoji at the cap boundary.
  • Why this is the smallest reliable guardrail: the regression hits execCommand at the retained-output source-of-truth boundary, covers both stdout and stderr, and also covers the default limit path where the same helper is now used.
  • Fix classification: Root cause fix
  • Maintainer-ready confidence: High. The diff is limited to two slicing call sites plus focused tests, and fresh verification is bound to the committed diff.
  • Patch quality notes: Patch quality warning considered: this is not a fallback, default-value change, catch/ignore path, or downstream masking step; it replaces unsafe slicing at the retention source.

Review findings addressed

No maintainer review findings are attached to this follow-up yet. This patch addresses the sibling gap left by #101513 in the extension exec output retention path.

Regression Test Plan

node scripts/test-projects.mjs src/agents/sessions/exec.test.ts

The regression suite now covers:

  • caller-capped retained stdout/stderr containing emoji at the truncation boundary;
  • default output-limit truncation when the cap would otherwise split an emoji surrogate pair;
  • existing independent stdout/stderr retention, process-tree cleanup, timeout, and stream-error behavior.

Merge risk

  • Risk labels considered: no merge-risk label is expected for this XS scoped agents/runtime retention patch; changed files are src/agents/sessions/exec.ts and src/agents/sessions/exec.test.ts only.
  • Risk explanation: retained output can become one UTF-16 code unit shorter at a surrogate boundary, but caps remain enforced and returned text becomes valid rather than partially split.
  • Why acceptable: this preserves the existing API and truncation behavior while enforcing the text-validity invariant already used by the shared helper.

Risk / Compatibility

Risk is low and localized to retained output truncation in execCommand.

  • API shape is unchanged.
  • Output caps remain enforced.
  • At a surrogate boundary, retained output may be one UTF-16 code unit shorter than the raw cap to avoid returning invalid text.
  • stdoutTruncatedChars and stderrTruncatedChars now count the actual number of removed UTF-16 code units after safe slicing.

What was not changed

  • No extension loader API changes.
  • No process spawning, timeout, abort, or process-tree cleanup behavior changes.
  • No model routing or network behavior changes.
  • No new truncation helper was added.

Root Cause

  • Root cause: execCommand used raw String.prototype.slice() to enforce retained output caps. JavaScript slicing is based on UTF-16 code units, so slicing at a cap boundary can return only one half of a surrogate pair.
  • Why this is root-cause fix: The fix replaces both affected slicing sites in appendCapturedOutput with the shared UTF-16-safe helper, covering tail retention for caller-supplied caps and head retention for default output-limit enforcement at the source where retained stdout/stderr is produced.

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jul 8, 2026
@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed July 8, 2026, 2:30 AM ET / 06:30 UTC.

Summary
The branch changes execCommand retained stdout/stderr truncation to use the shared UTF-16-safe slice helper and adds regression tests for caller-capped and default output-limit truncation.

PR surface: Source +4, Tests +60. Total +64 across 2 files.

Reproducibility: yes. Source inspection shows current appendCapturedOutput can retain a tail beginning at the low surrogate for A😀B with maxOutputChars: 2; I did not run tests because this cleanup review is read-only.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/agents/sessions/exec.test.ts, serialized state: src/agents/sessions/exec.ts, unknown-data-model-change: src/agents/sessions/exec.test.ts. Confirm migration or upgrade compatibility proof before merge.

Root-cause cluster
Relationship: canonical
Canonical: #102007
Summary: This PR is the canonical active item for the extension/custom-tool execCommand retained-output UTF-16 truncation gap; the merged sibling fixed the same class on different text surfaces.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Next step before merge

  • No ClawSweeper repair job is needed; the automerge-labeled PR has sufficient proof and no actionable code finding, so exact-head review and checks can gate merge.

Security
Cleared: No concrete security or supply-chain concern was found; the patch reuses an existing internal helper and does not change spawning, permissions, dependencies, secrets, or workflows.

Review details

Best possible solution:

Land the source-boundary fix after ordinary exact-head review and merge checks; any broader command-output decoding cleanup should stay separate.

Do we have a high-confidence way to reproduce the issue?

Yes. Source inspection shows current appendCapturedOutput can retain a tail beginning at the low surrogate for A😀B with maxOutputChars: 2; I did not run tests because this cleanup review is read-only.

Is this the best way to solve the issue?

Yes. Fixing appendCapturedOutput with the existing shared helper is the narrow source-boundary fix, and it is cleaner than sanitizing stdout/stderr downstream or adding a duplicate truncation helper.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against c46e76cff724.

Label changes

Label changes:

  • add status: 🚀 automerge armed: This PR is in ClawSweeper's automerge lane. Sufficient (terminal): The PR body includes after-fix terminal proof from a real execCommand child-process run plus focused test output showing valid retained stdout/stderr without dangling surrogates.
  • remove status: 👀 ready for maintainer look: Current PR status label is status: 🚀 automerge armed.

Label justifications:

  • P3: This is a small, low-blast-radius text-validity bug fix for retained exec output with no API, config, auth, persistence, or delivery contract change.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 🚀 automerge armed: This PR is in ClawSweeper's automerge lane. Sufficient (terminal): The PR body includes after-fix terminal proof from a real execCommand child-process run plus focused test output showing valid retained stdout/stderr without dangling surrogates.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body includes after-fix terminal proof from a real execCommand child-process run plus focused test output showing valid retained stdout/stderr without dangling surrogates.
Evidence reviewed

PR surface:

Source +4, Tests +60. Total +64 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 6 2 +4
Tests 1 63 3 +60
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 69 5 +64

What I checked:

  • Repository policy read: Root AGENTS.md and the scoped src/agents/AGENTS.md were read fully; their deep-review, current/source/test/history, and agent-test guidance shaped this review. (AGENTS.md:1, c46e76cff724)
  • Current main still has the bug surface: On current main, appendCapturedOutput keeps caller-capped tails and default-limit heads with raw combined.slice(...), so a cap can split a surrogate pair. (src/agents/sessions/exec.ts:67, c46e76cff724)
  • Latest release has the same raw slicing: The latest release tag also contains the same raw combined.slice(...) retained-output truncation, so current main has not already superseded the PR and the shipped path is not already fixed. (src/agents/sessions/exec.ts:66, e085fa1a3ffd)
  • PR source fix: The PR head imports sliceUtf16Safe, applies it at the retained-output boundary, and updates truncated counts from the actual safe retained text length. (src/agents/sessions/exec.ts:67, 904510504a08)
  • Regression coverage: The PR adds tests for caller-capped stdout/stderr with emoji at the truncation boundary, an exact-fit surrogate pair case, and default output-limit truncation without dangling surrogates. (src/agents/sessions/exec.test.ts:142, 904510504a08)
  • Helper contract: sliceUtf16Safe is an existing dependency-free helper that adjusts slice boundaries away from dangling UTF-16 surrogate halves, and the package already exports the subpath used by this PR. (packages/normalization-core/src/utf16-slice.ts:15, c46e76cff724)

Likely related people:

  • vincentkoc: History points to commit ba7af36 by Vincent Koc as the commit that added the current execCommand, appendCapturedOutput, extension exec delegate, and shared helper files on main. (role: introduced current behavior; confidence: medium; commits: ba7af3630673; files: src/agents/sessions/exec.ts, src/agents/sessions/exec.test.ts, src/agents/sessions/extensions/loader.ts)
  • wm0018: Authored merged PR fix(agents): keep exec auto-reviewer rationale truncation UTF-16 safe #101513, which applied the same UTF-16-safe truncation invariant to exec auto-reviewer and prompt-template text surfaces. (role: adjacent related fix contributor; confidence: medium; commits: e7d617d4d936; files: src/agents/exec-auto-reviewer.ts, src/agents/sessions/prompt-templates.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.
Review history (2 earlier review cycles)
  • reviewed 2026-07-08T04:00:08.222Z sha a7fb763 :: needs maintainer review before merge. :: none
  • reviewed 2026-07-08T04:08:01.514Z sha a7fb763 :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. labels Jul 8, 2026
@Takhoffman

Copy link
Copy Markdown
Contributor

@clawsweeper automerge

@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞🔧
ClawSweeper automerge is enabled.

Draft PRs stay fix-only until GitHub marks them ready for review. Pause with /clawsweeper stop.

Automerge progress:

  • 2026-07-08 06:07:21 UTC review queued a7fb763ef2f8 (queued)
  • 2026-07-08 06:20:49 UTC review queued 904510504a08 (after repair)
  • 2026-07-08 06:07:21 UTC review queued 904510504a08 (queued)

@clawsweeper clawsweeper Bot added the clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge label Jul 8, 2026
@clawsweeper clawsweeper Bot added the clawsweeper:human-review Needs maintainer review before ClawSweeper can continue label Jul 8, 2026
@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞✅
ClawSweeper is pausing this repair loop for human review.

Source: clawsweeper[bot]
Reason: - No ClawSweeper repair lane is needed because the PR has sufficient proof and no actionable code finding; maintainers can use normal review and merge checks.; Cleared: No concrete security or supply-chain concern was found; the diff reuses an existing internal helper and does not change spawning, permissions, dependencies, secrets, or workflows. (sha=a7fb763ef2f8cf5731cb5c44bc0d59db6324a239)

Why human review is needed:
This item has security-sensitive risk. ClawSweeper is pausing instead of making an autonomous change that could affect trust, credentials, permissions, or exposure.

What the maintainer can do as a next step:
If the maintainer accepts the current risk and wants ClawSweeper to continue merge gates, comment @clawsweeper approve. If the security-sensitive detail still needs changes, describe the safe path or push the fix, then comment @clawsweeper automerge. If the risk should not be automated, keep the PR paused for manual review or comment @clawsweeper stop.

I added clawsweeper:human-review and left the final call with a maintainer.

@clawsweeper clawsweeper Bot removed the clawsweeper:human-review Needs maintainer review before ClawSweeper can continue label Jul 8, 2026
@clawsweeper
clawsweeper Bot force-pushed the fix/followup-101513 branch from a7fb763 to 9045105 Compare July 8, 2026 06:20
@clawsweeper clawsweeper Bot added status: 🚀 automerge armed This PR is in ClawSweeper's automerge lane. clawsweeper:human-review Needs maintainer review before ClawSweeper can continue and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jul 8, 2026
@clawsweeper

clawsweeper Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

🦞✅
ClawSweeper is pausing this repair loop for human review.

Source: clawsweeper[bot]
Reason: - No ClawSweeper repair job is needed; the automerge-labeled PR has sufficient proof and no actionable code finding, so exact-head review and checks can gate merge.; Cleared: No concrete security or supply-chain concern was found; the patch reuses an existing internal helper and does not change spawning, permissions, dependencies, secrets, or workflows. (sha=904510504a089f404293092e2760b59fb9314fe0)

Why human review is needed:
This item has security-sensitive risk. ClawSweeper is pausing instead of making an autonomous change that could affect trust, credentials, permissions, or exposure.

What the maintainer can do as a next step:
If the maintainer accepts the current risk and wants ClawSweeper to continue merge gates, comment @clawsweeper approve. If the security-sensitive detail still needs changes, describe the safe path or push the fix, then comment @clawsweeper automerge. If the risk should not be automated, keep the PR paused for manual review or comment @clawsweeper stop.

I added clawsweeper:human-review and left the final call with a maintainer.

vincentkoc added a commit that referenced this pull request Jul 8, 2026
Consolidates #102007, #101818, and #101782.

Co-authored-by: 杨浩宇0668001029 <[email protected]>
Co-authored-by: NIO <[email protected]>
Co-authored-by: Cursor <[email protected]>
@vincentkoc vincentkoc self-assigned this Jul 8, 2026
@vincentkoc

Copy link
Copy Markdown
Member

Landed the canonical fix on main as 7e03242.

The valid execCommand UTF-16 truncation fix is included, and @mushuiyu886's authorship is preserved with a co-author trailer. I direct-landed one consolidated change because #102007, #101818, and #101782 fix the same boundary class across reachable owners, while keeping them separate would duplicate review/test work and #101782 conflicted with current main. The latest ClawSweeper-added exact-fit test is already covered by the shared helper contract and the included application-path regressions.

Proof on the landed patch:

Closing this PR as included in the landed main commit, not rejected.

@vincentkoc vincentkoc closed this Jul 8, 2026
giodl73-repo pushed a commit to giodl73-repo/openclaw that referenced this pull request Jul 8, 2026
Consolidates openclaw#102007, openclaw#101818, and openclaw#101782.

Co-authored-by: 杨浩宇0668001029 <[email protected]>
Co-authored-by: NIO <[email protected]>
Co-authored-by: Cursor <[email protected]>
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jul 9, 2026
Consolidates openclaw#102007, openclaw#101818, and openclaw#101782.

Co-authored-by: 杨浩宇0668001029 <[email protected]>
Co-authored-by: NIO <[email protected]>
Co-authored-by: Cursor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling clawsweeper:automerge Maintainer opted this PR into bounded ClawSweeper-reviewed automerge clawsweeper:human-review Needs maintainer review before ClawSweeper can continue P3 Low-priority cleanup, docs, polish, ergonomics, or speculative work. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: S status: 🚀 automerge armed This PR is in ClawSweeper's automerge lane.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants