Skip to content

fix(process): preserve UTF-8 at diagnostic truncation boundaries - #8907

Merged
morluto merged 2 commits into
morluto:mainfrom
Afloat16:fix/utf8-output-excerpt-boundaries
Nov 6, 2026
Merged

morluto merged 2 commits into
morluto:mainfrom
Afloat16:fix/utf8-output-excerpt-boundaries

Conversation

@Afloat16

@Afloat16 Afloat16 commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Byte-bounded command diagnostics can introduce replacement characters into valid UTF-8 output when the retained head or tail cuts through a multibyte character. For example, an eight-byte excerpt of A🙂...🙂Z shows broken emoji bytes around the omission marker. A one-shot StringDecoder.write() also hides some malformed bytes at the end of the retained head, such as 0xc0.

Changes

Decode the retained head after excluding only a suffix that could still become a valid UTF-8 code point. Preserve invalid lead bytes and already-invalid prefixes, including overlong encodings, surrogates, and values above U+10FFFF. Skip only the possible continuation-byte fragment at the cut edge of the tail. Raw retention stays within the existing byte budget.

Complete output is decoded after joining the head and tail. The non-truncated regression now uses A🙂, whose five-byte buffer is split inside the emoji by the three-byte head limit; every two-chunk split and bytewise append must reconstruct the exact original text.

The semantic owner is BoundedOutputExcerpt, shared by the bounded command runner's stdout/stderr diagnostics. This corrects an existing process-output boundary without adding an execution capability or changing byte limits, exit codes, or the tail policy.

Current validation

Head: 30aaca9bffbfad607bb008083f720120cf6024fc.

  • The repository runner (node scripts/run-development-tests.mjs local src/process/BoundedOutputExcerpt.test.ts src/process/ExecFileBoundedOutput.test.ts) passed 29 tests / 0 failures on Node 24.22.1 / Vitest 5.0.3: 25 module cases plus the existing four subprocess tests.
  • The same expanded tests against the previous head's source (ea26bd29891b3466297e86654e63978fb7de8a72) reported 9 failed / 20 passed, exposing the malformed-byte loss.
  • A real subprocess probe using the reported malformed head preserved A� … [output omitted] … BZ in both stdout and stderr after the fix, with exit code 7 and truncation unchanged. The prior source lost the replacement character.
  • Repository-pinned oxfmt 0.72.0 and oxlint 1.87.0, including the repository's custom lint plugins and --max-warnings 0, passed for both changed files. Targeted strict TypeScript 5.9.3 checking of the two modules and their tests, plus git diff --check, also passed.

The full project checks and real Hopper setup were not run locally. The hosted CI run 41866035849 has now completed successfully on this exact head, 30aaca9bffbfad607bb008083f720120cf6024fc.

Author readiness

Human owner: @Afloat16

  • I reviewed the final diff and relevant existing code, understand the change and its edge cases, and take responsibility for the complete PR, including AI-assisted work.
  • I checked the evidence above, confirmed it supports the current change, and recorded meaningful validation gaps.
  • I declare this PR ready for initial maintainer review.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-11-04T14:03:00.969396Z
We couldn’t complete this review. Comment @codex review to try again.
ea26bd2 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex usage limits have been reached for code reviews. Please check with the admins of this repo to increase the limits by adding credits.
Repo admins can enable using credits for code reviews in their settings.

@greptile-apps

greptile-apps Bot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5 Tier: apex

[Low impact] The PR appears safe to merge; both previous findings are fixed and no new actionable issues were found.

Summary

Preserves whole UTF-8 characters around the diagnostic omission marker while keeping malformed-byte evidence visible.

  • Truncated command diagnostics keep valid UTF-8 readable at the excerpt edges.

Reviews (2) · Last reviewed commit: "fix(process): preserve malformed UTF-8 e..." · Reviewed by Greptile

Comment thread src/process/BoundedOutputExcerpt.ts Resolved

Afloat16 commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Both CI and author readiness are now passing on current head 30aaca9bffbfad607bb008083f720120cf6024fc. The regression evidence and validation scope are recorded in the PR description. Could a maintainer review this for merge?

@morluto
morluto merged commit 3eff806 into morluto:main Nov 6, 2026
48 of 51 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants