[hooks_runner] Fix UTF-8 decoding of hook output on Windows by dzylikecode · Pull Request #3772 · dart-lang/native

dzylikecode · 2026-10-30T08:51:34Z

hooks_runner decoded each stdout/stderr chunk using systemEncoding. Dart
subprocesses emit UTF-8 to pipes, so a Windows system code page such as GBK
corrupts non-ASCII hook output (你好 becomes 浣犲ソ). Decoding individual
chunks also loses characters when a multibyte sequence spans pipe events.

Decode both streams with utf8.decoder before forwarding text to the logger
and captured output. Add regression coverage for Chinese text and emoji,
split byte sequences, both capture modes, logging and nonzero exit codes.
Also verify the real hook-failure diagnostic preserves stdout and exception
text, and include utility tests in the package's default test paths.

Validation on Windows x64, Chinese system code page, Dart 3.13.7:

All four new process regression cases fail against the original source.
All nine process tests pass with the fix.
The real failed-hook UTF-8 regression passes.
dart analyze pkgs/hooks_runner/lib pkgs/hooks_runner/test passes.
The repository CI generation, license, static analysis, format and dependency
validation stages pass (the format check was rerun after formatting the new
integration test).
Full-suite execution also exposes a separate UTF-8 decoding issue in
native_toolchain_c: it does not correctly decode the UTF-8 output requested
by vswhere.exe -utf8, causing localized JSON to fail parsing in
VisualStudioResolver.parseVswhere. This PR fixes the hooks_runner layer
only; it addresses part of the encoding problem, rather than all UTF-8
handling issues across these packages. The native_toolchain_c decoding
issue remains unresolved by this PR. [native_toolchain_c] Decode vswhere UTF-8 output correctly on Windows #3773

PR Checklist

I’ve reviewed the contributor guide and applied the relevant portions to this PR.
I've run dart tool/ci.dart --all locally and resolved all issues identified. This ensures the PR is formatted, has no lint errors, and ran all code generators. This applies to the packages part of the toplevel pubspec.yaml workspace.
All existing and new tests are passing. I added new tests to check the change I am making.
The PR is actually solving the issue. PRs that don't solve the issue will be closed. Please be respectful of the maintainers' time. If it's not clear what the issue is, feel free to ask questions on the GitHub issue before submitting a PR.
I have updated CHANGELOG.md for the relevant packages. (Not needed for small changes such as doc typos).
I have updated the pubspec package version if necessary.

google-cla · 2026-10-30T08:51:59Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

dcharkes

Thanks @dzylikecode!

codecov · 2026-11-03T15:02:44Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.85%. Comparing base (c9e8118) to head (71dc20e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3772      +/-   ##
==========================================
- Coverage   85.85%   85.85%   -0.01%     
==========================================
  Files         322      322              
  Lines       20867    20844      -23     
==========================================
- Hits        17916    17896      -20     
+ Misses       2951     2948       -3

Flag	Coverage Δ
native_pkgs_macos	`81.77% <100.00%> (+0.03%)`	⬆️
native_pkgs_ubuntu	`69.19% <100.00%> (+0.02%)`	⬆️
native_pkgs_windows	`72.13% <100.00%> (+0.02%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
code_assets	`95.31% <ø> (ø)`
data_assets	`91.56% <ø> (ø)`
ffi	`100.00% <ø> (ø)`
ffigen	`91.45% <ø> (ø)`
hooks	`86.40% <ø> (ø)`
hooks_runner	`90.84% <100.00%> (+0.31%)`	⬆️
jni	`66.15% <ø> (-0.33%)`	⬇️
jni_flutter	`∅ <ø> (∅)`
jni_util	`25.00% <ø> (ø)`
jnigen	`82.85% <ø> (ø)`
json_syntax_generator	`42.01% <ø> (ø)`
native_test_helpers	`85.71% <ø> (ø)`
native_toolchain_c	`95.99% <ø> (ø)`
objective_c	`79.38% <ø> (-0.35%)`	⬇️
pub_formats	`∅ <ø> (∅)`
record_use	`81.48% <ø> (ø)`
snippet_tool	`66.23% <ø> (ø)`
swift2objc	`92.95% <ø> (ø)`
swiftgen	`80.41% <ø> (ø)`
test_case_selector	`97.28% <ø> (ø)`

Files with missing lines	Coverage Δ
pkgs/hooks_runner/lib/src/utils/run_process.dart	`94.23% <100.00%> (+6.29%)`	⬆️

... and 3 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…ooks-runner-utf8-output

…o fix/hooks-runner-utf8-output

Revisions updated by `dart tools/rev_sdk_deps.dart`. dartdoc (https://github.com/dart-lang/dartdoc/compare/2f6847b..03ec6df): 03ec6df6 Thu Nov 5 11:12:45 2026 +0200 Sergey G. Grekhov Add tests for type parameters (dart-lang/dartdoc#4308) 14a91302 Tue Nov 3 04:24:32 2026 +0100 Moritz Migrate to ecosystem health publish check and setup-dart publish workflow (dart-lang/dartdoc#4310) cd8bf52c Mon Nov 2 14:19:25 2026 +0100 Sigurd Meldgaard Support standalone AOT compilation and embed resources in html_resources.g.dart (dart-lang/dartdoc#4298) cb1b1b2f Mon Nov 2 14:39:28 2026 +0200 Sergey G. Grekhov Add tests for parameters (dart-lang/dartdoc#4304) 07506415 Mon Nov 2 12:14:22 2026 +0100 dependabot[bot] Bump the github-actions group across 1 directory with 3 updates (dart-lang/dartdoc#4311) 494aec7f Mon Nov 2 12:14:05 2026 +0100 Sarah Zakarias Add a "Documentation inherited from" note to inherited doc comments (dart-lang/dartdoc#4309) 527502a9 Mon Nov 2 11:21:46 2026 +0100 Sarah Zakarias Give the mock SDK the version of the SDK running the tests (dart-lang/dartdoc#4312) http (https://github.com/dart-lang/http/compare/1f869d0..4646fe6): 4646fe6 Thu Nov 5 02:41:04 2026 +0530 Chandra ObulaReddy [http] Don't throw in RetryClient when `when` has read the response stream (dart-lang/http#2040) native (https://github.com/dart-lang/native/compare/1838637..6287745): 6287745ca Thu Nov 5 20:37:11 2026 +0800 Dz [hooks_runner] Fix UTF-8 decoding of hook output on Windows (dart-lang/native#3772) 0eada93f9 Thu Nov 5 19:18:09 2026 +0800 Dz [native_toolchain_c] Decode vswhere UTF-8 output correctly on Windows (dart-lang/native#3773) Change-Id: If15b14dac054320ded19f0633a71b77463b9b718 Reviewed-on: https://dart-review.googlesource.com/c/sdk/+/574640 Auto-Submit: Nate Bosch <[email protected]> Commit-Queue: Nicholas Shahan <[email protected]> Reviewed-by: Nicholas Shahan <[email protected]>

[hook runner] Fix UTF-8 decoding of hook output on Windows

d982855

github-actions Bot added the package:hooks_runner label Oct 30, 2026

Merge branch 'main' into fix/hooks-runner-utf8-output

3241115

dcharkes reviewed Nov 3, 2026

View reviewed changes

Comment thread pkgs/hooks_runner/test/build_runner/build_runner_failure_test.dart Resolved

dzylikecode changed the title ~~[hook runner] Fix UTF-8 decoding of hook output on Windows~~ [hook_runner] Fix UTF-8 decoding of hook output on Windows Nov 4, 2026

dzylikecode and others added 8 commits November 4, 2026 10:14

[hooks_runner] Move UTF-8 regression fixtures to test_data

54a037a

[hooks_runner] Simplify UTF-8 fixture to a failing hook

bc688ce

[hooks_runner] Add manual entry point for UTF-8 hook fixture

07d6c06

[hooks_runner] Compare direct hook invocation with runner output

954522d

[hooks_runner] Clarify UTF-8 regression test intent

f6ca918

Merge branch 'main' of https://github.com/dart-lang/native into fix/h…

ace2d37

…ooks-runner-utf8-output

Merge branch 'dart-lang:main' into fix/hooks-runner-utf8-output

ac83d46

Merge remote-tracking branch 'issue/fix/hooks-runner-utf8-output' int…

71dc20e

…o fix/hooks-runner-utf8-output

dzylikecode requested a review from dcharkes November 4, 2026 02:42

dcharkes changed the title ~~[hook_runner] Fix UTF-8 decoding of hook output on Windows~~ [hooks_runner] Fix UTF-8 decoding of hook output on Windows Nov 4, 2026

dcharkes approved these changes Nov 5, 2026

View reviewed changes

dcharkes merged commit 6287745 into dart-lang:main Nov 5, 2026
53 of 55 checks passed

Navigation Menu

Repository navigation

[hooks_runner] Fix UTF-8 decoding of hook output on Windows - #3772

[hooks_runner] Fix UTF-8 decoding of hook output on Windows#3772
dcharkes merged 10 commits into
dart-lang:mainfrom
dzylikecode:fix/hooks-runner-utf8-output

dzylikecode commented Oct 30, 2026 •

edited

Loading

Uh oh!

google-cla Bot commented Oct 30, 2026

Uh oh!

dcharkes left a comment

Uh oh!

Uh oh!

codecov Bot commented Nov 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

dzylikecode commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Checklist

Uh oh!

google-cla Bot commented Oct 30, 2026

Uh oh!

dcharkes left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

codecov Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dzylikecode commented Oct 30, 2026 •

edited

Loading

codecov Bot commented Nov 3, 2026 •

edited

Loading