Repository navigation
Encode the char after an unpaired high surrogate in writeUtf8(...) - #18252
Merged
normanmaurer merged 1 commit intoNov 7, 2026
Merged
Conversation
Motivation: When `ByteBufUtil.writeUtf8(...)` finds a high surrogate that is not followed by a low surrogate, it writes `?` and then writes the next char cast to a `byte`. That only works for ASCII. A non-ASCII char is truncated to its low byte, and a surrogate pair that follows is replaced by `??`. `String.getBytes(UTF_8)` writes `?` and then encodes the next char normally: ``` input String.getBytes(UTF_8) writeUtf8(...) "a\uD800éb" 61 3f c3a9 62 61 3f e9 62 (invalid UTF-8) "a\uD800中b" 61 3f e4b8ad 62 61 3f 2d 62 "a\uD800😀b" 61 3f f09f9880 62 61 3f 3f 3f 62 ``` `Unpooled.copiedBuffer(CharSequence, UTF_8)` and `TextWebSocketFrame(String)` go through this code. `QueryStringEncoder` has a copy of the same logic and percent-encodes only the low byte of the next char. Modification: - Write `?` for the unpaired high surrogate only and encode the next char in the next iteration, in all `writeUtf8(...)` variants, in `utf8Bytes(...)` and in `QueryStringEncoder`. - Add tests to `ByteBufUtilTest` and `QueryStringEncoderTest`. Result: `writeUtf8(...)`, `utf8Bytes(...)` and `QueryStringEncoder` agree with `String.getBytes(UTF_8)` for these inputs. Co-Authored-By: Claude Opus 5.5 <[email protected]>
🚨 All tests passed but jobs failed 🚨Failed Jobs without Test Failures❌ Build PR / linux-x86_64-java11-awslc build 🏷️ Commit: a2166b7 Rerun ControlsClick the checkbox to trigger a rerun:
Learn more about TestLens at testlens.app/docs. |
Contributor
|
Auto-port PR for 4.1: #18307 |
Contributor
|
Auto-port PR for 5.0: #18308 |
normanmaurer
pushed a commit
that referenced
this pull request
Nov 9, 2026
…iteUtf8(...) (#18307) Auto-port of #18252 to 4.1 Cherry-picked commit: 09b3836 --- Motivation: `ByteBufUtil.writeUtf8(...)` writes the char after an unpaired high surrogate as a single byte, so a non-ASCII char is truncated and a following surrogate pair is lost. `QueryStringEncoder` has the same logic. Modification: Write `?` for the unpaired surrogate only and encode the next char normally, as `String.getBytes(UTF_8)` does. Add tests. Result: Output matches `String.getBytes(UTF_8)`. I used Claude Code to write this change. Co-authored-by: Wei-Lun Li <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
normanmaurer
pushed a commit
that referenced
this pull request
Nov 9, 2026
…iteUtf8(...) (#18308) Auto-port of #18252 to 5.0 Cherry-picked commit: 09b3836 --- Motivation: `ByteBufUtil.writeUtf8(...)` writes the char after an unpaired high surrogate as a single byte, so a non-ASCII char is truncated and a following surrogate pair is lost. `QueryStringEncoder` has the same logic. Modification: Write `?` for the unpaired surrogate only and encode the next char normally, as `String.getBytes(UTF_8)` does. Add tests. Result: Output matches `String.getBytes(UTF_8)`. I used Claude Code to write this change. Co-authored-by: Wei-Lun Li <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
ByteBufUtil.writeUtf8(...)writes the char after an unpaired high surrogate as a single byte, so a non-ASCII char is truncated and a following surrogate pair is lost.QueryStringEncoderhas the same logic.Modification:
Write
?for the unpaired surrogate only and encode the next char normally, asString.getBytes(UTF_8)does. Add tests.Result:
Output matches
String.getBytes(UTF_8).I used Claude Code to write this change.