Skip to content

Encode the char after an unpaired high surrogate in writeUtf8(...) - #18252

Merged
normanmaurer merged 1 commit into
netty:4.2from
b84330808:writeutf8-unpaired-high-surrogate
Nov 7, 2026
Merged

normanmaurer merged 1 commit into
netty:4.2from
b84330808:writeutf8-unpaired-high-surrogate

Conversation

@b84330808

Copy link
Copy Markdown
Contributor

Motivation:

ByteBufUtil.writeUtf8(...) writes the char after an unpaired high surrogate as a single byte, so a non-ASCII char is truncated and a following surrogate pair is lost. QueryStringEncoder has the same logic.

Modification:

Write ? for the unpaired surrogate only and encode the next char normally, as String.getBytes(UTF_8) does. Add tests.

Result:

Output matches String.getBytes(UTF_8).

I used Claude Code to write this change.

Motivation:

When `ByteBufUtil.writeUtf8(...)` finds a high surrogate that is not
followed by a low surrogate, it writes `?` and then writes the next char
cast to a `byte`. That only works for ASCII. A non-ASCII char is
truncated to its low byte, and a surrogate pair that follows is replaced
by `??`. `String.getBytes(UTF_8)` writes `?` and then encodes the next
char normally:

```
input                   String.getBytes(UTF_8)   writeUtf8(...)
"a\uD800éb"        61 3f c3a9 62            61 3f e9 62 (invalid UTF-8)
"a\uD800中b"        61 3f e4b8ad 62          61 3f 2d 62
"a\uD800😀b"  61 3f f09f9880 62        61 3f 3f 3f 62
```

`Unpooled.copiedBuffer(CharSequence, UTF_8)` and
`TextWebSocketFrame(String)` go through this code. `QueryStringEncoder`
has a copy of the same logic and percent-encodes only the low byte of
the next char.

Modification:

- Write `?` for the unpaired high surrogate only and encode the next
  char in the next iteration, in all `writeUtf8(...)` variants, in
  `utf8Bytes(...)` and in `QueryStringEncoder`.
- Add tests to `ByteBufUtilTest` and `QueryStringEncoderTest`.

Result:

`writeUtf8(...)`, `utf8Bytes(...)` and `QueryStringEncoder` agree with
`String.getBytes(UTF_8)` for these inputs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@normanmaurer normanmaurer added this to the 4.2.21.Final milestone Nov 5, 2026
@testlens-app

testlens-app Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

🚨 All tests passed but jobs failed 🚨

Failed Jobs without Test Failures

❌ Build PR / linux-x86_64-java11-awslc build

🏷️ Commit: a2166b7
▶️ Tests: 167882 executed
⚪️ Checks: 30/30 completed

Rerun Controls

Click the checkbox to trigger a rerun:

  • Rerun jobs

Learn more about TestLens at testlens.app/docs.

@normanmaurer normanmaurer added needs-cherry-pick-4.1 This PR should be cherry-picked to 4.1 once merged. needs-cherry-pick-5.0 This PR should be cherry-picked to 5.0 once merged. labels Nov 6, 2026
@normanmaurer
normanmaurer merged commit 09b3836 into netty:4.2 Nov 7, 2026
30 of 31 checks passed
@netty-project-bot

Copy link
Copy Markdown
Contributor

Auto-port PR for 4.1: #18307

@netty-project-bot

Copy link
Copy Markdown
Contributor

Auto-port PR for 5.0: #18308

@github-actions github-actions Bot removed the needs-cherry-pick-5.0 This PR should be cherry-picked to 5.0 once merged. label Nov 7, 2026
normanmaurer pushed a commit that referenced this pull request Nov 9, 2026
…iteUtf8(...) (#18307)

Auto-port of #18252 to 4.1
Cherry-picked commit: 09b3836

---
Motivation:

`ByteBufUtil.writeUtf8(...)` writes the char after an unpaired high
surrogate as a single byte, so a non-ASCII char is truncated and a
following surrogate pair is lost. `QueryStringEncoder` has the same
logic.

Modification:

Write `?` for the unpaired surrogate only and encode the next char
normally, as `String.getBytes(UTF_8)` does. Add tests.

Result:

Output matches `String.getBytes(UTF_8)`.

I used Claude Code to write this change.

Co-authored-by: Wei-Lun Li <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
normanmaurer pushed a commit that referenced this pull request Nov 9, 2026
…iteUtf8(...) (#18308)

Auto-port of #18252 to 5.0
Cherry-picked commit: 09b3836

---
Motivation:

`ByteBufUtil.writeUtf8(...)` writes the char after an unpaired high
surrogate as a single byte, so a non-ASCII char is truncated and a
following surrogate pair is lost. `QueryStringEncoder` has the same
logic.

Modification:

Write `?` for the unpaired surrogate only and encode the next char
normally, as `String.getBytes(UTF_8)` does. Add tests.

Result:

Output matches `String.getBytes(UTF_8)`.

I used Claude Code to write this change.

Co-authored-by: Wei-Lun Li <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants