fix(ext/node): match Node's UTF-8 replacement for invalid bytes in Buffer decode#34947
Merged
Conversation
…ffer decode
`Buffer.prototype.toString("utf8")` (and therefore `node:string_decoder`)
produced different output than Node.js for buffers containing invalid UTF-8
bytes. For example a lone continuation byte (0x92) was not replaced with a
single U+FFFD the way Node does, leading to a different decoded string and
silent data corruption.
This regressed in #30213, which optimized `utf8Slice` to go through
`op_node_decode` -> `v8::String::new_from_utf8`. V8's `NewFromUtf8` does not
follow the WHATWG "maximal subpart" replacement that Node.js uses.
Decode invalid input with Rust's `String::from_utf8_lossy`, which implements
the WHATWG maximal-subpart replacement (one U+FFFD per ill-formed subsequence)
and matches Node. Valid UTF-8 is borrowed without allocating, so the fast path
is unaffected.
Co-Authored-By: Divy Srivastava <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes a regression where
Buffer.prototype.toString("utf8")— and thereforenode:string_decoder— produced different output than Node.js for buffers containing invalid UTF-8 bytes.Repro (from #34930)
The lone continuation byte
0x92was not replaced with a singleU+FFFDtheway Node does, leading to a different decoded string and silent data corruption.
Root cause
This regressed in #30213, which optimized
utf8Sliceto go throughop_node_decode→v8::String::new_from_utf8. V8'sNewFromUtf8does notfollow the WHATWG "maximal subpart" replacement that Node.js (and the previous
decoder) used for ill-formed UTF-8.
Fix
Decode invalid input with Rust's
String::from_utf8_lossy, which implementsthe WHATWG maximal-subpart replacement (one
U+FFFDper ill-formedsubsequence) and matches Node. Valid UTF-8 is borrowed without allocating, so
the optimized fast path is unaffected.
Added a regression test in
tests/unit_node/string_decoder_test.tscoveringboth
StringDecoder.writeandBuffer.prototype.toString("utf8").Closes #34930
Closes denoland/divybot#505