Skip to content

fix(ext/node): match Node's UTF-8 replacement for invalid bytes in Buffer decode#34947

Merged
littledivy merged 1 commit into
mainfrom
orch/divybot-505
Jun 6, 2026
Merged

fix(ext/node): match Node's UTF-8 replacement for invalid bytes in Buffer decode#34947
littledivy merged 1 commit into
mainfrom
orch/divybot-505

Conversation

@divybot

@divybot divybot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Fixes a regression where Buffer.prototype.toString("utf8") — and therefore node:string_decoder — produced different output than Node.js for buffers containing invalid UTF-8 bytes.

Repro (from #34930)

import { createHash } from "node:crypto";
import { StringDecoder } from "node:string_decoder";

const buffer = Buffer.from("R0ABkgEBAQE...AQE=", "base64"); // bytes: 47 40 01 92 01 01 ...
const text = new StringDecoder("utf-8").write(new Uint8Array(buffer));
console.log(createHash("sha256").update(text).digest("hex"));
// Node:   2637e4a587f78c8be9f18fec1ad0fcbe5ee9ca90e01c7c9ce61a7fe6ab7d6610
// Deno:   633b6f1cc3191287ce213370e649ed09700695a7485081914111f4bd553c8101 (before)

The lone continuation byte 0x92 was not replaced with a single U+FFFD the
way Node does, leading to a different decoded string and silent data corruption.

Root cause

This regressed in #30213, which optimized utf8Slice to go through
op_node_decodev8::String::new_from_utf8. V8's NewFromUtf8 does not
follow the WHATWG "maximal subpart" replacement that Node.js (and the previous
decoder) used for ill-formed UTF-8.

Fix

Decode invalid input with Rust's String::from_utf8_lossy, which implements
the WHATWG maximal-subpart replacement (one U+FFFD per ill-formed
subsequence) and matches Node. Valid UTF-8 is borrowed without allocating, so
the optimized fast path is unaffected.

Added a regression test in tests/unit_node/string_decoder_test.ts covering
both StringDecoder.write and Buffer.prototype.toString("utf8").

Closes #34930

Closes denoland/divybot#505

…ffer decode

`Buffer.prototype.toString("utf8")` (and therefore `node:string_decoder`)
produced different output than Node.js for buffers containing invalid UTF-8
bytes. For example a lone continuation byte (0x92) was not replaced with a
single U+FFFD the way Node does, leading to a different decoded string and
silent data corruption.

This regressed in #30213, which optimized `utf8Slice` to go through
`op_node_decode` -> `v8::String::new_from_utf8`. V8's `NewFromUtf8` does not
follow the WHATWG "maximal subpart" replacement that Node.js uses.

Decode invalid input with Rust's `String::from_utf8_lossy`, which implements
the WHATWG maximal-subpart replacement (one U+FFFD per ill-formed subsequence)
and matches Node. Valid UTF-8 is borrowed without allocating, so the fast path
is unaffected.

Co-Authored-By: Divy Srivastava <[email protected]>
@littledivy
littledivy merged commit 4b37f4a into main Jun 6, 2026
136 checks passed
@littledivy
littledivy deleted the orch/divybot-505 branch June 6, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

string_decoder produces different UTF-8 output than Node.js for identical Buffer input

2 participants