perf(workers): optimize message passing#35110
Conversation
|
Please do the second refactor too in this PR and provide some numbers |
|
@littledivy done — the second refactor (Rust-driven message delivery, eliminating the per-message How it works: Numbers — and a caveat I want to be upfront about. I only have a CI Linux x86_64 GCP VM, not the macOS arm64 machine from #11561. 100k round-trips, before = primitive fast path only, after = + Rust-driven dispatch:
On this VM I could not reproduce the ~12 µs/round-trip fixed gap — eliminating the per-message |
|
These results are not encouraging and not worth 700 additional lines of code. |
|
Agreed — reverted item 1 (the Rust-driven receive-loop dispatch). I couldn't demonstrate it moving the ping-pong number on the x86_64 Linux hardware I have, and it's not worth ~700 lines + a This PR is now just the primitive structured-clone fast path (item 2): ~JS-only, no |
|
@divybot what are new numbers then? |
|
@bartlomieju here are the numbers for just the primitive fast path (item 2 — item 1 stays reverted). Methodology: Ping-pong, 100k round-trips:
Fire-and-forget, 100k one-way:
So for the cases the fast path targets — primitive payloads — it's a consistent ~13% on ping-pong (~6.5 µs/round-trip, i.e. the two |
divybot
changed the title
|
Restored the second refactor in this PR and pushed the completed version in New release numbers from this x86_64 Linux VM compare the previous primitive-fast-path-only branch state against primitive fast path + Rust-driven dispatch. Each result is the average of 3 runs unless noted:
The dispatch refactor helps the latency-bound small-message cases by roughly 6-10% on top of the earlier primitive fast path here. The string row is larger because this revision also completes the string primitive fast path with raw UTF-16 code-unit encoding, so strings no longer fall back to V8 structured clone. The large-array case is effectively unchanged because structured clone dominates. Validation run:
|
d43ec46 to
103c260
Compare
|
Updated per the feedback here:
What it doesPrimitive message payloads ( NumbersRelease build, cross-thread
Variance across runs was <2%. The win scales with how serializer-bound the workload is: latency-bound ping-pong sees ~11%, while a tight fire-and-forget stream (where the serializer round-trip is a larger share of per-message cost) sees ~25%. |
|
@bartlomieju fresh numbers on the current head (the PR is now the primitive fast path only — the ~700-line Rust dispatch refactor is dropped, as agreed). Methodology: two release binaries built from this exact commit, differing only in whether the fast path is active (baseline forces Ping-pong, 100k round-trips (median ms):
Fire-and-forget, 100k one-way (median ms): So primitive payloads (the case the fast path targets) get ~12–14% on ping-pong and ~25% on fire-and-forget; strings benefit ~9.5% via the UTF-16 fast encoding. Object/array payloads fall back to V8 and pay ~2% (within variance) for the primitive-type check — the expected trade-off. JS-only, +264/−15, CI green. The PR description is updated to match this scope. |
Bypass V8's ValueSerializer/ValueDeserializer for primitive message payloads (undefined/null/boolean/number/string) on the no-transferables path -- the round-trip (two builtin op calls into C++ plus a buffer allocation) dominates the cost of latency-bound worker_threads ping-pong. Primitives are encoded with a tiny self-describing byte layout behind a 0xFE sentinel and decoded in pure JS with no serializer involved. V8's structured-clone format always begins with 0xFF (kVersionTag), so the two encodings can never collide: any buffer whose first byte isn't 0xFE is a regular V8 stream and goes through core.deserialize unchanged. Strings are encoded as raw UTF-16 code units so lone surrogates round-trip exactly; bigints/objects/functions fall back to V8. JS-only -- the message channel already carries an opaque buffer end to end, so no Rust changes are needed. All primitive send sites and their matching decode sites are routed through the new helpers. Refs #35025 Co-Authored-By: Divy Srivastava <[email protected]>
103c260 to
9bdc3eb
Compare
bartlomieju
left a comment
bartlomieju
left a comment
There was a problem hiding this comment.
Thorough review — the core design is sound and I verified the load-bearing correctness claims rather than trusting the description:
- Sentinel safety (
0xFEvs0xFF): confirmedop_serializecallsvalue_serializer.write_header()(libs/core/ops_builtin_v8.rs:877), so every V8 stream starts with0xFFand the two encodings genuinely can't collide. - Send/recv symmetry: traced all four paths (web
Worker, dedicated-worker host,MessagePort,node:worker_threads) — every send that can emit a0xFEbuffer is paired with a recv that branches on0xFE. The only remaining rawcore.serialize/core.deserializepair is the transferables path, which is symmetric and never usesserializeMessageData. - Buffer typing / endianness / numeric edge cases (
-0,NaN,±Infinity, int32 boundaries) / lone surrogates /markAsUncloneable: all check out.
No blockers. The substantive follow-ups are all about the string path (the most common non-trivial payload) — inline below. My recommendation is to land it with those as follow-ups.
Other perf notes (out of scope here): the dropped Rust dispatch refactor was the right call; and the per-message small-buffer alloc for undefined/null/bool can't be avoided with cached singletons because the post ops use #[buffer(detach)], which neuters a reused buffer.
| b[4] = (length >>> 16) & 0xFF; | ||
| b[5] = (length >>> 24) & 0xFF; | ||
| for (let i = 0, j = 6; i < length; i++, j += 2) { | ||
| const code = StringPrototypeCharCodeAt(value, i); |
There was a problem hiding this comment.
perf: this per-char charCodeAt + two byte-writes loop wins for short strings, but V8's ValueSerializer blits two-byte strings with a native memcpy — so for large strings the fast path likely regresses vs V8. Two suggestions:
- Latin1/ASCII 1-byte subpath (highest value): most real payloads (JSON, URLs, identifiers) are ASCII. Encoding 1 byte/char when all code units are
< 256halves both buffer size and write count, and keeps even large ASCII strings competitive. - Length cap: fall back to
core.serializeabove the crossover where the JS loop loses to native memcpy.
Worth a quick bench at 1KB/10KB/100KB to find the crossover.
| : length - i; | ||
| const codes = []; | ||
| for (let k = 0; k < chunkLength; k++, i++, j += 2) { | ||
| ArrayPrototypePush(codes, buffer[j] | (buffer[j + 1] << 8)); |
There was a problem hiding this comment.
perf nit: pre-size the array (const codes = new Array(chunkLength)) and assign by index instead of ArrayPrototypePush per char — avoids push overhead on the decode hot path.
| "\uD800", | ||
| "\uDC00", | ||
| "\uD83D\uDE00", | ||
| 123n, // bigint -> V8 fallback |
There was a problem hiding this comment.
test gap: FAST_STRING_CHUNK is 32768, but no value here exceeds it, so the multi-chunk while loop in fastDeserialize only ever runs one iteration. Add a "x".repeat(40000) case (and ideally one with a surrogate pair straddling the 32768 boundary) to exercise chunked decode.
Address review feedback on the string fast path: - Encode strings optimistically as Latin1 (1 byte/code unit), bailing to the two-byte path on the first code unit >= 256. ASCII/Latin1 payloads (JSON, URLs, identifiers) now use half the buffer and half the decode work. - Cap the two-byte path at 128 code units (FAST_STRING_2BYTE_MAX): the measured crossover on x86_64 Linux is ~128 units, beyond which V8's native two-byte memcpy beats the per-char JS loop. Longer two-byte strings fall back to core.serialize. - Decode: pre-size the code-unit array (new Array(chunkLength)) and index- assign instead of ArrayPrototypePush on the hot path. - Tests: chunked decode past FAST_STRING_CHUNK, high-Latin1 units, mid-string ASCII->two-byte bail, the cap boundary and its V8 fallback. Refs #35025 Co-Authored-By: Divy Srivastava <[email protected]>
Benchmarking the Latin1 subpath showed the per-char JS loop loses to V8's native string memcpy for both 1-byte and 2-byte strings past ~128 code units (roughly break-even at 256, +100%+ by a few KiB) -- V8 wins large strings regardless of encoding. So cap both paths at a single FAST_STRING_MAX = 128 and hand longer strings to core.serialize. With the cap below the old chunk size, the decode never needs to batch its String.fromCharCode.apply, so the chunked loop is removed: decode is now a single pre-sized apply per string. Tests updated for the unified cap (large strings validate the V8 fallback). Refs #35025 Co-Authored-By: Divy Srivastava <[email protected]>
|
Thanks for the thorough review — went ahead and did the string-path work in this PR. Summary of what changed and the bench that drove it: 1. Latin1/ASCII 1-byte subpath. Strings now encode optimistically as Latin1 (1 byte/code unit) in a single pass, bailing to the two-byte path on the first code unit 2. Length cap — I benchmarked the crossover, and it changed the plan. Two-byte
The key finding: the 1-byte path does not keep large ASCII competitive — a per-char JS loop can't beat V8's native memcpy regardless of 1 vs 2 bytes; it just moves the crossover from ~128 to ~256. Both lose hard past that. So rather than an asymmetric cap, I settled on a single 3. A consequence: with the cap at 128, no fast-path string ever approaches the old 4. Tests. Added coverage for every sub-path: high-Latin1 units ( Primitive ping-pong numbers (int −13.6%, bool −12.3%, fire-and-forget −25%) are unchanged from before — those paths didn't move. PR description updated; CI green. |
3 participants
Summary
Adds a structured-clone fast path for primitive message payloads to worker /
MessagePortmessage passing.undefined,null, booleans, numbers and stringssent on the no-transferables path skip V8's
ValueSerializer/ValueDeserializerround-trip (two builtin op calls into C++ plus a buffer allocation) and instead use
a tiny self-describing byte encoding, decoded back in pure JS. This is the dominant
fixed cost in latency-bound
worker_threadspatterns (request/response, ping-pong)identified in #11561 / #35025.
How it works
The first byte of the message buffer is a sentinel (
0xFE). V8's structured-cloneformat always begins with
0xFF(kVersionTag, written byValueSerializer::WriteHeader), so the two encodings can never be confused: anybuffer whose first byte isn't
0xFEis a regular V8 stream and goes throughcore.deserializeunchanged. Numbers use an int32 encoding when they round-tripexactly (else an f64 payload). Strings preserve JS string semantics exactly
(including lone surrogates): they encode optimistically as Latin1 (1 byte/code
unit) and fall to a two-byte encoding on the first code unit
>= 256. AboveFAST_STRING_MAX = 128code units a string is handed to V8'sValueSerializerinstead (see the crossover data below). Bigints, objects, functions and symbols
fall back to V8.
All primitive send sites (web
Worker, dedicated-worker host,MessagePort,node:worker_threads) and their matching decode sites route through the sharedserializeMessageData/deserializeMessageDatahelpers inext/web/13_message_port.js.Review follow-ups (addressed in this PR)
The string path was reworked per review:
Latin1/ASCII 1-byte subpath — strings encode 1 byte/code unit in a single
optimistic pass, bailing to two-byte on the first code unit
>= 256.Length cap, driven by a measured crossover — fast path vs V8 ping-pong,
"x".repeat(n)(1-byte) and"€".repeat(n)(2-byte), x86_64 Linux:A per-char JS loop cannot beat V8's native string memcpy at size regardless of
1 vs 2 bytes — Latin1 only moves the crossover from ~128 to ~256 — so a single
FAST_STRING_MAX = 128cap sends everything larger tocore.serialize. Thismatches the fast path's purpose: small latency-bound messages; large strings
are structured-clone-bound anyway.
Chunked decode removed — with the 128 cap, the old 32768-unit chunking was
dead code. Decode is now a pre-sized
new Array(length)+ index-assign + oneString.fromCharCode.applyper string.Tests cover every string sub-path: high-Latin1 units, mid-string
ASCII→two-byte bail, the cap boundary for both encodings (
.repeat(128)fastvs
.repeat(129)→ V8), large strings via the exact V8 fallback, plus-0,NaN,±Infinity, int32 boundaries, lone surrogates, and a bigint fallback.Numbers
node:worker_threadsping-pong / fire-and-forget, 100k iterations. Two releasebinaries that differ only in whether the fast path is active (baseline built
from the same commit with
fastSerializeforced to fall through tocore.serialize), run interleaved, median of 5 reps. Hardware: x86_64 Linux (notthe macOS arm64 from #11561), so treat these as a lower bound; the serializer cost
this removes is hardware-independent.
Ping-pong, 100k round-trips (median ms):
0(int)true(bool)null"…"(short string)Fire-and-forget, 100k one-way (median ms):
0(int)The string row was measured before the Latin1 subpath landed (short ASCII strings
now encode 1 byte/code unit, so it is a lower bound); the other rows are unaffected
by the string rework. For the cases the fast path targets — primitive payloads —
it's a consistent ~12–14% on ping-pong and ~25% on fire-and-forget. Strings above
the 128-code-unit cap, and object/array payloads, fall back to V8; the latter pay
~2% (within run-to-run variance) for the primitive-type check, the expected
trade-off. No correctness regressions.
Validation
cargo build --release --bin deno./target/debug/deno test -A --config tests/config/deno.json tests/unit/message_channel_test.ts./target/debug/deno test -A --config tests/config/deno.json tests/unit_node/worker_threads_test.ts./x test-spec workercargo fmt --check/dprint checkRefs #35025
Closes denoland/divybot#559