Repository navigation
Make --compile --bytecode executables work when cross-compiled (portable JSC bytecode cache) - #40270
Conversation
bun build --compile --bytecode --target=<other> embeds bytecode this machine encoded into an executable another OS/ABI decodes. The encoder's output is a pure function of the source, so one snapshot checked on every CI platform proves they all serialize identically; a mismatch on only some platforms is a portability bug, on all of them a format change.
… every source-reachable opcode features.js grows to cover every operator/jump form, property access and definition form, call form, sloppy-only construct, function/class shape and constant kind the front end has bytecode for; a generated big.js adds what only large functions produce (wide16 operands, a 32-bit metadata table, out-of-line jump targets, a >64 KB constant, a multi-page payload). Measured with llvm-cov on CachedTypes.cpp and --dumpGeneratedBytecodes: every encode path reachable from source, 173/194 opcodes; the test's header says why the other 21 cannot occur in a cached code block.
…t assume LF-only stderr
…rm-2d6835' into claude/jsc-bytecode-cross-platform-2d6835
|
Updated 2:32 AM PT - Aug 27th, 2026
⏳ @dylan-conway, your commit 49163b6 is still building in
|
- --compile appended up to 128 bytes of never-written buffer after each module's bytecode and counted them in its length; embed exactly the payload. - node:vm cachedData (Script#createCachedData, produceCachedData, SourceTextModule#createCachedData) is generated for the whole source with the default code-generation mode, instead of serializing whatever the VM's CodeCache held (nested functions only if they had already run; debugger / profiler opcodes if attached). The module variant is keyed as module code. - Thread-pool threads reserve the same 18 MB stack on every platform; JSC caps its recursion budget at 5 MB, and below that the depth at which the parser / bytecode generator give up depended on the platform. - Portability test: corpus gains the shapes whose encoding used to follow process state (Annex B block functions, >9 computed class fields, parse-time ** folding, an arrow ending in a multi-line template), and a test that the payload is the same under collectContinuously, useSourceProviderCache=0, a 64 KB heap, and through Bun.build()/vm.Script in a process that ran other JS first.
…ross-platform-2d6835
…age in the corpus; WebKit preview-pr-505-d3058e2d The golden-hash test is what guards the cache format's portability now, so its corpus has to reach everything: features.js gains the last user-reachable opcodes it was missing (top-level block functions, arguments.length, postfix increment), identical functions (shared arrays) and a shared string constant; lodash, acorn, react-dom, svelte's compiler, rollup and typescript from test/node_modules are bundled and hashed too (18 MB of payload in all), and lodash/acorn also go through vm.Script / vm.SourceTextModule.
…rm-2d6835' into claude/jsc-bytecode-cross-platform-2d6835
…nlines it as an absolute path); typescript through vm.Script
…ross-platform-2d6835
…ecord type), a fuller module.js, generated shapes.js sweeps; vm module cachedData is generated as strict code records.js walks the cache's record types -- strings/identifiers in every encoding, every constant kind and source representation, array-literal butterflies, regexps, template objects (incl. absent cooked strings), bigints, immediate/character/string switch tables, all four exception handler kinds, spread bit vectors and object-rest identifier sets, symbol tables of every scope kind, TDZ environment chains, every function parse mode / naming / parameter / source-position shape, class element definitions and executable rare data, object-literal definition forms, control flow, generator/async resume points, wide expression-info deltas, sloppy-only forms -- with several variations each. module.js covers every import/export form, top-level await, import.meta, dynamic import and strict block functions. shapes.js is generated: sweeps of string length (3 widths), integers around each encoding boundary, switch sizes, parameter and field counts, arrow nesting and closure depth. 21 payloads, 21 MB. NodeVM's getBytecode() generated module cachedData without StrictModeLexicallyScopedFeature (JSC's generateModuleBytecode sets it); a block-level function declaration in a vm.SourceTextModule then failed an assertion in the bytecode generator. Found by the new module.js.
bun build --compile --bytecode also embeds bytecode for the internal modules (node:*, bun:*) an app imports, generated through encodeBuiltinFunction from builtin-syntax source (@-intrinsics, private names, link-time constants) -- record and opcode paths user source never reaches. internal-for-testing gains internalModuleBytecode(index | source, name) returning exactly those bytes. The golden hash gains builtin.js, a builtin-syntax corpus (platform-independent source, so comparable across platforms). Bun's own internal modules have per-OS source (process.platform is inlined at codegen), so all ~200 of them go into the process-independence test instead: encoded in the test process and again in the busy subprocess, hashes must match.
…pared padding bytes)
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughThe PR refactors JavaScriptCore bytecode generation, exposes internal-module bytecode for testing, marks cross-compiled standalone bytecode, and adds portability coverage for bundled, VM, builtin, module, disk-cache, and executable paths. It also updates the WebKit version and adds a stack-trace regression test. ChangesBytecode portability and cached data
Stack trace source-location regression
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
Full details: Linked Issues checkExplanation The changes directly address issue Full details: Out of Scope Changes checkExplanation The WebKit update, VM cached-data changes, standalone executable flag, internal bytecode testing API, analytics reporting, and regression tests all support bytecode portability and validation for issue Full details: Description checkExplanation The description provides detailed context, implementation changes, verification steps, regression coverage, and the linked issue. Its headings differ from the template, but it fully covers the required information. Comment |
…t was encoded with (+ round-trip test in vm.test.ts); drop the executable cacheBytecode() no longer uses; run-from-cache and vm.Script tests concurrent; Buffer.alloc for generated ASCII runs; await the corpus script's own promise
…y VM) instead of per-VM PrivateSymbolImpls whose hash came from the process-wide symbol counter; builtin bytecode no longer depends on how many symbols the process created first. node:http2 back in the process-independence test.
…m:oven-sh/bun into claude/jsc-bytecode-cross-platform-2d6835
…er each call, as bun build does (on a JS thread it otherwise keeps WTFTimers registered with the main event loop)
What
bun build --compile --bytecode --target=<other OS>produced executables that segfault on launch (#18416): the embedded JSC bytecode cache held a per-function metadata offset table computed from the building machine'ssizeof(Op::Metadata), and the target decoded it with its own.oven-sh/WebKit#505 and oven-sh/WebKit#521 (both merged) make the cache format independent of the encoding platform — metadata stored as entry counts and laid out by the decoder; every serialized record compile-time-asserted to be padding-free with a fixed size; encoder output a pure function of the source (hash-table order, address order, host page size, host FPU NaN, SourceProviderCache re-parses all removed as inputs). This PR bumps WebKit to that build, adds the test that keeps it true, and fixes the Bun-side places that fed the encoder something non-deterministic.
Changes
7259739917cd.node:vmcachedData (NodeVM.cpp,NodeVMScript.cpp,NodeVMSourceTextModule.cpp):produceCachedData/createCachedData()now generate the complete code-block tree (recursivelyGenerateUnlinkedCodeBlockFor{Program,ModuleProgram}) instead of serializing whatever the VM's CodeCache happened to hold, so the data contains every nested function and no longer depends on what ran before or on an attached debugger; module cached data is produced and looked up with the key JSC uses for module code (strict,ModuleType), soSourceTextModulecached data round-trips. This makescachedDatalarger (all functions instead of the top level only) and a little slower to produce.Flags::CROSS_COMPILED_BYTECODEbit, set when--bytecodeoutput goes into an executable for a different os/arch/libc than the bun that built it; at startup it raises thecross_compiled_bytecodecrash-report feature so such crashes are identifiable.BunBuiltinNames.cpp): Bun's@namesymbols are now staticSymbolImpls with content-derived hashes, like JavaScriptCore's own, instead of per-VMPrivateSymbolImpls hashed by the process-wide symbol counter — so the bytecode generated for a builtin module no longer depends on how manySymbols the process created before (one was enough to reordernode:http2's constants). Also ~600 fewer allocations per VM; startup time and RSS unchanged within noise.--compilepreviously counted up to 128 never-written buffer bytes after each module's bytecode in its length; the executable now contains exactly the payload, so two builds of the same input are byte-identical.bun:internal-for-testing:internalModuleBytecode()exposes the builtin-module bytecode + external string table--compile --bytecodeembeds, for the test below.Test
test/bundler/bundler_bytecode_portable.test.tsencodes a corpus that reaches every kind of object the cache serializes and 170/194 opcodes —features.js,records.js(one section per record type), a generatedbig.js(32-bit metadata tables, wide operands, multi-page payloads),shapes.js,source-forms.js(hashbang, CRLF, U+2028/9,\u{}escapes, legacy octal — viavm.Script),module.js/esm/(module code), a builtin-syntax module, 7 individual libraries and a 64-library bundle fromtest/node_modules, andall.js(everything in one payload) — throughbun build --bytecode(±--minify),vm.Script,vm.SourceTextModuleand the builtin encoder, and compares one inline snapshot of the SHA-256s on every CI platform. Because the encoder's output depends only on the source, equal hashes on Linux/macOS/Windows × x64/arm64 (+ musl) mean every platform writes — and therefore reads — the same bytes. If the snapshot ever fails on only some platforms, that is a portability regression in JSC, not a snapshot to update (it caught one during this PR: x86 and ARM folding0/0to differently-signed NaNs, fixed in oven-sh/WebKit#521). The same file checks that the output is independent of the encoding process (GC timing, SourceProviderCache, a busy VM, internal modules), that every corpus build loads from its cache with identical output, that--compile --bytecodeexecutables (CJS and ESM, including a Worker entry) run from the embedded bytecode, and that an unusable.jscis a cache miss rather than a crash.Also:
vm.test.ts(SourceTextModule accepts its own cached data),capture-stack-trace.test.js(line numbers after a multi-line template literal, with and without the SourceProviderCache — one of the parser fixes in oven-sh/WebKit#505),bun-build-compile.test.ts(cross_compiled_bytecodeset only for a foreign target).Fixes #18416