Skip to content

Make --compile --bytecode executables work when cross-compiled (portable JSC bytecode cache) - #40270

Merged
dylan-conway merged 72 commits into
mainfrom
claude/jsc-bytecode-cross-platform-2d6835
Aug 27, 2026
Merged

dylan-conway merged 72 commits into
mainfrom
claude/jsc-bytecode-cross-platform-2d6835

Conversation

@dylan-conway

@dylan-conway dylan-conway commented Aug 24, 2026 •

Copy link
Copy Markdown
Member

What

bun build --compile --bytecode --target=<other OS> produced executables that segfault on launch (#18416): the embedded JSC bytecode cache held a per-function metadata offset table computed from the building machine's sizeof(Op::Metadata), and the target decoded it with its own.

oven-sh/WebKit#505 and oven-sh/WebKit#521 (both merged) make the cache format independent of the encoding platform — metadata stored as entry counts and laid out by the decoder; every serialized record compile-time-asserted to be padding-free with a fixed size; encoder output a pure function of the source (hash-table order, address order, host page size, host FPU NaN, SourceProviderCache re-parses all removed as inputs). This PR bumps WebKit to that build, adds the test that keeps it true, and fixes the Bun-side places that fed the encoder something non-deterministic.

Changes

  • WebKit → 7259739917cd.
  • node:vm cachedData (NodeVM.cpp, NodeVMScript.cpp, NodeVMSourceTextModule.cpp): produceCachedData / createCachedData() now generate the complete code-block tree (recursivelyGenerateUnlinkedCodeBlockFor{Program,ModuleProgram}) instead of serializing whatever the VM's CodeCache happened to hold, so the data contains every nested function and no longer depends on what ran before or on an attached debugger; module cached data is produced and looked up with the key JSC uses for module code (strict, ModuleType), so SourceTextModule cached data round-trips. This makes cachedData larger (all functions instead of the top level only) and a little slower to produce.
  • Standalone executables: one new Flags::CROSS_COMPILED_BYTECODE bit, set when --bytecode output goes into an executable for a different os/arch/libc than the bun that built it; at startup it raises the cross_compiled_bytecode crash-report feature so such crashes are identifiable.
  • Builtin private names (BunBuiltinNames.cpp): Bun's @name symbols are now static SymbolImpls with content-derived hashes, like JavaScriptCore's own, instead of per-VM PrivateSymbolImpls hashed by the process-wide symbol counter — so the bytecode generated for a builtin module no longer depends on how many Symbols the process created before (one was enough to reorder node:http2's constants). Also ~600 fewer allocations per VM; startup time and RSS unchanged within noise.
  • Exact-length bytecode embed: --compile previously counted up to 128 never-written buffer bytes after each module's bytecode in its length; the executable now contains exactly the payload, so two builds of the same input are byte-identical.
  • bun:internal-for-testing: internalModuleBytecode() exposes the builtin-module bytecode + external string table --compile --bytecode embeds, for the test below.

Test

test/bundler/bundler_bytecode_portable.test.ts encodes a corpus that reaches every kind of object the cache serializes and 170/194 opcodes — features.js, records.js (one section per record type), a generated big.js (32-bit metadata tables, wide operands, multi-page payloads), shapes.js, source-forms.js (hashbang, CRLF, U+2028/9, \u{} escapes, legacy octal — via vm.Script), module.js / esm/ (module code), a builtin-syntax module, 7 individual libraries and a 64-library bundle from test/node_modules, and all.js (everything in one payload) — through bun build --bytecode (± --minify), vm.Script, vm.SourceTextModule and the builtin encoder, and compares one inline snapshot of the SHA-256s on every CI platform. Because the encoder's output depends only on the source, equal hashes on Linux/macOS/Windows × x64/arm64 (+ musl) mean every platform writes — and therefore reads — the same bytes. If the snapshot ever fails on only some platforms, that is a portability regression in JSC, not a snapshot to update (it caught one during this PR: x86 and ARM folding 0/0 to differently-signed NaNs, fixed in oven-sh/WebKit#521). The same file checks that the output is independent of the encoding process (GC timing, SourceProviderCache, a busy VM, internal modules), that every corpus build loads from its cache with identical output, that --compile --bytecode executables (CJS and ESM, including a Worker entry) run from the embedded bytecode, and that an unusable .jsc is a cache miss rather than a crash.

Also: vm.test.ts (SourceTextModule accepts its own cached data), capture-stack-trace.test.js (line numbers after a multi-line template literal, with and without the SourceProviderCache — one of the parser fixes in oven-sh/WebKit#505), bun-build-compile.test.ts (cross_compiled_bytecode set only for a foreign target).

Fixes #18416

dylan-conway and others added 8 commits August 23, 2026 23:28
bun build --compile --bytecode --target=<other> embeds bytecode this machine
encoded into an executable another OS/ABI decodes. The encoder's output is a
pure function of the source, so one snapshot checked on every CI platform
proves they all serialize identically; a mismatch on only some platforms is a
portability bug, on all of them a format change.
… every source-reachable opcode

features.js grows to cover every operator/jump form, property access and
definition form, call form, sloppy-only construct, function/class shape and
constant kind the front end has bytecode for; a generated big.js adds what
only large functions produce (wide16 operands, a 32-bit metadata table,
out-of-line jump targets, a >64 KB constant, a multi-page payload). Measured
with llvm-cov on CachedTypes.cpp and --dumpGeneratedBytecodes: every encode
path reachable from source, 173/194 opcodes; the test's header says why the
other 21 cannot occur in a cached code block.
…rm-2d6835' into claude/jsc-bytecode-cross-platform-2d6835
@robobun

robobun commented Aug 24, 2026 •

Copy link
Copy Markdown
Collaborator
Updated 2:32 AM PT - Aug 27th, 2026

⏳ @dylan-conway, your commit 49163b6 is still building in Build #106714, but has 2 failures so far (All Failures):

dylan-conway and others added 5 commits August 24, 2026 01:17
- --compile appended up to 128 bytes of never-written buffer after each
  module's bytecode and counted them in its length; embed exactly the payload.
- node:vm cachedData (Script#createCachedData, produceCachedData,
  SourceTextModule#createCachedData) is generated for the whole source with the
  default code-generation mode, instead of serializing whatever the VM's
  CodeCache held (nested functions only if they had already run; debugger /
  profiler opcodes if attached). The module variant is keyed as module code.
- Thread-pool threads reserve the same 18 MB stack on every platform; JSC caps
  its recursion budget at 5 MB, and below that the depth at which the parser /
  bytecode generator give up depended on the platform.
- Portability test: corpus gains the shapes whose encoding used to follow
  process state (Annex B block functions, >9 computed class fields, parse-time
  ** folding, an arrow ending in a multi-line template), and a test that the
  payload is the same under collectContinuously, useSourceProviderCache=0, a
  64 KB heap, and through Bun.build()/vm.Script in a process that ran other JS
  first.
dylan-conway and others added 10 commits August 26, 2026 04:23
…age in the corpus; WebKit preview-pr-505-d3058e2d

The golden-hash test is what guards the cache format's portability now, so
its corpus has to reach everything: features.js gains the last user-reachable
opcodes it was missing (top-level block functions, arguments.length, postfix
increment), identical functions (shared arrays) and a shared string constant;
lodash, acorn, react-dom, svelte's compiler, rollup and typescript from
test/node_modules are bundled and hashed too (18 MB of payload in all), and
lodash/acorn also go through vm.Script / vm.SourceTextModule.
…rm-2d6835' into claude/jsc-bytecode-cross-platform-2d6835
…nlines it as an absolute path); typescript through vm.Script
…ecord type), a fuller module.js, generated shapes.js sweeps; vm module cachedData is generated as strict code

records.js walks the cache's record types -- strings/identifiers in every
encoding, every constant kind and source representation, array-literal
butterflies, regexps, template objects (incl. absent cooked strings),
bigints, immediate/character/string switch tables, all four exception
handler kinds, spread bit vectors and object-rest identifier sets, symbol
tables of every scope kind, TDZ environment chains, every function parse
mode / naming / parameter / source-position shape, class element
definitions and executable rare data, object-literal definition forms,
control flow, generator/async resume points, wide expression-info deltas,
sloppy-only forms -- with several variations each. module.js covers every
import/export form, top-level await, import.meta, dynamic import and strict
block functions. shapes.js is generated: sweeps of string length (3
widths), integers around each encoding boundary, switch sizes, parameter
and field counts, arrow nesting and closure depth. 21 payloads, 21 MB.

NodeVM's getBytecode() generated module cachedData without
StrictModeLexicallyScopedFeature (JSC's generateModuleBytecode sets it);
a block-level function declaration in a vm.SourceTextModule then failed an
assertion in the bytecode generator. Found by the new module.js.
bun build --compile --bytecode also embeds bytecode for the internal modules
(node:*, bun:*) an app imports, generated through encodeBuiltinFunction from
builtin-syntax source (@-intrinsics, private names, link-time constants) --
record and opcode paths user source never reaches. internal-for-testing gains
internalModuleBytecode(index | source, name) returning exactly those bytes.

The golden hash gains builtin.js, a builtin-syntax corpus (platform-independent
source, so comparable across platforms). Bun's own internal modules have
per-OS source (process.platform is inlined at codegen), so all ~200 of them
go into the process-independence test instead: encoded in the test process
and again in the busy subprocess, hashes must match.
@coderabbitai

coderabbitai Bot commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The PR refactors JavaScriptCore bytecode generation, exposes internal-module bytecode for testing, marks cross-compiled standalone bytecode, and adds portability coverage for bundled, VM, builtin, module, disk-cache, and executable paths. It also updates the WebKit version and adds a stack-trace regression test.

Changes

Bytecode portability and cached data

Layer / File(s) Summary
Unified program and module bytecode generation
src/jsc/bindings/NodeVM.*, src/jsc/bindings/NodeVMScript.cpp, src/jsc/bindings/NodeVMSourceTextModule.cpp
Node VM cached-data generation now serializes program and module bytecode directly from source types. Module decoding uses module-specific parser metadata. Failed generation returns null or raises a syntax error.
Internal-module bytecode testing API
src/js/internal-for-testing.ts, src/jsc/bindings/InternalModuleRegistry.*
The internalModuleBytecode API retrieves indexed builtin bytecode or compiles supplied module source. The native binding returns the module name, bytecode, and external string table.
Standalone compiler metadata and ABI detection
src/options_types/compile_target.rs, src/standalone_graph/StandaloneModuleGraph.rs, src/analytics/lib.rs, src/runtime/cli/run_command.rs, test/bundler/bun-build-compile.test.ts
Standalone graphs mark bytecode compiled for a different OS, architecture, or libc. Standalone execution records cross_compiled_bytecode. Parameterized compile tests cover host and cross-architecture targets.
Portable bytecode corpus
test/bundler/bundler_bytecode_portable.test.ts, test/bundler/bytecode-portability/*, scripts/build/deps/webkit.ts
The tests add deterministic JavaScript, builtin, and ESM corpora that exercise bytecode records, opcodes, source metadata, module behavior, workers, and external strings. The portability setup masks WebKit version fields and uses the updated WebKit commit.
Cross-platform cache and executable validation
test/bundler/bundler_bytecode_portable.test.ts, test/js/node/vm/vm.test.ts
The test harness compares fingerprints across platforms, checks process-independent output, validates disk-cache and compiled-executable loading, rejects modified or truncated payloads, and verifies matching and mismatched SourceTextModule cached data.

Stack trace source-location regression

Layer / File(s) Summary
Multiline template stack locations
test/js/node/v8/capture-stack-trace.test.js
A subprocess test checks the reported error line after a multiline template literal with the SourceProvider cache enabled and disabled.

Suggested reviewers: jarred-sumner, robobun

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The changes directly address issue #18416 by making serialized JSC bytecode independent of the build platform, updating cached-data generation, marking cross-compiled executables, and adding cross-pla…
Out of Scope Changes check ✅ Passed The WebKit update, VM cached-data changes, standalone executable flag, internal bytecode testing API, analytics reporting, and regression tests all support bytecode portability and validation for issu…
Title check ✅ Passed The title clearly identifies the main change: enabling cross-compiled --compile --bytecode executables through portable JSC bytecode caching.
Description check ✅ Passed The description provides detailed context, implementation changes, verification steps, regression coverage, and the linked issue. Its headings differ from the template, but it fully covers the require…
Full details: Linked Issues check

Explanation

The changes directly address issue #18416 by making serialized JSC bytecode independent of the build platform, updating cached-data generation, marking cross-compiled executables, and adding cross-platform regression tests.

Full details: Out of Scope Changes check

Explanation

The WebKit update, VM cached-data changes, standalone executable flag, internal bytecode testing API, analytics reporting, and regression tests all support bytecode portability and validation for issue #18416.

Full details: Description check

Explanation

The description provides detailed context, implementation changes, verification steps, regression coverage, and the linked issue. Its headings differ from the template, but it fully covers the required information.


Comment @coderabbitai help to get the list of available commands.

Comment thread src/jsc/bindings/NodeVMScript.cpp Resolved
…t was encoded with (+ round-trip test in vm.test.ts); drop the executable cacheBytecode() no longer uses; run-from-cache and vm.Script tests concurrent; Buffer.alloc for generated ASCII runs; await the corpus script's own promise
autofix-ci Bot and others added 5 commits August 27, 2026 08:34
…y VM) instead of per-VM PrivateSymbolImpls whose hash came from the process-wide symbol counter; builtin bytecode no longer depends on how many symbols the process created first. node:http2 back in the process-independence test.
…m:oven-sh/bun into claude/jsc-bytecode-cross-platform-2d6835
…er each call, as bun build does (on a JS thread it otherwise keeps WTFTimers registered with the main event loop)
@dylan-conway
dylan-conway merged commit 65362b5 into main Aug 27, 2026
9 of 10 checks passed
@dylan-conway
dylan-conway deleted the claude/jsc-bytecode-cross-platform-2d6835 branch August 27, 2026 09:37

@claude claude Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code review found no issues

No high-confidence issues detected in this change.

This was referenced Aug 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Executables cross-compiled with --bytecode segfault on launch if running on target OS

2 participants