Skip to content

Commit 0aacacd

Browse files
author
tsouth89
committed
fix(build): keep the dependency closure of external packages external
Real WSL testing on this branch found a case the hand-maintained list could not catch by inspection. node-gyp-build-optional-packages is external, so it is loaded from the real filesystem, so its own `require` resolves from the real filesystem too. It requires detect-libc, which was not on the list and therefore got bundled into the CLI bundle -- present only inside app.asar. The Windows primary reads that transparently under ELECTRON_RUN_AS_NODE and resolves it; plain node under WSL cannot. msgpackr-extract failed through the same chain. Measured under Ubuntu 24.04 with Linux node v24.18.0 against the packaged tree: before: MISSING (cjs) msgpackr-extract [MODULE_NOT_FOUND] detect-libc MISSING (cjs) node-gyp-build-optional-packages [MODULE_NOT_FOUND] after : no resolution failures The general rule is that an external package's entire runtime dependency closure must be external. That is not something to maintain by staring at a list, so it is now a test: it walks each runtime-external package's declared dependencies transitively and fails if any would be bundled away. Writing that test surfaced a distinction the single list had flattened. The Bun-only entries are external for a build-time reason -- they resolve `bun:*` specifiers that do not exist when bundling for Node -- and Node never loads them, so their closure genuinely does not need to be external. The native packages are external for a runtime reason and theirs does. The list is split along that line, and the closure test applies only to the runtime set. Adds 6 files to the installer (1,192 -> 1,198). Native binaries and installer size are unchanged.
1 parent ba61d1f commit 0aacacd

2 files changed

Lines changed: 85 additions & 4 deletions

File tree

‎scripts/lib/cli-external-packages.test.ts‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ import { assert, describe, it } from "@effect/vitest";
33
import {
44
CLI_EXTERNAL_PACKAGE_PREFIXES,
55
CLI_EXTERNAL_PACKAGE_UNPACK_GLOBS,
6+
CLI_RUNTIME_EXTERNAL_PREFIXES,
67
shouldBundleCliDependency,
78
} from "./cli-external-packages.ts";
89

@@ -62,3 +63,57 @@ describe("CLI_EXTERNAL_PACKAGE_UNPACK_GLOBS", () => {
6263
assert.include(CLI_EXTERNAL_PACKAGE_UNPACK_GLOBS, "node_modules/node-gyp-build*/**/*");
6364
});
6465
});
66+
67+
// The failure this guards is invisible on Windows and fatal under WSL.
68+
//
69+
// An external package is loaded from the real filesystem, so its own `require`
70+
// also resolves from the real filesystem. If one of its dependencies was
71+
// bundled away instead of left external, that dependency exists only inside
72+
// app.asar — which the Windows primary reads transparently under
73+
// ELECTRON_RUN_AS_NODE, and plain `node` under WSL cannot.
74+
//
75+
// Found the hard way: node-gyp-build-optional-packages requires detect-libc,
76+
// which was bundled. Windows was fine; WSL got MODULE_NOT_FOUND.
77+
describe("external package dependency closure", () => {
78+
// Must be runtime-external specifically: the dependency has to exist on disk.
79+
const isExternal = (name: string) =>
80+
CLI_RUNTIME_EXTERNAL_PREFIXES.some((prefix) => name.startsWith(prefix));
81+
82+
it("keeps every runtime dependency of an external package external too", async () => {
83+
const { createRequire } = await import("node:module");
84+
const require = createRequire(import.meta.url);
85+
86+
const violations: string[] = [];
87+
const seen = new Set<string>();
88+
// Only the runtime-external set. The build-only entries resolve `bun:*`
89+
// and are never loaded by Node, so their closure is irrelevant here.
90+
const queue: string[] = CLI_RUNTIME_EXTERNAL_PREFIXES.filter((prefix) => !prefix.endsWith("/"));
91+
92+
for (const name of queue) {
93+
if (seen.has(name)) continue;
94+
seen.add(name);
95+
96+
let manifest: { dependencies?: Record<string, string> };
97+
try {
98+
manifest = require(`${name}/package.json`);
99+
} catch {
100+
// Not installed on this platform (or reached only by prefix); nothing
101+
// to check. The globs still cover it if it does get installed.
102+
continue;
103+
}
104+
105+
for (const dependency of Object.keys(manifest.dependencies ?? {})) {
106+
if (!isExternal(dependency)) {
107+
violations.push(`${name} -> ${dependency}`);
108+
}
109+
if (!seen.has(dependency)) queue.push(dependency);
110+
}
111+
}
112+
113+
assert.deepStrictEqual(
114+
violations,
115+
[],
116+
`these dependencies of external packages would be bundled away, and fail to resolve under WSL: ${violations.join(", ")}`,
117+
);
118+
});
119+
});

‎scripts/lib/cli-external-packages.ts‎

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,8 +17,17 @@
1717
* a package's platform-specific siblings — `node-gyp-build` covers
1818
* `node-gyp-build-optional-packages`, `@yuuang/` covers every `ffi-rs-*` binding.
1919
*/
20-
export const CLI_EXTERNAL_PACKAGE_PREFIXES = [
21-
// Native addons (.node), and the JS wrappers that dlopen them by real path.
20+
/**
21+
* External because Node actually loads them from disk at runtime.
22+
*
23+
* Native addons (.node), the JS wrappers that dlopen them by real path, and —
24+
* critically — the ordinary JS packages those wrappers require. An external
25+
* package is loaded from the real filesystem, so its own `require` also
26+
* resolves from the real filesystem; a dependency that was bundled away exists
27+
* only inside app.asar and is unreachable there. This closure is enforced by a
28+
* test, not by inspection.
29+
*/
30+
export const CLI_RUNTIME_EXTERNAL_PREFIXES = [
2231
"node-pty",
2332
"ffi-rs",
2433
"@yuuang/",
@@ -28,12 +37,29 @@ export const CLI_EXTERNAL_PACKAGE_PREFIXES = [
2837
"msgpackr-extract",
2938
"node-gyp-build",
3039
"node-addon-api",
31-
// Bun-only entry points: reached through a runtime-conditional dynamic import
32-
// and resolving `bun:*` specifiers, which do not exist when bundling for Node.
40+
// Required by node-gyp-build-optional-packages. Not native, but in the
41+
// closure: without it, WSL gets MODULE_NOT_FOUND while Windows is fine.
42+
"detect-libc",
43+
] as const;
44+
45+
/**
46+
* External only so the bundler never has to resolve them.
47+
*
48+
* These are reached through a runtime-conditional dynamic import that Node
49+
* never takes, and they resolve `bun:*` specifiers that do not exist when
50+
* bundling for Node. Because Node never loads them, their dependency closure
51+
* does not need to be external — only the entry point must stay unbundled.
52+
*/
53+
export const CLI_BUILD_ONLY_EXTERNAL_PREFIXES = [
3354
"@effect/platform-bun",
3455
"@effect/sql-sqlite-bun",
3556
] as const;
3657

58+
export const CLI_EXTERNAL_PACKAGE_PREFIXES = [
59+
...CLI_RUNTIME_EXTERNAL_PREFIXES,
60+
...CLI_BUILD_ONLY_EXTERNAL_PREFIXES,
61+
] as const;
62+
3763
/** True when the CLI bundle should inline `id` rather than leave it external. */
3864
export function shouldBundleCliDependency(id: string): boolean {
3965
if (id.startsWith("node:")) return false;

0 commit comments

Comments
 (0)