Skip to content

fix(compile): respect npm registry sub-paths when flattening node_modules#34575

Merged
bartlomieju merged 3 commits into
denoland:mainfrom
lunadogbot:fix/compile-npm-registry-subpath
May 31, 2026
Merged

fix(compile): respect npm registry sub-paths when flattening node_modules#34575
bartlomieju merged 3 commits into
denoland:mainfrom
lunadogbot:fix/compile-npm-registry-subpath

Conversation

@lunadogbot

@lunadogbot lunadogbot commented May 30, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #26628. deno compile produced binaries that failed with ERR_MODULE_NOT_FOUND when the configured npm registry URL contained a sub-path (e.g. http://mirrors.example.com/npm/).

The on-disk cache layout for such a registry is <global_cache>/<host>/<sub>/<pkg>/..., but build_vfs_consolidating_global_npm_cache in cli/standalone/binary.rs flattened the cache by iterating each top-level entry under <global_cache> exactly one level deep — collapsing <global_cache>/<host>/<pkg> directly into the embedded localhost/. With a sub-path registry the loop instead picks up the intermediate <sub> directory and embeds packages at localhost/<sub>/<pkg>/..., which then can't be found at runtime (where the binary expects localhost/<pkg>/...).

This PR uses the known_registries_dirnames already tracked by NpmCacheDir (built from .npmrc resolution) to walk to each registry's actual package root before flattening, and to know which top-level host directories under the global cache are registry roots that should be dropped after their packages are moved into localhost/. The default no-sub-path case is unchanged.

To make this accessible from cli/standalone/binary.rs, the PR adds:

  • NpmCacheDir::known_registries_dirnames()
  • ManagedCliNpmResolver::known_registries_dirnames()

Manual repro

I verified end-to-end with a Deno-script proxy that mirrors registry.npmjs.org at a sub-path:

// proxy.ts - listens at http://localhost:4873/sub/path/npm/
const target = "https://registry.npmjs.org";
const prefix = "/sub/path/npm";
Deno.serve({ port: 4873 }, async (req) => {
  const url = new URL(req.url);
  if (!url.pathname.startsWith(prefix)) {
    return new Response("nf", { status: 404 });
  }
  const rest = url.pathname.slice(prefix.length) + url.search;
  const res = await fetch(target + rest, { redirect: "follow" });
  const ctype = res.headers.get("content-type") ?? "";
  if (!ctype.includes("application/json")) return res;
  const text = (await res.text()).replaceAll(
    "https://registry.npmjs.org/",
    \`http://localhost:4873${prefix}/\`,
  );
  const h = new Headers(res.headers);
  h.delete("content-encoding");
  h.delete("content-length");
  return new Response(text, { status: res.status, headers: h });
});
// main.ts
import { say } from "npm:[email protected]";
console.log(say({ text: "Hello from Deno!" }));

Before this PR:

$ NPM_CONFIG_REGISTRY="http://localhost:4873/sub/path/npm/" \
    deno compile --allow-read -o main main.ts
... .deno_compile_node_modules/localhost/sub/path/* ...
$ ./main
error: [ERR_MODULE_NOT_FOUND] Cannot find module
  'file:///tmp/deno-compile-main/.deno_compile_node_modules/localhost/cowsay/1.5.0/index.js'

After:

... .deno_compile_node_modules/localhost/cowsay/1.5.0/* ...
$ ./main
 __________________
< Hello from Deno! >
...

I also re-checked the default (no sub-path) registry case end-to-end to confirm no regression — embedded layout and runtime behaviour are unchanged.

A spec test is now included. handle_req_for_registry in tests/util/server/servers/npm_registry.rs strips a fixed /sub/path prefix before serving package files, so the test registry can be driven through a sub-path url. The spec at tests/specs/compile/npm_registry_subpath/ compiles a package cached through http://localhost:4260/sub/path/ and asserts the embedded layout flattens to localhost/@denotest/esm-basic/* rather than localhost/sub/path/..., then runs the binary to confirm it resolves at runtime.

Test plan

  • New compile/npm_registry_subpath spec asserts the flattened layout and runtime behaviour
  • Existing compile/npm/npmrc spec tests stay green (audit sub-path + global-cache compile specs verified locally)
  • Manual repro above before/after on a sub-path registry
  • Default registry (no sub-path) still works end-to-end

exe.dev user and others added 2 commits May 30, 2026 10:44
…ules

`deno compile` flattens each cached npm registry's package directories into
`.deno_compile_node_modules/localhost/<pkg>/...` so the embedded binary can
serve them from a single dummy registry at runtime. The flattening only
descended one directory level, which works when the registry URL has no
path (e.g. `https://registry.npmjs.org/`) but skips a level for registries
whose URL includes a sub-path (e.g. `http://mirrors.example.com/npm/`).
Packages end up at `localhost/<sub>/<pkg>/...` and the compiled binary
fails at runtime with `ERR_MODULE_NOT_FOUND`.

Use the known registry dir-names already tracked by `NpmCacheDir` (built
from `.npmrc` resolution) to walk to each registry's actual package root
before flattening, and only drop the registry host directories whose
packages were lifted into `localhost/`. The default no-sub-path case
behaves identically.

Fixes denoland#26628
Teaches the test npm registry server to strip a fixed `/sub/path` prefix
before serving package files, so tests can exercise a registry url with a
sub-path. Adds a compile spec that caches a package through such a registry
and asserts the embedded layout flattens to `localhost/<pkg>` rather than
`localhost/sub/path/<pkg>` (regression test for denoland#26628).
@bartlomieju

Copy link
Copy Markdown
Member

Added the deferred spec test in f00bebd:

  • Taught the test npm registry server (handle_req_for_registry) to strip a fixed /sub/path prefix before serving package files, so the registry can be driven through a sub-path url.
  • New spec tests/specs/compile/npm_registry_subpath/ caches @denotest/esm-basic through http://localhost:4260/sub/path/, asserts the embedded layout flattens to .deno_compile_node_modules/localhost/@denotest/esm-basic/* (not localhost/sub/path/...), and runs the binary to confirm it resolves.

Verified locally that the new spec passes and that the existing audit::subpath_registry and global-cache compile specs are unaffected. The earlier node_compat CI red was on macOS x86_64 debug and unrelated to this change (this PR only touches compile's npm-cache flattening + a test-only server tweak).

On Windows the compiled output is main.exe, so the embedded-files tree
root is main.exe rather than main. Use main[WILDLINE] like the other
compile specs so the assertion matches on all platforms.

@lunadogbot lunadogbot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(Comment review — can't approve as same auth account.) LGTM. Using NpmCacheDir's already-tracked known_registries_dirnames is the right hook — the cache layout is the source of truth, so deriving the flatten paths from it instead of assuming one-level depth fixes the regression without heuristics. The two new pass-throughs (NpmCacheDir::known_registries_dirnames, ManagedCliNpmResolver::known_registries_dirnames) are minimal.

Substantive notes:

  • registry_top_segments collects the first path segment of each known registry dirname and then drops any top-level dir under the global cache whose name matches. If two registries share the same host (e.g. localhost:4873/a and localhost:4873/b), only their explicit registry_dirname subtrees are flattened into localhost_entries, but the entire localhost top-level dir gets dropped. If a host dir ever contained packages cached under an unknown registry path on that same host, those would silently be lost. Realistically the global cache only gets populated through known registries, so I don't think this can happen — but a one-line comment in the new block explaining the assumption would save the next reader some time.

  • The duplicate-entry panic! is preserved from the previous code (so this PR doesn't make it worse), but with multiple registries flattening into the same localhost_entries map, the surface area for a real-world collision (e.g. same scoped package cached from two mirrors) is now higher. Not a blocker, but if a user reports a panic here in the wild, this is the place.

The handle_req_for_registry /sub/path strip is fine for the test; the spec asserts both the embedded layout (.deno_compile_node_modules/localhost/@denotest/esm-basic/*, not localhost/sub/path/...) and runtime resolution. Good regression guard.

@bartlomieju
bartlomieju merged commit 4937b0e into denoland:main May 31, 2026
136 checks passed
littledivy pushed a commit to crowlKats/deno that referenced this pull request Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compiled executable failed to find modules from custom npm registry containing subpath

2 participants