Skip to content

fix(npm): share copy-package variants via symlink for class identity#34691

Merged
littledivy merged 5 commits into
mainfrom
orch/divybot-404
Jun 2, 2026
Merged

fix(npm): share copy-package variants via symlink for class identity#34691
littledivy merged 5 commits into
mainfrom
orch/divybot-404

Conversation

@divybot

@divybot divybot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Summary

When deno install resolves an npm package through multiple peer-dep
contexts, deno_npm produces copy_index variants of the same
name+version (@[email protected], @[email protected]_1, ...). The local installer
filled each variant's node_modules/<pkg>/ with clone_dir_recursive
(per-file hardlinks at distinct directory paths). realpath() then
returned the variant path verbatim, so Deno's CJS Module._cache kept
one entry per variant and every variant produced a distinct class
object.

NestJS DI compares the constructor-parameter type captured by
@Injectable() decorator metadata to the registered provider class
via ===. With two ModuleRef classes alive the lookup fails:

[Nest] ... ERROR [ExceptionHandler] Nest can't resolve dependencies of
the TypeOrmHealthIndicator (?). Please make sure that the argument
ModuleRef at index [0] is available in the TerminusModule context.

That is #26427@nestjs/terminus triggers it because it
pulls @nestjs/core through a peer-dep cycle (core ↔ platform-express),
producing five physical copies of @nestjs/core in the user's
node_modules.

The fix symlinks each variant's package directory to the canonical
(copy_index=0) variant's package directory instead of cloning. The
variant's own node_modules/ neighbours (the peer-dep symlinks)
remain, but realpath() of any file inside the package collapses to a
single canonical path. Module._cache then has one entry per package
and the class identity is preserved across require chains, matching
Node + npm and pnpm semantics (first-peer-wins for the inner
peer-resolution).

The global cache install path (ensure_copy_package in
libs/npm_cache/lib.rs) is unchanged.

Test plan

  • Added regression spec tests/specs/npm/peer_dep_copy_class_identity
    with new fixture packages @denotest/[email protected]
    and @denotest/peer-dep-test-class-consumer@{1,2}.0.0: asserts a
    class exported from the (formerly duplicated) variant compares
    === regardless of which consumer (peer-dep context) reached it.
  • Updated existing
    tests/specs/npm/peer_deps_with_copied_folders_and_lockfile
    --node-modules-dir=auto outputs: the inner-peer values change
    from 1\n2 to 1\n1 because the previously distinct grandchild
    copies now share a module instance (first-peer-wins). Global
    cache runs are untouched and still produce 1\n2.
  • Manually verified upstream repro
    (lucassusanto/nestjs-deno-terminus-issue):
    [NestApplication] Nest application successfully started.
  • cargo test --test specs -- specs::npm:: — 276 passed; the 12
    pre-existing failures all baseline-fail without this change
    (tarball-integrity drift in the local test registry, unrelated).
  • cargo build --bin deno clean.

Refs #26427

Closes denoland/divybot#404

When deno install resolves an npm package through multiple peer-dependency
contexts, deno_npm produces copy_index variants of the same name+version
(`@[email protected]`, `@[email protected]_1`, ...). The local installer was filling each
variant's `node_modules/<pkg>/` with `clone_dir_recursive` (hardlinks at
distinct paths). Realpath then returned the variant path, so
`Module._cache` kept a separate entry per variant and exported a distinct
class object on each. NestJS DI compares the constructor-parameter type
captured by `@Injectable()` decorator metadata to the registered provider
class via `===`; with two `ModuleRef` classes alive the lookup failed
("Nest can't resolve dependencies of the TypeOrmHealthIndicator").

Symlink the variant's package directory to the canonical (copy_index=0)
copy instead. The variant directory keeps its own peer-dep symlink
neighbours, but realpath of any file inside the package collapses to one
path, matching Node + npm and pnpm semantics, so the class identity is
preserved across the require chain. Adds the regression spec
`peer_dep_copy_class_identity` and updates the existing
`peer_deps_with_copied_folders_and_lockfile` outputs whose
`--node-modules-dir` cases changed from `1\n2` to `1\n1` for the same
reason (first-peer-wins, as in Node).

Refs #26427

Closes denoland/divybot#404

Co-Authored-By: Divy Srivastava <[email protected]>

@nathanwhit nathanwhit left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is correct. I suspect this is a peer dep resolution bug and this just papers over it. Compare to what pnpm does on this exact case, and confirm that pnpm ALSO produces copy packages for this case. If it does not then we have an npm peer dependency resolution bug, and we should have deduplicated those copies. If pnpm DOES produce copy packages similarly, then you need to confirm that pnpm does this symlink thing too.

@divybot

divybot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

@nathanwhit you're right — I confirmed empirically that pnpm produces only one copy of @nestjs/[email protected] for the exact same package set (same lockfile-pinned @nestjs/{common,core,platform-express}@10.4.5 + @nestjs/[email protected] + reflect-metadata@^0.2), while deno install produces five (@[email protected] through @[email protected]_4). So this is an npm peer-dep resolution bug — these copies should have been deduplicated. I've reverted the install-time change (it was papering over the symptom, as you said).

Repro with pnpm 9.15.0:

$ pnpm install
$ ls node_modules/.pnpm | grep '@nestjs+core'
@[email protected]_@[email protected][email protected][email protected]__@nestjs+platform_v2fuvjwzi6rez7k3z2o4qmdrhy
$ readlink node_modules/@nestjs/core
../.pnpm/@[email protected]_..._v2fuvjwzi6rez7k3z2o4qmdrhy/node_modules/@nestjs/core
$ readlink node_modules/.pnpm/@nestjs+terminus*/node_modules/@nestjs/core
../../../@[email protected]_..._v2fuvjwzi6rez7k3z2o4qmdrhy/node_modules/@nestjs/core

Both the user-visible node_modules/@nestjs/core and @nestjs/terminus's nested @nestjs/core resolve through the same .pnpm entry → same realpath → same Module._cache entry → same ModuleRef class → NestJS DI succeeds.

For Deno, the existing dedup_peer_dependents pass in libs/npm/resolution/graph.rs is supposed to merge equivalent variants but doesn't fire here. The five @nestjs/[email protected] variants share the same nv and the same effective set of dep + peer-dep NVs (@nestjs/[email protected], @nestjs/[email protected], [email protected], [email protected]); each variant's nested NpmPackageId.peer_dependencies only differs in cycle-unrolling depth. is_superset_node compares children by name+nv and peer-deps by nv (not by NodeId), so the variants should be mutual subsets/supersets — but the pass leaves all five behind.

I'd appreciate guidance on whether the fix belongs in dedup_peer_dependents (extending it to handle the mutual-superset cycle case) or upstream in resolve_peers_of_node (not creating the spurious copies in the first place). Happy to take a swing once direction is clear.

@nathanwhit

Copy link
Copy Markdown
Member

it should probably avoid producing the duplicates (resolve_peers_of_node), though if that is intractable i think a fix in dedup_peer_dependents could work (it would need to be careful to only dedupe valid sets). i would suggest referencing pnpm's source code to see how they handle this.

`@nestjs/terminus` triggers a peer-dependency cycle (`@nestjs/core` ↔
`@nestjs/platform-express`) under which deno_npm's resolver historically
produced multiple `NpmPackageId` entries for the same name+version,
differing only in cycle-unrolling depth. Pnpm produces only one
canonical entry for the same package set. Once such a lockfile exists,
loading it via `snapshot_from_lockfile` preserves the spurious variants
because that path skips the resolver-side `dedup_peer_dependents` pass.
Each variant then becomes a separate `node_modules/.deno/...` directory
at install time, a separate `Module._cache` entry at runtime, and a
distinct class object — breaking the `===` identity comparisons that
NestJS DI relies on (the `ModuleRef` constructor metadata captured by
`@Injectable()` no longer matches the registered provider class).

Add a snapshot-level dedup pass, `dedup_equivalent_peer_variants`, that
merges packages of the same nv whose effective dependency set is
identical — same `dependencies` (mapping name → child `nv`),
`optional_dependencies`, `optional_peer_dependencies`, and `system`.
The shortest `NpmPackageId` in each equivalence class is kept as the
canonical entry; the others are dropped and all references (root
specifiers and per-package dependency ids) are rewritten. The pass
iterates so that merging the cores collapses references and exposes
their peer variants for the next round. Wire it into
`snapshot_from_lockfile` so stale lockfiles transparently load as the
deduplicated graph.

Variants with genuinely different peer resolutions (e.g. `[email protected]`
paired with `pe@1` vs `pe@2`) are left untouched — verified by
`test_dedup_keeps_distinct_peer_resolutions` and by the existing
`tests/specs/npm/peer_deps_with_copied_folders_and_lockfile` spec
continuing to pass.

Refs #26427

Closes denoland/divybot#404

Co-Authored-By: Divy Srivastava <[email protected]>
@divybot

divybot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

@nathanwhit reworked along your guidance.

Investigation:

  • Fresh resolution of this exact package set (no lockfile) on current main already produces a single @nestjs/[email protected]dedup_peer_dependents in the resolver does its job for live resolution.
  • The bug surfaces specifically when snapshot_from_lockfile loads a stale lockfile whose five @nestjs/[email protected]_… entries differ only in cycle-unrolling depth. That lockfile-load path constructs ValidSerializedNpmResolutionSnapshot directly from the deserialized data and never runs the resolver's dedup pass, so the spurious variants stick.
  • pnpm produces only one entry for the same package set; both the user-visible @nestjs/core and @nestjs/terminus's nested @nestjs/core symlink to that single store entry.

Fix in libs/npm/resolution/snapshot.rs:

  • Added SerializedNpmResolutionSnapshot::dedup_equivalent_peer_variants. Groups packages by nv, partitions each group by an effective-dependency signature — dependencies mapped to (name → child nv), plus optional_dependencies, optional_peer_dependencies, and system. Variants with the same signature are mutual supersets in is_superset_node terms, so they're collapsed onto the shortest serialized NpmPackageId and all references (root_packages and per-package dependency ids) are rewritten. Iterates because merging the core variants exposes the pe variants for the next round.
  • Called from snapshot_from_lockfile, so stale lockfiles transparently load as the deduplicated graph. No change to live-resolution paths or to the global cache install path.
  • "Careful to only dedupe valid sets": variants with genuinely different peer resolutions (e.g. [email protected]_pe@1 vs [email protected]_pe@2) are not merged. Verified by test_dedup_keeps_distinct_peer_resolutions and by the existing peer_deps_with_copied_folders_and_lockfile spec still passing unchanged.
  • Two new unit tests in snapshot.rs cover the cycle-unrolling collapse and the legitimate distinct-peer case.
  • Manually verified on lucassusanto/nestjs-deno-terminus-issue: only @[email protected] ends up under .deno/, and the NestJS app reaches [NestApplication] Nest application successfully started.

(I considered fixing this purely in resolve_peers_of_node per your preference, but my evidence shows the live resolver already produces a single variant — the leak is entirely through the lockfile-load path that bypasses the resolver. Snapshot-level dedup felt like the least invasive way to fix existing stale lockfiles in addition to whatever the live resolver does. Happy to move the logic if you'd rather have it elsewhere.)

CI flagged the unused `_nv` binding in
`dedup_equivalent_peer_variants`. Switch the iteration to
`HashMap::values()` since the key is never read.

Co-Authored-By: Divy Srivastava <[email protected]>
@littledivy
littledivy requested a review from nathanwhit June 2, 2026 03:46
@nathanwhit

Copy link
Copy Markdown
Member

Could we do the fixup on like deno install only? It's not great when people get random lockfile changes on deno run e.g.

Per review: don't dedup the lockfile snapshot on `deno run` — users
shouldn't see surprise lockfile rewrites while just running code.
Dedup now flows from a new `dedup_lockfile_peer_variants` option on
`NpmInstallerFactoryOptions`, threaded through
`NpmResolverManagedSnapshotOption::ResolveFromLockfile` and
`SnapshotFromLockfileParams::dedup_equivalent_peer_variants`. The flag
is set true only for `add`, `install` (local), `cache`, `ci`,
`remove`, and `approve-scripts`. LSP and all other entry points pass
false.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
@divybot

divybot commented Jun 2, 2026

Copy link
Copy Markdown
Contributor Author

@nathanwhit good call — gated. New commit 91df19a:

  • Added dedup_lockfile_peer_variants: bool to NpmInstallerFactoryOptions, threaded through NpmResolverManagedSnapshotOption::ResolveFromLockfile { lockfile, dedup_equivalent_peer_variants } and SnapshotFromLockfileParams::dedup_equivalent_peer_variants.
  • In cli/factory.rs the flag is set only for add, install (local), cache, ci, remove, and approve-scripts — same set that gets clean_on_install. Everything else (run, serve, test, bench, task, eval, LSP, …) passes false, so deno run against a stale lockfile loads it as-is and won't write back a deduped version.
  • The NestJS fix still applies the moment the user runs deno install / deno cache / deno ci — that path rewrites the lockfile cleanly and the installed node_modules ends up with a single physical copy.

@littledivy
littledivy merged commit 563aa7b into main Jun 2, 2026
136 checks passed
@littledivy
littledivy deleted the orch/divybot-404 branch June 2, 2026 09:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants