fix(npm): share copy-package variants via symlink for class identity#34691
Conversation
When deno install resolves an npm package through multiple peer-dependency contexts, deno_npm produces copy_index variants of the same name+version (`@[email protected]`, `@[email protected]_1`, ...). The local installer was filling each variant's `node_modules/<pkg>/` with `clone_dir_recursive` (hardlinks at distinct paths). Realpath then returned the variant path, so `Module._cache` kept a separate entry per variant and exported a distinct class object on each. NestJS DI compares the constructor-parameter type captured by `@Injectable()` decorator metadata to the registered provider class via `===`; with two `ModuleRef` classes alive the lookup failed ("Nest can't resolve dependencies of the TypeOrmHealthIndicator"). Symlink the variant's package directory to the canonical (copy_index=0) copy instead. The variant directory keeps its own peer-dep symlink neighbours, but realpath of any file inside the package collapses to one path, matching Node + npm and pnpm semantics, so the class identity is preserved across the require chain. Adds the regression spec `peer_dep_copy_class_identity` and updates the existing `peer_deps_with_copied_folders_and_lockfile` outputs whose `--node-modules-dir` cases changed from `1\n2` to `1\n1` for the same reason (first-peer-wins, as in Node). Refs #26427 Closes denoland/divybot#404 Co-Authored-By: Divy Srivastava <[email protected]>
nathanwhit
left a comment
There was a problem hiding this comment.
I don't think this is correct. I suspect this is a peer dep resolution bug and this just papers over it. Compare to what pnpm does on this exact case, and confirm that pnpm ALSO produces copy packages for this case. If it does not then we have an npm peer dependency resolution bug, and we should have deduplicated those copies. If pnpm DOES produce copy packages similarly, then you need to confirm that pnpm does this symlink thing too.
…dentity" This reverts commit cc1408a.
|
@nathanwhit you're right — I confirmed empirically that pnpm produces only one copy of Repro with pnpm 9.15.0: $ pnpm install
$ ls node_modules/.pnpm | grep '@nestjs+core'
@[email protected]_@[email protected][email protected][email protected]__@nestjs+platform_v2fuvjwzi6rez7k3z2o4qmdrhy
$ readlink node_modules/@nestjs/core
../.pnpm/@[email protected]_..._v2fuvjwzi6rez7k3z2o4qmdrhy/node_modules/@nestjs/core
$ readlink node_modules/.pnpm/@nestjs+terminus*/node_modules/@nestjs/core
../../../@[email protected]_..._v2fuvjwzi6rez7k3z2o4qmdrhy/node_modules/@nestjs/coreBoth the user-visible For Deno, the existing I'd appreciate guidance on whether the fix belongs in |
|
it should probably avoid producing the duplicates (resolve_peers_of_node), though if that is intractable i think a fix in dedup_peer_dependents could work (it would need to be careful to only dedupe valid sets). i would suggest referencing pnpm's source code to see how they handle this. |
`@nestjs/terminus` triggers a peer-dependency cycle (`@nestjs/core` ↔ `@nestjs/platform-express`) under which deno_npm's resolver historically produced multiple `NpmPackageId` entries for the same name+version, differing only in cycle-unrolling depth. Pnpm produces only one canonical entry for the same package set. Once such a lockfile exists, loading it via `snapshot_from_lockfile` preserves the spurious variants because that path skips the resolver-side `dedup_peer_dependents` pass. Each variant then becomes a separate `node_modules/.deno/...` directory at install time, a separate `Module._cache` entry at runtime, and a distinct class object — breaking the `===` identity comparisons that NestJS DI relies on (the `ModuleRef` constructor metadata captured by `@Injectable()` no longer matches the registered provider class). Add a snapshot-level dedup pass, `dedup_equivalent_peer_variants`, that merges packages of the same nv whose effective dependency set is identical — same `dependencies` (mapping name → child `nv`), `optional_dependencies`, `optional_peer_dependencies`, and `system`. The shortest `NpmPackageId` in each equivalence class is kept as the canonical entry; the others are dropped and all references (root specifiers and per-package dependency ids) are rewritten. The pass iterates so that merging the cores collapses references and exposes their peer variants for the next round. Wire it into `snapshot_from_lockfile` so stale lockfiles transparently load as the deduplicated graph. Variants with genuinely different peer resolutions (e.g. `[email protected]` paired with `pe@1` vs `pe@2`) are left untouched — verified by `test_dedup_keeps_distinct_peer_resolutions` and by the existing `tests/specs/npm/peer_deps_with_copied_folders_and_lockfile` spec continuing to pass. Refs #26427 Closes denoland/divybot#404 Co-Authored-By: Divy Srivastava <[email protected]>
|
@nathanwhit reworked along your guidance. Investigation:
Fix in
(I considered fixing this purely in |
CI flagged the unused `_nv` binding in `dedup_equivalent_peer_variants`. Switch the iteration to `HashMap::values()` since the key is never read. Co-Authored-By: Divy Srivastava <[email protected]>
|
Could we do the fixup on like deno install only? It's not great when people get random lockfile changes on deno run e.g. |
Per review: don't dedup the lockfile snapshot on `deno run` — users shouldn't see surprise lockfile rewrites while just running code. Dedup now flows from a new `dedup_lockfile_peer_variants` option on `NpmInstallerFactoryOptions`, threaded through `NpmResolverManagedSnapshotOption::ResolveFromLockfile` and `SnapshotFromLockfileParams::dedup_equivalent_peer_variants`. The flag is set true only for `add`, `install` (local), `cache`, `ci`, `remove`, and `approve-scripts`. LSP and all other entry points pass false. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
@nathanwhit good call — gated. New commit 91df19a:
|
Summary
When
deno installresolves an npm package through multiple peer-depcontexts, deno_npm produces
copy_indexvariants of the samename+version (
@[email protected],@[email protected]_1, ...). The local installerfilled each variant's
node_modules/<pkg>/withclone_dir_recursive(per-file hardlinks at distinct directory paths).
realpath()thenreturned the variant path verbatim, so Deno's CJS
Module._cachekeptone entry per variant and every variant produced a distinct class
object.
NestJS DI compares the constructor-parameter type captured by
@Injectable()decorator metadata to the registered provider classvia
===. With twoModuleRefclasses alive the lookup fails:That is #26427 —
@nestjs/terminustriggers it because itpulls
@nestjs/corethrough a peer-dep cycle (core ↔ platform-express),producing five physical copies of
@nestjs/corein the user'snode_modules.The fix symlinks each variant's package directory to the canonical
(copy_index=0) variant's package directory instead of cloning. The
variant's own
node_modules/neighbours (the peer-dep symlinks)remain, but
realpath()of any file inside the package collapses to asingle canonical path.
Module._cachethen has one entry per packageand the class identity is preserved across require chains, matching
Node + npm and pnpm semantics (first-peer-wins for the inner
peer-resolution).
The global cache install path (
ensure_copy_packageinlibs/npm_cache/lib.rs) is unchanged.Test plan
tests/specs/npm/peer_dep_copy_class_identitywith new fixture packages
@denotest/[email protected]and
@denotest/peer-dep-test-class-consumer@{1,2}.0.0: asserts aclass exported from the (formerly duplicated) variant compares
===regardless of which consumer (peer-dep context) reached it.tests/specs/npm/peer_deps_with_copied_folders_and_lockfile--node-modules-dir=autooutputs: the inner-peer values changefrom
1\n2to1\n1because the previously distinct grandchildcopies now share a module instance (first-peer-wins). Global
cache runs are untouched and still produce
1\n2.(
lucassusanto/nestjs-deno-terminus-issue):[NestApplication] Nest application successfully started.cargo test --test specs -- specs::npm::— 276 passed; the 12pre-existing failures all baseline-fail without this change
(tarball-integrity drift in the local test registry, unrelated).
cargo build --bin denoclean.Refs #26427
Closes denoland/divybot#404