Note that this is just a draft implementation. One thing that is puzzling me right now is that with the following expression, the output directory simply disappears after build:

pkgs.nodePackages.fetchNodeModules {
  src = pkgs.fetchFromGitHub {
    owner = "nttgin";
    repo = "BGPalerter";
    rev = "v1.28.1";
    sha256 = "sha256-Y0atkJfZxjUOGPQ3goXS/YD5SsX9ZjpbM0Nc5IuaFP4=";
  };

  makeTarball = false;
  exposeBin = true;
  sha256 = "sha256-T9W96csyA+/fwFs1eNccslx5qcbVTpAtqLM1Xd32tk4=";
}

Would like to know if this problem is reproducible by others and what the solution is.

What does this do that npmlock2nix (or one of the other 20 attempts to do npm things with nix) doesn't?

What does this do that npmlock2nix (or one of the other 20 attempts to do npm things with nix) doesn't?

I think npmlock2nix is doing import from derivation, which this tool does not.

fetchNodeModules output should be tested on different platforms if it produces different results. Also if it is stable on the same platform.

What does this do that npmlock2nix (or one of the other 20 attempts to do npm things with nix) doesn't?

I think npmlock2nix is doing import from derivation, which this tool does not.

Also this tool doesn't attempt to parse the package.json / package-lock.json or otherwise Nixify the dependencies at all - It simply delegates the fetching to the actual npm, since npm ci provides the reproducibility we need, which is why I compared it to fetchCargoTarball (used internally by buildRustPackage) which does the same thing but with cargo.

Pushed a fix to change the directory structure ($out/lib/node_modules) as well as the timestamps. The hashes in the examples have been updated.

Enforcement in Nix happens, but in a very narrow way. If you forget to update the hash, you will be using stale dependencies, unknowingly, until "hopefully" someone else tries to build it without your store or binary cache failing the fetcher, detecting the problem. Considering that these will end up on cache.nixos.org, these problems can linger for a long time.

This problem can be solved by adding the lock file to the dependencies as well and than diff the lock file from the fixed-derivation output with lock file delivered in the source.

It's also worth considering that these store paths will duplicate a lot of files that are shared store paths in case of other solutions. (hard link optimization only applies to a local nix store; nix-copy-closure and other forms of distribution such as docker images are still affected)

Having thousands of node derivations also comes with a cost: evaluation memory and nixpkgs download/unpack size.

The hashes in the examples have been updated.

We don't have a method for finding and updating these when a change logic inevitably requires this again. This is especially true when other projects call this function.

This might be alleviated somewhat by including a hash of relevant inputs in the name, forcing a new store path when those inputs change. However, the selection of those inputs is a trade-off between usability and correctness. Too many and you'll burden the user with unnecessary hash updates. Too few and we'll have reproducibility problems. This is not a choice we should have to make.

Specifically with npm we don't have to make this trade-off because it provides us with perfectly usable lock files that we can parse with Nix to fetch dependencies in a reliably reproducible way.

Does not really scale in nixpkgs. Dependency trees are so large for the average nodejs package that the added file size is an not acceptable overhead to store all lock files in nixpkgs. Otherwise we would have switched tools like https://github.com/nix-community/yarn2nix or similar. We cannot use import from derivation because this would increase the evaluation time.

To add another data sample. Peertubes yarn.lock file is 8000 lines. This is a huge amount for a single package even without the full dependency tree. I think this fetcher here is the only sane approach for nixpkgs.

I expect this fetcher to be implementable as a non-fixed output recursive derivation that

1. takes a fetchurled lock file as an input
2. fetches the modules using fetchurl during the build (during the derivation build in case of (general) recursive Nix or after the current derivation in case of ret-cont RFC40 recursive Nix)
3. creates a link farm in $out

Recursive Nix is implemented but experimental, so while we could try it today, we can't use it by default in Nixpkgs. Eelco has raised a concern

Probably we should also disallow instantiating/building fixed-output derivations (specifically, those that access the network[...])

However, I believe the impurity can be addressed without disallowing it. If for whatever reason that doesn't work out, this idea seems to be compatible with RFC40 ret-cont recursive Nix; a more restrictive form of recursion. Perhaps even RFC92, although I really can't confirm that now.

With this solution on the horizon, I believe we can use this PR in the interim without creating a structural problem for Nix's perceived reliability.

takes a fetchurled lock file as an input

That wont work for everything because upstream does not always provide an (up to date) lock file.

fetches the modules using fetchurl during the build

npm can take anything as a downloaded file and we already fetching random git repos in nodePackages right now.

takes a fetchurled lock file as an input

That wont work for everything because upstream does not always provide an (up to date) lock file.

While true, I do not think it's relevant to my comment. It is a more general problem but the solution is easy enough: make a PR to update the lock file, and fetch from the PR. Preferably tag it in the fork as well to avoid loss on rebase, but that's yet another topic.

fetches the modules using fetchurl during the build

npm can take anything as a downloaded file and we already fetching random git repos in nodePackages right now.

I intended fetchurl as an example. Any derivation can be used, including those of fetchgit.

I expect this fetcher to be implementable as a non-fixed output recursive derivation that

1. takes a `fetchurl`ed lock file as an input

2. fetches the modules using `fetchurl` during the build

3. creates a link farm in `$out`

The algorithm you are describing is effectively import-from-derivation. It is not a good idea to implement it this way because this would force a nix eval process to download all node packages just to evaluate nixpkgs. This is slow and takes a lot of disk space and moves computational work from builder to evaluator (bad for how Nix CIs currently work).
Correct me if I am wrong.

The algorithm you are describing is effectively import-from-derivation.

It is not. Evaluation for the (large number of) fetcher calls happens inside the sandbox thanks to recursive Nix. I've clarified the original comment.

It is not a good idea to implement it this way because this would force a nix eval process to download all node packages just to evaluate nixpkgs.

To the rest of Nixpkgs, a recursive Nix fetchNodeModules appears and performs like any other single derivation. Thanks to recursive Nix, the expensive parts are moved into the sandbox, where it can run later, in parallel and remotely.

This is slow and takes a lot of disk space and moves computational work from builder to evaluator (bad for how Nix CIs currently work).
Correct me if I am wrong.

Hercules CI has solved the user experience around import-from-derivation. It doesn't have such an evaluation bottleneck and it's clear when its evaluator is waiting for IFD. IFD is still sequential as of today, but that's not a problem in practice, unless you're building at the scale of Nixpkgs.

To the rest of Nixpkgs, a recursive Nix fetchNodeModules appears and performs like any other single derivation. Thanks to recursive Nix, the expensive parts are moved into the sandbox, where it can run later, in parallel and remotely.

Ok. But it is not clear when we will be able to use this feature. So far a nix release is just pushed forward year by year by adding more experimental features. To me it seems we could add it with a checksum and when time comes we could switch over to recursive nix by dropping the need for a checksum.

Hercules CI has solved the user experience around import-from-derivation. It doesn't have such an evaluation bottleneck and it's clear when its evaluator is waiting for IFD. IFD is still sequential as of today, but that's not a problem in practice, unless you're building at the scale of Nixpkgs.

That's nice however hercules evaluation is still slower than both plain nix and hydra (which uses multiple threads).

not clear when we will be able to use this feature.

and when time comes we could switch over to recursive nix by dropping the need for a checksum.

So we agree.

hercules evaluation is still slower than both plain nix and hydra

Thanks for letting me know. I'll start to optimize eval soon.

What does this do that npmlock2nix (or one of the other 20 attempts to do npm things with nix) doesn't?

I think npmlock2nix is doing import from derivation, which this tool does not.

Nope, it doesn't. It works in hydras restricted eval mode. You can use it with IFD if you don't want to copy the lock file into the repo.

Give that this PR proposes yet another big FOD I am really not in favor of this. We already had enough trouble in the Go and Rust world due to these issues.

I was originally planning to propose npmlock2nix for nixpkgs but I want it to be more mature first.

What does this do that npmlock2nix (or one of the other 20 attempts to do npm things with nix) doesn't?

I think npmlock2nix is doing import from derivation, which this tool does not.

Nope, it doesn't. It works in hydras restricted eval mode. You can use it with IFD if you don't want to copy the lock file into the repo.

Give that this PR proposes yet another big FOD I am really not in favor of this. We already had enough trouble in the Go and Rust world due to these issues.

I was originally planning to propose npmlock2nix for nixpkgs but I want it to be more mature first.

npmlock2nix is also not a solution just look how large a package-lock.json is: https://github.com/andig/evcc/blob/master/package-lock.json. That might work for projects that have the package-lock.json in the repo but certainly not for nixpkgs. It also still will increase evaluation time due to large dependency graph it needs to process. I don't see any other scalable solution but this fixed-input derivations and maybe recursive nix in the future.

today there was the new nix version released on nixpkgs-unstable

❯ nix --version
nix (Nix) 2.4pre20210908_3c56f62

just saying that the same problem of disappearing derivation happens with that new version as well.

In the interest of discussion I went ahead and did an implementation for yarn
#138233
(handles both yarn.lock and package-lock.json)
I would be happy to get feedback on it.
My thinking is that if it takes to get another 3 months before we get npmlock2nix in nixpkgs but we can get a fetcher in next week, then it's that much earlier that we can start making a transition for the js eco-system.

Have we considered the old haskellPackages-like workflow?

• contributors submit changes to the config file only
• update automation runs on a branch node-updates (like haskell-updates)
• hydra builds it and runs the relevant tests
• merge weekly

This is the old workflow though. Nowadays contributors do generate the exprs, because it's fast, but haskellPackages has done without that for a long time.
For this to be feasible on node, contributors would have to run with a minimal config file which is sufficient for their package only, so they can regenerate the expressions and test in a reasonable amount of time.

It does avoid the technical issues.

To move this forward, I have removed the exposeBin functionality and rebased the PR. Additionally, the following changes were made:

• The value of production is embedded in the output path, to alleviate the problem caused by forgetting to change the hash in this simple case so that it's guaranteed to cause a visible failure.
• package.json and package-lock.json are now injected into the output, so the consumer can ensure that the packages match the source it's building. As @roberth has pointed out, changes like this will silently break existing users, so we must ensure that the default behavior can satisfy most usecases and be expected to remain unchanged for the foreseeable future.

For the missing output problem, an issue should be created for Nix, once we figure out a minimal repro example.

I looked a bit more into the tarball reproducibility issue that @happysalada and @yu-re-ka mentioned, and it appears that the only differences come down to the permissions of the symlinks in the tar. For example:

│ │ -lrwxrwxrwx   0        0        0        0 1970-01-01 00:00:01.000000 node_modules/restify/node_modules/.bin/semver -> ../semver/bin/semver.js
│ │ +lrwxr-xr-x   0        0        0        0 1970-01-01 00:00:01.000000 node_modules/restify/node_modules/.bin/semver -> ../semver/bin/semver.js

(- is what I got, and + is @happysalada's tarball)

For reference, I'm using ZFS as my filesystem.

Thank you for your work on this!

I've tested again and here is the thing that works for me

{ nodePackages, fetchFromGitHub }: 

nodePackages.fetchNodeModules {
  name = "bgp-node-modules";
  src = fetchFromGitHub {
    owner = "nttgin";
    repo = "BGPalerter";
    rev = "v1.28.1";
    sha256 = "sha256-Y0atkJfZxjUOGPQ3goXS/YD5SsX9ZjpbM0Nc5IuaFP4=";
  };

  makeTarball = false;
  sha256 = "sha256-/x8XwHxIEYS3IxEuk1jp1wYyJIV11DdSXUU+DCkrQ1g=";
}

(just to confirm if we still have a different hash on different platforms)
(the closure size seems to be 30 mb, so not that much).

The good news is that the bug of the disappearing path after building is gone with this new update.

I think the main point is to figure out if the hash is still different on different platforms. If it is, what to do about it.

There are a couple of additional nits

• using mktemp instead of /tmp
• perhaps defaulting production to false ? I was thinking that most people wanting to use node_modules are going to use those to build something and so including dev modules seems the default to me.

@roberth I like the idea. People could generate their own node-packages to test when adding a package. When they are confident it works, they just commit the result of the expression, and then weekly the version update can be run. There might still be some hiccups, but it could be a short term solution. I think it would just be a matter of documenting the workflow properly.

@happysalada would you like to lead such an effort? The haskellPackages team has shown that a few committed members can have a great impact.
EDIT: Effort spent on testing will remain useful even after migrating to a fetching solution and having a focused team will ease the migration and improve maintenance afterwards.

@roberth thank you for your idea. I am a little at capacity right now, but I have added testing this approach on my task list for as soon as I get a bit more time.

I marked this as stale due to inactivity. → More info

Still important to me, can I do something to help @happysalada ?

What is your level of interest on this ?
It needs to be tested with several packages to see if the hash difference appears on different platforms for different packages. If it does, I don't think it's a problem, it's just a matter of making nix expressions that take into account the different hashes for different platform. (I think the vscode package has similar things).

if you are interested in doing 3 tests with js packages you are interested in, and reporting the results, that would be amazing.
After that, we can try to make a decision whether to merge or not. I'm personally in favour.

I've been using this downstream, but we should push this forward. I'll rebase the PR in a bit, and add the bgpalerter package in as well so it can be more easily tested.

What is your level of interest on this ? It needs to be tested with several packages to see if the hash difference appears on different platforms for different packages. If it does, I don't think it's a problem, it's just a matter of making nix expressions that take into account the different hashes for different platform. (I think the vscode package has similar things).

if you are interested in doing 3 tests with js packages you are interested in, and reporting the results, that would be amazing. After that, we can try to make a decision whether to merge or not. I'm personally in favour.

Between medium and high, I am not sure though I have different platforms alas (only x86_64-linux, but maybe I can ask for aarch64-linux community builder to test this platform), I can give it a try on some packages I know.

Hi all.

Over the past week or so, I've been working on a solution to this issue. (I actually didn't even know this PR was a thing before starting work on mine, until someone told me about it.)

It populates a cache directory to avoid the issue of different hashes occurring across platforms, and because of its superior reproducibility, is most likely the better solution to push forward.

There are more details in #189539 if you're interested in taking a look -- I welcome any and all comments/suggestions.

I haven't tested #189539 yet, but I think the approach there is less prone to breakages. Let's close this and focus our efforts on that.