This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/future-of-npm-packages-in-nixpkgs/14285/18

I have to admit, I'm not really sure if we want yet another FOD-based fetcher (which is essentially a big blackbox IMHO) in nixpkgs. For reference, see NixOS/nix#2270

(Please note that there isn't a final decision, I generally support this - and try to only use FOD for dependency fetching if there's no way around it - but there are a lot of people against this).

I didn't include the --ignore-scripts flag, as I'm still weighing pros and cons about it, happy to get more informations regarding it.

The problem is that it has sometimes scripts that download random stuff from the internet (e.g. electron IIRC) which would make the fixed-output hash unstable.

Result of nixpkgs-review pr 138233 at 85e36f3 run on x86_64-linux 1

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

Result of nixpkgs-review pr 138233 at 85e36f3 run on aarch64-linux 1

Note that build failures may predate this PR, and could be nondeterministic or hardware dependent.
Please exercise your independent judgement. Does something look off? Please file an issue or reach out on IRC.

I have to admit, I'm not really sure if we want yet another FOD-based fetcher (which is essentially a big blackbox IMHO) in nixpkgs. For reference, see NixOS/nix#2270

(Please note that there isn't a final decision, I generally support this - and try to only use FOD for dependency fetching if there's no way around it - but there are a lot of people against this).

I didn't include the --ignore-scripts flag, as I'm still weighing pros and cons about it, happy to get more informations regarding it.

The problem is that it has sometimes scripts that download random stuff from the internet (e.g. electron IIRC) which would make the fixed-output hash unstable.

To be honest for me a reviewing a 100K lines of node-packages.nix is the bigger blackbox. With FOD I can be sure at least the pull request author did not temper with the source after running the update script.

I didn't say that, my preferred solution is to read from an existing lockfile and re-use these results (such as at least import-cargo does it).

I think we all agree that npmlock2nix is a better solution with a lock file.

The idea here would be to use something like this until npmlock2nix gets merged into nixpkgs. It could be another 3 months or more until we have a better solution.

I still have to add the ignore-scripts flag on that PR and verify that the hash does not change accross platforms. (if it does it might make usage more problematic). At the moment the current ci failure is because of a hash mismatch on linux vs darwin.

I'm also thinking about opening an RFC to have a discussion more in the open and see what people would think. (the only reason I haven't done it yet, is that it seems to me that the discussion is only valid for 3 months, so the RFC might not be over before we have another solution merged. That said, npmlock2nix is only a solution for a package.lock (for now at least), so it might be good to open an RFC to discuss the alternatives for yarn. It might be good also to discuss an alternative if npm changes in such a way that npmlock2nix is not usable anymore. (which is what I think happened to the previous iteration called npm2nix))

That said, npmlock2nix is only a solution for a package.lock (for now at least), so it might be good to open an RFC to discuss the alternatives for yarn

I think that this shouldn't be Yarn-specific, but a more general decision about how we want to package language-specific stuff. I'm btw sure that there a a few more discussions about this that you may want to read before opening such an RFC, but in general I think that this is a great idea 👍 .

The idea here would be to use something like this until npmlock2nix gets merged into nixpkgs. It could be another 3 months or more until we have a better solution.

Well, as I said, using the lock-data from language-specific package managers in general is IMHO the ideal solution for the problem we have, so I have to admit I don't really understand your concern here. I don't think we should rush now with yet another fetcher (we have 2 yarn2nix implementations, npmlock2nix, node2nix and pnpm2nix and a similar amount of fetchers for Rust), but focus on getting a somewhat standardized approach that's applicable to different language-ecosystems.

In case of Yarn, we'll also have the question how to deal with the lockfile, though (in contrast to cargo using TOML, NPM using JSON and Go using something that doesn't seem too hard to parse in contrast to a yarn.lock), perhaps that's what you had in mind while talking about Yarn fetchers above. IIRC Yarn doesn't intend to change the format, so we'll either need to parse it in Nix (though that'll be very painful, but people have written parsers already such as https://github.com/mozilla/nixpkgs-mozilla/blob/master/lib/parseTOML.nix), try to get it in Nix as a builtin (don't think that's fairly likely) or discuss how to deal with FOD in such an RFC (cross-ref NixOS/nix#5253).

Please note that especially what I mentioned in the paragraph above is just some brainstorming and not completely thought through, but I thought it's still worth to share this here :)

After my change to add the --ignore-scripts, it seems there is still a hash mismatch between the different platforms. The hash on the same platform this was built (darwin-x86_64) does not even match, it doesn't bode well for this approach.

@Ma27 my approach here was to try to get something quickly that could be potentially used in case approaches with parsing the lock files do not work. The potential problems with approach parsing the lock file is that the time this lands in nixpkgs is still incertain and there are still cases where this is known to be impossible. In some cases, for some reason yarn doesn't even respect it's own format and doesn't include the integrity hashes. One such examle is the peertube PR attracting so much attention recently. Originally it included a patch to yarn.lock to add the missing integrity hashes. I've tried to re-generate the lock-file locally and there where no ways to get those hashes to show up.

I'm a little afraid that the parsing the lock file approach might take longer than we think, and up until now there is only a solution for package-lock.json. (npmlock2nix, the stronger contender in the race doesn't do yarn.lock as of now).

There is an attempt at the moment to parse several lock file formats, I'm going to see what I can do to help there.

I forgot to say.
I'm closing this for now as I don't have ideas as to how to fix the hash mismatch on the same platform as it was built on.
I'm keeping the branch around however in case there are new ideas or for reference for others.