@jbedo @SomeoneSerge @GaetanLepage

This could replace singularity-tools.mkLayer and prevent unnecessary waste of space. Could you guys help take a look?

Just add the test cases anyway. Let's see how OfBorg would run them.

Update: trivial-builders.nix tests are not library tests and OfBorg won't run them.

Update: Let's prevent the already-not-working test suite from growing more complex.

@ofborg test references

Awesome. I'll just note that I think bash is on its way of becoming a burden here. I would also really like to run some simple benchmarks because I don't have a very concrete understanding of how much of an issue this mkLayer is right now, and how much this change improves it, but slow as I am I wouldn't insist that you wait for my approval on merging

It's still short and understandable, so I think a shift away from bash would have more downsides than advantages.

I would also really like to run some simple benchmarks because I don't have a very concrete understanding of how much of an issue this mkLayer is right now, and how much this change improves it

I haven't done a serious benchmark yet. The reason why it helps is:

• mkLayer produces a package by copying all the files in the output path of packages in contents into its own output path.
• What buildImage needs is only a list of store paths of contents and their run-time dependencies (made with writeReferencesToFile), making the files mkLayer copies useless.
• writeMultipleReferencesToFile simply dump the list of store paths to its output file, preventing the redundant copying.

When dealing with large packages like root (CERN ROOT, a C++ data analysis framework), the use of mkLayer doubles the image build time and the disk space usage.

A temporary store would be neat!

rather than a shell to prime you can just set jobs to 0

I don't understand this part.

The purpose of nix-shell -A hello --run exit is to build all the dependencies before the benchmark. (I originally use sh -c ":" since I wasn't sure if it would be okay not to begin with an executable.)

In Nix, we have a default relation we refer to when talking about "the" closure, which is the relation of references, so I think the name writeClosure is fine.

I suppose "References" is a more explicit reference (jaja) to the the GC and dependency detection: you "writeReferencesToFile" to retain a runtime dependency on something you've seen at build time

For this use case, taking the direct references is sufficient. The GC knows that the whole closure is reachable from that starting set.
"Direct" is implied when talking about references, but I've had to use that term when I added writeDirectReferencesToFile, to make it distinct from the somewhat badly named writeReferencesToFile.

Just fixed a commit message. Sorry for the noise.

Does writeMultupleReferencesToFile not duplicate e.g. closureInfo?

@SomeoneSerge, sorry that I made it wrong. "${pkgs.writeClosure paths}" is a duplication of "${pkgs.closureInfo paths}".

I didn't realize that the whole closure of pkgs.linux is only itself.

"${pkgs.writeClosure paths}" is a duplication of "${pkgs.closureInfo paths}".

I thought we concluded someplace else that writeClosure had a different signature?

I thought we concluded somewhere else that writeClosure had a different signature

They have different signatures. pkgs.closureInfo is not a drop-in replacement of pkgs.writeClosure. As for their functions, one can use pkgs.closureInfo to achieve a functionality similar to that of pkgs.writeClosure.

I said they are "not even close" (#178717 (comment)) based on the content of the file "${pkgs.closureInfo { storePaths = [ pkgs.linux ]; }/store-paths". It contains only one line

nixpkgs_timeshift (9b54fb4) [$]
❯ cat /nix/store/sd8vsqzxhq9x4hyz3fvvggss0i4wkqvi-closure-info/store-paths
/nix/store/lgkpyy1b1zzh04n8b6andg8iw8vnr07m-linux-6.6.21

I thought the closure of pkgs.linux would be much bigger than that, but now I know I was wrong.