Draft
Conversation
Co-authored-by: Eelco Dolstra <[email protected]> Co-authored-by: Robert Hensing <[email protected]>
This fixes the double copy problem and improves performance
for expressions that don't force the whole source to be added to the
store.
Rules for fast expressions:
- Use path literals where possible
- import ./foo.nix
- Use + operator with slash in string
- src = fetchTree foo + "/src";
- Use source filtering, lib.fileset
- AVOID toString
- If possible, AVOID interpolations ("${./.}")
- If possible, move slashes into the interpolation to add less to the store
- "${./src}/foo" -> "${./src/foo}"
toString may be improved later as part of lazy-trees, so these
recommendations are a snapshot. Path values are quite nice though.
This allows clever editors/IDEs to discern the path more easily for Ctrl+Click navigate to functionality, e.g. when building .?ref=HEAD
This showPath is getting a little too ad hoc, but it works for now.
A string is only allowed to create one path component; containing no slashes.
github-actions
bot
added
new-cli
This was referenced Aug 25, 2024
Contributor
PosixAccessor calls assertNoSymlinks too often?[ RUN ] PrimOpTest.unsafeGetAttrPos
unknown file: Failure
C++ exception with description "error:
… while calling the 'unsafeGetAttrPos' builtin
at «string»:1:1:
1| builtins.unsafeGetAttrPos "y" (import <nix/foo.nix>)
| ^
… while calling the 'import' builtin
at «string»:1:32:
1| builtins.unsafeGetAttrPos "y" (import <nix/foo.nix>)
| ^
(stack trace truncated; use '--show-trace' to show the full, detailed trace)
error: path '/home/tom/.nix-defexpr/channels' is a symlink" thrown in the test body.
[ FAILED ] PrimOpTest.unsafeGetAttrPos (0 ms)findFile prefix/suffix separator is ":" or "/"+(lang.sh:61) bash lang/eval-fail-derivation-name.postprocess lang/eval-fail-derivation-name
--- lang/eval-fail-derivation-name.err.exp 2024-08-25 17:27:55.317195410 -0400
+++ lang/eval-fail-derivation-name.err 2024-08-26 00:34:52.148460570 -0400
@@ -1,20 +1,20 @@
error:
… while evaluating the attribute 'outPath'
- at <nix/derivation-internal.nix>:<number>:<number>:
+ at <nix:derivation-internal.nix>:<number>:<number>:
<number>| value = commonAttrs // {
<number>| outPath = builtins.getAttr outputName strict;
| ^
<number>| drvPath = strict.drvPath;
… while calling the 'getAttr' builtin
- at <nix/derivation-internal.nix>:<number>:<number>:
+ at <nix:derivation-internal.nix>:<number>:<number>:
<number>| value = commonAttrs // {
<number>| outPath = builtins.getAttr outputName strict;
| ^
<number>| drvPath = strict.drvPath;
… while calling the 'derivationStrict' builtin
- at <nix/derivation-internal.nix>:<number>:<number>:
+ at <nix:derivation-internal.nix>:<number>:<number>:
<number>|
<number>| strict = derivationStrict drvAttrs;
| ^SourceAccessor is relative to root not to absolute path++(toString-path.sh:8) nix eval --raw --impure --expr 'builtins.readFile (builtins.toString (builtins.fetchTree { type = "path"; path = "/tmp/nix-shell.yXAp0S/nix-test/toString-path/foo"; } + "/bar"))'
error:
… while calling the 'readFile' builtin
at «string»:1:1:
1| builtins.readFile (builtins.toString (builtins.fetchTree { type = "path"; path = "/tmp/nix-shell.yXAp0S/nix-test/toString-path/foo"; } + "/bar"))
| ^
error: opening file '/bar': No such file or directory
+(toString-path.sh:8) [[ '' = bla ]]
++(toString-path.sh:8) onError
++(/home/tom/nix/lazy-paths/build/src/nix-functional-tests/common/functions.sh:237) set +x
toString-path.sh: test failed at:
main in toString-path.sh:8
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
117/183 nix-functional-tests:main / read-only-store Absolute path is used relative to root++(toString-path.sh:8) nix eval --raw --impure --expr 'builtins.readFile (builtins.toString (builtins.fetchTree { type = "path"; path = "/tmp/nix-shell.yXAp0S/nix-test/toString-path/foo"; } + "/bar"))'
error:
… while calling the 'readFile' builtin
at «string»:1:1:
1| builtins.readFile (builtins.toString (builtins.fetchTree { type = "path"; path = "/tmp/nix-shell.yXAp0S/nix-test/toString-path/foo"; } + "/bar"))
| ^
error: opening file '/bar': No such file or directory
+(toString-path.sh:8) [[ '' = bla ]]
++(toString-path.sh:8) onError
++(/home/tom/nix/lazy-paths/build/src/nix-functional-tests/common/functions.sh:237) set +x
toString-path.sh: test failed at:
main in toString-path.sh:8
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
117/183 nix-functional-tests:main / read-only-store |
Contributor
Unable to find input at specified path+(follow-paths.sh:80) nix flake metadata /tmp/nix-shell.yXAp0S/nix-test/flakes/follow-paths/follows/flakeA
warning: Git tree '/tmp/nix-shell.yXAp0S/nix-test/flakes/follow-paths/follows/flakeA' is dirty
error:
… while updating the lock file of flake 'git+file:///tmp/nix-shell.yXAp0S/nix-test/flakes/follow-paths/follows/flakeA'
… while updating the flake input 'B'
… while fetching the input 'path:./flakeB'
error: path 'flakeB' does not exist
++(follow-paths.sh:80) onError
++(/home/tom/nix/lazy-paths/build/src/nix-functional-tests/common/functions.sh:237) set +x
follow-paths.sh: test failed at:
main in follow-paths.sh:80 |
Member
Author
This is probably the right behavior for a SourceAccessor, but we shouldn't use a SourceAccessor directly, or not its low level methods or not for this purpose. |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
Contributor
|
Next steps:
|
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
|
This pull request has been mentioned on NixOS Discourse. There might be relevant details there: |
This was referenced Apr 5, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(This is #10252, but as an upstream branch for easier collaboration)
Motivation
Improve performance, and make the
fetchTreeinterface more capable while keeping it clean.Description
This makes
fetchTreereturn lazyInputAccessor-basedSourcePaths instead of "cowardly" fetching them to the store and returning absolute "system" paths.It stays close to existing path semantics, including support for
readFile "${toString p}/..", which some expressions rely on.It does not go as far as lazy-trees, but judging from the amount of change I could reuse, and how little of my own I had to add, lazy-trees will be a natural extension of this PR.
Done:
nixpkgs#hello, and 1s reduction onnixosTests.simpleConclusion so far:
Viable
TODO:
narHashin some observable placestoString pathbehavior to lazy-trees(fetchTree x).outPathflake.outPathContext
Priorities and Process
Add 👍 to pull requests you find important.
The Nix maintainer team uses a GitHub project board to schedule and track reviews.