`lib.fileset.trace`, `lib.fileset.traceVal`: init by infinisil · Pull Request #256417 · NixOS/nixpkgs

infinisil · 2023-09-20T23:58:45Z

Description of changes

Another split-off from the main file set combinators PR. This one adds two functions for debugging file sets:

lib.fileset.trace: Trace a file set and return a value

trace (unions [ ./Makefile ./src ./tests/run.sh ]) null

lib.fileset.traceVal: Same, but it returns the file set itself

toSource {
  root = ./.;
  fileset = traceVal (unions [
    ./Makefile
    ./src
    ./tests/run.sh
  ]);
}

These both print the file set like this:

trace: /home/user/src/myProject
trace: - Makefile (regular)
trace: - src (all files in directory)
trace: - tests
trace:   - run.sh (regular)

This work is sponsored by Antithesis ✨

Things done

Implementation
- Tested
- Commented
Added user documentation

lib/fileset/internal.nix

infinisil · 2023-09-27T20:02:30Z

Removed the seq foldl' hack now that #256544 is merged.

fricklerhandwerk

The weird thing is that the stack overflow avoidance tests fail on my NixOS 23.05 with Nix 2.13.3.

test case at lib/fileset/tests.sh:523 failed: toSource { root = ./.; fileset = unions (
mapAttrsToList (name: _: ./. + "/${name}/a") (builtins.readDir ./.)); } failed to evalu
ate, but it was expected to succeed

lib/fileset/tests.sh: line 124: 18590 Segmentation fault      (core dumped) nix-instant
iate --eval --show-trace --expr "$prefixExpression (trace $config $expr)" > /dev/null 2
> "$tmp"/stderrTrace
lib/fileset/tests.sh: line 124: 18596 Segmentation fault      (core dumped) nix-instant
iate --eval --show-trace --expr "$prefixExpression (trace $config $expr)" > /dev/null 2
> "$tmp"/stderrTraceVal
test case at lib/fileset/tests.sh:635 failed: ./. should have traced this:

/run/user/1000/tmp.Ue0Y2xun3k/work (all files in directory)

but this was actually traced:

lib/fileset/tests.sh

lib/fileset/default.nix

lib/fileset/internal.nix

infinisil · 2023-09-28T01:10:53Z

lib/fileset/tests.sh

@fricklerhandwerk (using this thread to reply to #256417 (review))

The weird thing is that the stack overflow avoidance tests fail on my NixOS 23.05 with Nix 2.13.3.

Interesting, does it happen both with nix-build lib/tests/release.nix and lib/fileset/tests.sh?

I tried reproducing this with

diff --git a/lib/tests/release.nix b/lib/tests/release.nix index c8d6b810122e..02067a79f9a6 100644 --- a/lib/tests/release.nix +++ b/lib/tests/release.nix @@ -2,7 +2,12 @@ # Don't test properties of pkgs.lib, but rather the lib in the parent directory pkgs ? import ../.. {} // { lib = throw "pkgs.lib accessed, but the lib tests should use nixpkgs' lib path directly!"; }, nix ? pkgs.nix, - nixVersions ? [ pkgs.nixVersions.minimum nix pkgs.nixVersions.unstable ], + nixVersions ? [ + pkgs.nixVersions.minimum + pkgs.nixVersions.nix_2_13 + nix + pkgs.nixVersions.unstable + ], }: let

and nix-build lib/tests/release.nix -A paths.1, but no luck. I also tried downgrading from the current 2.13.5 to 2.13.3 by reverting bcb02bd, but that also didn't cause it to break.

I can kind of see how it could happen though: It looks like Nix segfaults if there's too little stack space, though for me that only happens for <44*1024 bytes:

$ ulimit -s 44 $ nix-env --version nix-env (Nix) 2.15.1 $ ulimit -s 43 $ nix-env --version zsh: segmentation fault (core dumped) nix-env --version

Could you test whether you can run the tests on master? Because there's already another test doing this too:

nixpkgs/lib/fileset/tests.sh

Lines 471 to 473 in efb617b

# Locally limit the maximum stack size to 100 * 1024 bytes

# If unions was implemented recursively, this would stack overflow

ulimit -s 100

@roberth Testing the stack like this might be machine dependent I guess, might be better to drop it

@infinisil it doesn't fail with or without ulimit when using nix-build, but does the same as previously when running test script directly. Note though that the output store path when using nix-build is /nix/store/7gf7wnp28h77gl5y1k2pw3n2pmxg6b57-nixpkgs-lib-tests-nix-2.17.0.

infinisil · 2023-09-29T19:22:31Z

I was able to not make the new test here rely on ulimit -s, I also created #258080 to remove the other use of ulimit -s.

lib/fileset/default.nix

lib/fileset/tests.sh

lib/fileset/default.nix

roberth · 2023-10-03T16:27:44Z

Probably needs adaptation for

lib.fileset: Representation for empty file sets without a base path #257351

infinisil · 2023-10-03T19:42:24Z

Rebased to fix conflicts and removed the {} argument from trace/traceVal as suggested. This even makes this more consistent with builtins.trace and lib.debug.traceVal, as those also can't be customised.

Here's the easily-viewable diff of the changes: https://github.com/tweag/nixpkgs/compare/55c1b0b462d28d404f78695e7d8d8885f81fe080..53c053be347989bff8c08769f08561ced200b21f

roberth · 2023-10-04T09:35:15Z

Thanks for the diff link! That helps a lot.

I don't see a test case for trace (unions [ ]). Could you add it?

infinisil · 2023-10-04T14:34:50Z

@roberth Oh good catch, I didn't think of the interaction after #257351, it indeed didn't work at all and needed to be fixed! That's done now, diff here (this one GitHub also links to)

infinisil requested review from fricklerhandwerk and roberth September 20, 2023 23:58

github-actions bot added 8.has: documentation This PR adds or changes documentation 6.topic: lib The Nixpkgs function library labels Sep 20, 2023

infinisil mentioned this pull request Sep 20, 2023

[WIP] File set combinators #222981

Closed

7 tasks

infinisil force-pushed the fileset.trace branch from adee15d to 174051a Compare September 21, 2023 00:10

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Sep 21, 2023

infinisil force-pushed the fileset.trace branch from 174051a to ef36ede Compare September 21, 2023 11:33

github-actions bot removed the 8.has: documentation This PR adds or changes documentation label Sep 21, 2023

infinisil marked this pull request as ready for review September 21, 2023 11:33

roberth reviewed Sep 21, 2023

View reviewed changes

lib/fileset/internal.nix Outdated Show resolved Hide resolved

infinisil force-pushed the fileset.trace branch 2 times, most recently from 08c75df to 0fc1619 Compare September 27, 2023 19:53

fricklerhandwerk reviewed Sep 27, 2023

View reviewed changes

infinisil commented Sep 28, 2023

View reviewed changes

infinisil force-pushed the fileset.trace branch from 0fc1619 to dbc0a5d Compare September 29, 2023 15:48

infinisil mentioned this pull request Sep 29, 2023

lib.fileset: Don't use non-reproducible ulimit for stack overflow testing #258080

Merged

1 task

infinisil force-pushed the fileset.trace branch from d642d82 to dbc0a5d Compare September 29, 2023 18:36

infinisil force-pushed the fileset.trace branch 3 times, most recently from 941dfb3 to 37a5ca7 Compare October 2, 2023 20:55

infinisil requested review from fricklerhandwerk and roberth October 2, 2023 20:56

fricklerhandwerk approved these changes Oct 2, 2023

View reviewed changes

delroth added the 12.approvals: 1 This PR was reviewed and approved by one person. label Oct 2, 2023

roberth reviewed Oct 3, 2023

View reviewed changes

lib/fileset/default.nix Outdated Show resolved Hide resolved

lib/fileset/tests.sh Outdated Show resolved Hide resolved

roberth reviewed Oct 3, 2023

View reviewed changes

lib/fileset/default.nix Outdated Show resolved Hide resolved

infinisil added 2 commits October 3, 2023 21:27

lib.fileset: Order noEval last

b3c2281

lib.fileset: _simplifyTree -> _normaliseTreeFilter

86802e1

infinisil force-pushed the fileset.trace branch from 37a5ca7 to 53c053b Compare October 3, 2023 19:38

infinisil requested review from fricklerhandwerk and roberth October 3, 2023 23:24

infinisil added 7 commits October 4, 2023 16:28

lib.fileset: Add internal helpers for pretty-printing

efbcf59

lib.fileset: Make expectEqual check more

ac2c8d3

lib.fileset.trace: init

467e428

lib.fileset.traceVal: init

704452f

lib.fileset: Mention trace functions in the file set evaluation error

5bbe67b

lib.fileset: Refactor inotify testing to be reusable

692b752

lib.fileset: Don't use ulimit for testing tracing

5b4e53a

infinisil force-pushed the fileset.trace branch from 53c053b to 5b4e53a Compare October 4, 2023 14:32

roberth approved these changes Oct 4, 2023

View reviewed changes

infinisil merged commit 5db719f into NixOS:master Oct 4, 2023

infinisil deleted the fileset.trace branch October 4, 2023 15:39

infinisil mentioned this pull request Nov 9, 2023

File set library tracking issue and feature requests #266356

Open

	# Locally limit the maximum stack size to 100 * 1024 bytes
	# If unions was implemented recursively, this would stack overflow
	ulimit -s 100

Uh oh!

Comments

Conversation

infinisil commented Sep 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Things done

Uh oh!

Uh oh!

infinisil commented Sep 27, 2023

Uh oh!

fricklerhandwerk left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

infinisil Sep 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fricklerhandwerk Oct 2, 2023

Choose a reason for hiding this comment

Uh oh!

infinisil commented Sep 29, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roberth commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

infinisil commented Oct 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

roberth commented Oct 4, 2023

Uh oh!

infinisil commented Oct 4, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

infinisil commented Sep 20, 2023 •

edited

Loading

infinisil Sep 28, 2023 •

edited

Loading

roberth commented Oct 3, 2023 •

edited

Loading

infinisil commented Oct 3, 2023 •

edited

Loading