pkgs/top-level/release-outpaths-parallel.nix 5x faster eval checks · Pull Request #269403 · NixOS/nixpkgs

ghost · 2023-11-23T09:35:48Z

This PR is implemented entirely in pure Nix code and a single call to xargs.

It has already been used to build the entire release packageset at larger-than-Hydra scale.
- This is a complete, working solution, not a component or a theoretical possibility.
It works with any Nix interpreter, and is not cppnix-specific like Hydra is.
- It does not need to link against any cppnix shared libraries.
- It does not use cppnix's internal interfaces.
It is in-tree as part of nixpkgs
- It does not add additional out-of-repo dependencies with their own gatekeeping committees.

Results

On a 32-core machine, the following command takes 4m58s:

NIX_BUILD_CORES=32 time pkgs/top-level/release-outpaths-parallel.sh false

and produces exactly the same output as this command (the one ofborg uses), which takes 24m10s:

time nix-env -qaP --no-name --out-path --arg checkMeta false -f pkgs/top-level/release-outpaths.nix | sort

... in 1/5th of the wall-clock time. So the eval-check latency is reduced by 5x. With checkMeta=true the ratio is almost identical: 37m54s with the current method, 7m38s with release-outpaths-parallel.

There is room to improve even further:

Half of the 4m58s is spent in the (completely serial) release-attrpaths-superset.nix process. However that process produces output incrementally, meaning that the parallel part of the eval-checks can be started as soon as the first line of release-attrpaths-superset.nix output is produced -- this should give another 25% improvement.
Right now the 4*NUM_CORES chunks are scheduled blindly, by simply taking alphabetical chunks of job attrpaths. This results in a huge amount of duplicated work. Simple dependency-based scheduling (possibly using the previous commit's attrpath dependency graph as an approximation of the current one) should give another large boost, possibly even 50%.

Includes

Ensure all jobset attributes evaluate, and check that in CT #269356

Artturin · 2023-11-23T16:06:22Z

pkgs/top-level/release-outpaths-parallel.sh

Did you try nix-eval-jobs, it does parallel but doesn't help with memory usage AFAIK

It helps with memory usage as well because it restarts workers if they take too much memory. The chunking at the moment might be less ideal because it doesn't chunk the list into X parts and gives this to threads but instead starts to iterate the list from the beginning and give each thread an attribute in lexical key order. I think it would be faster if attributes closer to each other would be evaluated by the same thread to avoid evaluating the same thing twice (i.e. x86_64-linux packages evaluated on thread1 and aarch64-linux packages on thread2).

However from my tests I would say that having more threads than CPUs will actually slow you down because you end up evaluating the same thing more. I usually have less threads than cores for large package sets like nixpkgs to avoid duplicating work and rather trait it for memory.

cole-h · 2023-11-27T13:48:53Z

(Sorry, GitHub mobile fat finger...... Love how there's no confirmation...)

adisbladis · 2023-11-28T03:01:36Z

I would like to note that using this in ofborg would mean giving up the performance report, at least as it is today.

adisbladis · 2023-11-28T03:02:44Z

Conceptually this PR works similarly to how nix-eval-jobs works, just implemented as a pile of shell/nix instead of using the evaluator directly.
I think it's better to make ofborg use nix-eval-jobs.

cc NixOS/ofborg#440

…tributes

This commit vendors `outpaths.nix` from ofborg commit 74f38efa7ef6f0e8e71ec3bfc675ae4fb57d7491

Partly taken from NixOS#352808 and NixOS#269403

nixos-discourse · 2024-11-17T08:19:54Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025/37

nixos-discourse · 2024-11-17T08:52:26Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025/39

Motivated by ofborg struggling [1] and its evaluations taking too long, inspired by Jörg's initial PR [2] and Adam's previous attempt to parallelise Nixpkgs evaluation [3], this PR contains initial work to relief ofborg from its evaluation duty by using GitHub Actions to evaluate Nixpkgs. For now this doesn't take care of all of what ofborg does, such as requesting appropriate reviewers or labeling mass rebuilds, but this can be follow-up work. [1]: https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025?u=infinisil [2]: NixOS#352808 [3]: NixOS#269403 Co-Authored-By: Jörg Thalheim <[email protected]> Co-Authored-By: Adam Joseph <[email protected]>

* [WIP] Parallel GH actions workflow for Nixpkgs eval Partly taken from NixOS#352808 and NixOS#269403 * ci/parallel.nix: nixfmt * ci/eval: drop swap Turns out we don't actually need it * ci/eval: reformat with shfmt the file was using mixed indentation (2 vs 4 spaces) * ci/eval: depend on coreutils nproc is only part of coreutils and not present on macOS * ci/eval: allow to override cores this is useful to control how much RAM is used during evaluation * ci/eval: use xargs instead of parallel saves us from having to install perl for this script. * ci/eval: add quick test flag * ci/eval: seperate different outputs in json * WIP * Update README.md --------- Co-authored-by: Silvan Mosberger <[email protected]> Co-authored-by: Jörg Thalheim <[email protected]> Co-authored-by: Silvan Mosberger <[email protected]>

Motivated by ofborg struggling [1] and its evaluations taking too long, inspired by Jörg's initial PR [2] and Adam's previous attempt to parallelise Nixpkgs evaluation [3], this PR contains initial work to relief ofborg from its evaluation duty by using GitHub Actions to evaluate Nixpkgs. For now this doesn't take care of all of what ofborg does, such as requesting appropriate reviewers or labeling mass rebuilds, but this can be follow-up work. [1]: https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025?u=infinisil [2]: NixOS#352808 [3]: NixOS#269403 Co-Authored-By: Jörg Thalheim <[email protected]> Co-Authored-By: Adam Joseph <[email protected]>

Motivated by ofborg struggling [1] and its evaluations taking too long, inspired by Jörg's initial PR [2] and Adam's previous attempt to parallelise Nixpkgs evaluation [3], this PR contains initial work to relief ofborg from its evaluation duty by using GitHub Actions to evaluate Nixpkgs. For now this doesn't take care of all of what ofborg does, such as requesting appropriate reviewers or labeling mass rebuilds, but this can be follow-up work. [1]: https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025?u=infinisil [2]: #352808 [3]: #269403 Co-Authored-By: Jörg Thalheim <[email protected]> Co-Authored-By: Adam Joseph <[email protected]> (cherry picked from commit fbbe972)

Motivated by ofborg struggling [1] and its evaluations taking too long, inspired by Jörg's initial PR [2] and Adam's previous attempt to parallelise Nixpkgs evaluation [3], this PR contains initial work to relief ofborg from its evaluation duty by using GitHub Actions to evaluate Nixpkgs. For now this doesn't take care of all of what ofborg does, such as requesting appropriate reviewers or labeling mass rebuilds, but this can be follow-up work. [1]: https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025?u=infinisil [2]: NixOS#352808 [3]: NixOS#269403 Co-Authored-By: Jörg Thalheim <[email protected]> Co-Authored-By: Adam Joseph <[email protected]>

Motivated by ofborg struggling [1] and its evaluations taking too long, inspired by Jörg's initial PR [2] and Adam's previous attempt to parallelise Nixpkgs evaluation [3], this PR contains initial work to relief ofborg from its evaluation duty by using GitHub Actions to evaluate Nixpkgs. For now this doesn't take care of all of what ofborg does, such as requesting appropriate reviewers or labeling mass rebuilds, but this can be follow-up work. [1]: https://discourse.nixos.org/t/infrastructure-announcement-the-future-of-ofborg-your-help-needed/56025?u=infinisil [2]: NixOS/nixpkgs#352808 [3]: NixOS/nixpkgs#269403 Co-Authored-By: Jörg Thalheim <[email protected]> Co-Authored-By: Adam Joseph <[email protected]>

ghost mentioned this pull request Nov 23, 2023

Ensure all jobset attributes evaluate, and check that in CT #269356

Merged

ghost changed the title ~~pkgs/top-level/release-outpaths-parallel.{nix,sh}: init 5x faster eval checks~~ pkgs/top-level/release-outpaths-parallel.{nix,sh}: 5x faster eval checks Nov 23, 2023

Artturin reviewed Nov 23, 2023

View reviewed changes

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Nov 23, 2023

NixOS locked and limited conversation to collaborators Nov 27, 2023

NixOS unlocked this conversation Nov 27, 2023

github-actions bot removed 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 6.topic: steam Steam game store/launcher (store.steampowered.com) labels Nov 30, 2023

ghost mentioned this pull request Nov 30, 2023

[DONTMERGE] stabilization branch for gcc13 upgrade #271101

Closed

Adam Joseph added 6 commits November 30, 2023 17:30

test.cuda: fix broken eval

8f99d0a

pkgs/top-level/all-packages.nix: remove self-referential top-level at…

2ce6305

…tributes

pkgs/top-level/release-outpaths.nix: vendor from ofborg

8e0624f

This commit vendors `outpaths.nix` from ofborg commit 74f38efa7ef6f0e8e71ec3bfc675ae4fb57d7491

pkgs/top-level/release-outpaths.nix: adjust default path value

f73063f

pkgs/top-level/release-outpaths.nix: add attrNamesOnly option

90d2150

pkgs/top-level/release-outpaths.nix: simplify invocation

13f65d2

infinisil added a commit to tweag/nixpkgs that referenced this pull request Nov 14, 2024

[WIP] Parallel GH actions workflow for Nixpkgs eval

4874faf

Partly taken from NixOS#352808 and NixOS#269403

infinisil mentioned this pull request Nov 14, 2024

Parallel GH actions workflow for Nixpkgs eval #356023

Merged

2 tasks

This pull request was closed.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

pkgs/top-level/release-outpaths-parallel.nix 5x faster eval checks#269403

pkgs/top-level/release-outpaths-parallel.nix 5x faster eval checks#269403
ghost wants to merge 26 commits intomasterfrom
unknown repository

ghost commented Nov 23, 2023 •

edited by ghost

Loading

Uh oh!

Artturin Nov 23, 2023

Uh oh!

Mic92 Dec 3, 2023

Uh oh!

Mic92 Dec 3, 2023

Uh oh!

cole-h commented Nov 27, 2023

Uh oh!

adisbladis commented Nov 28, 2023

Uh oh!

adisbladis commented Nov 28, 2023

Uh oh!

nixos-discourse commented Nov 17, 2024

Uh oh!

nixos-discourse commented Nov 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Uh oh!

Conversation

ghost commented Nov 23, 2023 • edited by ghost Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Results

Uh oh!

Artturin Nov 23, 2023

Choose a reason for hiding this comment

Uh oh!

Mic92 Dec 3, 2023

Choose a reason for hiding this comment

Uh oh!

Mic92 Dec 3, 2023

Choose a reason for hiding this comment

Uh oh!

cole-h commented Nov 27, 2023

Uh oh!

adisbladis commented Nov 28, 2023

Uh oh!

adisbladis commented Nov 28, 2023

Uh oh!

nixos-discourse commented Nov 17, 2024

Uh oh!

nixos-discourse commented Nov 17, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

ghost commented Nov 23, 2023 •

edited by ghost

Loading