Skip to content

llvmPackages.bolt: add experimental bolt hook#375435

Open
RossComputerGuy wants to merge 4 commits intoNixOS:stagingfrom
RossComputerGuy:feat/llvm-bolt-hook
Open

llvmPackages.bolt: add experimental bolt hook#375435
RossComputerGuy wants to merge 4 commits intoNixOS:stagingfrom
RossComputerGuy:feat/llvm-bolt-hook

Conversation

@RossComputerGuy
Copy link
Member

Things done

Experimental concept, this adds a hook which calls bolt on the installed binaries. The idea is this will optimize the installed binaries. We'll test with just hello however all packages can be tried by enabling useBolt when specifying the cross platform.

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added 6.topic: stdenv Standard environment 6.topic: lib The Nixpkgs function library 6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related labels Jan 21, 2025
@RossComputerGuy RossComputerGuy force-pushed the feat/llvm-bolt-hook branch 2 times, most recently from 1b7ad73 to 27d0f64 Compare January 21, 2025 06:14
@github-actions github-actions bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Jan 21, 2025
@RossComputerGuy RossComputerGuy force-pushed the feat/llvm-bolt-hook branch 2 times, most recently from fd1b9b5 to 6554b12 Compare January 21, 2025 06:37
@github-actions github-actions bot added 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. and removed 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 11-100 This PR causes between 11 and 100 packages to rebuild on Linux. labels Jan 21, 2025
@RossComputerGuy RossComputerGuy force-pushed the feat/llvm-bolt-hook branch 2 times, most recently from 0ba380f to e1a994c Compare January 21, 2025 07:19
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we really need this? I'm not sure every possible thing you can set on a platform needs its own package set, and I especially don't think we need Hydra to build every single-change stdenv.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can drop it before merging but for the purposes of testing this PR, I added it in.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this sort of thing is useful to have. For what it's worth. I don't think its existence needs to promise that it will work, much the same as for pkgsStatic and friends as well. It is quite powerful to have a simple name for this thing to reach for from the shell.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that these things also make it way too easy to use them within Nixpkgs, and then we end up with a layering violation where a single package is going through a whole distinct package set out of convenience.

But that's a general problem that needs to be solved, by making these inaccessible from within Nixpkgs while still being a convenience layer on top, so if this is really useful it can stay. I think there's no justification for building this set on Hydra, though.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that these things also make it way too easy to use them within Nixpkgs, and then we end up with a layering violation where a single package is going through a whole distinct package set out of convenience.

We can add in a config option similar to the aliases one which enables the subsets. Then have it disabled in Hydra / CI.

But that's a general problem that needs to be solved, by making these inaccessible from within Nixpkgs while still being a convenience layer on top, so if this is really useful it can stay. I think there's no justification for building this set on Hydra, though.

I kinda agree but its also good to ensure things work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that these things also make it way too easy to use them within Nixpkgs, and then we end up with a layering violation where a single package is going through a whole distinct package set out of convenience.

We can add in a config option similar to the aliases one which enables the subsets. Then have it disabled in Hydra / CI.

Oh that's not a bad idea!

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is done in #400351

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unresolved this thread, because "We can drop it before merging" should still be taken care of eventually. (Or via the variants stuff, but not the current form)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought it was resolved because it sounded like Alyssa agreed to the variants overlay.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't really like adding new ones, but not enough to really object if it can't be used within Nixpkgs.

@RossComputerGuy
Copy link
Member Author

hello with gcc - 77kb binary size:

0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 1584maxresident)k
4520inputs+0outputs (12major+81minor)pagefaults 0swaps

hello with llvm - 49kb binary size:

0.00user 0.00system 0:00.00elapsed 0%CPU (0avgtext+0avgdata 2640maxresident)k
0inputs+0outputs (0major+95minor)pagefaults 0swaps

hello with llvm + bolt - 4MB binary size:

0.00user 0.00system 0:00.00elapsed 16%CPU (0avgtext+0avgdata 1568maxresident)k
5192inputs+0outputs (19major+91minor)pagefaults 0swaps

@RossComputerGuy RossComputerGuy marked this pull request as ready for review January 22, 2025 05:32
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use replaceVars instead, see #395275.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unresolved this thread, because "We can drop it before merging" should still be taken care of eventually. (Or via the variants stuff, but not the current form)

@ofborg ofborg bot added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 24, 2025
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 24, 2025
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't mkDerivation do this?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Idk, I couldn't get the hook to appear for whatever reason.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if there's a more precise verb than "use" that would work better here…

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure

@github-project-automation github-project-automation bot moved this to In Progress in Stdenv Apr 26, 2025
@RossComputerGuy RossComputerGuy changed the base branch from master to staging April 27, 2025 04:58
@ofborg ofborg bot added 2.status: merge conflict This PR has merge conflicts with the target branch and removed 2.status: merge conflict This PR has merge conflicts with the target branch labels Apr 27, 2025
@github-actions github-actions bot added 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. and removed 10.rebuild-darwin: 1-10 This PR causes between 1 and 10 packages to rebuild on Darwin. 10.rebuild-linux: 1-10 This PR causes between 1 and 10 packages to rebuild on Linux. labels Apr 27, 2025
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Jun 9, 2025
@nixpkgs-ci nixpkgs-ci bot added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Oct 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 6.topic: lib The Nixpkgs function library 6.topic: llvm/clang Issues related to llvmPackages, clangStdenv and related 6.topic: stdenv Standard environment 10.rebuild-darwin: 11-100 This PR causes between 11 and 100 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 5001+ This PR causes many rebuilds on Linux and must target the staging branches.

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

5 participants