llvmPackages_13: build from filtered monorepoSrc#347887
Merged
sternenseemann merged 1 commit intoNixOS:masterfrom Oct 14, 2024
Merged
llvmPackages_13: build from filtered monorepoSrc#347887sternenseemann merged 1 commit intoNixOS:masterfrom
sternenseemann merged 1 commit intoNixOS:masterfrom
Conversation
github-actions
bot
added
the
6.topic: llvm/clang
sternenseemann
force-pushed
the
llvm-13-monorepo-src
branch
2 times, most recently
from
70dd0be to
7df8f7c
Compare
ofborg
bot
requested review from
Ericson2314,
RossComputerGuy,
alyssais,
dtzWill,
lovek323 and
rrbutani
ofborg
bot
added
11.by: package-maintainer
rrbutani
approved these changes
Oct 11, 2024
Contributor
rrbutani
left a comment
rrbutani
left a comment
There was a problem hiding this comment.
looks good, thanks for doing this!
Contributor
There was a problem hiding this comment.
is this TODO about merging the cmake and third-party optionals when you make your src/fetchurl changes (since we'll be eating a rebuild then anyways)? 🙂
Member
Author
There was a problem hiding this comment.
Yes, exactly. I'll bundle this with some complete LLVM rebuild. This change could just be merged into master.
This change implements a leftover task from NixOS#307211, namely passing monorepoSrc to the different llvmPackages_13 package expressions. Before this change, all packages llvmPackages_13 would be built from a subdirectory of the full LLVM monorepo tree. After this change only the relevant directories are made available at build time. This - reduces the size of the source that needs to be made available to the builder. - prevents LLVM from sidestepping our instructions and including extra sources from other directories it shouldn't. Since LLVM 12 and 13 don't have the `cmake` directory at the top level, the runCommand expressions filtering the source need to be adjusted, but this causes no rebuild for any other LLVM version (ofborg should confirm this). The only problem encountered was in lld: - We need to make the patch to the inclusion of libunwind headers unconditional now. lld needs this on non-darwin as well. In the full monorepo, LLVM_MAIN_SRC_DIR would be set correctly, so the patch wasn't necessary. - The substitute mechanism for LLVM 12 and 13 can't be unified yet since LLVM 12 still uses a non monorepo build, so we come up with a different LLVM_MAIN_SRC_DIR. Change was tested by building the following expression on x86_64-linux. with import ./. {}; builtins.removeAttrs llvmPackages_13 [ "lldb" "lldbPlugins" ]' lld was also tested on aarch64-darwin.
sternenseemann
force-pushed
the
llvm-13-monorepo-src
branch
from
7df8f7c to
db57787
Compare
rrbutani
added
12.approvals: 1
rrbutani
approved these changes
Oct 14, 2024
RossComputerGuy
approved these changes
Oct 14, 2024
sternenseemann
added a commit
to sternenseemann/nixpkgs
that referenced
this pull request
Nov 8, 2024
As mentioned in NixOS#305146, keeping LLVM 12 is a source of pain because it is the only version to be built from individual release tarball instead of the LLVM monorepo. This commit makes LLVM 12 start from the monorepo as well, simplifying all common LLVM expressions in the process. With NixOS#347887, some quirks in the expressions for LLVM <14 were ironed out, so building LLVM through from the monorepo is quite simple now. - Most expressions only required minor changes, mostly removing the special casing for `sourceRoot`. - The patch lists from llvm/12/default.nix were ported to common/default.nix. This only required a few extra conditionals which could be reduced via a rebuild also involving other LLVM versions. Outstanding tasks of little urgency have been noted in TODO comments. I have verified that the patch lists stay the same for all packages except LLVM where merely the order changes. An extra set of eyes is appreciated, of course. - clang: The expression was reworked to use the same symlink location for clang-tools-extra for all versions including LLVM 12. This required adjusting the ad hoc patching of the clangd cmake files slightly. - libunwind: We no longer need to make the libcxx sources available manually. We can rely on the monorepo source instead. - lld: We no longer need to make the libunwind sources available manually. - llvm: We no longer need to make the polly sources available manually - On Darwin, we need to bypass CMake's C++ compiler for libcxx and libunwind now. It isn't a 100% clear why, probably because we've started to use Darwin's bootstrapStdenv for libcxx in the common expression compared to LLVM 12 on master [1]. The reordering of flags for wasm causes a rebuild for some packages like firefox, but this should be tolerable on staging. [1]: https://github.com/NixOS/nixpkgs/blob/665ebfb253caba7b85c2affefe2a92b305def4e6/pkgs/development/compilers/llvm/12/default.nix#L392-L430
github-actions bot
pushed a commit
that referenced
this pull request
Nov 17, 2024
As mentioned in #305146, keeping LLVM 12 is a source of pain because it is the only version to be built from individual release tarball instead of the LLVM monorepo. This commit makes LLVM 12 start from the monorepo as well, simplifying all common LLVM expressions in the process. With #347887, some quirks in the expressions for LLVM <14 were ironed out, so building LLVM through from the monorepo is quite simple now. - Most expressions only required minor changes, mostly removing the special casing for `sourceRoot`. - The patch lists from llvm/12/default.nix were ported to common/default.nix. This only required a few extra conditionals which could be reduced via a rebuild also involving other LLVM versions. Outstanding tasks of little urgency have been noted in TODO comments. I have verified that the patch lists stay the same for all packages except LLVM where merely the order changes. An extra set of eyes is appreciated, of course. - clang: The expression was reworked to use the same symlink location for clang-tools-extra for all versions including LLVM 12. This required adjusting the ad hoc patching of the clangd cmake files slightly. - libunwind: We no longer need to make the libcxx sources available manually. We can rely on the monorepo source instead. - lld: We no longer need to make the libunwind sources available manually. - llvm: We no longer need to make the polly sources available manually - On Darwin, we need to bypass CMake's C++ compiler for libcxx and libunwind now. It isn't a 100% clear why, probably because we've started to use Darwin's bootstrapStdenv for libcxx in the common expression compared to LLVM 12 on master [1]. The reordering of flags for wasm causes a rebuild for some packages like firefox, but this should be tolerable on staging. [1]: https://github.com/NixOS/nixpkgs/blob/665ebfb253caba7b85c2affefe2a92b305def4e6/pkgs/development/compilers/llvm/12/default.nix#L392-L430 (cherry picked from commit ee9eacf)
vcunat
pushed a commit
that referenced
this pull request
Nov 30, 2024
As mentioned in #305146, keeping LLVM 12 is a source of pain because it is the only version to be built from individual release tarball instead of the LLVM monorepo. This commit makes LLVM 12 start from the monorepo as well, simplifying all common LLVM expressions in the process. With #347887, some quirks in the expressions for LLVM <14 were ironed out, so building LLVM through from the monorepo is quite simple now. - Most expressions only required minor changes, mostly removing the special casing for `sourceRoot`. - The patch lists from llvm/12/default.nix were ported to common/default.nix. This only required a few extra conditionals which could be reduced via a rebuild also involving other LLVM versions. Outstanding tasks of little urgency have been noted in TODO comments. I have verified that the patch lists stay the same for all packages except LLVM where merely the order changes. An extra set of eyes is appreciated, of course. - clang: The expression was reworked to use the same symlink location for clang-tools-extra for all versions including LLVM 12. This required adjusting the ad hoc patching of the clangd cmake files slightly. - libunwind: We no longer need to make the libcxx sources available manually. We can rely on the monorepo source instead. - lld: We no longer need to make the libunwind sources available manually. - llvm: We no longer need to make the polly sources available manually - On Darwin, we need to bypass CMake's C++ compiler for libcxx and libunwind now. It isn't a 100% clear why, probably because we've started to use Darwin's bootstrapStdenv for libcxx in the common expression compared to LLVM 12 on master [1]. The reordering of flags for wasm causes a rebuild for some packages like firefox, but this should be tolerable on staging. [1]: https://github.com/NixOS/nixpkgs/blob/665ebfb253caba7b85c2affefe2a92b305def4e6/pkgs/development/compilers/llvm/12/default.nix#L392-L430 (cherry picked from commit ee9eacf) (cherry picked from commit c4e9f17)
hsjobeki
pushed a commit
to hsjobeki/nixpkgs
that referenced
this pull request
Dec 3, 2024
As mentioned in NixOS#305146, keeping LLVM 12 is a source of pain because it is the only version to be built from individual release tarball instead of the LLVM monorepo. This commit makes LLVM 12 start from the monorepo as well, simplifying all common LLVM expressions in the process. With NixOS#347887, some quirks in the expressions for LLVM <14 were ironed out, so building LLVM through from the monorepo is quite simple now. - Most expressions only required minor changes, mostly removing the special casing for `sourceRoot`. - The patch lists from llvm/12/default.nix were ported to common/default.nix. This only required a few extra conditionals which could be reduced via a rebuild also involving other LLVM versions. Outstanding tasks of little urgency have been noted in TODO comments. I have verified that the patch lists stay the same for all packages except LLVM where merely the order changes. An extra set of eyes is appreciated, of course. - clang: The expression was reworked to use the same symlink location for clang-tools-extra for all versions including LLVM 12. This required adjusting the ad hoc patching of the clangd cmake files slightly. - libunwind: We no longer need to make the libcxx sources available manually. We can rely on the monorepo source instead. - lld: We no longer need to make the libunwind sources available manually. - llvm: We no longer need to make the polly sources available manually - On Darwin, we need to bypass CMake's C++ compiler for libcxx and libunwind now. It isn't a 100% clear why, probably because we've started to use Darwin's bootstrapStdenv for libcxx in the common expression compared to LLVM 12 on master [1]. The reordering of flags for wasm causes a rebuild for some packages like firefox, but this should be tolerable on staging. [1]: https://github.com/NixOS/nixpkgs/blob/665ebfb253caba7b85c2affefe2a92b305def4e6/pkgs/development/compilers/llvm/12/default.nix#L392-L430 (cherry picked from commit ee9eacf) (cherry picked from commit c4e9f17)
13 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change implements a leftover task from #307211, namely passing monorepoSrc to the different llvmPackages_13 package expressions. Before this change, all packages llvmPackages_13 would be built from a subdirectory of the full LLVM monorepo tree. After this change only the relevant directories are made available at build time. This
Since LLVM 12 and 13 don't have the
cmakedirectory at the top level, the runCommand expressions filtering the source need to be adjusted, but this causes no rebuild for any other LLVM version (ofborg should confirm this).Things done
nix.conf? (See Nix manual)sandbox = relaxedsandbox = truenix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)Add a 👍 reaction to pull requests you find important.