cudaPackages.setupCudaHook: init#233581
Merged
ConnorBaker merged 4 commits intoNixOS:masterfrom Jul 25, 2023
Merged
Conversation
ofborg
bot
added
10.rebuild-darwin: 11-100
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
5 times, most recently
from
c345719 to
c237d6a
Compare
Member
|
Will this obviate the need for the |
Contributor
Author
Only partially: for the projects that use CMake and |
Contributor
|
Well, magma built, and comparing the diffs it looks like it worked! $ diff --text --minimal --ignore-all-space --side-by-side ./magma-master.log ./magma-serge.log
acl-2.3.1 108.9K 31.3M acl-2.3.1 108.9K 31.3M
attr-2.5.1 78.8K 31.2M attr-2.5.1 78.8K 31.2M
bash-5.2-p15 1.6M 32.7M bash-5.2-p15 1.6M 32.7M
binutils-2.40 28.2M 69.8M binutils-2.40 28.2M 69.8M
binutils-2.40-lib 2.7M 33.9M binutils-2.40-lib 2.7M 33.9M
binutils-wrapper-2.40 50.1K 84.7M binutils-wrapper-2.40 50.1K 84.7M
blas-3 53.3M 121.1M blas-3 53.3M 121.1M
coreutils-9.1 1.4M 41.1M coreutils-9.1 1.4M 41.1M
cuda_cudart-11.7.60 6.3M 6.3M cuda_cudart-11.7.60 6.3M 6.3M
cuda_cupti-11.7.50 88.4M 88.6M cuda_cupti-11.7.50 88.4M 88.6M
cuda_nvcc-11.7.64 114.8M 395.5M | cuda_nvcc-11.7.64 114.8M 401.8M
cuda_nvprof-11.7.50 9.9M 129.6M cuda_nvprof-11.7.50 9.9M 129.6M
cuda-native-redist-11.7 63.0K 2.1G <
expand-response-params 16.4K 31.1M expand-response-params 16.4K 31.1M
gcc-11.3.0 183.0M 244.6M gcc-11.3.0 183.0M 244.6M
gcc-11.3.0-lib 7.4M 46.3M gcc-11.3.0-lib 7.4M 46.3M
gcc-11.3.0-libgcc 114.3K 114.3K gcc-11.3.0-libgcc 114.3K 114.3K
gcc-12.2.0-lib 7.5M 38.8M gcc-12.2.0-lib 7.5M 38.8M
gcc-12.2.0-libgcc 139.3K 139.3K gcc-12.2.0-libgcc 139.3K 139.3K
gcc-wrapper-11.3.0 58.6K 280.7M gcc-wrapper-11.3.0 58.6K 280.7M
gfortran-12.2.0-lib 10.5M 41.7M gfortran-12.2.0-lib 10.5M 41.7M
gfortran-12.2.0-libgcc 139.3K 139.3K gfortran-12.2.0-libgcc 139.3K 139.3K
glibc-2.37-8 28.8M 31.1M glibc-2.37-8 28.8M 31.1M
glibc-2.37-8-bin 2.7M 33.8M glibc-2.37-8-bin 2.7M 33.8M
glibc-2.37-8-dev 2.2M 42.1M glibc-2.37-8-dev 2.2M 42.1M
gmp-with-cxx-6.2.1 729.2K 39.5M gmp-with-cxx-6.2.1 729.2K 39.5M
gmp-with-cxx-6.2.1 730.4K 39.5M gmp-with-cxx-6.2.1 730.4K 39.5M
gnugrep-3.7 773.1K 32.4M gnugrep-3.7 773.1K 32.4M
isl-0.20 2.5M 41.9M isl-0.20 2.5M 41.9M
libcublas-11.10.1.25 1.1G 1.2G libcublas-11.10.1.25 1.1G 1.2G
libcusparse-11.7.3.50 488.1M 488.2M libcusparse-11.7.3.50 488.1M 488.2M
libidn2-2.3.4 350.4K 2.1M libidn2-2.3.4 350.4K 2.1M
libmpc-1.3.1 273.4K 40.5M libmpc-1.3.1 273.4K 40.5M
libunistring-1.1 1.8M 1.8M libunistring-1.1 1.8M 1.8M
linux-headers-6.2 6.1M 6.1M linux-headers-6.2 6.1M 6.1M
magma-2.7.1 233.6M 2.4G magma-2.7.1 233.6M 2.4G
mpfr-4.2.0 774.0K 40.2M mpfr-4.2.0 774.0K 40.2M
openblas-0.3.21 26.1M 67.8M openblas-0.3.21 26.1M 67.8M
pcre-8.45 514.4K 31.6M pcre-8.45 514.4K 31.6M
> setup-cuda-paths-hook 3.2K 280.7M
xgcc-12.2.0-libgcc 139.3K 139.3K xgcc-12.2.0-libgcc 139.3K 139.3K
zlib-1.2.13 129.6K 31.2M zlib-1.2.13 129.6K 31.2MI'll try to run an ML workload with the new magma just to verify it still works. |
Contributor
|
I failed to build PyTorch (because $ cat ~/.config/nixpkgs/config.nix
{
allowUnfree = true;
enableUnfree = true;
cudaSupport = true;
cudaCapabilities = [ "8.6" ];
cudaForwardCompat = false;
}
$ nix build --impure -L .#cudaPackages.cudatoolkit
cudatoolkit> Sourcing auto-add-opengl-runpath-hook
cudatoolkit> Using autoAddOpenGLRunpathPhase
cudatoolkit> Sourcing setup-cuda-paths-hook
cudatoolkit> Executing setupCUDAToolkitCompilers
cudatoolkit> unpacking sources
cudatoolkit> Creating directory pkg
cudatoolkit> patching sources
cudatoolkit> configuring
cudatoolkit> no configure script, doing nothing
cudatoolkit> building
cudatoolkit> no Makefile or custom buildPhase, doing nothing
cudatoolkit> glibPreInstallPhase
cudatoolkit> installing
cudatoolkit> renamed '/nix/store/s10xiaawf1dmfpkkw7jjhnl0l6qj5zv3-cudatoolkit-11.7.0/lib64/libcudart.so' -> '/nix/store/ybz3nr5lwq1rf4xjn69pl9wijzi0qa8m-cudatoolkit-11.7.0-lib/lib/libcudart.so'
cudatoolkit> renamed '/nix/store/s10xiaawf1dmfpkkw7jjhnl0l6qj5zv3-cudatoolkit-11.7.0/lib64/libcudart.so.11.0' -> '/nix/store/ybz3nr5lwq1rf4xjn69pl9wijzi0qa8m-cudatoolkit-11.7.0-lib/lib/libcudart.so.11.0'
cudatoolkit> renamed '/nix/store/s10xiaawf1dmfpkkw7jjhnl0l6qj5zv3-cudatoolkit-11.7.0/lib64/libcudart.so.11.7.60' -> '/nix/store/ybz3nr5lwq1rf4xjn69pl9wijzi0qa8m-cudatoolkit-11.7.0-lib/lib/libcudart.so.11.7.60'
cudatoolkit> renamed '/nix/store/s10xiaawf1dmfpkkw7jjhnl0l6qj5zv3-cudatoolkit-11.7.0/lib64/libcudart_static.a' -> '/nix/store/ybz3nr5lwq1rf4xjn69pl9wijzi0qa8m-cudatoolkit-11.7.0-lib/lib/libcudart_static.a'
cudatoolkit> /nix/store/qamh89j2h8xa0dgl6g8lsa8vzynsz8k9-mark-for-cudatoolkit-root-hook/nix-support/setup-hook: line 5: out lib doc: invalid variable name
error: builder for '/nix/store/19i1yf9zh3rsq1rwym1bw1vnb7ry9499-cudatoolkit-11.7.0.drv' failed with exit code 1How have you been debugging the setup hooks @SomeoneSerge? Is there any way to get a look at the interpolated scripts to see what exactly is being substituted? |
SomeoneSerge
commented
May 31, 2023
pkgs/development/compilers/cudatoolkit/hooks/mark-for-cudatoolkit-root-hook.sh
Outdated
Show resolved
Hide resolved
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
from
c237d6a to
d07a482
Compare
Contributor
Author
😫 We got to do something about gstreamer already |
Contributor
Author
|
This needs to be refactored and documented, but I really cannot afford to right now. I can either just mark this ready as is, or I'm also fine with someone else adopting this PoC and opening a new cleaner PR |
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
from
d07a482 to
89ed623
Compare
SomeoneSerge
changed the title
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
3 times, most recently
from
b683b28 to
bb6acce
Compare
4 tasks
Contributor
Author
|
I added some more comments, but eventually (in a separate PR) we need to update the cuda section of the manual. |
This comment was marked as outdated.
This comment was marked as outdated.
Contributor
Author
|
Result of 59 packages marked as broken and skipped:
79 packages failed to build:
672 packages built:
|
Contributor
Author
Failed derivationsDetails
|
Contributor
Author
|
ConnorBaker
reviewed
Jul 17, 2023
samuela
reviewed
Jul 18, 2023
Member
samuela
left a comment
samuela
left a comment
There was a problem hiding this comment.
I love the idea of getting rid of all the symlinkJoins when working with redist packages!
I'm not sure I understand what each of these hooks does and why they are all necessary. Do you think you could give a quick breakdown of what's happening here and how these 3 new hooks interface with CMAKE?
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
4 times, most recently
from
0b60813 to
ab4b05e
Compare
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
from
ab4b05e to
92ed31a
Compare
SomeoneSerge
commented
Jul 19, 2023
Comment on lines
57
to
64
Contributor
Author
There was a problem hiding this comment.
I removed the setup hook from nvcc and merged it here instead, so we have one big setupCudaHook. This is still more complex than I hoped for, and we'll have to revisit the paths soon when implementing cross-compilation
Contributor
Author
There was a problem hiding this comment.
Whelp, I broke it. Cudatoolkit uses this hook, so for cudaPackages_10_1 it's infinite recursion
Contributor
Author
There was a problem hiding this comment.
My feeling is that deps<foo><bar>Propagated is just a bad idea
Contributor
Author
There was a problem hiding this comment.
Reverted to when cuda_nvcc had its own hook for cudart. I'll open an issue for removing the hook depsTargetTargetPropagated (and maybe removing backendStdenv) in favour of explicit
mkDerivation {
# ...
nativeBuildInputs = [
# ...
cudaPackages.setupCudaHook
cudaPackages.cuda_nvcc
];
# ...
}The reason to not do that in this PR is that this would require touching many downstream packages, making the diff and the blame messy, and pinging more people.
SomeoneSerge
changed the title
Contributor
Author
There's now just one "public" hook, namely Its main purpose is to scan through the new package's The main hook also sets some nvcc flags, mostly to make EDIT: Because of bootstrapping issues, I temporarily moved parts of |
12 tasks
SomeoneSerge
force-pushed
the
CUDAToolkit_ROOT
branch
from
92ed31a to
251d316
Compare
Contributor
Author
|
Alright, I think this ready to be merged |
Contributor
Author
|
Result of 69 packages marked as broken and skipped:
203 packages failed to build:
814 packages built:
|
Contributor
|
@SomeoneSerge that seems like a lot of breakages for stuff that this didn't touch, right? Would you try running |
Contributor
Author
Failed derivationsDetails
|
Contributor
Author
I can schedule one but tbh it's going to be just energy waste. Skimming briefly through the logs of the second run, I don't think we'll see anything different from the first:) (nixpkgs-review comparing against a database of previous build results when, automatic estimates for the probability of reproducing a failure when) EDIT2: ...OOM-ed |
Contributor
Author
|
Result of 59 packages marked as broken and skipped:
93 packages failed to build:
659 packages built:
|
Contributor
Author
|
@ConnorBaker the last list is from --checkout commit, and I think it aligns with the original |
ConnorBaker
approved these changes
Jul 25, 2023
Contributor
ConnorBaker
left a comment
ConnorBaker
left a comment
There was a problem hiding this comment.
I think this is in a good enough state that it can be merged, and any nits/additional work can be done in a followup PR.
Contributor
|
Thank you again for working on this @SomeoneSerge, it's much appreciated! |
12 tasks
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of changes
Experiments towards #224291
State: tried rewriting
magmawithoutsymlinkJoinand it seems to build fineThings done
sandbox = trueset innix.conf? (See Nix manual)nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage./result/bin/)