Skip to content

Backport 271078 to staging 23.11#273040

Merged
ConnorBaker merged 38 commits intoNixOS:staging-23.11from
SomeoneSerge:backport-271078-to-staging-23.11
Dec 15, 2023
Merged

Backport 271078 to staging 23.11#273040
ConnorBaker merged 38 commits intoNixOS:staging-23.11from
SomeoneSerge:backport-271078-to-staging-23.11

Conversation

@SomeoneSerge
Copy link
Contributor

@SomeoneSerge SomeoneSerge commented Dec 8, 2023

Description of changes

Supersedes #272336. Incorporates #272063, #273048, #272964 and #272812

This probably goes into staging before #272784

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added the 6.topic: python Python is a high-level, general-purpose programming language. label Dec 8, 2023
@ofborg ofborg bot added 8.has: clean-up This PR removes packages or removes other cruft 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 501-1000 This PR causes many rebuilds on Linux and should normally target the staging branches. labels Dec 9, 2023
@SomeoneSerge SomeoneSerge added the 6.topic: cuda Parallel computing platform and API label Dec 10, 2023
@SomeoneSerge
Copy link
Contributor Author

I should probably reorder the commits so that blender won't be broken in the intermediate commits either

SomeoneSerge and others added 20 commits December 11, 2023 21:06
also fix the incompatible gcc error by adding cuda_nvcc to
nativeBuildInputs (cudaPackages.cudatoolkit in buildInputs is
insufficient because that doesn't propagate the hook)

(cherry picked from commit 7c97d5f)
(cherry picked from commit 2381384)
…ctures (compute_35)

(cherry picked from commit 5c2a368)
(cherry picked from commit 361d7da)
(cherry picked from commit 31f1b51)
The GitHub repo has been archived, the new thing is
[cccl](https://github.com/nvidia/cccl)

(cherry picked from commit 9cc210a)
(cherry picked from commit 3e37f3c)
(cherry picked from commit 0f047c2)
(cherry picked from commit 4c6d2b8)
SomeoneSerge and others added 14 commits December 11, 2023 21:06
(cherry picked from commit 18a2e51)
This is useful for the cuda variants of packages like opencv and pytorch,
whose xxxxConfig.cmake files do find_package(CUDAToolkit REQUIRED)
regardless of whether they actually use it. With the propagated hook,
we no longer have to manually add cuda dependencies into torch/opencvs
reverse dependencies

cudaPackages.cuda_nvcc: fix setupCudaHook propagation

(cherry picked from commit be9c779)
(cherry picked from commit 55af932)
(cherry picked from commit 44611c4)
@SomeoneSerge SomeoneSerge force-pushed the backport-271078-to-staging-23.11 branch from c8287ce to d8ed2ba Compare December 11, 2023 21:07
@ConnorBaker
Copy link
Contributor

ConnorBaker commented Dec 13, 2023

Note

Template nixpkgs-review command:

PR=273040; \
SYSTEM="aarch64-linux"; \
CUDA_SUPPORT="true"; \
CUDA_CAPABILITIES='[ "7.5" ]'; \
nixpkgs-review pr "$PR" \
  --system "$SYSTEM" \
  --no-shell \
  --checkout commit \
  --allow aliases \
  --build-args "--max-jobs 1" \
  --extra-nixpkgs-config "{
    allowUnfree = true;
    allowBroken = false;
    cudaSupport = ${CUDA_SUPPORT:-false};
    cudaCapabilities = ${CUDA_CAPABILITIES:-[]};
  }"

🕐 aarch64-darwin

In progress.

🕐 x86_64-darwin

In progress.

🕐 aarch64-linux

✅ Jetson

Unsupported pending merge of #272784.

✅ Non-Jetson

Unsupported pending merge of #272784.

🕐 Non-CUDA

In progress.

x86_64-linux

🕐 Non-Jetson

In progress.

❌ Non-CUDA

Failure is due to my explicitly passing an empty list for cudaCapabilities, which should not be done as it overrides the default prior to #272784, causing derivations which assume they will always be built with CUDA support (like magma-cuda) to fail with an out of bounds error when indexing into the empty list.

Details
$ PR=273040; SYSTEM="x86_64-linux"; CUDA_SUPPORT="false"; CUDA_CAPABILITIES='[ ]'; nixpkgs-review pr "$PR"   --system "$SYSTEM"   --no-shell   --checkout commit   --allow aliases   --build-args "--max-jobs 1"   --extra-nixpkgs-config "{
    allowUnfree = true;
    allowBroken = false;
    cudaSupport = ${CUDA_SUPPORT:-false};
    cudaCapabilities = ${CUDA_CAPABILITIES:-[]};
  }"
$ git -c fetch.prune=false fetch --no-tags --force https://github.com/NixOS/nixpkgs staging-23.11:refs/nixpkgs-review/0 pull/273040/head:refs/nixpkgs-review/1
$ git worktree add /home/connorbaker/.cache/nixpkgs-review/pr-273040-2/nixpkgs d8ed2baa7c48a75470e2093e6dfdf497ffa31d4c
Preparing worktree (detached HEAD d8ed2baa7c48)
Updating files: 100% (38204/38204), done.
HEAD is now at d8ed2baa7c48 cudaPackages.cudatoolkit: propagate the hook to nativeBuildInputs correctly
error:
       … while evaluating attribute 'magma-cuda'

         at /home/connorbaker/.cache/nixpkgs-review/pr-273040-2/nixpkgs/lib/attrsets.nix:488:14:

          487|     value:
          488|     { inherit name value; };
             |              ^
          489|

       … while evaluating attribute 'broken'

         at /nix/store/3mzsbqwxz6va3f0fwsbsvkahpp15w62y-nixpkgs-review/lib/python3.11/site-packages/nixpkgs_review/nix/evalAttrs.nix:39:20:

           38|           {
           39|             inherit exists broken;
             |                    ^
           40|             path = if !broken then maybePath.value else null;

       (stack trace truncated; use '--show-trace' to show the full trace)

       error: list index 0 is out of bounds

       at «none»:0: (source not available)
https://github.com/NixOS/nixpkgs/pull/273040 failed to build: nix --extra-experimental-features nix-command no-url-literals --system x86_64-linux eval --nix-path nixpkgs=/home/connorbaker/.cache/nixpkgs-review/pr-273040-2/nixpkgs nixpkgs-overlays=/tmp/tmpbi17390e --json --impure --no-allow-import-from-derivation --expr (import /nix/store/3mzsbqwxz6va3f0fwsbsvkahpp15w62y-nixpkgs-review/lib/python3.11/site-packages/nixpkgs_review/nix/evalAttrs.nix { attr-json = /tmp/tmpbb0h7tim; }) failed to run, /tmp/tmpbb0h7tim was stored inspection

@ConnorBaker ConnorBaker self-requested a review December 13, 2023 17:44
@ConnorBaker ConnorBaker self-assigned this Dec 13, 2023
@ConnorBaker
Copy link
Contributor

Just waiting for ofborg-eval to finish before merging...

@ConnorBaker ConnorBaker merged commit 9e16663 into NixOS:staging-23.11 Dec 15, 2023
@vcunat
Copy link
Member

vcunat commented Jan 18, 2024

@vcunat
Copy link
Member

vcunat commented Jan 18, 2024

At a glance, I'd say that such large changes aren't even suitable for backporting to stable.

@SomeoneSerge
Copy link
Contributor Author

Cctag definitely was affected by the opencvConfig issue on master at the time, as were two dozen other cudaSupport=true packages in the release branch (link tbd, writing from a mobile). The backport is huge partly because I tried to merge and reorder fixes from several correlated PRs. I'll look into the hydra failure, thx

@vcunat
Copy link
Member

vcunat commented Jan 18, 2024

For reference, target PR #279139

vcunat referenced this pull request Jan 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

6.topic: cuda Parallel computing platform and API 6.topic: python Python is a high-level, general-purpose programming language. 8.has: clean-up This PR removes packages or removes other cruft 10.rebuild-darwin: 101-500 This PR causes between 101 and 500 packages to rebuild on Darwin. 10.rebuild-linux: 501-1000 This PR causes many rebuilds on Linux and should normally target the staging branches. 10.rebuild-linux: 501+ This PR causes many rebuilds on Linux and should normally target the staging branches.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants