🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/121956

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit a4a5d05 with merge base 5ea956a ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Hmm, I think it needs some discussion, because if we are to stop building/testing CUDA-11.8, it means we are loosing Keplers, don't we? @atalman is there a doc whether this is intended

why not mantain 11.8 as cuda 11 and 12.4 as cuda 12? And skip 12.1. I mean maintain always two versions of cuda,
for example if cuda 13 is out, the newer versions it would be 12 and 13

We discussed for a short term, we would have 11.8, 12.1, and 12.4. I will need to refactor this PR to add back 11.8.

when will be merged? 😋

12.4 workflows are failing. Still working on coming up with a fix.

12.4 update 1 is out:
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/

@ptrblck @nWEIdia nvidia cudnn9 now is available nvidia-cudnn-cu12 9.0.0.312
https://pypi.org/project/nvidia-cudnn-cu12/

@johnnynunez Yes, it is! We will focus on 12.4 in this PR and follow up with the cuDNN update separately to avoid creating confusing issues pointing to the CUDA and cuDNN update.

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

Successfully rebased cuda_124_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_124_ci && git pull --rebase)

Warning: Unknown label ciflow/inductor-perf-test-nightly.
Currently recognized labels are

• ciflow/binaries
• ciflow/binaries_conda
• ciflow/binaries_libtorch
• ciflow/binaries_wheel
• ciflow/inductor
• ciflow/inductor-perf-compare
• ciflow/inductor-micro-benchmark
• ciflow/linux-aarch64
• ciflow/mps
• ciflow/nightly
• ciflow/periodic
• ciflow/rocm
• ciflow/slow
• ciflow/trunk
• ciflow/unstable
• ciflow/xpu
• ciflow/torchbench

Please add the new label to .github/pytorch-probot.yml

Successfully rebased cuda_124_ci onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout cuda_124_ci && git pull --rebase)

@malfet Could you please help take another look?
I am composing torchinductor 12.4 issues in here.
Thanks!

Hi @nWEIdia please disable the failing tests. We will follow up on this in the issue you opened

@pytorchmergebot merge -f "All required tests are pasing"

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

For the context, after this change lands in trunk, the new CUDA 12.4 build starts to fails on newly created open PyTorch PR. Here is what happens:

• This PR includes the fix for the build failure in caffe2/CMakeLists.txt
• After this lands in trunk, newer PRs will now start running CUDA 12.4 build coming from the trunk version of workflow. We now have 2 cases:
  1. If the newer PRs has https://hud.pytorch.org/pytorch/pytorch/commit/0902929d582879caa926705f283f5f55864bd7bf, they will build fine because they have the fix
  2. If the newer PRs don't have the above commit, they will fail without the fix, for example Default TreadPool size to number of physical cores #125963

This is not an ideal rollout, but the way for now is to ask folks to rebase onto main

Sorry for the mishaps.

The PR went in 05/23 1:37pm, @malfet issued a "pytorch rebase" at 2:22pm on the #125963 PR, the result is based on #126976 (10:31am)

I guess the lesson is we should request an immediate push to viable/strict for future occurrences like this.