[CD] Upgrade to CUDA 13.0.2 for nightly binaries #165470

tinglvv · 2025-10-14T21:12:23Z

13.0.U2 is posted, adding to nightlies
Why we want to upgrade: CUDA 13.0.U2 included a new release from cuBLAS that

Enabled opt-in fixed-point emulation for FP64 matmuls (D/ZGEMM) which improves performance and power-efficiency.
Improved performance on NVIDIA DGX Spark for FP16/BF16 and FP8 GEMMs.
adds BF16x9 FP32 emulation support for SYRK and HERK routines.
Reference: https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#cublas-release-13-0-update-2

cc @atalman @malfet @ptrblck @nWEIdia

pytorch-bot · 2025-10-14T21:12:28Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165470

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 24 New Failures, 1 Cancelled Job, 4 Unrelated Failures

As of commit 269473d with merge base 23c55c5 ():

NEW FAILURES - The following jobs have failed:

Build libtorch docker images / libtorch-cxx11-builder:cpu (gh)
Final attempt failed. Child_process exited with error code 1
Build libtorch docker images / libtorch-cxx11-builder:cuda12.6 (gh)
Final attempt failed. Child_process exited with error code 1
Build libtorch docker images / libtorch-cxx11-builder:cuda12.8 (gh)
Final attempt failed. Child_process exited with error code 1
Build libtorch docker images / libtorch-cxx11-builder:cuda12.9 (gh)
Final attempt failed. Child_process exited with error code 1
Build libtorch docker images / libtorch-cxx11-builder:cuda13.0 (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.4-cudnn9-py3-gcc11) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc11) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc9-inductor-benchmarks) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3-gcc9) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3.10-clang12) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3.10-linter) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda12.8-cudnn9-py3.12-gcc11-vllm) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-cuda13.0-cudnn9-py3-gcc11) (gh)
Final attempt failed. Timeout of 5400000ms hit
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-clang12-onnx) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3.10-clang12) (gh)
Final attempt failed. Timeout of 5400000ms hit
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3.10-gcc11) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3.12-halide) (gh)
Final attempt failed. Timeout of 5400000ms hit
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3.12-triton-cpu) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3.13-clang12) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-rocm-n-py3-benchmarks) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-rocm-n-py3) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-xpu-n-1-py3) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-xpu-n-py3) (gh)
Final attempt failed. Child_process exited with error code 1
docker-builds / docker-build (linux.12xlarge, pytorch-linux-noble-rocm-n-py3) (gh)
Final attempt failed. Child_process exited with error code 1

CANCELLED JOB - The following job was cancelled. Please retry:

docker-builds / docker-build (linux.12xlarge, pytorch-linux-jammy-py3-gcc11-inductor-benchmarks) (gh)
##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

Build manywheel docker images / manylinuxaarch64-builder:cuda13.0 (gh) (detected as infra flaky with no log or failing log classifier)
trunk / linux-jammy-rocm-py3.10 / test (default, 1, 2, linux.rocm.gpu.gfx942.1) (gh) (disabled by #165872 but the issue was closed recently and a rebase is needed to make it pass)
test/test_cuda.py::TestCudaMallocAsync::test_allocator_backend
windows-binary-wheel / wheel-py3_13-xpu-build (gh) (matched win rule in flaky-rules.json)
##[error]The operation was canceled.
windows-binary-wheel / wheel-py3_14-xpu-build (gh) (matched win rule in flaky-rules.json)
##[error]The operation was canceled.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman · 2025-10-17T18:29:22Z

HI @tinglvv please create a PR for updating the dependencies here: https://github.com/pytorch/test-infra/blob/main/s3_management/update_dependencies.py#L11

tinglvv · 2025-10-21T18:18:25Z

@pytorchbot rebase

pytorchmergebot · 2025-10-21T18:19:58Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-10-21T18:20:01Z

Successfully rebased 13.0-u2-nightly onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout 13.0-u2-nightly && git pull --rebase)

atalman

lgtm

atalman · 2025-10-27T16:11:00Z

@pytorchbot merge

pytorchmergebot · 2025-10-27T16:16:44Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-10-27T22:11:24Z

The merge job was canceled or timed out. This most often happen if two merge requests were issued for the same PR, or if merge job was waiting for more than 6 hours for tests to finish. In later case, please do not hesitate to reissue the merge command
For more information see pytorch-bot wiki.

tinglvv · 2025-10-28T00:19:34Z

@pytorchbot merge

pytorchmergebot · 2025-10-28T00:21:17Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-10-28T02:20:15Z

@pytorchmergebot revert -m "Sorry reverting for now, to restore trunk health" -c nosignal

pytorchmergebot · 2025-10-28T02:21:45Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-10-28T02:21:51Z

@tinglvv your PR has been successfully reverted.

This reverts commit 5e769ff. Reverted #165470 on behalf of https://github.com/atalman due to Sorry reverting for now, to restore trunk health ([comment](#165470 (comment)))

atalman · 2025-10-28T15:12:14Z

@pytorchbot merge -f "docker build issue is resolved, merging this"

pytorchmergebot · 2025-10-28T15:13:52Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-11-03T17:48:03Z

@pytorchmergebot revert -m "reverting broke Docker nightly builds" -c nosignal

pytorchmergebot · 2025-11-03T17:49:47Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-11-03T17:49:49Z

Reverting PR 165470 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit 544b443ea1d1a9b19e65f981168a01cb87a2d333 returned non-zero exit code 1

Auto-merging .ci/docker/common/install_cuda.sh
Auto-merging .github/scripts/generate_binary_build_matrix.py
CONFLICT (content): Merge conflict in .github/scripts/generate_binary_build_matrix.py
Auto-merging .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-aarch64-binary-manywheel-nightly.yml
Auto-merging .github/workflows/generated-linux-binary-manywheel-nightly.yml
CONFLICT (content): Merge conflict in .github/workflows/generated-linux-binary-manywheel-nightly.yml
error: could not revert 544b443ea1d... [CD] Upgrade to CUDA 13.0.2 for nightly binaries (#165470)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

)" This reverts commit 544b443.

tinglvv requested review from a team and jeffdaily as code owners October 14, 2025 21:12

pytorch-bot bot added the topic: not user facing topic category label Oct 14, 2025

tinglvv changed the title ~~Upgrade to CUDA 13.0.2 for nightly binaries~~ [CD] Upgrade to CUDA 13.0.2 for nightly binaries Oct 14, 2025

pytorchbot added the open source label Oct 14, 2025

tinglvv mentioned this pull request Oct 14, 2025

Enable CUDA 13.0 binaries #159779

Closed

15 tasks

mikaylagawarecki requested a review from atalman October 17, 2025 16:05

mikaylagawarecki added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Oct 17, 2025

tinglvv added 2 commits October 21, 2025 18:19

Upgrade to CUDA 13.0.2

cc1e06e

modify the installation name

269473d

pytorchmergebot force-pushed the 13.0-u2-nightly branch from 06f6698 to 269473d Compare October 21, 2025 18:20

tinglvv added the ciflow/binaries Trigger all binary build and upload jobs on the PR label Oct 21, 2025

atalman approved these changes Oct 27, 2025

View reviewed changes

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 27, 2025

pytorchmergebot added the merging label Oct 27, 2025

pytorchmergebot added the Merged label Oct 28, 2025

pytorchmergebot closed this in 5e769ff Oct 28, 2025

pytorchmergebot removed the merging label Oct 28, 2025

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Oct 28, 2025

pytorchmergebot reopened this Oct 28, 2025

atalman mentioned this pull request Oct 28, 2025

All Docker build failed due to Ubuntu archive outage #166363

Closed

pytorchmergebot added the merging label Oct 28, 2025

pytorchmergebot closed this in 544b443 Oct 28, 2025

pytorchmergebot removed the merging label Oct 28, 2025

atalman added a commit to atalman/pytorch that referenced this pull request Nov 3, 2025

Revert "[CD] Upgrade to CUDA 13.0.2 for nightly binaries (pytorch#165470

66549ee

)" This reverts commit 544b443.

This was referenced Nov 3, 2025

Revert "[CD] Upgrade to CUDA 13.0.2 for nightly binaries (#165470)" #166893

Closed

Docker release devel image. Use ubuntu22.04 as base instead of nvidia/cuda:CUDA_VERSION-devel-ubuntu22.04 #166897

Open

[CD] Upgrade to CUDA 13.0.2 for nightly binaries #165470

[CD] Upgrade to CUDA 13.0.2 for nightly binaries #165470

Uh oh!

Conversation

tinglvv commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/165470

❌ 24 New Failures, 1 Cancelled Job, 4 Unrelated Failures

Uh oh!

atalman commented Oct 17, 2025

Uh oh!

tinglvv commented Oct 21, 2025

Uh oh!

pytorchmergebot commented Oct 21, 2025

Uh oh!

pytorchmergebot commented Oct 21, 2025

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

atalman commented Oct 27, 2025

Uh oh!

pytorchmergebot commented Oct 27, 2025

Merge started

Uh oh!

pytorchmergebot commented Oct 27, 2025

Uh oh!

tinglvv commented Oct 28, 2025

Uh oh!

pytorchmergebot commented Oct 28, 2025

Merge started

Uh oh!

atalman commented Oct 28, 2025

Uh oh!

pytorchmergebot commented Oct 28, 2025

Uh oh!

pytorchmergebot commented Oct 28, 2025

Uh oh!

atalman commented Oct 28, 2025

Uh oh!

pytorchmergebot commented Oct 28, 2025

Merge started

Uh oh!

atalman commented Nov 3, 2025

Uh oh!

pytorchmergebot commented Nov 3, 2025

Uh oh!

pytorchmergebot commented Nov 3, 2025

Reverting PR 165470 failed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

tinglvv commented Oct 14, 2025 •

edited

Loading

pytorch-bot bot commented Oct 14, 2025 •

edited

Loading