🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/132202

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 4 Unrelated Failures

As of commit 29e15e9 with merge base 8624a57 ():

This comment was automatically generated by Dr. CI and updates every 15 minutes.

What's the goal of updating the minor version? Are there important perf / bug fixes? Does it update the minimum driver version or requirements? If we are doing this we should also consider updating NCCL and CUDNN in a concurrent PR probably.

What's the goal of updating the minor version? Are there important perf / bug fixes? Does it update the minimum driver version or requirements? If we are doing this we should also consider updating NCCL and CUDNN in a concurrent PR probably.

The update originally comes from the suggestion of @malfet and @ptrblck to always use the maximum minor version of CUDA 12.4 which is CUDA 12.4.1.
At least it would fix nanogpt smoke test regression, see https://github.com/pytorch/pytorch/blob/main/.ci/pytorch/test.sh#L640

The Linux minimum driver version (550.54.15) needed by CUDA 12.4.1 was already adopted when enabling CUDA 12.4.0 [yeah, we probably should have just enabled CUDA 12.4.1 then]. Only the Windows side, the driver may need a bump.

NCCL and CUDNN with CUDA 12.4.1 may work similar to CUDA 12.4.0, we could consider updating them though.

For Pip Binary/jobs:
https://ossci-raw-job-status.s3.amazonaws.com/log/28186744872 error below
pip install /final_pkgs/torch-2.5.0.dev20240731+cu124-cp310-cp310-linux_x86_64.whl --index-url https://download.pytorch.org/whl/nightly/cu124
No matching distribution found for nvidia-cuda-nvrtc-cu12==12.4.127; platform_system == "Linux" and platform_machine == "x86_64"
says the pypi packages have to be copied and hosted from aws s3 pytorch (https://download.pytorch.org/whl/nightly/cu124).

For Conda Binary/jobs:
Linking: pytorch/builder#1947

@atalman Can help with the nightly packages

Also pinging @malfet . @nWEIdia Are they are any older versions of CUDA we do should do minor version bumps to? Like 11.8 or 12.1?

@nWEIdia @Skylion007 uploaded packages for 12.4 and 12.4 split build.

Also pinging @malfet . @nWEIdia Are they are any older versions of CUDA we do should do minor version bumps to? Like 11.8 or 12.1?

https://developer.nvidia.com/cuda-toolkit-archive shows CUDA 11.8 does not have patch versions. It looks like 12.1 indeed has an update 1. Fortunately, upstream pytorch already comes with 12.1.1 as indicated by https://github.com/pytorch/pytorch/blob/main/.github/scripts/generate_binary_build_matrix.py#L21

Successfully rebased enable_cuda_12.4.1 onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout enable_cuda_12.4.1 && git pull --rebase)

Looks like the signal is green for CUDA tests but the 6 pending jobs are due to long queue of runner linux.9xlarge.ephemeral

cc @atalman @malfet

For this kind of PRs, it would be great if we can retain the test results from 653b4ac

Update: the 653b4ac is a rebased commit from https://hud.pytorch.org/pytorch/pytorch/commit/77532344e35f7bd8d6e4c221e1b6807e17734204
fortunately, the commit before the rebase and test results are still available!

How to get to the old commit? Post a job link in this PR (e.g. https://github.com/pytorch/pytorch/actions/runs/10295868135/job/28500435981 ) this can serve as index to find the old commit ID.

@pytorchbot merge -i

Merge started

Your change will be merged while ignoring the following 11 checks: trunk / macos-py3-arm64 / test (default, 1, 3, macos-m1-stable), trunk / macos-py3-arm64 / test (default, 2, 3, macos-m1-stable), trunk / macos-py3-arm64 / test (default, 3, 3, macos-m1-stable), linux-binary-manywheel / manywheel-py3_11-rocm6_0-build / build, linux-binary-manywheel / manywheel-py3_12-rocm6_0-build / build, linux-binary-manywheel / manywheel-py3_9-rocm6_0-build / build, linux-binary-manywheel / manywheel-py3_10-rocm6_0-build / build, linux-binary-libtorch-cxx11-abi / libtorch-rocm6_0-shared-with-deps-cxx11-abi-build / build, linux-binary-libtorch-pre-cxx11 / libtorch-rocm6_0-shared-with-deps-pre-cxx11-build / build, periodic / win-vs2019-cuda11.8-py3 / test (default, 3, 4, windows.g5.4xlarge.nvidia.gpu), periodic / linux-focal-rocm6.1-py3.8 / test (distributed, 1, 3, linux.rocm.gpu, unstable)

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Post Merge:
Step 1: check https://hud.pytorch.org/pytorch/pytorch/commit/333890b701fc30065e35b6ae6f24908ecb9d81f8
Observed job failures: https://github.com/pytorch/pytorch/actions/runs/10476660034/job/29016052870

Update: https://github.com/pytorch/pytorch/actions/workflows/build-manywheel-images.yml shows that failure is existing on trunk https://github.com/pytorch/pytorch/actions/runs/10460666895 and https://github.com/pytorch/pytorch/actions/runs/10463836139

Step 1: TLDR, watch for simultaneous pytorch/test-infra AMI updates that could break in-between green signals and final landing. e.g. the above incident.

1st bad build: https://github.com/pytorch/pytorch/actions/runs/10433835042/job/28936166917

2024-08-19T09:45:31.8316859Z #15 421.6 Installing : fipscheck-1.4.1-6.el7.x86_64 2/73
2024-08-19T09:52:25.7608721Z #15 421.7 Installing : fipscheck-lib-1.4.1-6.el7.x86_64 3/73

See the 7 minutes gap!!

Last known good build: https://github.com/pytorch/pytorch/actions/runs/10425348037/job/28876036438
The installation of these two are instant!

Could the runner be from a different region? If not, the multiple "RUN sed -i s/^#.baseurl=http/baseurl=http/g /etc/yum.repos.d/.repo" seems suspicious.

cc @atalman @malfet @ptrblck

@nWEIdia Looks like this is maybe related to AMI update:
Last successful build: https://github.com/pytorch/pytorch/actions/runs/10424302444/job/28872834620 uses AMI Name: amzn2-ami-hvm-2.0.20240306.2-x86_64-ebs

Current build takes 5hrs and uses:
https://github.com/pytorch/pytorch/actions/runs/10463836139/job/28976474949
AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64

cc: @ZainRizvi

yeah, this does seem related to the new Amazon Linux 2023 ami upgrade. The green builds all use the Amazon 2 AMI while the red ones are on Amazon 2023. @atalman is working on a mitigation PR

Post merge Step 4:
Missed steps: aarch64 binary validation (building was ok, validation of the binary in builder repo was not)

https://github.com/pytorch/builder/actions/runs/10776424717/job/29883178793#step:12:4025

Above aarch64 build failure is seen on cuda aarch64.
Error https://github.com/pytorch/builder/actions/runs/10776424717/job/29883178793#step:12:4025

++ python3 ./test/smoke_test/smoke_test.py --package torchonly
Traceback (most recent call last):
  File "/pytorch/builder/./test/smoke_test/smoke_test.py", line 9, in <module>
    import torch._dynamo
  File "/opt/conda/envs/conda-env-10794919545/lib/python3.9/site-packages/torch/_dynamo/__init__.py", line 3, in <module>
    from . import convert_frame, eval_frame, resume_execution
  File "/opt/conda/envs/conda-env-10794919545/lib/python3.9/site-packages/torch/_dynamo/convert_frame.py", line 53, in <module>
    from . import config, exc, trace_rules
  File "/opt/conda/envs/conda-env-10794919545/lib/python3.9/site-packages/torch/_dynamo/trace_rules.py", line 45, in <module>
    from .utils import getfile, hashable, NP_SUPPORTED_MODULES, unwrap_if_wrapper
ImportError: cannot import name 'NP_SUPPORTED_MODULES' from 'torch._dynamo.utils' (/opt/conda/envs/conda-env-10794919545/lib/python3.9/site-packages/torch/_dynamo/utils.py)

Will need to see if this is due to CUDA 12.4.1 upgrade and fix it for 2.5.0 release

cc @atalman