Repository navigation
[Windows] Depend on PyPI CUDA runtime packages in wheels - #194056
akarale-nv wants to merge 6 commits into
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/194056
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit d8bb29a with merge base d2f3a9d ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
802e898 to
52229ae
Compare
|
@pytorchbot label "module: windows" |
janeyx99
added
the
triaged
09e7057 to
bbc4d95
Compare
bbc4d95 to
2403632
Compare
2403632 to
64a045d
Compare
|
Hi maintainers, gentle ping for review when you have bandwidth. This PR significantly reduces Windows CUDA wheel size (~277 MB) and avoids duplicate CUDA DLLs across packages, improving mixed-package deployments. Feedback on the packaging approach and Windows CI validation would be greatly appreciated. Thank you! @Skylion007 I have fixed the linter errors as well. The windows native version of lintrunner keeps failing silently, hence missed this. Can you please help re-trigger the pipelines? |
64a045d to
83e29fe
Compare
atalman
added
ciflow/binaries
|
|
|
The following ciflow label(s) have been added but CI has not been triggered yet because the workflows are awaiting approval:
Once a maintainer approves the workflows (scroll to the bottom of the PR page), the corresponding CI jobs will be triggered automatically. Please ping one of the reviewers if you do not have access to approve and run workflows. |
|
HI @akarale-nv we need to make sure all the build and test pass |
83e29fe to
a7ca8be
Compare
|
Hi @atalman, seems like between the rebases I missed updating the pypi CuDNN version for Windows. It should be good to go now. Good thing we have a test case to verify that :D |
b9ff57f to
7a8a7d8
Compare
|
The current 13.4 test failures are due to the mismatch in CTK patch version of AMI and the wheel dependency. Since the new AMI is on CTK 13.4.2, re-running this after rebasing should fix the issue. This issue will also be fixed for good next CTK release onwards |
7a8a7d8 to
0b630ec
Compare
atalman
left a comment
atalman
left a comment
There was a problem hiding this comment.
lgtm, please merge once windows builds are passing. We should target this to 2.16 release
Windows CUDA wheels locate their runtime DLL directories from installed NVIDIA wheel packages before configuring the DLL search path.
Test Plan:
```
python -c 'compile(open("torch/__init__.py", encoding="utf-8").read(), "torch/__init__.py", "exec")'
git diff --cached --check
```
This commit was authored with assistance from an AI assistant.
Configure the Windows CUDA wheel matrix with versioned NVIDIA runtime requirements, validate those requirements in smoke tests, and avoid duplicating the runtime package when CMake writes the wheel metadata. Test Plan: ``` python .github/scripts/generate_binary_build_matrix.py git diff --check git diff --cached --check ``` This commit was authored with assistance from an AI assistant.
Load CUDA runtime DLLs from NVIDIA PyPI packages through cuda-pathfinder, and keep Windows binary wheel requirements aligned with the CUDA installers by using cuDNN 9.26.0.51. CUDA installations without cuda-pathfinder used to fail while importing torch; warn and continue to the existing DLL loader when the pathfinder import fails. Test Plan: ```powershell .\.venv\Scripts\python.exe -m py_compile torch/__init__.py wsl.exe --cd /mnt/c/akarale/git/pytorch bash -lc "source .venv-wsl/bin/activate && lintrunner -a --skip FLAKE8 torch/__init__.py" git diff --check ``` An isolated check of the pathfinder block with mocked imports verified both the available and missing-pathfinder branches. FLAKE8 is skipped in WSL because the /mnt/c mount reports this non-executable file as executable. This commit was authored with assistance from an AI assistant.
The Windows wheel matrix inherited CUDA 13.4.1 from the Linux wheel pin,
while the Windows CI installer uses 13.4.2. Pin the Windows wheel to the
installer version and regenerate the Windows workflow. Keep the CUDA 13.2
wheel pin at 13.2.2, matching the updated installer on main.
Test Plan:
```powershell
.\.venv\Scripts\python.exe -m py_compile .github/scripts/generate_binary_build_matrix.py
.\.venv\Scripts\python.exe -c "import runpy; x=runpy.run_path('.github/scripts/generate_binary_build_matrix.py')['PYTORCH_EXTRA_INSTALL_REQUIREMENTS']; assert [x[v]['windows'].split('==')[1].split(';')[0] for v in ('13.2','13.4')] == ['13.2.2','13.4.2']"
wsl.exe --cd /mnt/c/akarale/git/pytorch bash -lc "source .venv-wsl/bin/activate && lintrunner -a .github/scripts/generate_binary_build_matrix.py && lintrunner -a --skip EXEC .github/workflows/generated-windows-binary-wheel-nightly.yml"
git diff --check
```
This commit was authored with assistance from an AI assistant.
0b630ec to
605d6f2
Compare
|
@pytorchbot merge |
|
Pull workflow has not been scheduled for the PR yet. It could be because author doesn't have permissions to run those or skip-checks keywords were added to PR/commits, aborting merge. Please get/give approval for the workflows and/or remove skip ci decorators before next merge attempt. If you think this is a mistake, please contact PyTorch Dev Infra. |
|
The Nvidia Windows OOT CI has discovered 2 corner cases:
|
a3045c6 to
701ff12
Compare
701ff12 to
d8bb29a
Compare
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Windows CUDA wheels built on main depend on the cuda-toolkit pip package and no longer bundle the CUDA or cuDNN DLLs (#194056), so cmake/PostBuildSteps.cmake never copies zlibwapi.dll into them. Drop that block and the install_zlib step in cuda_install.bat that downloads the 1.2.3 DLL into System32 for it. Related to #201626. Please open this PR as a draft and mark it ready for review once the description and code are ready to be looked at. Before submitting, please review: - [PR lifecycle](https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#pr-lifecycle) in the contributing guide - [AI-Assisted Development](https://github.com/pytorch/pytorch/blob/main/AI_POLICY.md) policy Go through the following checklist - [ ] Passes lint (`spin fixlint`) - [ ] Added/updated tests - [ ] Updated documentation (if applicable) - [ ] Included benchmark results (for PRs impacting perf) Created with Claude Pull Request resolved: #201873 Approved by: https://github.com/Skylion007, https://github.com/huydhn
Summary
This PR teaches Windows PyTorch CUDA wheels to locate their CUDA runtime DLLs from installed NVIDIA PyPI packages (via cuda-pathfinder) rather than bundling them inside torch/lib/. It adds per-OS wheel dependency metadata, updates the build matrix generator, adjusts CMake to skip bundling when using PyPI-sourced runtimes, and adds a Windows smoke test that validates the declared cuda-toolkit version matches the build.
torch/__init__.py: On Windows CUDA builds, cuda-pathfinder is called to load each CUDA libraryBuild matrix / workflows: PYTORCH_EXTRA_INSTALL_REQUIREMENTS is restructured from flat strings to {linux, windows} dicts; Windows CUDA entries gain cuda-toolkit, cuda-pathfinder, nvidia-cudnn-cuXX, nvidia-cusparselt-cuXX, and nvidia-nvjitlink-cuXX requirements. CPU and macOS entries that incorrectly carried Linux-scoped requirements are cleaned up.
Windows and Linux use different CUDA Toolkit patch versions and dependency sets, so the binary-workflow generator now represents those requirements independently.
cmake/PostBuildSteps.cmake: CUDA DLL bundling is conditionally skipped for PyPI wheel builds; NvToolsExt install is moved outside that guard, so it is always included for CUDA builds regardless of packaging mode.
smoke_test.py: The Windows smoke test checks the full
major.minor.patchCUDA Toolkit version because older CUPTI DLL naming does not provide compatible discovery across all patch versions. This is addressed upstream with future CTK versions.Motivation
This reduces Windows torch wheel size to 277 MB and avoids cross-DLL-boundary problems when multiple Python packages in one process ship their own CUDA DLL copies. Relying on shared PyPI CUDA packages avoids duplicate runtime copies and improves mixed-package deployments.
Test Plan
smoke_test.py.cc @peterjc123 @mszhanyi @skyline75489 @nbcsm @iremyux @Blackhex @nkhasbag-nv