Skip to content

[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) #166480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
eqy wants to merge 2 commits into from
Closed

[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) #166480

eqy wants to merge 2 commits into from
+2 −2

Conversation

@eqy
Copy link
Collaborator

@eqy eqy commented Oct 29, 2025 •
edited by pytorch-bot bot
Loading

@eqy eqy added module: cudnn Related to torch.backends.cudnn, and CuDNN support module: cuda Related to torch.cuda, and CUDA support in general module: convolution Problems related to convolutions (THNN, THCUNN, CuDNN) open source topic: not user facing topic category labels Oct 29, 2025
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 29, 2025 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/166480

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit ab4e945 with merge base afaaaa3 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added the module: cpu CPU specific problem (e.g., perf, algorithm) label Oct 29, 2025
Skylion007
Skylion007 previously approved these changes Oct 29, 2025
View reviewed changes
@Skylion007
Copy link
Collaborator

Skylion007 commented Oct 29, 2025 •
edited
Loading

Shouldn't there be a CUDNN_FRONTEND version guard though instead of deleting the code?

@malfet
Copy link
Contributor

malfet commented Oct 29, 2025

@eqy but can you add a test, to make sure we'll not regress again?

@malfet malfet dismissed Skylion007’s stale review October 29, 2025 18:20

I think feedback about cudnn frontend verison guard is valid as well as unit test

@eqy
Copy link
Collaborator Author

eqy commented Oct 29, 2025

Test for this case was already checked in here: e2817ac#diff-31c5c90e1292af7427be151ba6c4aca280793122d3a2010698aeb18a4f69a508
Will just update the runtime version guard for now

@malfet
Copy link
Contributor

malfet commented Oct 29, 2025

@eqy IMO instead of deleting the check completely, you need to change the cudnn_version here (as with dynamic linking dev can choose to install pytorch with older cudnn (or newer))

@eqy eqy added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 30, 2025
@eqy eqy mentioned this pull request Oct 30, 2025
Open
Skylion007
Skylion007 reviewed Oct 30, 2025
View reviewed changes
@eqy
Copy link
Collaborator Author

eqy commented Oct 30, 2025

@pytorchmergebot merge

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 30, 2025

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@eqy eqy mentioned this pull request Oct 31, 2025
Open
BoyuanFeng pushed a commit that referenced this pull request Oct 31, 2025
@eqy @BoyuanFeng
[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) (#…
1a67a1e 
…166480)

Pull Request resolved: #166480
Approved by: https://github.com/Skylion007, https://github.com/malfet
@ngimel ngimel added this to the 2.9.1 milestone Nov 1, 2025
@Lucaskabela
Copy link
Contributor

Lucaskabela commented Nov 3, 2025

@pytorchbot cherry-pick --onto release/2.9 --fixes "4x performance regressions for 3d convs with AMP" -c regression

@Lucaskabela Lucaskabela linked an issue Nov 3, 2025 that may be closed by this pull request
4x performance regression for 3D convs with AMP on torch 2.9.0 #166122
Open
pytorchbot pushed a commit that referenced this pull request Nov 3, 2025
@eqy @pytorchbot
[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) (#…
55af469 
…166480)

Pull Request resolved: #166480
Approved by: https://github.com/Skylion007, https://github.com/malfet

(cherry picked from commit df71b70)
@pytorchbot
Copy link
Collaborator

pytorchbot commented Nov 3, 2025

Cherry picking #166480

The cherry pick PR is at #166908 and it is linked with issue 4x performance regressions for 3d convs with AMP. The following tracker issues are updated:

Details for Dev Infra team Raised by workflow job

@pytorchbot pytorchbot mentioned this pull request Nov 3, 2025
Closed
@eqy
Copy link
Collaborator Author

eqy commented Nov 4, 2025

@Lucaskabela we cannot cherrypick this without a cuDNN version bump...
Otherwise we will dispatch to broken kernels. This issue should also be reflected in existing CI tests.

@Lucaskabela
Copy link
Contributor

Lucaskabela commented Nov 4, 2025

Okay I will close this for now - once we have that version bump please submit the cherry pick :)

@eqy
Copy link
Collaborator Author

eqy commented Nov 4, 2025

@Lucaskabela we will discuss this in the core team sync meeting tomorrow... don't think we can bump a cuDNN backend version in a patch release.

etaf pushed a commit to etaf/pytorch-inductor-xpu that referenced this pull request Nov 4, 2025
@eqy @etaf
[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) (p…
137ca8d 
…ytorch#166480)

Pull Request resolved: pytorch#166480
Approved by: https://github.com/Skylion007, https://github.com/malfet
@eqy
Copy link
Collaborator Author

eqy commented Nov 4, 2025

@Lucaskabela OK, after discussion in the meeting I think you can proceed with the cherrypick, as the reenablement is guarded based on cuDNN runtime version.
We don't upgrade the cuDNN runtime version in the release matrix but will recommend users who face the performance regression to upgrade their local cuDNN package as a workaround.

@eqy eqy mentioned this pull request Nov 4, 2025
Open
atalman pushed a commit that referenced this pull request Nov 4, 2025
@pytorchbot @eqy
[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) (#…
e0c8ff1 
…166908)

[cuDNN][conv] Re-enable cuDNN for 3D convolutions (fixed in 9.15+) (#166480)

Pull Request resolved: #166480
Approved by: https://github.com/Skylion007, https://github.com/malfet

(cherry picked from commit df71b70)

Co-authored-by: Eddie Yan <[email protected]>
@atalman atalman mentioned this pull request Nov 10, 2025
Open
38 tasks
jovan2009 referenced this pull request in comfyanonymous/ComfyUI Nov 14, 2025
@comfyanonymous
Pytorch is stupid. (#10398)
b4f30bd
@jovan2009 jovan2009 mentioned this pull request Nov 14, 2025
Open
@jovan2009 jovan2009 mentioned this pull request Nov 21, 2025
Closed
1 task
@saberrroool saberrroool mentioned this pull request Nov 27, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Skylion007 Skylion007 Skylion007 approved these changes

@malfet malfet malfet approved these changes

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged module: convolution Problems related to convolutions (THNN, THCUNN, CuDNN) module: cpu CPU specific problem (e.g., perf, algorithm) module: cuda Related to torch.cuda, and CUDA support in general module: cudnn Related to torch.backends.cudnn, and CuDNN support open source topic: not user facing topic category

Projects

None yet

Milestone

2.9.1

Development

Successfully merging this pull request may close these issues.

4x performance regression for 3D convs with AMP on torch 2.9.0

7 participants

@eqy @Skylion007 @malfet @pytorchmergebot @Lucaskabela @pytorchbot @ngimel