[CI][CUDA][cuSPARSELt] cusparselt 0.6.3 and cu121 related cleanups #145696
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/145696
Note: Links to docs will display an error until the docs builds have been completed. ❌ 2 New Failures, 6 Cancelled Jobs, 72 PendingAs of commit 52354fd with merge base b808774 ( NEW FAILURES - The following jobs have failed:
CANCELLED JOBS - The following jobs were cancelled. Please retry:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Skylion007
approved these changes
Jan 27, 2025
Collaborator
Author
|
I may need to get rid of lingering cu121 jobs before merging this |
Skylion007
reviewed
Jan 27, 2025
| CUSPARSELT_NAME="libcusparse_lt-linux-${arch_path}-0.5.2.1-archive" | ||
| CUSPARSELT_NAME="libcusparse_lt-linux-${arch_path}-0.6.2.3-archive" | ||
| curl --retry 3 -OLs https://developer.download.nvidia.com/compute/cusparselt/redist/libcusparse_lt/linux-${arch_path}/${CUSPARSELT_NAME}.tar.xz | ||
| elif [[ ${CUDA_VERSION:0:4} == "11.8" ]]; then |
Collaborator
There was a problem hiding this comment.
This matching error would have been caught a bit easier if you made sure the last else in this chain raised an error.
Collaborator
Author
There was a problem hiding this comment.
Sorry, could you please clarify? cuda matching error, or cusparselt matching error?
Where is the "matching"?
Collaborator
There was a problem hiding this comment.
Oh where no CUSPARSELT_NAME name is selected, and there it's blank. IE: None of the if else are matched
Collaborator
There was a problem hiding this comment.
I think he is referring to if none of the "if condition" matched, need to raise an CUSPARLT_NAME not defined error in a separate "else" clause.
Collaborator
Author
|
cc @huydhn for help reviewing test_trymerge and test_filter* changes, thanks! |
huydhn
approved these changes
Jan 27, 2025
Contributor
huydhn
left a comment
huydhn
left a comment
There was a problem hiding this comment.
There is still one failed test https://hud.pytorch.org/pr/pytorch/pytorch/145696#36245113047, but overall LGTM! Please fix the test before landing
nWEIdia
changed the title
eqy
approved these changes
Jan 27, 2025
(pytorch#104312) to test the API. Since the old PR has cu121, I cannot remove cu121 occurences here. Some other cases, it is just testing remove suffix. So it is fine to not use cu121 but cu124, well let's just use cu126
31 tasks
…pytorch#145568) Will be used in inductor. Pull Request resolved: pytorch#145568 Approved by: https://github.com/malfet, https://github.com/Skylion007
…)" This reverts commit c7b2f7d. Reverted pytorch#136780 on behalf of https://github.com/izaitsevfb due to internal regression ([comment](pytorch#136780 (comment)))
…#145488) pytorch#144989 This fixes tts_angular model on torchbench for `--export-aot-inductor` I put meta function in cpp, as shape calculation requires cudnn API calls. I've extracted shape calculation to be used in implementation as this logic has some non-trivial actions and comments. ``` └─ $ python benchmarks/dynamo/torchbench.py --only tts_angular --accuracy --no-translation-validation --inference --bfloat16 --export-aot-inductor --disable-cudagraphs --device cuda loading model: 0it [00:00, ?it/s]WARNING:common:Model tts_angular does not support bfloat16, running with amp instead loading model: 0it [00:01, ?it/s] WARNING:common:Model tts_angular does not support bfloat16, running with amp instead cuda eval tts_angular WARNING:common:Model tts_angular does not support bfloat16, running with amp instead pass ``` Pull Request resolved: pytorch#145488 Approved by: https://github.com/eqy, https://github.com/zou3519
Let TORCHINDUCTOR_FX_GRAPH_CACHE=0 being respected in unit test. This is helpful if I want the compilation to happen for testing. Setting INDUCTOR_TEST_DISABLE_FRESH_CACHE to 1 is not the same, since that will cause the generated wrapper file being deleted. But we may want to check those files after running a test. Pull Request resolved: pytorch#141195 Approved by: https://github.com/masnesral, https://github.com/desertfire
Signed-off-by: Edward Z. Yang <[email protected]> Pull Request resolved: pytorch#145579 Approved by: https://github.com/gchanan
…torch#145572) Summary: I think skipping these tests is suboptimal. If we categorize as expected failures, then we'll see test failures when they start passing, which means they're more likely to be removed. As a skip, they quietly continue to skip. Pull Request resolved: pytorch#145572 Approved by: https://github.com/Skylion007, https://github.com/zou3519
Test Plan: CI D68155591 Pull Request resolved: pytorch#144773 Approved by: https://github.com/izaitsevfb
nWEIdia
requested review from
Chillee,
albanD,
bdhirsh,
ezyang,
fmassa,
jbschlosser,
justinchuby,
kshitij12345,
kulinseth,
malfet,
mruberry,
shubhambhokare1,
soulitzer,
soumith,
sraikund16,
titaiwangms,
wschin and
zou3519
as code owners
pytorch-bot
bot
added
ciflow/inductor
ciflow/mps
Collaborator
Author
|
Oops, sorry for the disaster. Closing. |
pytorchmergebot
pushed a commit
that referenced
this pull request
Jan 28, 2025
…145793) Make ci cusparselt installation be consistent with nightly binary Remove cu121 related docker build jobs and inductor runs Update test failures relating to cu121 Retry of #145696 Pull Request resolved: #145793 Approved by: https://github.com/eqy, https://github.com/tinglvv
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fix inconsistency between CD (nightly binary) and CI (what ci jobs install for cu126)
Remove remaining cu121 ci/docker jobs
cc @H-Huang @awgu @kwen2501 @wanchaol @fegin @fduwjj @wz337 @wconstab @d4l3k @c-p-i-o @mcarilli @ptrblck @leslie-fang-intel @jgong5 @voznesenskym @penguinwu @EikanWang @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov @LucasLLC @MeetVadakkanchery @mhorowitz @pradeepfn @ekr0 @xmfan @atalman @malfet @eqy @tinglvv