-
Notifications
You must be signed in to change notification settings - Fork 26.3k
[CUDA] fix indexing on large tensor causing nvalid configuration argument #164049
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/164049
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 611a148 with merge base a0136f1 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
@pytorchbot merge |
|
Pull workflow has not been scheduled for the PR yet. It could be because author doesn't have permissions to run those or skip-checks keywords were added to PR/commits, aborting merge. Please get/give approval for the workflows and/or remove skip ci decorators before next merge attempt. If you think this is a mistake, please contact PyTorch Dev Infra. |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Proper fix for #164048, fixes gather too, reverts #164049 Pull Request resolved: #166974 Approved by: https://github.com/eqy
Proper fix for #164048, fixes gather too, reverts #164049 Pull Request resolved: #166974 Approved by: https://github.com/eqy
Proper fix for #164048, fixes gather too, reverts #164049 Pull Request resolved: #166974 Approved by: https://github.com/eqy
…ment (#164049) Summary: Fixes #164048 Pull Request resolved: #164049 Approved by: https://github.com/eqy ============================================================================== PyTorch commit - 48a5470 Internal bug report - https://fb.workplace.com/groups/1405155842844877/permalink/26039640882303031/ Test Plan: Tested by pushing this and the next diff to https://www.internalfb.com/code/pytorch-pytorch/[llama4-flagship-bugfix][history]/ branch which is on top of the current llama4-flagship branch. Then built a conda finetune env using feedstock xlformers_finetune_conda_feedstock:0c420acf6878b60ab87c1cc4544e56a0 which was used to successfully run ARPG job https://fburl.com/mlhub/btt2cs8o. This job fails with the current conda env. Also updated the build.toml file to use the new branch and created a diff that ran the existing unit and e2e tests (D86340156) with it. Rollback Plan: Reviewers: xdwang, haoyc, kunalb, ngimel Reviewed By: xdwang Differential Revision: https://phabricator.intern.facebook.com/D86383742
Summary: Proper fix for #164048, fixes gather too, reverts #164049 Pull Request resolved: #166974 Approved by: https://github.com/eqy ======================================================================== PyTorch commit - 4271ffe There is one difference where CUDA_KERNEL_ASSERT is used instead of CUDA_KERNEL_ASSERT_VERBOSE in IndexKernelUtils.cu because that required porting a few more commits on 2.8. Test Plan: See previous diff (D86383742) for testing done Rollback Plan: Reviewers: xdwang, kunalb, haoyc Reviewed By: xdwang Differential Revision: https://phabricator.intern.facebook.com/D86383794
Fixes #164048
cc @ptrblck @msaroufim @eqy @jerryzh168