Skip to content

[reland #136389] Skip kernel saving if already existed #137073

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
muchulee8 wants to merge 4 commits into from
Closed

[reland #136389] Skip kernel saving if already existed #137073

muchulee8 wants to merge 4 commits into from

Conversation

@muchulee8
Copy link
Contributor

@muchulee8 muchulee8 commented Oct 1, 2024 •
edited
Loading

Stack from ghstack (oldest at bottom):

Summary:
We skip the save_gpu_kernel if kernel is being saved already.
This would give us a more accurate Triton profiling result. The
following trace shows before/after the change for a benchmarking of a
trivial addmm:

Before:
Screenshot 2024-09-23 at 10 26 53 AM

After:
Screenshot 2024-09-23 at 10 27 03 AM

We can see that before the change, the benchmarking includes two parts,
(1) The overhead of our triton_heuristic call, which includes the
save/get, and the (expensive) hash computation.
(2) The exact computation of Triton kernel.

We see that (1) accounts >50% of time, which makes kernel selection
for profiling choosing aten kernels over Triton kernels.

Test Plan:
Existing OSS CI
python test/inductor/test_cuda_cpp_wrapper.py

Reviewers:

Subscribers:

Tasks:

Tags:

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @ColinPeppler @amjames @desertfire @chauhang

@muchulee8
[reland #136389] Skip kernel saving if already existed
1deafab 
Summary:
We skip the save_gpu_kernel if kernel is being saved already.
This would give us a more accurate Triton profiling result. The
following trace shows before/after the change for a benchmarking of a
trivial addmm:

Before:
After:

We can see that before the change, the benchmarking includes two parts,
   (1) The overhead of our triton_heuristic call, which includes the
   save/get, and the (expensive) hash computation.
   (2) The exact computation of Triton kernel.

   We see that (1) accounts >50% of time, which makes kernel selection
   for profiling choosing aten kernels over Triton kernels.

Test Plan:
Existing OSS CI

Reviewers:

Subscribers:

Tasks:

Tags:

[ghstack-poisoned]
@pytorch-bot
Copy link

pytorch-bot bot commented Oct 1, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137073

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (5 Unrelated Failures)

As of commit a76c1f5 with merge base d725758 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@muchulee8 muchulee8 requested a review from desertfire October 1, 2024 04:04
@muchulee8
Update on "[reland #136389] Skip kernel saving if already existed"
8595c80 
Summary:
We skip the save_gpu_kernel if kernel is being saved already.
This would give us a more accurate Triton profiling result. The
following trace shows before/after the change for a benchmarking of a
trivial addmm:

Before:
<img width="1255" alt="Screenshot 2024-09-23 at 10 26 53 AM" src="https://github.com/user-attachments/assets/5aea05ef-6ef0-464c-8da9-17b31c97b43a">

After:
<img width="910" alt="Screenshot 2024-09-23 at 10 27 03 AM" src="https://github.com/user-attachments/assets/488b7d4f-268f-41cf-8553-cb16ceeae118">

We can see that before the change, the benchmarking includes two parts,
   (1) The overhead of our triton_heuristic call, which includes the
   save/get, and the (expensive) hash computation.
   (2) The exact computation of Triton kernel.

   We see that (1) accounts >50% of time, which makes kernel selection
   for profiling choosing aten kernels over Triton kernels.

Test Plan:
Existing OSS CI
python test/inductor/test_cuda_cpp_wrapper.py

Reviewers:

Subscribers:

Tasks:

Tags:

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng ColinPeppler amjames desertfire chauhang

[ghstack-poisoned]
muchulee8 added a commit that referenced this pull request Oct 1, 2024
@muchulee8
[reland #136389] Skip kernel saving if already existed
0035b2d 
Summary:
We skip the save_gpu_kernel if kernel is being saved already.
This would give us a more accurate Triton profiling result. The
following trace shows before/after the change for a benchmarking of a
trivial addmm:

Before:
After:

We can see that before the change, the benchmarking includes two parts,
   (1) The overhead of our triton_heuristic call, which includes the
   save/get, and the (expensive) hash computation.
   (2) The exact computation of Triton kernel.

   We see that (1) accounts >50% of time, which makes kernel selection
   for profiling choosing aten kernels over Triton kernels.

Test Plan:
Existing OSS CI

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: b920462
Pull Request resolved: #137073
@muchulee8
Copy link
Contributor Author

muchulee8 commented Oct 1, 2024

@pytorchbot rebase -b main

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 1, 2024

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

@muchulee8
Update
1d2795e 
[ghstack-poisoned]
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 1, 2024

Successfully rebased gh/muchulee8/37/orig onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/137073)

@muchulee8
Copy link
Contributor Author

muchulee8 commented Oct 2, 2024

@pytorchbot rebase -b main

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 2, 2024

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

@muchulee8
Update
a76c1f5 
[ghstack-poisoned]
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 2, 2024

Successfully rebased gh/muchulee8/37/orig onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/137073)

pytorchmergebot pushed a commit that referenced this pull request Oct 2, 2024
@muchulee8
[reland #136389] Skip kernel saving if already existed
181e30a 
Summary:
We skip the save_gpu_kernel if kernel is being saved already.
This would give us a more accurate Triton profiling result. The
following trace shows before/after the change for a benchmarking of a
trivial addmm:

Before:
After:

We can see that before the change, the benchmarking includes two parts,
   (1) The overhead of our triton_heuristic call, which includes the
   save/get, and the (expensive) hash computation.
   (2) The exact computation of Triton kernel.

   We see that (1) accounts >50% of time, which makes kernel selection
   for profiling choosing aten kernels over Triton kernels.

Test Plan:
Existing OSS CI

Reviewers:

Subscribers:

Tasks:

Tags:

ghstack-source-id: b182949
Pull Request resolved: #137073
@muchulee8
Copy link
Contributor Author

muchulee8 commented Oct 2, 2024

@pytorchbot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Oct 2, 2024
@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Oct 2, 2024

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

AnantGulati pushed a commit to AnantGulati/pytorch that referenced this pull request Oct 2, 2024
@muchulee8 @AnantGulati
[reland pytorch#136389] Skip kernel saving if already existed (pytorc…
34749ce 
…h#137073)

Summary:
We skip the save_gpu_kernel if kernel is being saved already.
This would give us a more accurate Triton profiling result. The
following trace shows before/after the change for a benchmarking of a
trivial addmm:

Before:
<img width="1255" alt="Screenshot 2024-09-23 at 10 26 53 AM" src="https://github.com/user-attachments/assets/5aea05ef-6ef0-464c-8da9-17b31c97b43a">

After:
<img width="910" alt="Screenshot 2024-09-23 at 10 27 03 AM" src="https://github.com/user-attachments/assets/488b7d4f-268f-41cf-8553-cb16ceeae118">

We can see that before the change, the benchmarking includes two parts,
   (1) The overhead of our triton_heuristic call, which includes the
   save/get, and the (expensive) hash computation.
   (2) The exact computation of Triton kernel.

   We see that (1) accounts >50% of time, which makes kernel selection
   for profiling choosing aten kernels over Triton kernels.

Test Plan:
Existing OSS CI
python test/inductor/test_cuda_cpp_wrapper.py

Reviewers:

Subscribers:

Tasks:

Tags:

Pull Request resolved: pytorch#137073
Approved by: https://github.com/desertfire
@github-actions github-actions bot deleted the gh/muchulee8/37/head branch November 6, 2024 02:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@desertfire desertfire desertfire approved these changes

Assignees

No one assigned

Labels

ciflow/inductor ciflow/slow ciflow/trunk Trigger trunk jobs on your pull request Merged module: inductor release notes: inductor

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@muchulee8 @pytorchmergebot @desertfire