[Profiler] Add CUDA Overhead to Auto-trace #142271
Closed
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/142271
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 40845a2 with merge base d3d1a78 ( UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D66904879 |
✅ Deploy Preview for chimerical-cranachan-793287 ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
ngimel
approved these changes
Dec 6, 2024
sraikund16
force-pushed
the
export-D66904879
branch
from
e52cb9b to
02e2c3e
Compare
pytorch-bot bot
pushed a commit
that referenced
this pull request
Dec 10, 2024
Summary:
We already have CUDA OVERHEAD events enabled in on-demand so we should also add them to auto-trace
Test Plan:
Tested using servicelab and found no performance difference:
kineto_benchmark
duration_ms: 21668
number_of_events: 26542
profiler_prepare_call_duration_us: 970
profiler_enable_call_duration_us: 616474
profiling_window_duration_us: 2188525
profiler_disable_call_duration_us: 148628
parse_kineto_call_duration_us: 1672536
function_events_build_tree_call_duration_us: 285939
kineto_benchmark
duration_ms: 21718
number_of_events: 26556
profiler_prepare_call_duration_us: 885
profiler_enable_call_duration_us: 7037
profiling_window_duration_us: 1772481
profiler_disable_call_duration_us: 174122
parse_kineto_call_duration_us: 1983683
function_events_build_tree_call_duration_us: 333582
Differential Revision: D66904879
Contributor
|
This pull request was exported from Phabricator. Differential Revision: D66904879 |
Contributor
Author
|
@pytorchbot rebase |
Collaborator
|
@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here |
Collaborator
Contributor
Author
|
@pytorchbot rebase -b main |
Collaborator
|
@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here |
Summary:
We already have CUDA OVERHEAD events enabled in on-demand so we should also add them to auto-trace
Test Plan:
Tested using servicelab and found no performance difference:
kineto_benchmark
duration_ms: 21668
number_of_events: 26542
profiler_prepare_call_duration_us: 970
profiler_enable_call_duration_us: 616474
profiling_window_duration_us: 2188525
profiler_disable_call_duration_us: 148628
parse_kineto_call_duration_us: 1672536
function_events_build_tree_call_duration_us: 285939
kineto_benchmark
duration_ms: 21718
number_of_events: 26556
profiler_prepare_call_duration_us: 885
profiler_enable_call_duration_us: 7037
profiling_window_duration_us: 1772481
profiler_disable_call_duration_us: 174122
parse_kineto_call_duration_us: 1983683
function_events_build_tree_call_duration_us: 333582
Differential Revision: D66904879
Collaborator
|
Successfully rebased |
pytorchmergebot
force-pushed
the
export-D66904879
branch
from
02e2c3e to
40845a2
Compare
Contributor
Author
|
@pytorchmergebot merge |
Collaborator
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
mori360
pushed a commit
to mori360/pytorch
that referenced
this pull request
Dec 11, 2024
Summary: We already have CUDA OVERHEAD events enabled in on-demand so we should also add them to auto-trace Test Plan: Tested using internal performance suites and found no noticeable performance change Differential Revision: D66904879 Pull Request resolved: pytorch#142271 Approved by: https://github.com/ngimel
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary: We already have CUDA OVERHEAD events enabled in on-demand so we should also add them to auto-trace
Test Plan: Tested using internal performance suites and found no noticeable performance change
Differential Revision: D66904879