Add cuda graph implementation for NV TRT RTX EP #25787
Merged
+267
−137
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…there is no impact on perf
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds comprehensive CUDA Graph support to the NV TensorRT RTX Execution Provider to improve inference performance by reducing per-kernel launch overhead and enabling better GPU throughput for repeated inference runs.
- Implements graph annotation ID-based CUDA graph management for multi-graph support
- Adds automatic detection and disabling of CUDA graphs for unsupported scenarios (shape tensors)
- Refactors stream management to support both user-provided and internally created streams
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| nv_execution_provider.h | Adds CUDA graph method declarations and per-thread context data structures |
| nv_execution_provider.cc | Implements core CUDA graph logic with capture/replay functionality and stream management |
| cuda_graph.h | Adds overloaded Replay method signature for sync flag support |
| cuda_graph.cc | Implements sync flag support in CUDA graph replay functionality |
| nv_provider_options.h | Updates CUDA graph enable option name for consistency |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc
Outdated
Show resolved
Hide resolved
onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc
Outdated
Show resolved
Hide resolved
Contributor
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
Contributor
Author
|
@jywu-msft can you mark this for 1.23 release? |
umangb-09
force-pushed
the
umang/cuda_graph_msr_main
branch
from
08ad1d7 to
49a8b7f
Compare
Contributor
Author
FIxed |
Contributor
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
chilo-ms
reviewed
Aug 22, 2025
chilo-ms
reviewed
Aug 22, 2025
umangb-09
force-pushed
the
umang/cuda_graph_msr_main
branch
from
aa82e20 to
a376b3c
Compare
gaugarg-nv
reviewed
Aug 25, 2025
Contributor
|
Please help resolve the conflicts |
umangb-09
force-pushed
the
umang/cuda_graph_msr_main
branch
from
4f3aa35 to
c4fd393
Compare
Contributor
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,Windows x64 QNN CI Pipeline |
|
Azure Pipelines successfully started running 5 pipeline(s). |
chilo-ms
approved these changes
Aug 26, 2025
Contributor
|
open and reopen for the specific CI to run |
Contributor
Author
|
@microsoft-github-policy-service agree company="NVIDIA" |
snnn
pushed a commit
that referenced
this pull request
Aug 28, 2025
### Description This change adds CUDA Graph support to the NV TensorRT RTX Execution Provider (EP). ### Motivation and Context Integrating CUDA Graphs into the NV TRT RTX EP provides: Lower latency by minimizing per-kernel launch overhead. Better throughput for repeated inference runs. Improved efficiency on GPUs with high kernel launches overhead sensitivity. --------- Co-authored-by: Maximilian Mueller <[email protected]> Co-authored-by: Gaurav Garg <[email protected]>
snnn
pushed a commit
that referenced
this pull request
Aug 29, 2025
- **Relax WeightBiasQuantization constraint for larger QDQ node group (#25673)** - **Add cuda graph implementation for NV TRT RTX EP (#25787)** - **python GPU IO Bindings for NVIDIA (#25776)** - **Fixes for DynamicQuantizeMatMul and Attention3D tests (#25814)** - **Fix a long standing bug on file memory mapping on windows. (#25833)** - **Add API for precompiled model compatibility check using just the compat info (#25841)** - **Enable ABSL_FLAGS flag registration for onnxruntime_perf_test for mobile build (#25849)** - **Add default constructor to Ort::Status. (#25860)** - #25871 - #25878 - #25884 - #25886 - #25866
Contributor
|
The change is added to the release branch |
gedoensmax
added a commit
to gedoensmax/onnxruntime
that referenced
this pull request
Sep 2, 2025
### Description This change adds CUDA Graph support to the NV TensorRT RTX Execution Provider (EP). ### Motivation and Context Integrating CUDA Graphs into the NV TRT RTX EP provides: Lower latency by minimizing per-kernel launch overhead. Better throughput for repeated inference runs. Improved efficiency on GPUs with high kernel launches overhead sensitivity. --------- Co-authored-by: Maximilian Mueller <[email protected]> Co-authored-by: Gaurav Garg <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This change adds CUDA Graph support to the NV TensorRT RTX Execution Provider (EP).
Motivation and Context
Integrating CUDA Graphs into the NV TRT RTX EP provides:
Lower latency by minimizing per-kernel launch overhead.
Better throughput for repeated inference runs.
Improved efficiency on GPUs with high kernel launches overhead sensitivity.