Skip to content

[Executor] Default use CUDAGraph #3594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jiang-Jia-Jun merged 50 commits into from
Oct 21, 2025
Merged

[Executor] Default use CUDAGraph #3594

Jiang-Jia-Jun merged 50 commits into from
Oct 21, 2025

Conversation

@gongshaotian
Copy link
Collaborator

@gongshaotian gongshaotian commented Aug 25, 2025 •
edited
Loading

Summary

  1. CUDAGraph has been enabled by default in some scenarios at the beginning of this PR.
  2. CUDAGraph will be automatically closed for functions that are not compatible with CUDAGraph (speculative decoding, RL training, multi-mode model). Added some GraphOptConfig related startup parameter checks:

User interface changes

The --use-cudagraph startup parameter is deleted, you can also manually control the CUDAGraph by setting --graph-optimization-config .

--graph-optimization-config '{"use_cudagraph":false}'

Detailed description reference graph_optimization.md

@paddle-bot
Copy link

paddle-bot bot commented Aug 25, 2025

Thanks for your contribution!

@gongshaotian gongshaotian changed the title [Config] Add GraphOptConfig start intercept [Config] Add GraphOptConfig start interception Aug 25, 2025
gzy19990617
gzy19990617 reviewed Aug 25, 2025
View reviewed changes
Copy link
Collaborator

@gzy19990617 gzy19990617 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

YuanRisheng
YuanRisheng previously approved these changes Aug 25, 2025
View reviewed changes
@codecov-commenter
Copy link

codecov-commenter commented Aug 25, 2025 •
edited
Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (develop@43d5bd6). Learn more about missing BASE report.

Additional details and impacted files 
@@            Coverage Diff             @@
##             develop    #3594   +/-   ##
==========================================
  Coverage           ?   62.50%           
==========================================
  Files              ?        2           
  Lines              ?       16           
  Branches           ?        7           
==========================================
  Hits               ?       10           
  Misses             ?        4           
  Partials           ?        2
Flag Coverage Δ
diff 62.50% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@gongshaotian gongshaotian changed the title [Config] Add GraphOptConfig start interception [Executor] Default use CUDAGraph Aug 25, 2025
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
gongshaotian
gongshaotian commented Aug 27, 2025
View reviewed changes
@gongshaotian gongshaotian self-assigned this Aug 28, 2025
@gongshaotian gongshaotian mentioned this pull request Sep 1, 2025
Closed
yuanlehome
yuanlehome approved these changes Oct 21, 2025
View reviewed changes
Copy link
Collaborator

@yuanlehome yuanlehome left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM👍

@gongshaotian
Copy link
Collaborator Author

gongshaotian commented Oct 21, 2025 •
edited
Loading

fastdeploy/model_executor/graph_optimization/utils.py：新增显存 Debug 工具，无须单测
fastdeploy/worker/gpu_model_runner.py 和 fastdeploy/config.py：新增防御性代码，非报错无法覆盖
申请豁免覆盖率@Jiang-Jia-Jun

@Jiang-Jia-Jun Jiang-Jia-Jun merged commit 775edcc into PaddlePaddle:develop Oct 21, 2025
23 of 28 checks passed
chang-wenbin
chang-wenbin reviewed Oct 21, 2025
View reviewed changes
benchmarks/yaml/eb45-32k-wint4-a800-tp4.yaml
max_model_len: 32768
max_num_seqs: 96
gpu_memory_utilization: 0.9
gpu_memory_utilization: 0.85
Copy link
Collaborator

@chang-wenbin chang-wenbin Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gpu_memory_utilization修改 也不会导致性能下降吗？

Copy link
Collaborator Author

@gongshaotian gongshaotian Oct 21, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gpu_memory_utilization修改 也不会导致性能下降吗？

从测试结果来看是上升的

This was referenced Oct 21, 2025
Merged
Merged
Merged
Merged
@gongshaotian gongshaotian deleted the start_intercept branch November 3, 2025 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chang-wenbin chang-wenbin chang-wenbin left review comments

@gzy19990617 gzy19990617 gzy19990617 left review comments

@yuanlehome yuanlehome yuanlehome approved these changes

@YuanRisheng YuanRisheng YuanRisheng left review comments

Assignees

@gongshaotian gongshaotian

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@gongshaotian @codecov-commenter @yuanlehome @YuanRisheng @chang-wenbin @gzy19990617 @Jiang-Jia-Jun