Repository navigation
[TPU/Compile] Fix warmup slowdown from CPython 3.12 frame-stack churn - #1037
Merged
Merged
Conversation
wyzhang
requested review from
Bob-Chen222,
alexnails,
angelayi,
colin2328,
hlFu,
pyc96,
qizzzh,
yliu120 and
zou3519
as code owners
October 4, 2026 22:39
wyzhang
force-pushed
the
wyzhang/exp/1001-compile-fast
branch
from
October 4, 2026 22:48
d787e56 to
0548a54
Compare
Collaborator
Author
|
@zou3519 This should probably address the issue you raised regarding slower precompile due to stack depth. |
qizzzh
approved these changes
Oct 4, 2026
zou3519
approved these changes
Oct 5, 2026
zou3519
left a comment
Collaborator
There was a problem hiding this comment.
nice.
Btw do we have compile time regression tests in sglang-torchtpu? (could we get someone to add some if not?)
Issue ----- Warmup compile time depended on Python stack depth: on CPython 3.12, a hot call that lands on a frame-stack chunk boundary allocates and frees memory on every call (python/cpython#142183), so an unrelated extra wrapper frame, such as the one #814 added, could slow precompile. What ---- Run each forward-pass warmup (prefill and decode graphs) inside a reserved frame-stack chunk so its speed no longer depends on the caller's stack depth. Results: Qwen3-30B warmup is 5.7% faster with an unlucky stack layout.
wyzhang
force-pushed
the
wyzhang/exp/1001-compile-fast
branch
from
October 5, 2026 01:45
0548a54 to
8de20b5
Compare
Collaborator
Author
Good point. Opened #1041 |
pyc96
approved these changes
Oct 5, 2026
4 of 5 tasks
dhansen-nvidia
added a commit
to dhansen-nvidia/TensorRT-LLM
that referenced
this pull request
Nov 12, 2026
CPython 3.11+ keeps frames in per-thread data-stack chunks. When a hot call sits at a chunk boundary, every call maps a fresh chunk and every return unmaps it (gh-142183). On GB200 each munmap also pays a Grace SMMU invalidation once NCCL has peer-mapped memory, so the churn costs milliseconds per executor step: MiniMax-M3 piecewise CUDA graph prefill steps spent about half their host time in munmap, all on the executor thread. CPython fixed this in 3.13.13 and 3.14.4; 3.12 will not get the fix. Add run_with_frame_stack_reserve() and call PyExecutor.event_loop through it, so the loop starts near the beginning of a large data-stack chunk and ordinary call depths never reach a boundary. The reserve is applied only on affected CPython versions and can be disabled with TLLM_DISABLE_FRAME_STACK_RESERVE=1. Cover the version gate, the disable switch, the churn itself and the executor wiring with cpu_only unit tests. Background: - python/cpython#142183 (the issue) - python/cpython#145789 (the fix; backports #145829 / #146123 for 3.13, #145828 for 3.14) - python/cpython#88756 (3.11 change that moved frames onto the data stack) - sgl-project/sglang-torchtpu#1037 (same technique) Signed-off-by: Dan Hansen <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue
Warmup compile time depended on Python stack depth: on CPython 3.12, a
hot call that lands on a frame-stack chunk boundary allocates and frees
memory on every call (python/cpython#142183), so an unrelated extra
wrapper frame, such as the one #814 added, could slow precompile.
What
Run each forward-pass warmup (prefill and decode graphs) inside a
reserved frame-stack chunk so its speed no longer depends on the
caller's stack depth.
Results: Qwen3-30B warmup is 5.7% faster with an unlucky stack layout.
CI States
Latest PR Test (Base): ✅ Run #37252699043
Latest PR Test (Extra): ❌ Blocked --
run-ciis required first.Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.