Skip to content

[TPU/Compile] Fix warmup slowdown from CPython 3.12 frame-stack churn - #1037

Merged
wyzhang merged 1 commit into
mainfrom
wyzhang/exp/1001-compile-fast
Oct 5, 2026
Merged

wyzhang merged 1 commit into
mainfrom
wyzhang/exp/1001-compile-fast

Conversation

@wyzhang

@wyzhang wyzhang commented Oct 4, 2026 •

Copy link
Copy Markdown
Collaborator

Issue

Warmup compile time depended on Python stack depth: on CPython 3.12, a
hot call that lands on a frame-stack chunk boundary allocates and frees
memory on every call (python/cpython#142183), so an unrelated extra
wrapper frame, such as the one #814 added, could slow precompile.

What

Run each forward-pass warmup (prefill and decode graphs) inside a
reserved frame-stack chunk so its speed no longer depends on the
caller's stack depth.

Results: Qwen3-30B warmup is 5.7% faster with an unlucky stack layout.


CI States

Latest PR Test (Base): ✅ Run #37252699043
Latest PR Test (Extra): ❌ Blocked -- run-ci is required first.
Latest PR Test (AMD ROCm 10): ➖ No AMD PR run found for this commit.

@wyzhang

wyzhang commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator Author

@zou3519 This should probably address the issue you raised regarding slower precompile due to stack depth.

@zou3519 zou3519 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice.

Btw do we have compile time regression tests in sglang-torchtpu? (could we get someone to add some if not?)

Issue
-----
Warmup compile time depended on Python stack depth: on CPython 3.12, a
hot call that lands on a frame-stack chunk boundary allocates and frees
memory on every call (python/cpython#142183), so an unrelated extra
wrapper frame, such as the one #814 added, could slow precompile.

What
----
Run each forward-pass warmup (prefill and decode graphs) inside a
reserved frame-stack chunk so its speed no longer depends on the
caller's stack depth.

Results: Qwen3-30B warmup is 5.7% faster with an unlucky stack layout.
@wyzhang

wyzhang commented Oct 5, 2026

Copy link
Copy Markdown
Collaborator Author

nice.

Btw do we have compile time regression tests in sglang-torchtpu? (could we get someone to add some if not?)

Good point. Opened #1041

@wyzhang
wyzhang merged commit 329f1c0 into main Oct 5, 2026
48 checks passed
@wyzhang
wyzhang deleted the wyzhang/exp/1001-compile-fast branch October 5, 2026 03:09
dhansen-nvidia added a commit to dhansen-nvidia/TensorRT-LLM that referenced this pull request Nov 12, 2026
CPython 3.11+ keeps frames in per-thread data-stack chunks. When a hot
call sits at a chunk boundary, every call maps a fresh chunk and every
return unmaps it (gh-142183). On GB200 each munmap also pays a Grace SMMU
invalidation once NCCL has peer-mapped memory, so the churn costs
milliseconds per executor step: MiniMax-M3 piecewise CUDA graph prefill
steps spent about half their host time in munmap, all on the executor
thread. CPython fixed this in 3.13.13 and 3.14.4; 3.12 will not get the
fix.

Add run_with_frame_stack_reserve() and call PyExecutor.event_loop through
it, so the loop starts near the beginning of a large data-stack chunk and
ordinary call depths never reach a boundary. The reserve is applied only on
affected CPython versions and can be disabled with
TLLM_DISABLE_FRAME_STACK_RESERVE=1. Cover the version gate, the disable
switch, the churn itself and the executor wiring with cpu_only unit tests.

Background:
- python/cpython#142183 (the issue)
- python/cpython#145789 (the fix; backports
  #145829 / #146123 for 3.13, #145828 for 3.14)
- python/cpython#88756 (3.11 change that moved
  frames onto the data stack)
- sgl-project/sglang-torchtpu#1037 (same technique)

Signed-off-by: Dan Hansen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants