[Spec] Fix fa3 EAGLE draft-decode expand page_table scatter OOB for topk>1 + page_size>1#27360
Merged
Conversation
hnyls2002
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
hebiao064,
ispobock and
merrymercy
as code owners
June 5, 2026 07:54
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request prevents out-of-bounds writes in the draft-decode page table scatter by slicing cache_loc to the active decode_length. It also introduces an optional async assertion (_ASSERT_DRAFT_EXPAND_OOB) to guard against out-of-bounds errors. The reviewer pointed out that if positions is empty (e.g., during empty batches), calling .max() on it will raise a runtime error, and suggested using torch.nn.functional.pad to safely handle empty tensors without causing graph breaks under torch.compile.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Collaborator
Author
|
/rerun-test test_self_e2e_pd_perturb.py |
Contributor
|
Results for 🚀 |
5 tasks
hnyls2002
added a commit
to Zhiy-Zhang/sglang
that referenced
this pull request
Jun 8, 2026
5 tasks
ch-wan
added a commit
that referenced
this pull request
Jun 13, 2026
_apply_cuda_graph_metadata has no callers since init_forward_metadata_out_graph converged onto the use_bound seam (its body was inlined verbatim as the use_bound=True path of _compute_forward_metadata). Pure deletion. Repoint the debug-only PR #27360 revert toggle (SGLANG_DEBUG_REVERT_PR, never active in normal runs) from the removed _apply_cuda_graph_metadata to _compute_forward_metadata, where the matched `cache_loc = cache_loc[:, :decode_length]` line now lives unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ch-wan
added a commit
that referenced
this pull request
Jun 14, 2026
_apply_cuda_graph_metadata has no callers since init_forward_metadata_out_graph converged onto the use_bound seam (its body was inlined verbatim as the use_bound=True path of _compute_forward_metadata). Pure deletion. Repoint the debug-only PR #27360 revert toggle (SGLANG_DEBUG_REVERT_PR, never active in normal runs) from the removed _apply_cuda_graph_metadata to _compute_forward_metadata, where the matched `cache_loc = cache_loc[:, :decode_length]` line now lives unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ch-wan
added a commit
that referenced
this pull request
Jun 20, 2026
_apply_cuda_graph_metadata has no callers since init_forward_metadata_out_graph converged onto the use_bound seam (its body was inlined verbatim as the use_bound=True path of _compute_forward_metadata). Pure deletion. Repoint the debug-only PR #27360 revert toggle (SGLANG_DEBUG_REVERT_PR, never active in normal runs) from the removed _apply_cuda_graph_metadata to _compute_forward_metadata, where the matched `cache_loc = cache_loc[:, :decode_length]` line now lives unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ch-wan
added a commit
that referenced
this pull request
Jun 20, 2026
_apply_cuda_graph_metadata has no callers since init_forward_metadata_out_graph converged onto the use_bound seam (its body was inlined verbatim as the use_bound=True path of _compute_forward_metadata). Pure deletion. Repoint the debug-only PR #27360 revert toggle (SGLANG_DEBUG_REVERT_PR, never active in normal runs) from the removed _apply_cuda_graph_metadata to _compute_forward_metadata, where the matched `cache_loc = cache_loc[:, :decode_length]` line now lives unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ch-wan
added a commit
that referenced
this pull request
Jun 20, 2026
_apply_cuda_graph_metadata has no callers since init_forward_metadata_out_graph converged onto the use_bound seam (its body was inlined verbatim as the use_bound=True path of _compute_forward_metadata). Pure deletion. Repoint the debug-only PR #27360 revert toggle (SGLANG_DEBUG_REVERT_PR, never active in normal runs) from the removed _apply_cuda_graph_metadata to _compute_forward_metadata, where the matched `cache_loc = cache_loc[:, :decode_length]` line now lives unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
ch-wan
added a commit
that referenced
this pull request
Jun 20, 2026
_apply_cuda_graph_metadata has no callers since init_forward_metadata_out_graph converged onto the use_bound seam (its body was inlined verbatim as the use_bound=True path of _compute_forward_metadata). Pure deletion. Repoint the debug-only PR #27360 revert toggle (SGLANG_DEBUG_REVERT_PR, never active in normal runs) from the removed _apply_cuda_graph_metadata to _compute_forward_metadata, where the matched `cache_loc = cache_loc[:, :decode_length]` line now lives unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FlashAttentionMultiStepBackendbuilds the EAGLE draft-decode expand metadata viadraft_decode_set_expand_metadata, which scatters compacted page indices into a step-tight(decode_length + 1)-columnpage_table. Thepage_size > 1branch scattered the fullnum_steps-widecache_locinstead of slicing it todecode_length(thepage_size == 1branch already slices). So fortopk > 1the per-branch draft slots span more distinct pages than the row holds, andscatter_writes out of bounds — silently corrupting the cuda-graph pool and surfacing later as an illegal memory access / NaN logits.Fix: slice
cache_loc[:, :decode_length]at the (non-compiled) cuda-graph metadata call site, matching thepage_size == 1path, so the scatter fills at mostdecode_lengthof thedecode_length + 1columns. An always-on size assert (cache_loc.shape[1] <= page_tablecolumns) right before the scatter makes any future regression fail fast at the source instead of silently corrupting memory — no env flag, notorch.compilegraph break.Repro
Standalone repro (real spec server, EAGLE3
topk=8+page_size=2+ fa3): https://gist.github.com/hnyls2002/909e761ff0c7b67b824a88e59b198874page_size=2makes one branch's contiguous draft slots span more distinct pages than thedecode_length + 1row, so a single request trips the first draft-decode step.SGLANG_DEBUG_REVERT_PR=27360 python repro_fa3_draft_expand_page_table_overflow.pyreverse-applies the fix; the always-on assert then fires deterministically:AssertionError: draft expand page_table too narrow: cache_loc width 5 > 2 columns (decode_length + 1); page_size=2, topk=8, num_steps=5.CI States
Latest PR Test (Base): 🚫 Run #27055762654
Latest PR Test (Extra): ✅ Run #27055762627