[Bug] Fix EAGLE draft CUDA-graph kv_indices under-allocation for topk > 1#27338
Merged
Conversation
hnyls2002
requested review from
Fridge003,
HaiShaw,
Qiaolin-Yu,
hebiao064,
ispobock and
merrymercy
as code owners
June 5, 2026 05:21
Contributor
|
Warning You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again! |
Collaborator
Author
|
/rerun-test test_spec_eagle_topk.py test_spec_eagle_stress.py |
Contributor
|
Results for 🚀 🚀 |
Contributor
|
Results for 🚀 🚀 |
kv_indices under-allocation for topk > 1
Contributor
|
Results for 🚀 🚀 |
Collaborator
Author
|
/tag-and-rerun-ci extra |
hnyls2002
added a commit
that referenced
this pull request
Jun 6, 2026
5 tasks
This was referenced Jun 6, 2026
This was referenced Jun 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
cuda_graph_kv_indicesbuffer by the missingtopkfactor (max_bs * max_context_len->max_bs * topk * max_context_len), matching the eagerinit_forward_metadatasizing.common_templateso an undersized row fails fast instead of silently corrupting memory.Bug
generate_draft_decode_kv_indicespackstopkper-branch sequences (prefix + draft) into each row, so a row needstopk * seq_lenentries; the CUDA-graph buffer omitted thetopkfactor that the eager path already has.speculative_eagle_topk > 1, onceseq_lens_sum * topk > max_bs * max_context_lenthe kernel writes past the row into adjacent graph-pool memory. The corruption is usually silent (wrong-but-in-vocab draft tokens); intermittently the stomped int32 indices land on the int64 draft token list, producing out-of-vocab ids that crash the target verify embedding gather (vectorized_gather_kernelindex-out-of-bounds).topkpacking, not page layout; demonstrated atpage_size = 1+topk = 8(see Repro). Addresses a flashinfertopk > 1draft coredump tracked in CUDA Coredump Tracker #26340.Fix
topkfactor (eager-equivalent).seq_lens_sum * topk + bs * num_steps <= row_widthhost-side before the kernel launch -- a cheap always-on check that turns the silent overflow into a deterministic, clearly-messaged failure and guards future sizing regressions.Repro
Isolates the bug to this buffer (flashinfer EAGLE
topk>1,page_size=1concurrent burst): https://gist.github.com/hnyls2002/3d20584c6d2f2af97fc9ff0e80472780With this PR the burst passes (buffer sized
max_bs * topk * max_context_len). Revert the* topkininit_cuda_graph_state(keeping the size invariant) and the first batch trips the invariant deterministically:page_size=1keeps the repro on the draftkv_indicesbuffer; a single request cannot overflow it, hence the concurrent burst.CI States
Latest PR Test (Base): 🚫 Run #27002608851
Latest PR Test (Extra): ❌ Run #27002608748