Skip to content

[Bug] Fix EAGLE draft CUDA-graph kv_indices under-allocation for topk > 1#27338

Merged
hnyls2002 merged 6 commits into
mainfrom
lsyin/flashinfer-draft-kv-indices-topk
Jun 5, 2026
Merged

[Bug] Fix EAGLE draft CUDA-graph kv_indices under-allocation for topk > 1#27338
hnyls2002 merged 6 commits into
mainfrom
lsyin/flashinfer-draft-kv-indices-topk

Conversation

@hnyls2002

@hnyls2002 hnyls2002 commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Widen the multi-step draft CUDA-graph cuda_graph_kv_indices buffer by the missing topk factor (max_bs * max_context_len -> max_bs * topk * max_context_len), matching the eager init_forward_metadata sizing.
  • Add an always-on size invariant in common_template so an undersized row fails fast instead of silently corrupting memory.

Bug

  • generate_draft_decode_kv_indices packs topk per-branch sequences (prefix + draft) into each row, so a row needs topk * seq_len entries; the CUDA-graph buffer omitted the topk factor that the eager path already has.
  • For speculative_eagle_topk > 1, once seq_lens_sum * topk > max_bs * max_context_len the kernel writes past the row into adjacent graph-pool memory. The corruption is usually silent (wrong-but-in-vocab draft tokens); intermittently the stomped int32 indices land on the int64 draft token list, producing out-of-vocab ids that crash the target verify embedding gather (vectorized_gather_kernel index-out-of-bounds).
  • The undersizing is about the topk packing, not page layout; demonstrated at page_size = 1 + topk = 8 (see Repro). Addresses a flashinfer topk > 1 draft coredump tracked in CUDA Coredump Tracker #26340.

Fix

  • Size the CUDA-graph buffer with the topk factor (eager-equivalent).
  • Invariant: assert seq_lens_sum * topk + bs * num_steps <= row_width host-side before the kernel launch -- a cheap always-on check that turns the silent overflow into a deterministic, clearly-messaged failure and guards future sizing regressions.

Repro

Isolates the bug to this buffer (flashinfer EAGLE topk>1, page_size=1 concurrent burst): https://gist.github.com/hnyls2002/3d20584c6d2f2af97fc9ff0e80472780

With this PR the burst passes (buffer sized max_bs * topk * max_context_len). Revert the * topk in init_cuda_graph_state (keeping the size invariant) and the first batch trips the invariant deterministically:

AssertionError: EAGLE draft kv_indices row too small: need 84280 but row width is 32768 (topk=8, num_seqs=7, seq_lens_sum=10500, num_steps=5)

page_size=1 keeps the repro on the draft kv_indices buffer; a single request cannot overflow it, hence the concurrent burst.


CI States

Latest PR Test (Base): 🚫 Run #27002608851
Latest PR Test (Extra): ❌ Run #27002608748

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@hnyls2002

hnyls2002 commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator Author

/rerun-test test_spec_eagle_topk.py test_spec_eagle_stress.py
(3 tries)

@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_spec_eagle_topk.py test_spec_eagle_stress.py:

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle_topk.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle_stress.py

@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_spec_eagle_topk.py test_spec_eagle_stress.py:

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle_topk.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle_stress.py

@hnyls2002 hnyls2002 changed the title Fix EAGLE multi-step draft CUDA graph kv_indices under-allocation for topk>1 [Bug] Fix EAGLE draft CUDA-graph kv_indices under-allocation for topk > 1 Jun 5, 2026
@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Results for /rerun-test test_spec_eagle_topk.py test_spec_eagle_stress.py:

🚀 1-gpu-5090 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle_topk.py

🚀 1-gpu-h100 (1 test): ✅ View workflow run

cd test/ && python3 registered/spec/eagle/test_spec_eagle_stress.py

@hnyls2002

Copy link
Copy Markdown
Collaborator Author

/tag-and-rerun-ci extra

@hnyls2002
hnyls2002 merged commit 2c2a4f2 into main Jun 5, 2026
124 of 133 checks passed
@hnyls2002
hnyls2002 deleted the lsyin/flashinfer-draft-kv-indices-topk branch June 5, 2026 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant