You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Bug]: DFlash2 spec-decode with a sliding-window drafter: admission (full_sequence_must_fit) and execution allocation both scale the drafter group's KV requirement with FULL sequence length -> admission deadlock above a block-size-dependent bound (~15.5K tokens at block-size 2304), engine crash on admission-clamp workarounds #55800
With a sliding-window drafter, the spec-decode admission check sums per-KV-group requirements min(cdiv(N, group_block_size), cap) against the UNIFIED free-block pool. The drafter's
SlidingWindowManager (block_size 16) contributes cdiv(N,16) — full-sequence-scaled, not window-scaled.
Above a pool-size-dependent bound (N ≥ ~15,537 at block-size 2304 with a 1,012-block pool), a single
request can never admit: it sits in WAITING for 600s (Running: 0, Waiting: 1, KV 0.0%) until the
request times out. The true sliding-window need is only cdiv(2048,16)+2 = 130 blocks.
Clamping the ADMISSION requirement to the window need (130) removes the wedge but the engine then
crashes EXECUTION-side: allocate_new_blocks computes num_required_blocks = cdiv(num_tokens, bs)
RAW for the same manager — mid-prefill of 15,787 tokens it demanded 475 new blocks (held 513) vs
~459 free → ValueError: Cannot get 475 free blocks from the pool → engine death. The remove_skipped_blocks mechanism IS wired and called per chunk with correct values (frees 383–406
blocks/call), but execution re-consumes blocks at full-sequence scale every chunk — held never drops.
Pool block-count scales with the flag at constant BYTES (~2.33M block-tokens); the drafter's
requirement is flag-invariant in TOKENS → the token bound moves with block size (non-monotone
in raw block terms: 7 blocks@2304 wedge, 15–17 blocks@1024 pass, because the pool also changes).
Group0 (indexer attention) block-size follows the flag WITH A FLOOR (1024 → group0 2048).
The fix (validated)
For KV managers with sliding_window set, cap the requirement at the window need in BOTH paths:
Validated on the repro stack (block-2304, dflash n=7): smokes PASS at 8,528 / 15,787 / 25,344 prompt
tokens AND 63,734 / 127,952 / 256,388 tokens (the practical context ceiling of a 262,144-token
deployment), all with coherent output, engine exit=0, zero errors. Execution trace post-fix: SW manager
cdiv=130, new=130 (pre-fix: 513/988 at 15.8K; raw cdiv 16,028 at 256K). Quality beyond 4 short coherent
completions not yet evaluated (smoke-level sanity only).
Expected behavior
A sliding-window drafter's KV requirement should scale with the window (≈130 blocks for sliding_window=2048 at bs=16), not with full sequence length — admission should not deadlock
and execution should not exhaust the pool for requests well within max-model-len.
Reproduced on a stack with DFlash2 registration present (stock vLLM images lack DFlash2DraftModel,
so the drafter cannot boot without it — verified). The PINNED MECHANISM
(full_sequence_must_fit vs unified pool; SlidingWindowManager full-sequence-scaled execution
allocation) lives in core vLLM v1 KV-cache code (kv_cache_utils.py, single_type_kv_cache_manager.py), not in model-registration code. A stock-upstream repro requires
a vLLM-main source build with DFlash2 (not yet done).
All raws / traces / patched files are public: canada-quant/glm53-w4a16-fp8-mtp — bench/raw/louse-bench/df2w*/ (28 legs, incl. df2w26 validation + patched single_type_kv_cache_manager.py and df2w27 full-context validation), findings chain findings/df2w*-2026-09-04.md (24+ legs).
Activity