Skip to content

[Bug]: DFlash2 spec-decode with a sliding-window drafter: admission (full_sequence_must_fit) and execution allocation both scale the drafter group's KV requirement with FULL sequence length -> admission deadlock above a block-size-dependent bound (~15.5K tokens at block-size 2304), engine crash on admission-clamp workarounds #55800

Open

Description

@pasta-paul

Environment

  • vLLM v1, version 0.1.dev20051+g487ecf187 (image vllm/vllm-openai:glm53-flash-x86_64-cu130@sha256:2e771fa615452282cc331eb418b3ef21636fce355bea0491fca89e6d362ab703)
  • DFlash2 speculative decoding (dflash, num_speculative_tokens=7), drafter config sliding_window=2048, use_sliding_window=true
  • 8×H100, TP=4, --max-model-len 262144, default --max-num-batched-tokens, --no-enable-prefix-caching
  • Model: GLM-5.3-class MoE + DFlash2 drafter (windowed drafter)

Summary

With a sliding-window drafter, the spec-decode admission check sums per-KV-group requirements
min(cdiv(N, group_block_size), cap) against the UNIFIED free-block pool. The drafter's
SlidingWindowManager (block_size 16) contributes cdiv(N,16) — full-sequence-scaled, not window-scaled.
Above a pool-size-dependent bound (N ≥ ~15,537 at block-size 2304 with a 1,012-block pool), a single
request can never admit: it sits in WAITING for 600s (Running: 0, Waiting: 1, KV 0.0%) until the
request times out. The true sliding-window need is only cdiv(2048,16)+2 = 130 blocks.

Clamping the ADMISSION requirement to the window need (130) removes the wedge but the engine then
crashes EXECUTION-side: allocate_new_blocks computes num_required_blocks = cdiv(num_tokens, bs)
RAW for the same manager — mid-prefill of 15,787 tokens it demanded 475 new blocks (held 513) vs
~459 free → ValueError: Cannot get 475 free blocks from the pool → engine death. The
remove_skipped_blocks mechanism IS wired and called per chunk with correct values (frees 383–406
blocks/call), but execution re-consumes blocks at full-sequence scale every chunk — held never drops.

Measured bounds (actual prompt tokens, deterministic generator)

block size last PASS first WEDGE pool blocks (free) predicted bound
2304 15,523 ~15,556 (est.) 1,012 (1,011) N ≥ 15,537 ✓
1024 17,404 ~17,600 (est.) ~2,000 ~17,537 ✓ (measured bracket contains it)
1536 clears 16K–24K (16 points, zero wedges) — 1,516 SW cap 1,153 saturates first ✓

Pool block-count scales with the flag at constant BYTES (~2.33M block-tokens); the drafter's
requirement is flag-invariant in TOKENS → the token bound moves with block size (non-monotone
in raw block terms: 7 blocks@2304 wedge, 15–17 blocks@1024 pass, because the pool also changes).
Group0 (indexer attention) block-size follows the flag WITH A FLOOR (1024 → group0 2048).

The fix (validated)

For KV managers with sliding_window set, cap the requirement at the window need in BOTH paths:

# get_num_blocks_to_allocate (admission) AND allocate_new_blocks (execution):
if getattr(self, "sliding_window", None):
    num_required_blocks = min(num_required_blocks,
                              cdiv(self.sliding_window, self.block_size.block_size) + 2)

Validated on the repro stack (block-2304, dflash n=7): smokes PASS at 8,528 / 15,787 / 25,344 prompt
tokens AND 63,734 / 127,952 / 256,388 tokens (the practical context ceiling of a 262,144-token
deployment), all with coherent output, engine exit=0, zero errors. Execution trace post-fix: SW manager
cdiv=130, new=130 (pre-fix: 513/988 at 15.8K; raw cdiv 16,028 at 256K). Quality beyond 4 short coherent
completions not yet evaluated (smoke-level sanity only).

Expected behavior

A sliding-window drafter's KV requirement should scale with the window (≈130 blocks for
sliding_window=2048 at bs=16), not with full sequence length — admission should not deadlock
and execution should not exhaust the pool for requests well within max-model-len.

Related upstream work (swept; none covers this)

Disclosure

Reproduced on a stack with DFlash2 registration present (stock vLLM images lack DFlash2DraftModel,
so the drafter cannot boot without it — verified). The PINNED MECHANISM
(full_sequence_must_fit vs unified pool; SlidingWindowManager full-sequence-scaled execution
allocation) lives in core vLLM v1 KV-cache code (kv_cache_utils.py,
single_type_kv_cache_manager.py), not in model-registration code. A stock-upstream repro requires
a vLLM-main source build with DFlash2 (not yet done).

All raws / traces / patched files are public: canada-quant/glm53-w4a16-fp8-mtp —
bench/raw/louse-bench/df2w*/ (28 legs, incl. df2w26 validation + patched
single_type_kv_cache_manager.py and df2w27 full-context validation), findings chain
findings/df2w*-2026-09-04.md (24+ legs).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions