👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors.

You ask your reviewers to trigger select CI tests on top of fastcheck CI.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

🚀

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jmamou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @jmamou.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@benchislett @tomasruizt

Update: Padding Fix and Async DSL Sync support

Changes since initial submission

Two issues were addressed after the initial benchmarks were posted:

1. Draft-token padding fix (commit e9c5588a3 — "trim scheduler draft slots to actual early-exit count"):
   When DSL exited early (fewer than num_speculative_tokens tokens drafted), the CPU token list
   returned to the scheduler still contained the full padded width. This caused the verifier to
   process padding positions that were never actually drafted, wasting verification compute on every
   early exit. The GPU tensor is intentionally kept at [B, K] (the model runner uses a fixed
   stride of num_speculative_tokens when indexing into the flattened draft-token tensor), but
   the CPU list is now trimmed to the actual early-exit count before being handed to the scheduler.

2. Async early-exit loop (commits 1447128dc + current HEAD):
   The original ConfidenceThresholdPolicy called .item() on a GPU scalar at each draft step,
   causing blocking host–device synchronisations per request. The new design moves all
   comparison state to GPU tensors (update() uses only masked_fill_ / logical_or_) and
   uses the Isolating While Condition
   pattern to both eliminate blocking syncs:

   • update() performs GPU tensor ops then kicks off an async 1-byte D→H copy of the exit
     flag (_exited_gpu) to a pinned CPU buffer on a dedicated CUDA stream.
   • The loop checks policy.exited at the top of the next iteration. Python loop overhead
     (slot mapping, metadata rebuild) is sufficient for the copy to complete, so
     event.synchronize() has near-zero additional latency.
   • On exit, the draft-token tensor is F.pad-ded back to [B, K] so that
     _prepare_input_ids keeps its fixed num_speculative_tokens stride.

   This reduces blocking syncs to near-free async syncs.

Updated Benchmarks

Hardware: 1× NVIDIA A100 80 GB PCIe
vLLM version: v0.16.0rc2.dev1480+gee9dc6d87 (branch dsl, commit d35682337 — async early-exit loop)
Server flags: --max-num-seqs 1 --gpu-memory-utilization 0.95 --max-model-len 2048
Metric: Output token throughput (tok/s)

Note on metric: These results report output token throughput only (matching vllm bench serve
default output). The original PR description reported total token throughput (input + output).
Speedup ratios are unaffected since all modes share the same input.

Set A — mt-bench, 10 prompts, max_concurrency=1, output_len=1024

Model pair 1: meta-llama/Llama-3.1-8B-Instruct + meta-llama/Llama-3.2-1B-Instruct

Best DSL configuration: 20 tokens, threshold=0.8 → 163.72 tok/s, 1.80× vs target, 2.74× vs SD

SD/5 is slower than target (0.66×) at output_len=1024 — a known characteristic of speculative
decoding at long outputs where the overhead of generating and verifying K draft tokens
outweighs the gains. DSL/20/0.8 avoids this by exiting early, achieving 1.80× vs target.

Model pair 2: facebook/opt-6.7b + facebook/opt-125m

Best DSL configuration: 20 tokens, threshold=0.8 → 210.56 tok/s, 2.20× vs target, 1.09× vs SD

Set B — mt-bench, 80 prompts, output_len=512, max_concurrency=1

Model pair 1: meta-llama/Llama-3.1-8B-Instruct + meta-llama/Llama-3.2-1B-Instruct

Best DSL configuration: 10 tokens, threshold=0.6 → 102.00 tok/s, 2.52× vs target, 1.39× vs SD

With output_len=512 at single concurrency, SD/5 already gives a large boost (1.81×) because
mt-bench prompts are highly predictable. DSL improves further by exiting early once confidence
drops, recouping the overhead of larger draft budgets.

Set C — random synthetic, 80 prompts, input_len=128, output_len=128, max_concurrency=8

Note: The server is configured with --max-num-seqs 1, which serialises all requests
regardless of client concurrency. The async early-exit loop significantly improves high-k
DSL performance even in this setting by skipping up to K−k_valid draft forward passes per
step. A rerun with --max-num-seqs ≥ max_concurrency would show further gains from
batch-level confidence averaging.

Model pair 1: meta-llama/Llama-3.1-8B-Instruct + meta-llama/Llama-3.2-1B-Instruct

Set D — mt-bench, 80 prompts, output_len=512, max_concurrency=4

Note: Same --max-num-seqs 1 caveat as Set C applies here. As in Set C, the async
early-exit loop substantially improves high-k DSL configurations.

Model pair 1: meta-llama/Llama-3.1-8B-Instruct + meta-llama/Llama-3.2-1B-Instruct

Best DSL configuration: 15 tokens, threshold=0.8 → 222.73 tok/s, 1.91× vs target, 1.44× vs SD

Observations

• Async early-exit impact (Sets D & E): The most dramatic improvement from the early-exit
  loop is visible at high concurrency with large K. Set C DSL/20/0.6 improves from 21.36 → 219.00
  tok/s (+925%); Set D DSL/15/0.8 improves from 34.42 → 222.73 tok/s (+547%). At k=20
  with typical exits at step 3–5, the loop now skips 15–17 forward passes per decode step — the
  dominant cost in these configurations. The 1-byte async D→H copy adds negligible overhead.

• Single-concurrency scenarios (Sets A & C): DSL consistently beats SD at c=1.
  Set B: 2.52× vs target, 1.39× vs SD (k=10/τ=0.6). Set A Llama: DSL/20/0.8 at
  1.80× vs target, 2.74× vs SD — note SD/5 is itself slower than target (0.66×) at
  output_len=1024, showing that DSL's early exit rescues speculative decoding at long outputs.

• OPT: DSL beats SD at mt-bench: Set A OPT DSL/20/0.8 achieves 2.20× vs target,
  1.09× vs SD (210.56 tok/s).

• Best confirmed speedups (output tok/s):

  • Set C Llama random (c=8): 3.00× vs target, 1.57× vs SD (DSL k=20/τ=0.6)
  • Set D Llama mt-bench (c=4): 1.91× vs target, 1.44× vs SD (DSL k=15/τ=0.8)
  • Set B Llama mt-bench (c=1): 2.52× vs target, 1.39× vs SD (DSL k=10/τ=0.6)
  • Set A Llama mt-bench (c=1, long): 1.80× vs target, 2.74× vs SD (DSL k=20/τ=0.8)
  • Set A OPT mt-bench (c=1): 2.20× vs target, 1.09× vs SD (DSL k=20/τ=0.8)

• Threshold / token-count sensitivity: DSL/20/0.6 excels on short-output high-concurrency
  workloads (Set C: 3.00×) where early exit triggers quickly and skips most drafts. High
  thresholds (0.8) with moderate K (15) are the most consistently safe choice across diverse
  workloads.

The speedups are super interesting! A couple of questions:

• Synthetic random inputs don't really reflect acceptance lengths under realistic conditions, right? Not sure about how much weight metrics with these inputs should carry. Let me know if you see it differently.
• How did you decide for K=5 for SD? Depending on models and dataset type different Ks are optimal. E.g. in some configs SD is slower than vanilla decoding, using lower K would have helped here. It seems that users still are expected to tune num_spec_tokens K and the threshold c, so the baseline SD should also choose optimal K for a fair comparison.
• Did you analyze performance over changing batch size? If not, why?
• What is the mental model for the importance of output_length? Less time spent in prefill?
  If answers to my questions are in the papers, just let me know 👍

@tomasruizt thanks for the valuable comments!

• Synthetic random inputs don't really reflect acceptance lengths under realistic conditions, right? Not sure about how much weight metrics with these inputs should carry. Let me know if you see it differently.

Correct. Synthetic inputs are used here primarily to measure throughput under controlled load. They do understate acceptance rates compared to real workloads. I plan to run additional exp on real workloads.

• How did you decide for K=5 for SD? Depending on models and dataset type different Ks are optimal. E.g. in some configs SD is slower than vanilla decoding, using lower K would have helped here. It seems that users still are expected to tune num_spec_tokens K and the threshold c, so the baseline SD should also choose optimal K for a fair comparison.

I agree the baseline should also be evaluated at its optimal K. Note that that I reported also vanilla (target row). Anyway, I plan to add a sweep over K for baseline SD.

• Did you analyze performance over changing batch size? If not, why?

Our benchmarks were run at different concurrencies but I have not yet swept batch size. I will add a batch size sweep.

• What is the mental model for the importance of output_length? Less time spent in prefill?

Correct, for short outputs, prefill dominates and DSL savings are diluted. For long outputs, decode dominates and DSL's per-step savings compound over more steps, making the speedup more visible. This is why we report output_length as a variable — it shows where DSL's benefit saturates and is most relevant.

@jmamou About the output_length point: If you compare the TPOT metric rather than tok/s you can cleanly separate the decoding performance gains of DSL from the prefill phase (which is captured in the TTFT metric).

@tomasruizt I ran additional experiments with optimal K for SD, real workloads and added TPOT.

I added evaluations on ShareGPT and mt-bench to complement the random-input results (see below).
I ran also a K sweep (K=3,4,5,6,7) on a held-out train split (20 prompts, seed=42) for each dataset to select the optimal K for SD, then evaluated on a separate test set (80 prompts, seed=123).

Train-phase K sweep (total token throughput, tok/s):

We applied the same train/test discipline for DSL, running a grid search over K∈{5,10,15,20} × τ∈{0.4,0.6,0.8} on the same train split:

• ShareGPT optimal: K=15, τ=0.8
• mt-bench optimal: K=10, τ=0.6

Test-set results (80 prompts, seed=123, c=1, output_len=512, Llama-3.1-8B + Llama-3.2-1B)

DSL with train-optimised (K, τ) outperforms best-K SD by 34–42% in throughput and 6–18% in TPOT on held-out test data across both real-text datasets.

DSL still requires tuning (K, τ), so the number of hyperparameters is the same as SD+K. The practical advantage is that DSL with a reasonably large K (e.g. K=15) and a moderate τ (e.g. 0.6–0.8) is robust across datasets: the confidence threshold self-regulates at runtime by exiting early when the draft model is uncertain. A fixed-K SD with suboptimal K has no such recourse — e.g. SD/K=7 drops to 0.65× vanilla throughput on ShareGPT test (slower than vanilla decoding), while DSL/K=15/τ=0.8 is at 1.65×.

@jmamou Thanks for the quick reply! Two questions:

• In table 1, SD achieves 222 tok/s throughput on ShareGPT, but then only 116 tok/s on table 2. What am I missing?
• Also in table 1, mt-bench: the crash from K=6 to K=7 is really surprising! Do you know what is going on?

@tomasruizt

• In table 1, SD achieves 222 tok/s throughput on ShareGPT, but then only 116 tok/s on table 2. What am I missing?

The vLLM server has enable_prefix_caching=True. Train-phase job (table 1) ran vllm bench serve
twice on the same 20 ShareGPT prompts (seed=42) without restarting the server:

I picked the last (2nd invacation) value, so the cache-inflated 222 tok/s was reported.

The test run (table 2, 80 diverse prompts, seed=123) also has enable_prefix_caching=True but sees
0.0% hits — 80 prompts (~16K input tokens) fill and evict the cache as they run, so there
is nothing to reuse. That gives the clean 116 tok/s.

The K-selection remains valid: all K values in the train sweep share the same warm-cache
second-run bias, so the relative ranking (K=7 best for ShareGPT) is unaffected. The train
numbers should not be reported as absolute throughput.

• Also in table 1, mt-bench: the crash from K=6 to K=7 is really surprising! Do you know what is going on?

The same issue — K=6 and K=7 ran in the same server session, so their prefix cache states diverged:

The TPOT is almost identical — the entire 2.4× gap comes from TTFT. K=6 got a warm cache (65 ms
prefill); K=7's extra draft-token allocation pressure evicted the cached KV blocks during
warmup, leaving it with cold prefill (4913 ms).

The K=6 selection from the train sweep is not confirmed. The lower TPOT for K=7 (6.15 ms vs 6.47 ms) and
higher accepted length (4.73 vs 3.88) suggest K=7 might actually be better on a clean
measurement.

@jmamou Thanks for being so transparent! I suggest using vllm serve ... --no-enable-prefix-caching and focusing on the TPOT metric. Otherwise, we are mixing up speedups with KV-cache hits, and looking at inflated throughput metrics.

I have looked through the benchmarks here and I feel there is more noise than signal. Let's align directly, either via DM or in the spec decoding SIG meeting. My high-level concerns are:

• Unfair comparisons against draft model setups with unusual choices of K, inexhaustive sweep is leaving out a lot of nuance.
• No comparisons with EAGLE3
• Unreliable benchmarking methodology (synthetic data, ShareGPT are poor choices of dataset)
• Synchronizations, even when offset by one iteration of drafting, will not be enough overlap to hide all the CPU overhead (e.g. prepare_inputs). In such cases, this may not be sufficient to provide speedup compared to optimal-K EAGLE speculative decoding at low batch sizes.
• At higher batch sizes, I feel like the mean K per turn will be fairly consistent due to the large sample set of which we are taking the average (admittedly, this is assuming a somewhat homogeneous workload)

@jmamou Hi jmamou, thank you for your excellent work. I am currently conducting tests based on your commits, running the Llama-8B + Eagle-3 configuration on vLLM version 0.18.1. The results from two specific experimental groups—one with draft_tokens=10 and DSL_rate=0.0, and the other with draft_tokens=10 and DSL_rate=0.7—show virtually no speedup. Would it be possible for you to perform a comparative experiment on the Llama-8B + Eagle-3 setup? Thank you.

@jmamou Hi jmamou, thank you for your excellent work. I am currently conducting tests based on your commits, running the Llama-8B + Eagle-3 configuration on vLLM version 0.18.1. The results from two specific experimental groups—one with draft_tokens=10 and DSL_rate=0.0, and the other with draft_tokens=10 and DSL_rate=0.7—show virtually no speedup. Would it be possible for you to perform a comparative experiment on the Llama-8B + Eagle-3 setup? Thank you.

@easonfzw, is my understanding correct that vanilla SD (DSL_rate=0.0) doesn't speed up autoregressive in your setup? Have you been able to measure any speedup of DSL (DSL_rate=0.7) over vanilla SD (DSL_rate=0.0) for draft_tokens=10?

@keyboardAnt Thank you for responding to my issue. Based on my current experiments—conducted using Eagle3 with draft_tokens=10—the vanilla SD configuration (DSL_rate=0.0) demonstrates a speedup compared to running without SD. Furthermore, the results for vanilla SD with DSL_rate=0.0 and DSL_rate=0.7 are comparable; specifically, setting DSL_rate=0.7 did not yield any additional speedup relative to DSL_rate=0.0. @jmamou Hi could you help on this? thanks a lot.

@keyboardAnt Thank you for responding to my issue. Based on my current experiments—conducted using Eagle3 with draft_tokens=10—the vanilla SD configuration (DSL_rate=0.0) demonstrates a speedup compared to running without SD. Furthermore, the results for vanilla SD with DSL_rate=0.0 and DSL_rate=0.7 are comparable; specifically, setting DSL_rate=0.7 did not yield any additional speedup relative to DSL_rate=0.0. @jmamou Hi could you help on this? thanks a lot.

@easonfzw, can you please try to rerun your benchmark with a larger cap on the number of drafts (e.g., draft_tokens=100)?

@easonfzw Thanks for running this — the result is consistent with what we'd expect for DSL in this regime, and it's a useful data point.

DSL's speedup comes from skipping draft forward passes when the draft model's confidence collapses early. The savings scale roughly as (K_max − k_effective) × draft_forward_cost. With Eagle3 at draft_tokens=10 two effects compress that gap:

1. Each Eagle3 draft step is much cheaper than a separate draft-model forward pass — it's a single transformer layer reusing the target's hidden state, no full prefill/decode pipeline. So even when DSL correctly skips drafts, the wall-clock saving per skipped draft is small, and closer in magnitude to DSL's own bookkeeping (extra logits read, mask ops, async D→H copy, padding back to [B, K]).
2. Eagle3's logit confidences may not calibrate to acceptance the way a standard draft model's do. DSL's exit signal is essentially softmax(logits).max() > τ, used as a proxy for acceptance probability. But Eagle3 is trained to predict target hidden states, not to produce well-calibrated next-token distributions, so the threshold may fire at the wrong points relative to actual verifier rejections.

So at K=10 with Eagle3, DSL ≈ SD is the expected outcome.

To be upfront: DSL is a weaker fit for Eagle than for draft-model SD. The cheap-draft point (1) caps DSL's upper-bound benefit regardless of how well it's tuned, and the calibration point (2) just means τ values that work for draft-model SD probably don't transfer — Eagle3's logits still correlate with acceptance (they go through the target LM head), they're just noisier as an exit signal, so τ would need to be re-swept for Eagle3 specifically. Where DSL could still help with Eagle is at very large K_max (50, 100) where many skips × small per-skip savings can still beat the bookkeeping overhead, on long-tail prompts where Eagle's accepted length collapses, or at high batch sizes where unused draft positions cost the verifier. At moderate K and low batch size — the regime you're testing — DSL has little headroom by construction.

To see whether DSL recovers anything in your setup, two things would help:

• Increase draft_tokens (e.g. 20, 50, 100, as @keyboardAnt already suggested). DSL only pays off when K_max is materially above the natural optimum so there is wasted-draft headroom for it to reclaim. At Eagle3's natural K (≈3–6 on most workloads), DSL has little to do by construction.

• Disable prefix caching and report TPOT, not throughput:

  vllm serve ... --no-enable-prefix-caching

  Otherwise prefix-cache hit-rate drift between runs can dominate the throughput delta and obscure the actual decode-side effect (this came up earlier in the thread with the draft-model setup).

If you can share TPOT numbers for:

• (a) Eagle3 vanilla SD at its optimal K, vs
• (b) Eagle3 + DSL at (K, τ) swept over K ∈ {10, 20, 50} × τ ∈ {0.4, 0.6, 0.8},

all with prefix caching off and a fixed seed, that would be the cleanest comparison.

Could you also share which device you ran on? The DSL bookkeeping vs. draft-skip trade-off shifts with device compute/memory bandwidth, so that affects the interpretation.