Repository navigation
[Bug]: VLLM_BATCH_INVARIANT=1 returns non-identical logprobs across repeats of a byte-identical concurrent workload at ~44 co-resident sequences #51187
Description
Summary
The batch invariance docs state that the feature "ensures that the output of a model is deterministic and independent of the batch size or the order of requests in a batch."
The clause at issue here is deterministic. With VLLM_BATCH_INVARIANT=1, repeating a byte-identical concurrent workload against the same unchanged server process produces different logprobs once enough sequences are co-resident. Five repeats of one workload produced three distinct outputs.
Setting --max-num-seqs 8 on the identical workload makes it fully reproducible (5/5 identical). That is the only knob I found that changes the outcome.
This is a correctness/reproducibility report, not a security report. I could not determine the mechanism from outside the engine, so much of this issue is what I was able to rule out, which I hope is the useful part.
Reproduction
A client sends N requests concurrently (thread pool), waits for all, and repeats 5 times against the same server. Between repeats nothing changes: same prompts, same token ids, same submission order, same sampling params, warm cache, no restart. A discarded priming pass with byte-identical traffic runs before repeat 0, so repeat 0 is not the first time the server sees this workload.
- greedy only,
temperature=0,seed=0,max_tokens=24,logprobs=5 - observable: emitted token ids, plus the chosen-token logprob and top-5 alternatives with their ids, compared as exact doubles after JSON round-trip
- comparison is all-pairs across the 5 repeats, not first-vs-rest
Two workloads reproduce it. Both are 31-32 requests sharing a 16-token prefix, plus 13 unrelated filler requests, so 44 and 45 sequences are in flight:
| workload | co-resident | distinct outputs / 5 repeats | repeat pairs that differ |
|---|---|---|---|
| 31 requests + 13 fillers | 44 | 3 of 5 | (1,2) (1,3) (1,4) (2,4) (3,4) |
| 32 requests + 13 fillers | 45 | 2 of 5 | (1,2) (2,3) (2,4) |
The differing pairs are later-vs-later, so this is not a first-batch or warmup effect: repeats 1, 2, 3 and 4 disagree with each other.
- max logprob delta: 4.685e-02 (31-request case), 3.906e-02 (32-request case)
- no token divergence at any position; emitted ids are identical, only logprobs move
- 188 and 128 diverging (request, repeat-pair) observations respectively
It is intermittent across server lifetimes. The same configuration in a second lifetime was fully reproducible (1 distinct output of 5 for every workload), so it does not reproduce on every run.
The one knob that changes it
Identical client, identical workload, only the server flag differs:
--max-num-seqs |
sequences co-resident | distinct outputs / 5 |
|---|---|---|
| 8 | 8 | 1 of 5 (reproducible) |
| 64 | 45 | not reproducible |
| 128 (default) | 45 | not reproducible (intermittent) |
Negative controls
These are why I do not think this is a client bug:
- Sequential submission is clean. One request in flight at a time, same client code, same server: 7/7 workloads reproducible.
- Default mode fails the same observable 7/7, including a first-token divergence at position 23. The client demonstrably detects nondeterminism when it is present.
--max-num-seqs 8is clean with the client completely unchanged. Only a server flag differs between the clean and non-clean runs.- Smaller batches are clean. 4 filler requests (36 co-resident) was reproducible across 3 server lifetimes, 21/21 workloads.
Ruled out
- CUDA graphs. All runs use
--enforce-eager; the server config line confirmsenforce_eager=True, cudagraph_mode=CUDAGraphMode.NONE. - Prefix caching. Divergence persists with
--no-enable-prefix-caching. - KV cache pressure. vLLM reports
GPU KV cache size: 27,744 tokens; this workload needs roughly 2,700. - Chunked prefill via budget. Default
max_num_batched_tokensis 2048; the largest workload here totals 994 prompt tokens, so the budget is never approached. Raising it to 16384 changes nothing (reproduced in 2 lifetimes). - Request order. Holding the request multiset byte-identical and rotating only submission order behaves the same as not rotating it, so order is not the variable.
- Warmup / first batch. Priming with byte-identical traffic before repeat 0 removed a separate first-batch effect that affected small workloads; the large-workload divergence survives priming, and the differing pairs are later-vs-later.
Not determined
- The mechanism. I cannot see inside the engine and would rather not guess.
- Whether there is a threshold. Divergence was never observed at or below 36 co-resident and was observed at 44-45. Whether that is a threshold or a probability rising with resident batch size I could not establish; my sampling at intermediate widths was one lifetime per point, which is too few given the intermittency.
Only the --max-num-seqs comparison is a controlled single-variable result. Everything else is observational.
Environment
vLLM 0.26.0
torch 2.11.0+cu130
GPU NVIDIA GeForce RTX 4060 Laptop (sm_89, 8 GB)
driver 595.84
model Qwen3-0.6B, dtype float16
server --enforce-eager --max-model-len 1024 --block-size 16
--gpu-memory-utilization 0.55
env VLLM_BATCH_INVARIANT=1
Happy to re-run anything, add instrumentation, or test a patch. I can also try this on an A100 if hardware specificity is the first question, though I wanted to file before spending that.
Activity
All-pairs comparison, and the vLLM finding it narrowed
Re-run every claim at a clean revision, and publish both graph modes
Reproduce the filed finding at a clean revision
SyaOtiLan commentedon Aug 8, 2026
I reproduced this on a single RTX 4090 (sm_89).
Environment:
--enforce-eagerVLLM_BATCH_INVARIANT=1Using the current LockStep harness with warm cache, fixed filler width 13,
identical order, and 5 repeats, I reproduced the divergence in two independent
server lifetimes with the default FlashInfer sampler enabled.
The other five boundary cases were clean in both lifetimes, and all seven cases
actually formed a batch. Emitted token IDs remained identical; only the returned
logprobs differed.
I also ran two controls:
--max-num-seqs 8, all 7/7 cases were clean and the measured maximumnumber of co-resident sequences was 8.
VLLM_USE_FLASHINFER_SAMPLER=0, two independent server lifetimes were both7/7 clean. The 44- and 45-sequence batches still formed, so this did not
merely reduce the batch size.
There is an important caveat to the second observation: this workload is fully
greedy (
temperature=0), and vLLM's all-greedy path appears to bypass theFlashInfer top-k/top-p sampling call. I therefore would not identify the
FlashInfer sampling kernel itself as the root cause yet. Enabling the sampler
may instead change initialization, loaded kernels, or allocation layout. Still,
the repeatable A/B difference may be useful for narrowing the issue.
This was TP=1, so it appears separate from #51290 / #51292, whose proposed fix
targets the fused all-reduce + RMSNorm path under TP>1.
I have the five JSON artifacts and corresponding server logs and can provide
them if useful.
SyaOtiLan commentedon Aug 8, 2026
This now looks very likely to be the RMSNorm block-size issue already fixed by #48391 (merged as b6cbba8).
The mechanism matches this reproduction closely: before that fix, the C++ RMSNorm kernels selected a different reduction block size when
num_tokenscrossed 256. That changes the floating-point reduction order even withVLLM_BATCH_INVARIANT=1. #48391 pins the block size in batch-invariant mode.I verified this on the same RTX 4090 setup at two levels.
End-to-end
With the original v0.26.0 stack:
the 44- and 45-sequence cases diverged in two independent server lifetimes.
Keeping that Python/dependency stack unchanged but replacing the stable C extension with the current nightly extension containing the RMSNorm fix produced:
The full current nightly build (
0.26.1rc1.dev403+g821717118) was also clean in 5/5 independent lifetimes.Focused RMSNorm check
I compared the shared first 255 rows between 255-token and 300-token launches, using FP16, hidden sizes 512 and 4096, and 16 seeds:
rms_normfailuresfused_add_rms_normfailuresThe replacement extension contains changes beyond #48391, so this is not a literal single-commit binary bisect. However, the focused operator result, the exact 256-token threshold, and the end-to-end result all point strongly to #48391 as the fix.
Since the report uses v0.26.0, which predates that merged fix, could this issue be linked to #48391 and closed as already fixed?
u7k4rs6 commentedon Aug 10, 2026
Thanks, this is more than I could get from outside the engine. The end-to-end
and operator results both look convincing to me.
One thing before I close it: the operator test shows batch-size dependence, and
this issue is repeat-to-repeat variance at a fixed workload. The link is that
per-step num_tokens must vary across repeats. I have the server logs and can
check that directly. Re-running on nightly on my 4060 too. Will report back.
LioEinaudi commentedon Aug 18, 2026
v0.24.0 reproduces the eager-mode divergence on both sm89 (RTX 4090 D) and sm90 (H20) with a single-vs-batched bitwise probe; v0.27.0 with #48391 is clean on both. Happy to share the probe harness if useful. Fuller measurements (determinism tax + kernel attribution on these two GPUs) posted in #27433.