Repository navigation
[Feature]: Batch Invariant Feature and Performance Optimization #27433
Description
🚀 The feature, motivation and pitch
We have basically support Batch Invariant based on https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
Batch-invariant Inference (view)
But there are still some work to be done, so here is the issue to track the work
TODOs:
-
Basic framework Kernel-override Determinism [1/n] #25603 @bwasti
-
Flashinfer support [unrevert] Add batch invariant kernel override for FlashInfer backend [2/n] #26373 @bwasti
-
Deepseek-v3 Deepseek-v3 Batch Invariant on 8xH100 #26609 @bwasti
-
DeepGEMM on Blackwell [Feature] Batch Invariant: Support DeepGEMM and Blackwell #27127 @yewentao256
-
Batch Invariant for R1 TP 8 on Blackwell [Feature] Batch Invariant for R1 TP 8 on Blackwell #27229 @yewentao256
-
Torch compile & Cuda Graph support [Feature] Batch invariant torch.compile #27660 @PaulZhang12
-
Usability & Documentation @bwasti Batch invariance doc #27839
-
an RL example @bwasti https://github.com/bwasti/spirl
-
Adds Batch invariant tests to CI [CI] Add batch invariant test to ci #27842 @yewentao256
-
TRITON_MLA support [Feature] Batch invariant: Enable
TRITON_MLAwithout prefix-caching #29125 @yewentao256 -
FLASHINFER_MLA support 🙋Help needed, context: [Feature Request]
trtllm_batch_decode_with_kv_cache_mlaBatch Invariant support flashinfer-ai/flashinfer#2107 -
Optimize the batch invariant performance
- BMM optimization [Perf] Optimize batch invariant BMM, 18.1% Throughput improvement, 10.7% TTFT improvement #29345 @yewentao256
- [Kernel] Batch invariant NVFP4 linear using cutlass #39912 @jzakrzew
- [Kernel] Batch invariant NVFP4 MoE using cutlass #40372 @jzakrzew
- [Perf] Optimize batch invariant with fused rms norm, 2.1% E2E latency improvement #40413 @yewentao256
- [Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement #40408 @yewentao256
- [Kernel][Perf] Per-architecture tuned configs for batch-invariant persistent matmul (~3x decode kernels on RTX 4090D/H20) #53247 @LioEinaudi
- [Perf] Autotune batch invariance triton kernel in blackwell, 33.6% E2E latency reduction #53649 @yewentao256
- 🙋Help needed
Nice to have:
- Prefix caching support
- AMD support
- XPU support
- Speculative decoding support (this might be hard)
- (Out of scope) DP Support
[Feature]: Batch Invariant Feature in DP+EP #30321
Model coverage
🙋Help needed for validations of more models.
- Test a model using the script in https://github.com/vllm-project/vllm/tree/main/tests/v1/determinism
- Submit a PR updating the document
Activity
PaulZhang12 commentedon Oct 25, 2025
Happy to look into the torch.compile batch invariance
bwasti commentedon Oct 31, 2025
documentation done! happy to take more of these on
anxiang1836 commentedon Nov 14, 2025
Will this feature support on GPU L20 (Ada Lovelace) in vllm 0.11.1？
quanliu1991 commentedon Nov 14, 2025
We are currently implementing Batch-invariant Inference for FlashAttention2 on A800 and L20 GPUs. The work is now in the testing phase, and we plan to submit the code once the tests are successfully completed.
bi1101 commentedon Nov 25, 2025
Hi, I enabled batch invariant in 0.11.2 with
ELVISIO/Qwen3-30B-A3B-Instruct-2507-AWQ. And tested the output for determinism, but it didn't seem to work. The result still varies. I'm testing on 5090, which is supported and has compute capability above 9.0. Does Batch Invariant work for awq quantized model?100 remaining items
yifanFengg commentedon Oct 4, 2026
Validated three more models on GB300 (Qwen3-32B, QwQ-32B, Qwen3-235B-A22B, all 22/22) and found that the Qwen3-Next entry in the tested list no longer starts in batch-invariant mode since #40193. PR: #59972
yifanFengg commentedon Oct 4, 2026
While validating large models on GB300 I found that zai-org/GLM-4.5-Air is not batch invariant with VLLM_BATCH_INVARIANT=1 (12 of 22 suite tests fail). The cause is GateLinear's specialized router GEMM tiers, which bypass the batch-invariant linear path and dispatch on token count. Fix with a regression test: #59985. Docs update from the same validation runs (six models added, Qwen3-Next and GPT-OSS entries corrected): #59972.
yifanFengg commentedon Oct 9, 2026
Status update on the batch-invariance fixes from my side, in case it helps the tracker:
init_attn_backendnow raises the selector's "batch_invariant mode is not supported" error for every attention backend, so models that pick their own backend (MiniMax-M3 and similar) fail fast instead of silently violating invariance.VLLM_BATCH_INVARIANT=1(DeepSeek-V2-Lite, DeepSeek-V3.1) #60007); the PR routes prefill rows through the per-row decode kernel in batch-invariant mode. Validated on 2x A10 and on DeepSeek-V3.1 (TP 4, GB300), including FP8 KV cache.torch.matmuloverride not reached from the compiled forward) and GPTQ (Marlin) linear layers; fixes [Bug]: VLLM_BATCH_INVARIANT=1 is not batch-invariant for AWQ models on sm8x in the default mode #59086. A10 and B300 evidence.Known gaps I have not touched: GPTQ/AWQ MoE experts on Marlin (
AutoGPTQMoEMethod), the FA4 scheduler on Blackwell (forced to FA2), and INT8 W8A8 on SM 10.x (CUTLASS has no kernel there, unrelated to invariance).mawong-amd commentedon Oct 9, 2026
AMD support update on #52231: rebased on current main, green in upstream CI on both pipelines — the determinism suite passes on MI250, MI300 and MI355, including the distributed steps at TP=4 (with and without custom all-reduce) and DP=4 with expert parallelism, and on the ROCm 10 image. Back under review.
Also updating the DP note above, since it's outdated: DP+EP is supported now — #45683 (merged) made the MoE combine deterministic, and @yifanFengg validated one config above (DP=2×TP=2 with EP, bitwise equal on GB300). On AMD, #45683's fixed-root
reducemeasures batch-variant on RCCL at 4 ranks, so #52231 routes the MoE combine (and reduce-scatter generally) through an all-to-all plus a fixed-order rank sum instead; the DP=4+EP arm (OLMoE, online quantization) is bitwise invariant with it at 4 ranks. Dense DP needs no support — ranks run decoupled — and #52231 absorbs the MoE DP padding (1→200 rows across replicas). A custom-kernel reduce-scatter stays a perf follow-up.Laurent-Zhang commentedon Oct 14, 2026
PR submitted: #61575 — validated Qwen/Qwen2.5-Coder-1.5B-Instruct on RTX 4090 (sm_89). All 6 test variants passed (FLASH_ATTN / TRITON_ATTN / FLEX_ATTENTION × default / vllm_c).
[CI] Deflake HF-vs-vLLM logprobs comparisons with near-tie tolerance
chf003 commentedon Oct 23, 2026
Hi! I validated
Qwen/Qwen3-4Bfor batch invariance using the official test suite on a single NVIDIA A100 80GB (BF16, TP=1, default compiled mode).I opened #63348 to add the model to the tested-models list.
[CI] Deflake HF-vs-vLLM logprobs comparisons with near-tie tolerance
yifanFengg commentedon Nov 3, 2026
Model coverage note for the list:
zai-org/GLM-5.3(DSA sparse MLA) cannot run underVLLM_BATCH_INVARIANT=1today (no sparse backend opts in), and with a measurement-only opt-in it is not batch invariant on 4x GB300: 6/6 suite ids fail in all three backend/KV configs, and a short request's first-token logprob changes with its position in a 16-request batch while every layer-0 projection stays bitwise identical; the first divergence is inside the layer-0 sparse-MLA attention core. Kernel probes, model-level logs, the per-module bisection and the opt-in diff are in #65278.imthomp commentedon Nov 4, 2026
Proposal for the Hopper/Blackwell Linear path (under "Optimize the batch invariant performance"): #65569 . On H100, routing UnquantizedLinearMethod to cuBLAS under batch-invariant mode cut batch-32 decode cost from 63.1% to 0.2% on Qwen2.5-7B, with identical outputs across cache and batch conditions. Feedback welcome before I open a PR.
imthomp commentedon Nov 12, 2026
For the SM100 part of this tracker, in case it helps whoever upgrades vLLM's cuBLAS: cuBLAS 13.8's experimental batch-invariance mode (
CUBLAS_BATCH_INVARIANCE_FLAGS=8, AUTO) is bitwise batch invariant on a B200 in my tests (batch sizes 1-8192; 25 shapes including the gpt-oss router, odd widths and large projections; bf16, fp16, fp32), but only if batch-invariant mode's PyTorch reduced-precision setting is not also applied. Withallow_{fp16,bf16}_reduced_precision_reduction = (False, False)(how the mode disables split-k through PyTorch), odd widths (97, 127, 129) differ; with only cuBLASLt preferred and that setting left at its default, 0 of 408,828 comparisons differ. Pinning the alignment class instead does not help (ALIGN_2B makes it worse; ALIGN_16B rejects odd widths). So when vLLM's pinned cuBLAS includes the mode, enabling it on SM100 would mean skipping that PyTorch setting there. Details and raw data in #65569 and https://github.com/imthomp/vllm-batch-invariance (PROTOCOL.md, Phases 1e/1f).mosafariuk commentedon Nov 14, 2026
CPU backend proposal for
VLLM_BATCH_INVARIANT=1: #67696. Today the flag refuses to start on CPU_ATTN. With a local prototype, one request's top-5 logprobs were bit-identical alone and in batches forfacebook/opt-125mon x86 (fp32, bf16) and macOS, and on Linux arm64 up to 77 prompt tokens. The issue lists five sources and the open design question (the matmul kernel). Is CPU in scope for this tracker?