[Bugfix][KV Connector] Reap expired NIXL leases behind a heartbeated head by gokay-ai · Pull Request #58292 · vllm-project/vllm

gokay-ai · 2026-09-23T07:04:59Z

Purpose

The NIXL send-side lease reaper treated _reqs_to_send as expiry-ordered and stopped at the first still-live entry. Heartbeats update that map in place, so a request that keeps getting renewed can sit at the head forever and block reclaim of every expired entry behind it. That is the case in #58222: A is heartbeated, B and C expire, and neither is ever released.

Delete+reinsert on heartbeat is not enough either — max(old, now + extension) can still leave a later-inserted longer TTL in front of an already-expired entry. The reaper now scans the whole map. Expired send-lease IDs are always in _reqs_to_process (stamped only after that set membership check), so the reaper keeps _reqs_to_process.remove().

Fixes #58222.

Not a duplicate of #50789: that is a stale draft from August against an older get_finished() layout, and it does not target this issue. This change is against current main, extracts a testable helper, and pins the issue's stdlib repro as a unit test.

Test Plan

.venv/bin/python -m pytest tests/v1/kv_connector/unit/test_nixl_heartbeat.py -v
uvx ruff check vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py tests/v1/kv_connector/unit/test_nixl_heartbeat.py

No GPU. No model eval — reclaim path only, serving output unchanged.

Test Result

tests/v1/kv_connector/unit/test_nixl_heartbeat.py
11 passed in 1.39s

Including:

test_reaper_reclaims_expired_leases_behind_heartbeated_head (issue [Bug] NIXL lease reaper strands expired KV blocks behind a heartbeated head entry #58222 repro; also reaps A when now == expires)
test_reaper_reclaims_when_now_equals_expiry (now >= expires boundary)
test_reaper_reclaims_shorter_lease_behind_later_deadline
existing heartbeat tests

Ruff check and format: passed on the two changed files.

AI assistance was used to locate the reaper/heartbeat code and write the regression tests. I reviewed the change.

…head The send-side lease reaper assumed _reqs_to_send was expiry-ordered and stopped at the first live entry. Heartbeats renew expiry in place, so a long-lived head entry stranded every expired request behind it. Scan the whole map instead, and discard from _reqs_to_process so reaping previously unreachable IDs cannot KeyError. Fixes vllm-project#58222 Signed-off-by: GokayAI <[email protected]> Co-authored-by: GokayAI <[email protected]>

github-actions · 2026-09-23T07:07:01Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Reviewers with write access and configured trusted contributors can comment /ci run for upstream CI or /amd-ci run for AMD CI only whenever CI signals are needed.

Once the PR is approved or has the ready label, the PR author can also use the corresponding /ci run, /ci retry, and /ci cancel commands, or their /amd-ci variants. New commits do not start upstream CI automatically.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

liuzijing2014 · 2026-09-23T20:44:38Z

Thanks for the PR! @NickLucche @njhill do you mind taking a look? Would be great to have this merged soon as we are also facing the exact same issue in production.

snadampal · 2026-09-23T21:43:31Z

+
+    now = {"t": 0.0}
+    monkeypatch.setattr(base_worker.time, "perf_counter", lambda: now["t"])
+    for t in (20.0, 40.0, 60.0, 80.0):


also test the matching timestamp, now==expiry

Thanks — added coverage for that boundary. test_reaper_reclaims_when_now_equals_expiry reaps when now == expires, and the existing heartbeated-head test now also reaps A at now == 100 so now >= expires is checked on equality.

snadampal · 2026-09-23T22:01:09Z

+                req_id,
+                count,
+            )
+            self._reqs_to_process.discard(req_id)


I'm curious why you have changed from remove to discard here. did you run into any scenario where the request was present in _reqs_to_send but not in _reqs_to_process?

We didn’t hit that case. _reqs_to_send is only written after the req is already in _reqs_to_process (pull/push start_load_kv), and the abort path asserts the opposite. discard was defensive — restored .remove(req_id).

Cover the now >= expires boundary in tests, and restore _reqs_to_process.remove() for expired send leases. Those IDs are always added to _reqs_to_process before _reqs_to_send, so discard() was not needed. Signed-off-by: GokayAI <[email protected]> Co-authored-by: GokayAI <[email protected]>

njhill

Thanks @gokay-ai! LGTM, just couple of nits

Address njhill review nits on vllm-project#58292: do not pass now as an arg, and early-return when _reqs_to_send is empty before building the expired list. Signed-off-by: GokayAI <[email protected]>

gokay-ai · 2026-09-25T10:18:16Z

Addressed both nits in 9ee47c6 — clock stays inside _reap_expired_send_leases, and empty _reqs_to_send returns early. Thanks @njhill.

njhill

Thanks @gokay-ai

github-actions · 2026-09-25T15:42:29Z

✅ @gokay-ai, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
Your branch must contain every commit currently on its upstream target branch. Merge or rebase onto the latest target branch, then rerun the command. Append --allow-stale to a run command to test an outdated branch at your own risk.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

njhill · 2026-09-25T15:42:31Z

/ci run

github-actions · 2026-09-25T15:44:16Z

✅ Triggered Buildkite CI #91226 for commit c97a25abce67.

* [Bugfix][XPU] store the pointer raw bit pattern instead of its numeric value (#54514) Signed-off-by: Lai, Yejing <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> * [Attention][CPU] Run Zen CPU encoder attention on zentorch SDPA (#54508) Signed-off-by: priyansh jain <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [MRV2] Validate MRV2 entrypoint logits processors (#57728) Signed-off-by: Taneem Ibrahim <[email protected]> * [Bugfix][Rust Frontend] Prevent MM timing from enabling debug tracing (#58378) Co-authored-by: Bugen Zhao <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: Bugen Zhao <[email protected]> * [XPU][UT] Align HF and vLLM inputs for Qwen2 embedding test by preventing Sentence Transformers from applying chat template (#58117) Signed-off-by: RyanMa29 <[email protected]> * [CPU] Gate the AVX10.2 paths on compiler support (#58133) Signed-off-by: R <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Li, Jiang <[email protected]> * [Perf][Frontend] Offload streaming derender detokenization (#57528) Signed-off-by: Shrey Gajjar <[email protected]> * [Multimodal] Reuse the supplied tokenizer in the MiniMax-M3 VL processor (#58460) Signed-off-by: Zijing Liu <[email protected]> * [XPU] enable XPU GRAPH by default (#51600) Signed-off-by: zhenwei-intel <[email protected]> * [ROCm][DSv4.1][Perf] Emit MXFP8 from the sparse decode reduce and run wo_a as a grouped FP8 GEMM (#58456) Signed-off-by: fai <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * Remove `.gemini/` and `CLAUDE.md` (#58541) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Bugfix][Pooling] Fix JinaVL label configuration and restore multimodal tests (#57347) Signed-off-by: Linze-Shi <[email protected]> * [Chore] Use Transformers v5 names and drop redundant processor `use_fast` (#58550) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Refactor] Remove dead or duplicate tests (#58446) Signed-off-by: yewentao256 <[email protected]> * [Perf][Attention] Bound FlashInfer prefill dequantization scratch (#57918) Signed-off-by: Yueh-Ting Chen <[email protected]> * fix(config): apply presence_penalty/frequency_penalty from override-generation-config (#50769) Signed-off-by: Chenglun Hu <[email protected]> Signed-off-by: hclsys <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Harry Mellor <[email protected]> * [Bugfix] Resolve the Hub revision once per repo (#56092) Signed-off-by: Wauplin <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Frontend] Remove the slow tokenizer mode (#58545) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * Revert "[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)" (#58484) * [transformer] RMSNorm matching for alternative rsqrt (#54461) Signed-off-by: Thomas Ortner <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Harry Mellor <[email protected]> * [Bugfix] Count unsplit Idefics3 image patches (#48760) Signed-off-by: nightcityblade <[email protected]> Co-authored-by: nightcityblade <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> * [Bugfix] Keep JIT warmup under enforce-eager when fault tolerance is on (#58593) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Kimi <[email protected]> * [Core] Skip JIT monitor when JIT warmup is disabled (#58590) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Kimi Code <[email protected]> * [Fast Start] Wait for weight cache daemon readiness (#58370) * [Bugfix][Quantization] Add Humming to the W4A8 (INT4xFP8) MoE oracle (#58427) Signed-off-by: mgoin <[email protected]> * [Refactor] Move auxiliary files out of the repository root (#58572) Signed-off-by: mgoin <[email protected]> * [Cleanup] Remove online quantization support in `fp8.py` in favor of online shorthands (#53585) Signed-off-by: Felix Marty <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm] Fix misrouting race-condition in multi-decode P/D disagg with mori-io (#51681) Signed-off-by: Vincent Cave <[email protected]> Signed-off-by: Shiksha Patel <[email protected]> Co-authored-by: Shiksha Patel <[email protected]> Co-authored-by: Douglas Lehr <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Perf] DiffusionGemma: constrained reads over the request's logprob_token_ids (#58216) Signed-off-by: Matt Mastracci <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Misha Goin <[email protected]> * [Bugfix] Pass quant_config to DiffusionGemma's ParallelLMHead (#48521) Signed-off-by: Aaron Kang <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [ROCm][CI] skip the ROCm MRV1 default where MRV1 cannot serve the config (#58535) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [DFlash] Capture the context K/V precompute in the draft CUDA graph (#57632) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Bugfix][Outlines] Fix EOS termination and unconstrained masks after rejected drafts (#58612) * [Bugfix][KV Cache] Fix incremental multimodal block hashing (#51694) Signed-off-by: Jellow <[email protected]> Signed-off-by: Jellow <[email protected]> * [XPU][CI] enable prompt embeds tests on XPU (#58283) Signed-off-by: Lin, Fanli <[email protected]> Signed-off-by: Fanli Lin <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> * [CI] Report to CRCR after all jobs finish, gated on the build's long pole (#58628) * [PD][PushConnector] Record last activity of remotes on the D side (#52245) Signed-off-by: Sunita Nadampalli <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> * [BUGFIX] fix ovis2_5 multimodal tokens (#52623) Signed-off-by: Milosz Grunwald <[email protected]> * [Bugfix][Core] Keep every multimodal feature in the partial-block KV event (#58288) Signed-off-by: haosenwang1018 <[email protected]> Co-authored-by: haosenwang1018 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> * [ROCm][CI] Mirror the DSv4-Flash disaggregated DP EP group on MI355 (#58558) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Bugfix][Quantization] Give LM heads standard linear metadata (#58444) Signed-off-by: mgoin <[email protected]> * [Bugfix][Mamba] Restore prompt-tail prefix-cache hits with MTP (#58368) Signed-off-by: Netanel Haber <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Benjamin Chislett <[email protected]> * [Perf] Parallelize registered CUDA Triton kernel warmup at startup (#58582) Signed-off-by: mgoin <[email protected]> Co-authored-by: Codex <[email protected]> * [KV Connector] Fix DecodeBench fp8 fill values and add a startup fill mode (#58472) Signed-off-by: Zijing Liu <[email protected]> Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: Yifan Qiao <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Release prompt_embeds tensor when its InputBatch slot is freed (#57988) Signed-off-by: khushali9 <[email protected]> * [Bugfix][KVConnector] Finalize saves on steps without a forward (#57775) Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Kimi Code <[email protected]> * [Bugfix][Frontend] Count reasoning tokens for Harmony, DeepSeek-V3 and Step3 parsers (#58626) Signed-off-by: Samyabrata Maji <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Bugfix] GLM-5.3-Flash: launch the kpool paged MQA logits in the varlen mode its schedule was built with (#55270) Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> * [Bugfix] Accept EOS after grammar finish in outlines backend; reject json_object at validation (#57743) Signed-off-by: SIDDARTHA REDDY <[email protected]> * [Perf] Batch Mamba2 prefill SSM state saves, removing GPU<->CPU syncs (#49371) Signed-off-by: samuelkim7 <[email protected]> Co-authored-by: Claude <[email protected]> * [CI] Run DFlash2 NVFP4 acceptance test on B200; skip it on H200 35GB MIG (#58496) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [Bugfix][MRV2] Treat padded prompt tails as spec-decode rows for hybrid models (#58434) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Kimi-K3][Perf] Dispatch GEMM for vision patch embedder (#58527) Signed-off-by: Thien Tran <[email protected]> * [Minimax-M3][Perf] Use triton_mrope for vision tower + int64 offset fix for triton_mrope (#58526) Signed-off-by: Thien Tran <[email protected]> Co-authored-by: Kimi <[email protected]> Co-authored-by: Isotr0py <[email protected]> * [Bugfix][Quantization] Refresh online NVFP4 scales before reload post-processing (#57954) Signed-off-by: S1ro1 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: aoshen02 <[email protected]> * [Perf][DSv4.1] Restore the fused query RMSNorm + MXFP8 quantization path (#57679) Signed-off-by: Juntian Liu <[email protected]> * [Bugfix][LogitsProcessor] Validate ':' separator in custom logits processor FQCN (#56020) Signed-off-by: 100milliongold <[email protected]> * [ROCm][CI][AITER Coverage] Harden MoE sorting-backend/dispatch env-var test matrix (#58393) Signed-off-by: Divakar Verma <[email protected]> * [gRPC] Fix ping tolerance so long non-streaming RPCs are not dropped (#55102) Signed-off-by: Wei Gong <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> * [Perf][Rust Frontend] Make histogram observations lock-free (#58574) Co-authored-by: jthomson04 <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Signed-off-by: Bugen Zhao <[email protected]> * [ROCm][Perf] MXFP8 GEMM on native 32x32 block scales for gfx950 (#58510) Signed-off-by: fai <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Bugfix][Qwen4Exp] Keep pinned PLE prefetch ids out of the CUDA graph pool (#58489) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Perf][Engram] Serialize offloaded lookups and pack host tables into huge pages (#56926) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Kimi Code <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix][Quantization] Fix MXFP8 startup crash on layers below mm_mxfp8 shape limits (#54223) Signed-off-by: samuelkim7 <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]> * [ROCm] Cut 69 wasted contiguous copies per decode step from the skinny GEMM path (#58566) Signed-off-by: lifulu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [ROCm][Build] Filter crate tags from vLLM version detection (#57744) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: reidliu41 <[email protected]> * [Perf][Distributed] Add low-SM multimem reduce-scatter for SM100/SM103 (#55072) Signed-off-by: Yongye Zhu <[email protected]> Signed-off-by: Summer Yang <[email protected]> Co-authored-by: Summer Yang <[email protected]> * [PP][XPU]Add the flag to control microbatch feature on MRV2+PP (#55145) Signed-off-by: yisheng <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> * [Feature] Triton kernel dispatcher (#43048) Signed-off-by: wangxiyuan <[email protected]> * [ROCm] Fix CI runtime and tests for MI355 DPX (#58244) Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Mahesh Kunreddi <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [CI][ROCm] Prevent Model Executor apt stalls (#58607) Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> * [Qwen4Exp][ROCm] PLE n-gram table CPU offload (#57497) Signed-off-by: Mathew Odden <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: opencode+deepseek-v4-flash+vllm <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [MM] Add Triton kernel for mm_input_normal. (#56798) Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> * [Structured Outputs] Parse Lark grammars natively in the xgrammar backend (#58321) Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Bugen Zhao <[email protected]> * [ROCm] Credit ROCm/aiter for the block32 GEMM's packed kernel and in-launch split-K (#58659) Signed-off-by: Lingpeng Jin <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Frontend] Handle Disable Thinking in /v1/messages (#58613) Signed-off-by: jryberg <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: jryberg <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Bugfix] Stop leaking the internal field name in the max_tokens validation error (#58336) Signed-off-by: shallow10 <[email protected]> * [Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA (#55528) Signed-off-by: lz <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Perf][DSv4] Fuse inverse RoPE + FP8 quant into FlashInfer sparse MLA (#58621) Signed-off-by: Yongye Zhu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [UX][Frontend] Introduce `vllm preload` cli for fast restart (#56680) Signed-off-by: Isotr0py <[email protected]> * [MoE] Defer the TRTLLM-Gen top-k finalize on the modular path (#58635) Signed-off-by: Yongye Zhu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Docs] Add return annotation to `fused_mm_input_norm_triton` (#58687) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [CI] Shard (H100) Helion Kernels five ways (#58645) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Core] Model console logging as CLI configuration (#57205) Add `--logging-config` CLI argument which can be supplied as JSON or using dotted arguments. The `--log-level` argument is provided for convenience, and `--log-config-file` is deprecated in favor of `--logging-config.pylogging_config_file`. Signed-off-by: Mark McLoughlin <[email protected]> Co-authored-by: AI Assistant <[email protected]> * [ROCm][CI] Pass weight_shape in MXFP8 block32 linear tests (#58698) Signed-off-by: Djordje Ramic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [SpecDecode] Add LiLiCorr drafter (#57934) Signed-off-by: Andrii Skliar <[email protected]> Signed-off-by: Andrii Skliar <[email protected]> Co-authored-by: Andrii Skliar <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Benjamin Chislett <[email protected]> * [MRV2] Minor model_runner.py code cleanup (#58610) Signed-off-by: Nick Hill <[email protected]> * [Bugfix] Fix generative scoring body cancellation (#57729) Signed-off-by: Taneem Ibrahim <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Pooling] Preserve reranker tokenization with document limits (#57666) Signed-off-by: Taneem Ibrahim <[email protected]> Co-authored-by: Wentao Ye <[email protected]> * [Kernel][Perf] Register-resident path for per-token-group 8-bit quant (#55330) Signed-off-by: chao.huan <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [Perf][Kernel] Vectorized flat abs-max for dynamic per-tensor FP8 quantization (#58194) Signed-off-by: Monishver Chandrasekaran <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [Core][Logging] Fix JSON logging process decoration (#57957) Signed-off-by: Mark McLoughlin <[email protected]> * [Bugfix] Stop allocator fragmentation from shrinking the KV cache during memory profiling (#58430) Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Bugfix][Frontend] Keep logprobs of parser-suppressed streaming chunks (#58583) Signed-off-by: errmakov <[email protected]> Co-authored-by: Yanxiao Zhao <[email protected]> Co-authored-by: Prakhar Agarwal <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Bugfix][GLM-5.3-Flash] SM90 sparse MLA: index_kpool mismatch leads to corruption via unread query token (#58704) Signed-off-by: Matt Mastracci <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Benchmark] Record model_id in bench latency/throughput --output-json (#58112) Signed-off-by: yashasvi <[email protected]> * [Bugfix][GLM-5.3-Flash] kpool corruption with speculative decoding (#58454) Signed-off-by: Matt Mastracci <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Yifan Qiao <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [GLM5.3 Perf] Optimize glm 5.3 metadata op, 1.6~4.8x kernel level performance improvement (#58450) Signed-off-by: yewentao256 <[email protected]> * [Bugfix][KV Connector] Reap expired NIXL leases behind a heartbeated head (#58292) Signed-off-by: GokayAI <[email protected]> Co-authored-by: GokayAI <[email protected]> * [ROCm][Kimi-K3] Optimize low-concurrency speculative KDA (#58045) Signed-off-by: jiacao-amd <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Cover the AITER MQA logits dispatch on gfx950 (#58724) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Add quantized MoE serving test for gfx950 (#58748) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Test AMD DeepSeek V4 MoE routing against a PyTorch reference (#58740) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Perf] DiffusionGemma: one-pass sampler statistics kernel (#58226) Signed-off-by: Matt Mastracci <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> * [ROCm][CI] Expand single-GPU coverage on MI355 DPX (#57599) Signed-off-by: Sheral Kumar <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [CI] [MRV2] Restore MRV2 pp dp coverage (#57735) Signed-off-by: Taneem Ibrahim <[email protected]> * [Bugfix][CI] Report subprocess test skips as skips, not passes (#58701) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Run the MLA attention+quant fusion test on ROCm (#58717) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm] Bump torch 2.13, triton 3.8, torchaudio, torchvision (#50605) Signed-off-by: Rohan Potdar <[email protected]> Signed-off-by: Rohan138 <[email protected]> Signed-off-by: jpvillam <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: jpvillam <[email protected]> * [Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957) Signed-off-by: Yousaf shah <[email protected]> Co-authored-by: sfeng33 <[email protected]> * [Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop (#58551) Signed-off-by: errmakov <[email protected]> Co-authored-by: Du Bin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Flora Feng <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix][Frontend] Document 404 response for `/generative_scoring` (#58788) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Kimi Code <[email protected]> * [Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams (#56067) Signed-off-by: Cheng Rui <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Bugfix] Default missing detail for Responses API input images (#57241) Signed-off-by: Yifan Zong <[email protected]> Co-authored-by: Ben Browning <[email protected]> * [watermarking] golden tests for backwards compatibility (#56809) Signed-off-by: Raphael Rialland <[email protected]> Signed-off-by: Simon Veitner <[email protected]> Co-authored-by: Simon Veitner <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix] Don't drop the rest of the allocator config when toggling expandable segments (#57982) Signed-off-by: Oxana Korzh <[email protected]> * [AuxOutput] Only require Model Runner V2 on GPU platform (#58205) Signed-off-by: Linkun Chen <[email protected]> * [Pooling] Preserve BERT-family heads for raw logits (#57664) Signed-off-by: Taneem Ibrahim <[email protected]> Co-authored-by: Wentao Ye <[email protected]> * fix: perf: use startswith(x, i) instead of string slicing to avoid O(N^2) (#52580) Signed-off-by: Ricardo-M-L <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * [Bugfix] V1: fix allowed_token_ids_mask aliasing in InputBatch.swap_states (#48419) Signed-off-by: Rui Zhu <[email protected]> Co-authored-by: Claude <[email protected]> * [Bugfix][Reasoning] Count Kimi K3 reasoning tokens (#58372) Signed-off-by: Elvir Crncevic <[email protected]> Signed-off-by: Flora Feng <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [CI] Split (H200 MIG/MI300) Basic Correctness into named jobs (#57054) Signed-off-by: Thang Nguyen <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> * [Bugfix][Frontend] Fix Inkling tool name leaking into content after reasoning (#58792) Signed-off-by: Baljinder Hothi <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Core] Bound draft-token RPC waits by the execute-model timeout (#58779) Signed-off-by: VS Chandra Mourya <[email protected]> Co-authored-by: VS Chandra Mourya <[email protected]> * [Perf][DSv4.1] Shard the Engram wkv projection across TP ranks (#58678) Signed-off-by: Shuolei Wang <[email protected]> * [RL] Add sharding-aware NCCL M2N weight transfer (#51520) Signed-off-by: Ke Wen <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: aoshen02 <[email protected]> * [Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support (#57071) Signed-off-by: Xiao Yu <[email protected]> Co-authored-by: Cursor <[email protected]> * [Bugfix][Frontend][Rust Frontend] Update DeepSeek V4.1 Flash reasoning effort mappings (#58316) Co-authored-by: Claude <[email protected]> Co-authored-by: Bugen Zhao <[email protected]> Signed-off-by: zhec <[email protected]> * [CI] Only isolate the registry tests that need a fresh process (#58764) Signed-off-by: aarushjain29 <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263) Signed-off-by: zixi-qi <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [CI] Split (B200) Miscellaneous Kernels into mHC, FLA Ops and Misc named jobs (#58609) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> * [Security] Gate per-request multimodal processor kwargs (#58830) Signed-off-by: Juan Pérez de Algaba <[email protected]> Co-authored-by: Cursor <[email protected]> * [Elastic EP] Fix EPLB load statistics during scaling (#58473) Signed-off-by: Itay Alroy <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> * [Refactor] Remove dead tests utils (#58803) Signed-off-by: yewentao256 <[email protected]> * [Perf][MoE] Index expert mapping lookups in RoutedExperts.load_weights (#58720) Signed-off-by: Willian <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]> * [Bugfix] Fix Anthropic Thinking Disabled with P/D (#58786) Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template (#58754) Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046) Signed-off-by: Ashraf Bhuiyan <[email protected]> * [CI] Reduce CUDA graph mode test overhead (#58749) * [GLM5.3 Bug] Fix sparse indexer attn topk backend selection (#58594) Signed-off-by: yewentao256 <[email protected]> * [CI] Stabilize batch submission in full CUDA graph tests (#58810) * [Bugfix][DSV4.1] Avoid host sync in ViT CUDA graph replay metadata (#58499) * [Security] Harden message sanitization (#58832) Signed-off-by: DarkLight1337 <[email protected]> * [Kernel][DSV4.1] Fuse MoE finalize into the TP all-reduce + mHC boundary (#58586) Signed-off-by: Yongye Zhu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [PCP][DCP] Support DCP target model with non-DCP Dspark (#56723) Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Codex <[email protected]> * [Kernel] Bump FlashKDA to keep the recurrent state in fp32 (#58846) Signed-off-by: Simon Veitner <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Metrics][KV Offload] Add Prometheus metrics for SimpleCPUOffloadConnector (#57251) Signed-off-by: Vincent <[email protected]> * [mooncake] support CUSTOM_MEM_POOL in vllm (#49300) Signed-off-by: bruce.xu <[email protected]> Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: bruce.xu <[email protected]> Co-authored-by: Yifan Qiao <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Qwen3.8-Flash-Next] Avoid memory fragmentation in QSA indexer logits workspace (#57105) Signed-off-by: Thien Tran <[email protected]> * [Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test (#56377) Signed-off-by: LioEinaudi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm][Kimi-K3] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4 (#58201) Signed-off-by: Hongxia Yang <[email protected]> * [Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL (#50047) Signed-off-by: Yannik Hinteregger <[email protected]> Co-authored-by: xijiade.aihemaiti <[email protected]> * [Bugfix] V1: clear stale allowed_token_ids mask in InputBatch.condense (#43931) Signed-off-by: Varshith <[email protected]> Co-authored-by: Claude <[email protected]> * [Perf][DSv4.1] Fuse small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 (#58634) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Attention][CPU] Use zentorch SDPA for CPU MLA prefill (#54967) Signed-off-by: Rakul Chauhan <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Cursor <[email protected]> * [Perf][Attention] Remove D2H sync from FlashInfer SM90 sparse MLA plan under async scheduling (#58684) Signed-off-by: NickLucche <[email protected]> * [CI] Split (H200 MIG 35GB) Spec Decode Speculators + MTP into 4 named jobs (#57237) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Thang Nguyen <[email protected]> Co-authored-by: Kimi Code <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [ROCm][Perf] Enable layer-aware CSA2 multi-stream overlap for DeepSeek-V4.1-Flash (#57407) Signed-off-by: Shanshan Shen <[email protected]> * [Perf][Pooling] Avoid blocking seq_lens GPU-to-CPU copy for pooling in FlashInfer metadata builder (#57214) Signed-off-by: frankwang28 <[email protected]> Co-authored-by: Misha Goin <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Vadim Gimpelson <[email protected]> * [Bugfix] Support repsonse_format + tool_choice=auto (#56086) Signed-off-by: Artem Perevedentsev <[email protected]> Co-authored-by: Yufeng He <[email protected]> Co-authored-by: pablopupo <[email protected]> Co-authored-by: hubunt <[email protected]> Co-authored-by: Claude Sonnet 5 <[email protected]> * [Fast Start] Add `/health` endpoint for the weight cache daemon (#58552) Signed-off-by: Xun Sun <[email protected]> Co-authored-by: Isotr0py <[email protected]> * [Perf][MoE] Use fused MiniMax2 routing with non-unit routed scaling (#58880) Signed-off-by: zjy0516 <[email protected]> * [ROCm][Bugfix] Fall back to default GEMM for CPU tensors on ROCm builds (#58923) Signed-off-by: fai <[email protected]> * [Bugfix][KV Connector] Retry Mooncake bootstrap registration on timeout (reopens #55763) (#58919) * [Frontend] Switch Python Harmony dependency to oss-harmony (#55128) Signed-off-by: Anton Peganov <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <[email protected]> * [ROCm] Bump AITER to v0.1.23 (#58867) Signed-off-by: fai <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Bugfix][Frontend] Count Responses reasoning tokens per tool round (#58927) Signed-off-by: sfeng33 <[email protected]> * [KimiViT][Perf] Fuse per-layer QK RoPE into one in-place kernel (#58651) * [Mypy] Fix mypy typing for Ultravox and Unlimited-OCR models (#58239) Signed-off-by: Ashraf Bhuiyan <[email protected]> * [Bugfix][Frontend] Sample batched chat completions from the adjusted requests (#58929) Signed-off-by: sfeng33 <[email protected]> * [Bugfix][Frontend] Use a fresh parser per choice in non-streaming chat completions (#58939) Signed-off-by: sfeng33 <[email protected]> * [Bugfix][EPD] Skip sampling for encoder-only async steps (#58490) Signed-off-by: Tianyu Guo <[email protected]> * [CPU] Build CPU wheels on Ubuntu 22.04 with AMX-FP8 support (#58515) Signed-off-by: zhejiangxiaomai <[email protected]> Signed-off-by: jiang1.li <[email protected]> Co-authored-by: jiang1.li <[email protected]> * [Test][ROCm] Stabilize the mixed OLMoE LoRA test (#58945) Signed-off-by: Andreas Karatzas <[email protected]> * [Model] Enable LoRA support for RobertaForSequenceClassification (#58884) Signed-off-by: jz-yolo <[email protected]> Signed-off-by: Jane Zhu <[email protected]> Co-authored-by: jz-yolo <[email protected]> * [Bugfix][Frontend] Apply Harmony adjust_request in batched chat completions (#58958) Signed-off-by: sfeng33 <[email protected]> * [Minimax-M3] Add Encoder CUDA graph support (#58673) Signed-off-by: Thien Tran <[email protected]> Co-authored-by: Codex <[email protected]> * [Perf][MRV2] Reuse Mamba/GDN metadata across KV cache groups (#58762) Signed-off-by: Luca Motz <[email protected]> Co-authored-by: Xin Yang <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> * [Bugfix] Fix the two multimodal root tests that fail on main (OpenPangu-VL embed merge, MiMo sink test fixture) (#58900) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Benchmark] Add Responses API backend to vllm bench serve (#54628) Signed-off-by: QHarshil <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Chauncey <[email protected]> * [CI] Allowlist-shrink batch 1: wire 17 root-level tests + drop 5 stale watermarking entries into misc.yaml (#58055) Signed-off-by: Turner Jabbour <[email protected]> Co-authored-by: Claude Sonnet 5 <[email protected]> * [CPU] Use accelerator memory API in DiffusionGemma (#58964) Signed-off-by: zhejiangxiaomai <[email protected]> Co-authored-by: Li, Jiang <[email protected]> * [CPU] Add video inferencing via torchcodec on s390x (#58693) Signed-off-by: Rehan Khan <[email protected]> Co-authored-by: Li, Jiang <[email protected]> * [Skills] Update kernel-microbenchmark to include ROCm (#58646) Signed-off-by: Thien Tran <[email protected]> Co-authored-by: Kimi <[email protected]> * [Misc] Name each backend and its kernel block sizes in block-size errors (#58557) Signed-off-by: Eugenio "Jay" Zuccarelli <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [CI] Shard (H200 MIG 35GB / MI355 DPX) Entrypoints Integration (Pooling) into named jobs (#58652) Signed-off-by: Thang Nguyen <[email protected]> Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: khluu <[email protected]> * [CI] Split Dynamic Shapes out of (H200 MIG 35GB) PyTorch Compilation + (MI300) mirror (#58451) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Bugfix][Qwen4Exp] Release the profiling KV cache held by QSA key views (#58961) Signed-off-by: Zihua Wu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [CPU] Include vLLM Recipes tooling in release image to deploy models using vLLM Recipes (#58796) Signed-off-by: louie-tsai <[email protected]> * Revert "[CI] Shard (H200 MIG 35GB / MI355 DPX) Entrypoints Integration (Pooling) into named jobs (#58652)" (#59011) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Perf][Qwen4Exp] Fuse HC down projection and SiLU on NVIDIA (#58957) Signed-off-by: Thien Tran <[email protected]> * [Bugfix][Model] Gemma4: register aliased embedding scalars as buffers (#54213) Signed-off-by: Yannick Schnider <[email protected]> * [Bugfix][Spec Decode] Implement get_top_tokens() on the ROCm DeepSeek V4 MTP drafter (#57568) Signed-off-by: BaoYunkai <[email protected]> * [Perf][Qwen3.8] Reduce PLE metadata construction overhead (#58114) Signed-off-by: zjy0516 <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [ROCm][Refactor] Move DeepSeek-V4/V4.1 multi-stream overlap gate to ROCm platform (#58983) Signed-off-by: Shanshan Shen <[email protected]> * [Multimodal] Avoid extra d2d for encoder cudagraph with fused input norm (#56711) Signed-off-by: Isotr0py <[email protected]> * [Bugfix][Logging] Preserve application log record factories (#58747) Signed-off-by: mgoin <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Bugfix] Use the correct repository revision for secondary artifact loaders (#57461) Signed-off-by: Clinton Thomas <[email protected]> Co-authored-by: Lucas Bourtoule <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm][Perf] Kimi-K3 Enable sharded latent MoE up-projection under EP (#54956) Signed-off-by: Xavier Aguilar <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Feature] Add fixed-token prefill scoring (#54335) Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> * [LoRA] Support variable num_labels for sequence classification (#57766) Signed-off-by: linitra24 <[email protected]> Co-authored-by: Jee Jee Li <[email protected]> * [ROCm][Perf] Replace torch.topk in DSA candidate block selection (#58208) Signed-off-by: fai <[email protected]> Co-authored-by: TJian <[email protected]> Co-authored-by: Cursor Agent <[email protected]> * [ROCm]Keep LMCache OpenTelemetry on the image's 1.40 stack (#59056) Signed-off-by: Micah Williamson <[email protected]> * [Bugfix][Kimi-K3] Refresh DSpark context KV cache pointers after the KV cache is re-bound (#58814) Signed-off-by: Oxana Korzh <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [KVConnector][NIXL] Support packed MLA KV layouts in pipeline-parallel push prefill (#50499) Signed-off-by: zixi-qi <[email protected]> * [Quant] Use canonical N-first weight format for CT WNA16 MoE (#52798) Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: HDCharles <{"message":"Not Found","documentation_url":"https://docs.github.com/rest/users/emails#list-email-addresses-for-the-authenticated-user","status":"404"}> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [CI][Bugfix] Relax packed_qk_rope_ correctness test to one ULP (#59008) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Pin OpenTelemetry to LMCache's cap in the ROCm images (#59051) Signed-off-by: Rohan Potdar <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [ROCm][CI] Increase timeout for Entrypoints Unit (#59076) Signed-off-by: Djordje Ramic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][MLA] Add an AITER ASM round-robin decode route for DCP multi-token verify (#56861) Signed-off-by: Xiaohu Guo <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: seungrokj <[email protected]> Co-authored-by: Cursor <[email protected]> * [Bugfix] Don't sync-police or retry FlashInfer all-reduce workspace creation in eager mode (#58498) Signed-off-by: khluu <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [Bugfix] Fix resumable request + async scheduling handoff race (#58259) Signed-off-by: Yifan Zong <[email protected]> * [Model Runner V2][Spec Decode] Support spec decode with draft model (#43091) Signed-off-by: Icey <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Perf][MRV2] Allow FULL decode graphs for one-token prompt tails (#58400) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Yongye Zhu <[email protected]> * [Bugfix][Scheduler] Preserve logprobs across streaming continuations (#57447) Signed-off-by: 0xsensei <[email protected]> * [Mypy] Fix mypy typing for Voxtral and vision models (#58251) Signed-off-by: Ashraf Bhuiyan <[email protected]> Co-authored-by: Wentao Ye <[email protected]> * [Core] Rework scheduler `skipped_waiting` queue (#58947) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Bugfix][MRV2][Spec Decode] Reject draft slots that were never proposed (#58784) Signed-off-by: zixi-qi <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [ROCm][CI] Add missing test coverage for upstream parity (#50519) Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Stefan Koncarevic <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Stefan Koncarevic <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Bugfix][Scheduler] Refresh max tokens for streaming continuations (#57676) Signed-off-by: 0xsensei <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Perf][Spec Decode] Avoid triton recompiles in the acceptance estimator (#57107) Signed-off-by: Giancarlo Delfin <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]> * [CI/Build] Skip the snapshot runtime on CUDA 12.x images (#59118) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit aedaba8664f67d8ae1538e5e5cec2b1ce3f258dd) * [XPU][CI]Skip test_abort_timeout_on_prefiller in nightly (#58307) Signed-off-by: zengxian <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> (cherry picked from commit 09c47db1ca080793dc2351144cb39513fd7984ca) * [Bugfix][Engram] Keep THP tables private when resolving shared memory (#59068) Signed-off-by: Ren Yuzhou <[email protected]> Co-authored-by: Claude <[email protected]> (cherry picked from commit ec5e0c352f079c2cb8f46752fd0317a2745050bf) * [ROCm][CI] Expand MI355 mirrors and route MIG-sized jobs to DPX (#59137) Signed-off-by: Andreas Karatzas <[email protected]> (cherry picked from commit 6ebb5bd64f5ef75dbe1471ee88c009003b3d03ec) * [Bugfix][Mamba] Keep the prompt-end prefill checkpoint under sparse retention (#59146) Signed-off-by: Jared Wen <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Jiangyun Zhu <[email protected]> (cherry picked from commit d882bddbeab6b4a0d5861dfcb171bf61ce2109d6) * [Core][BugFix] Tag prefix-cache extra keys by source (#51899) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Lucas Bourtoule <[email protected]> Co-authored-by: Tai An <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 765872e7ed5b6f372f0898a67e48d3365fa28139) * [Bugfix][Frontend] Reject LoRA adapters named after a served model (#59286) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 105a4e097b905127b1d5f09a7e93bd1095e87a6c) * [Dependency] Upgrade FlashInfer to 0.7.0.post1 (#59323) Signed-off-by: wzhao18 <[email protected]> (cherry picked from commit 678baf53724e06cef7f07628ae8fd4cc6c96f11a) * [Bugfix][HiSparse] Resolve MTP verification rows with a union residency kernel (#59235) Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 2798f668608155bc8a8c74cb97e3ddd0d3053085) * [Bugfix][HiSparse] Never allocate GPU pages without host backing (#59036) Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 90e13fc757cffc11b55689d9cc33d844d9606516) * [CPU][Whisper] Support W4A16 quantized Whisper on the CPU WNA16 kernel (#58268) Signed-off-by: Harshal Adhav <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Li, Jiang <[email protected]> (cherry picked from commit 5faf81a4297921c699f9aa58f9c6395e95ede718) * [Core] Bound UniProc EngineCore startup threads to available CPUs (#58946) Signed-off-by: Andreas Karatzas <[email protected]> (cherry picked from commit 866fa130fac1e1252330a680fbed45ad05eba64f) * [KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA (#57652) (cherry picked from commit 5463fe4962785cdc3383477bf3af6533e7647dfd) * [Bugfix][HiSparse] Preserve host prefix publication after request completion (#59007) Signed-off-by: Yueh-Ting Chen <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> (cherry picked from commit ff1b87cca25690fef6bd12667fd0d26690d949af) * [Bugfix][HiSparse] Adopt GPU prefix copies after the hit's allocation (#59282) Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit 3eb6cec22ad9bb098393021b956feaa97081fc85) * [Bugfix][HiSparse] Stop the host pool feeding device KV cache residency metrics (#58725) Signed-off-by: Yueh-Ting Chen <[email protected]> Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 73c7cae4d746f64c22677348d1cc120eea6f7439) * [Bugfix][Core] Fix mamba prefill checkpoint block reservation and prompt-end eviction in align mode (#59175) Signed-off-by: kliuae <[email protected]> Co-authored-by: Nick Hill <[email protected]> (cherry picked from commit 7e583e615c20ee4ff0cd82aac592c7d6310aa7c3) * [Core] Include the LoRA path in prefix-cache block hashes (#59335) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit c055c1e075ed461cff922aa710d387281f651940) * [Perf][PP] Skip sampled-token broadcasts whose requests leave the engine (#58542) Signed-off-by: LostFox11 <[email protected]> Signed-off-by: Nick Hill <[email protected]> Co-authored-by: LostFox11 <[email protected]> Co-authored-by: Nick Hill <[email protected]> (cherry picked from commit 7314360c9eebb3b4b460ec49f6606ef0a08fcae2) * [CPU][Zen] Add DA8W4 (W4A8) int4 support for dense and MoE layers (#54024) Signed-off-by: R <[email protected]> Signed-off-by: Ganesh R <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit e12291d7332db897885fdc0e6ed61b28c969d743) * [Model Runner V2] Support randomized dummy inputs (#58411) Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Nick Hill <[email protected]> (cherry picked from commit e5e38ba9b7d18f9746d989e389a51a94b0f96f6f) * [Bugfix][HiSparse] Fix a chunked-prefill preemption livelock (#59494) Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 3a6963664537ed21172e2ec12e96e3a2dcd3718c) * [Bugfix][HiSparse] Size the KV cache from the groups HiSparse allocates (#59450) Signed-off-by: Matthew Bonanni <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> (cherry picked from commit f7999d2e4489126f1c21868699f3890021f01b73) * [Bugfix][HiSparse] Fix MTP acceptance collapse under FULL graphs with a saturated GPU pool (#59309) Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> (cherry picked from commit 4056c8ac1f8a7e8fb50cf1c56ff96649fcadccc6) * [CI] Drop a test that depends on #57930 from the #59309 backport Resolving the #59309 cherry-pick conflict in tests/v1/kv_connector/unit/test_hisparse_connector.py pulled in test_scheduled_prefix_hit_publishes_adopted_copies from #57930, which is not on this branch; it imports _allocate_scheduled from tests.v1.core.test_prefix_caching and fails on every platform. After this change the file matches the upstream #59309 diff. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: khluu <[email protected]> * [Bugfix] Fix minimax-m3 multimodal processor compatability with Transformers v5.18 (#59613) Signed-off-by: Isotr0py <[email protected]> (cherry picked from commit b558f160a2c0abcb5902acc3c91a14c38a4af173) * [Misc] Add Transformers version upper bound in requirements (#59614) Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Harry Mellor <[email protected]> (cherry picked from commit 58b3298457dde7b4554b3b4e20b238c0ac2c3a65) * Forward-port the tkv/turbo-attn vLLM seam onto upstream v0.31.0 Squashes arbicity/vllm-turbo main @969c125e9 (upstream v0.28.0 + the seam: b0a14f2eb #28, 76a3e6d72, 6d4c75611 #29) into one commit and replays it onto upstream v0.31.0 (db9527a46, the commit vllm/vllm-openai:v0.31.0 is built from) as a 3-way merge against the real base. Upstream's v0.29-v0.31 KV-cache layout refactor (vllm#51718 and follow-ups) replaced the allocator the seam patched: one backing allocation, per-layer [B, H, N, C] views placed by the engine, page geometry read off the spec, and AttentionBackend.customize_spec applied to every layer's spec by both model runners. The seam is re-expressed on that and shrinks from 49 files to 18. Kept (re-applied on upstream's structure): - plugin KV-cache dtype registry; --kv-cache-dtype choices/type - TURBO_ATTN backend slot (now a distinct enum value: two None members made CUSTOM an alias of TURBO_ATTN), turbo-attn spelling, auto-default for plugin dtypes (#29), CUDA candidate once registered - lifecycle hooks on_model_loaded / on_draft_model_loaded / on_kv_cache_initialized / adjust_kv_budget, fail-loud dispatch to every backend in use - MLA wrapping in the selector; MLA chunked-context _get_gather_op - _tq_layer_idx injection for tkv layers - aggregated_layer_count: fused (composite) pages, now one shared page per fused set in upstream's single-allocation planner Moved to upstream's extension points (turbo-attn plugin side): - get_kv_cache_spec_class (Attention, MLAAttention, hybrid alignment) -> AttentionBackend.customize_spec - KVCacheSpec.get_manager_class -> KVCacheSpecRegistry MRO lookup - get_supported_kv_cache_dtypes -> supported_kv_cache_dtypes ClassVar - get_kv_cache_shape(kv_cache_spec=...) passthrough and the backend-managed-dtype shape coercion -> spec-driven views Dropped: - spec_decode_warmup.py: upstream registers the same kernels with its JIT warmup registry (aed894c19, vllm#56323) - turbo_attn_warmup.py, utils/cutedsl_cache.py: superseded by turbo-attn's own prefill prewarm (on_kv_cache_initialized) and CuTeDSL cache - per-group BlockPools, sampler reserve, padded-page block fill, drain hook / on_kv_manager_created, KVBlockZeroer clamp: conflict with upstream's single-pool allocator; no turbo-attn consumer for the hooks - rotary fast-path registry: upstream guards the import (1f60771c7, vllm#42679) - --kv-cache-dtype-skip-layers-dtype, VLLM_KV_CACHE_SKIP_LAYERS_DTYPE, Triton fused fp8 GEMM hook, gsm8k startup waits, notify-turbo-attn workflow, draft-backend inheritance: unused or superseded - FA2 varlen paged split-K patch: never reached the overlay image (the image installs no compiled _vllm_fa2_C) and the FA pin moved PROTOCOL.md rewritten for the v0.31.0 seam. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Signed-off-by: Lai, Yejing <[email protected]> Signed-off-by: priyansh jain <[email protected]> Signed-off-by: Taneem Ibrahim <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: Bugen Zhao <[email protected]> Signed-off-by: RyanMa29 <[email protected]> Signed-off-by: R <[email protected]> Signed-off-by: Shrey Gajjar <[email protected]> Signed-off-by: Zijing Liu <[email protected]> Signed-off-by: zhenwei-intel <[email protected]> Signed-off-by: fai <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: Linze-Shi <[email protected]> Signed-off-by: yewentao256 <[email protected]> Signed-off-by: Yueh-Ting Chen <[email protected]> Signed-off-by: Chenglun Hu <[email protected]> Signed-off-by: hclsys <[email protected]> Signed-off-by: Wauplin <[email protected]> Signed-off-by: Thomas Ortner <[email protected]> Signed-off-by: nightcityblade <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Felix Marty <[email protected]> Signed-off-by: Vincent Cave <[email protected]> Signed-off-by: Shiksha Patel <[email protected]> Signed-off-by: Matt Mastracci <[email protected]> Signed-off-by: Aaron Kang <[email protected]> Signed-off-by: Stefan Koncarevic <[email protected]> Signed-off-by: Juntian Liu <[email protected]> Signed-off-by: Jellow <[email protected]> Signed-off-by: Jellow <[email protected]> Signed-off-by: Lin, Fanli <[email protected]> Signed-off-by: Fanli Lin <[email protected]> Signed-off-by: Sunita Nadampalli <[email protected]> Signed-off-by: Milosz Grunwald <[email protected]> Signed-off-by: haosenwang1018 <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Yifan Qiao <[email protected]> Signed-off-by: khushali9 <[email protected]> Signed-off-by: Samyabrata Maji <[email protected]> Signed-off-by: SIDDARTHA REDDY <[email protected]> Signed-off-by: samuelkim7 <[email protected]> Signed-off-by: khluu <[email protected]> Signed-off-by: Thien Tran <[email protected]> Signed-off-by: S1ro1 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Signed-off-by: 100milliongold <[email protected]> Signed-off-by: Divakar Verma <[email protected]> Signed-off-by: Wei Gong <[email protected]> Signed-off-by: lifulu <[email protected]> Signed-off-by: Yongye Zhu <[email protected]> Signed-off-by: Summer Yang <[email protected]> Signed-off-by: yisheng <[email protected]> Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Mathew Odden <[email protected]> Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Lingpeng Jin <[email protected]> Signed-off-by: jryberg <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: shallow10 <[email protected]> Signed-off-by: lz <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Thang Nguyen <[email protected]> Signed-off-by: Mark McLoughlin <[email protected]> Signed-off-by: Djordje Ramic <[email protected]> Signed-off-by: Andrii Skliar <[email protected]> Signed-off-by: Andrii Skliar <[email protected]> Signed-off-by: chao.huan <[email protected]> Signed-off-by: Monishver Chandrasekaran <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: errmakov <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: yashasvi <[email protected]> Signed-off-by: GokayAI <[email protected]> Signed-off-by: jiacao-amd <[email protected]> Signed-off-by: Sheral Kumar <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Rohan Potdar <[email protected]> Signed-off-by: Rohan138 <[email protected]> Signed-off-by: jpvillam <[email protected]> Signed-off-by: Yousaf shah <[email protected]> Signed-off-by: Cheng Rui <[email protected]> Signed-off-by: Yifan Zong <[email protected]> Signed-off-by: Raphael Rialland <[email protected]> Signed-off-by: Simon Veitner <[email protected]> Signed-off-by: Oxana Korzh <[email protected]> Signed-off-by: Linkun Chen <[email protected]> Signed-off-by: Ricardo-M-L <[email protected]> Signed-off-by: Rui Zhu <[email protected]> Signed-off-by: Elvir Crncevic <[email protected]> Signed-off-by: Flora Feng <[email protected]> Signed-off-by: Baljinder Hothi <[email protected]> Signed-off-by: VS Chandra Mourya <[email protected]> Signed-off-by: Shuolei Wang <[email protected]> Signed-off-by: Ke Wen <[email protected]> Signed-off-by: Xiao Yu <[email protected]> Signed-off-by: zhec <[email protected]> Signed-off-by: aarushjain29 <[email protected]> Signed-off-by: zixi-qi <[email protected]> Signed-off-by: Juan Pérez de Algaba <[email protected]> Signed-off-by: Itay Alroy <[email protected]> Signed-off-by: Willian <[email protected]> Signed-off-by: Ashraf Bhuiyan <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Vincent <[email protected]> Signed-off-by: bruce.xu <[email protected]> Signed-off-by: LioEinaudi <[email protected]> Signed-off-by: Hongxia Yang <[email protected]> Signed-off-by: Yannik Hinteregger <[email protected]> Signed-off-by: Varshith <[email protected]> Signed-off-by: Rakul Chauhan <[email protected]> Signed-off-by: NickLucche <[email protected]> Signed-off-by: Shanshan Shen <[email protected]> Signed-off-by: frankwang28 <[email protected]> Signed-off-by: Artem Perevedentsev <[email protected]> Signed-off-by: Xun Sun <[email protected]> Signed-off-by: zjy0516 <[email protected]> Signed-off-by: Anton Peganov <[email protected]> Signed-off-by: sfeng33 <[email protected]> Signed-off-by: Tianyu Guo <[email protected]> Signed-off-by: zhejiangxiaomai <[email protected]> Signed-off-by: jiang1.li <[email protected]> Signed-off-by: jz-yolo <[email protected]> Signed-off-by: Jane Zhu <[email protected]> Signed-off-by: Luca Motz <[email protected]> Signed-off-by: QHarshil <[email protected]> Signed-off-by: Turner Jabbour <[email protected]> Signed-off-by: Rehan Khan <[email protected]> Signed-off-by: Eugenio "Jay" Zuccarelli <[email protected]> Signed-off-by: Zihua Wu <[email protected]> Signed-off-by: louie-tsai <[email protected]> Signed-off-by: Yannick Schnider <[email protected]> Signed-off-by: BaoYunkai <[email protected]> Signed-off-by: Clinton Thomas <[email protected]> Signed-off-by: Xavier Aguilar <[email protected]> Signed-off-by: aoshen02 <[email protected]> Signed-off-by: linitra24 <[email protected]> Signed-off-by: Micah Williamson <[email protected]> Signed-off-by: Stefan Kon…

* [Bugfix][XPU] store the pointer raw bit pattern instead of its numeric value (#54514) Signed-off-by: Lai, Yejing <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> * [Attention][CPU] Run Zen CPU encoder attention on zentorch SDPA (#54508) Signed-off-by: priyansh jain <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [MRV2] Validate MRV2 entrypoint logits processors (#57728) Signed-off-by: Taneem Ibrahim <[email protected]> * [Bugfix][Rust Frontend] Prevent MM timing from enabling debug tracing (#58378) Co-authored-by: Bugen Zhao <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: Bugen Zhao <[email protected]> * [XPU][UT] Align HF and vLLM inputs for Qwen2 embedding test by preventing Sentence Transformers from applying chat template (#58117) Signed-off-by: RyanMa29 <[email protected]> * [CPU] Gate the AVX10.2 paths on compiler support (#58133) Signed-off-by: R <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Li, Jiang <[email protected]> * [Perf][Frontend] Offload streaming derender detokenization (#57528) Signed-off-by: Shrey Gajjar <[email protected]> * [Multimodal] Reuse the supplied tokenizer in the MiniMax-M3 VL processor (#58460) Signed-off-by: Zijing Liu <[email protected]> * [XPU] enable XPU GRAPH by default (#51600) Signed-off-by: zhenwei-intel <[email protected]> * [ROCm][DSv4.1][Perf] Emit MXFP8 from the sparse decode reduce and run wo_a as a grouped FP8 GEMM (#58456) Signed-off-by: fai <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * Remove `.gemini/` and `CLAUDE.md` (#58541) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Bugfix][Pooling] Fix JinaVL label configuration and restore multimodal tests (#57347) Signed-off-by: Linze-Shi <[email protected]> * [Chore] Use Transformers v5 names and drop redundant processor `use_fast` (#58550) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Refactor] Remove dead or duplicate tests (#58446) Signed-off-by: yewentao256 <[email protected]> * [Perf][Attention] Bound FlashInfer prefill dequantization scratch (#57918) Signed-off-by: Yueh-Ting Chen <[email protected]> * fix(config): apply presence_penalty/frequency_penalty from override-generation-config (#50769) Signed-off-by: Chenglun Hu <[email protected]> Signed-off-by: hclsys <[email protected]> Co-authored-by: Claude Opus 4.8 <[email protected]> Co-authored-by: Harry Mellor <[email protected]> * [Bugfix] Resolve the Hub revision once per repo (#56092) Signed-off-by: Wauplin <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Harry Mellor <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Frontend] Remove the slow tokenizer mode (#58545) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * Revert "[DSpark] Support pipeline-parallel targets in aggregated serving (#56956)" (#58484) * [transformer] RMSNorm matching for alternative rsqrt (#54461) Signed-off-by: Thomas Ortner <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Harry Mellor <[email protected]> * [Bugfix] Count unsplit Idefics3 image patches (#48760) Signed-off-by: nightcityblade <[email protected]> Co-authored-by: nightcityblade <[email protected]> Co-authored-by: Cyrus Leung <[email protected]> * [Bugfix] Keep JIT warmup under enforce-eager when fault tolerance is on (#58593) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Kimi <[email protected]> * [Core] Skip JIT monitor when JIT warmup is disabled (#58590) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Kimi Code <[email protected]> * [Fast Start] Wait for weight cache daemon readiness (#58370) * [Bugfix][Quantization] Add Humming to the W4A8 (INT4xFP8) MoE oracle (#58427) Signed-off-by: mgoin <[email protected]> * [Refactor] Move auxiliary files out of the repository root (#58572) Signed-off-by: mgoin <[email protected]> * [Cleanup] Remove online quantization support in `fp8.py` in favor of online shorthands (#53585) Signed-off-by: Felix Marty <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm] Fix misrouting race-condition in multi-decode P/D disagg with mori-io (#51681) Signed-off-by: Vincent Cave <[email protected]> Signed-off-by: Shiksha Patel <[email protected]> Co-authored-by: Shiksha Patel <[email protected]> Co-authored-by: Douglas Lehr <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Perf] DiffusionGemma: constrained reads over the request's logprob_token_ids (#58216) Signed-off-by: Matt Mastracci <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Misha Goin <[email protected]> * [Bugfix] Pass quant_config to DiffusionGemma's ParallelLMHead (#48521) Signed-off-by: Aaron Kang <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [ROCm][CI] skip the ROCm MRV1 default where MRV1 cannot serve the config (#58535) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [DFlash] Capture the context K/V precompute in the draft CUDA graph (#57632) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Bugfix][Outlines] Fix EOS termination and unconstrained masks after rejected drafts (#58612) * [Bugfix][KV Cache] Fix incremental multimodal block hashing (#51694) Signed-off-by: Jellow <[email protected]> Signed-off-by: Jellow <[email protected]> * [XPU][CI] enable prompt embeds tests on XPU (#58283) Signed-off-by: Lin, Fanli <[email protected]> Signed-off-by: Fanli Lin <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> * [CI] Report to CRCR after all jobs finish, gated on the build's long pole (#58628) * [PD][PushConnector] Record last activity of remotes on the D side (#52245) Signed-off-by: Sunita Nadampalli <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> * [BUGFIX] fix ovis2_5 multimodal tokens (#52623) Signed-off-by: Milosz Grunwald <[email protected]> * [Bugfix][Core] Keep every multimodal feature in the partial-block KV event (#58288) Signed-off-by: haosenwang1018 <[email protected]> Co-authored-by: haosenwang1018 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> * [ROCm][CI] Mirror the DSv4-Flash disaggregated DP EP group on MI355 (#58558) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Bugfix][Quantization] Give LM heads standard linear metadata (#58444) Signed-off-by: mgoin <[email protected]> * [Bugfix][Mamba] Restore prompt-tail prefix-cache hits with MTP (#58368) Signed-off-by: Netanel Haber <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Benjamin Chislett <[email protected]> * [Perf] Parallelize registered CUDA Triton kernel warmup at startup (#58582) Signed-off-by: mgoin <[email protected]> Co-authored-by: Codex <[email protected]> * [KV Connector] Fix DecodeBench fp8 fill values and add a startup fill mode (#58472) Signed-off-by: Zijing Liu <[email protected]> Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: Yifan Qiao <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * Release prompt_embeds tensor when its InputBatch slot is freed (#57988) Signed-off-by: khushali9 <[email protected]> * [Bugfix][KVConnector] Finalize saves on steps without a forward (#57775) Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Kimi Code <[email protected]> * [Bugfix][Frontend] Count reasoning tokens for Harmony, DeepSeek-V3 and Step3 parsers (#58626) Signed-off-by: Samyabrata Maji <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Bugfix] GLM-5.3-Flash: launch the kpool paged MQA logits in the varlen mode its schedule was built with (#55270) Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> * [Bugfix] Accept EOS after grammar finish in outlines backend; reject json_object at validation (#57743) Signed-off-by: SIDDARTHA REDDY <[email protected]> * [Perf] Batch Mamba2 prefill SSM state saves, removing GPU<->CPU syncs (#49371) Signed-off-by: samuelkim7 <[email protected]> Co-authored-by: Claude <[email protected]> * [CI] Run DFlash2 NVFP4 acceptance test on B200; skip it on H200 35GB MIG (#58496) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [Bugfix][MRV2] Treat padded prompt tails as spec-decode rows for hybrid models (#58434) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Kimi-K3][Perf] Dispatch GEMM for vision patch embedder (#58527) Signed-off-by: Thien Tran <[email protected]> * [Minimax-M3][Perf] Use triton_mrope for vision tower + int64 offset fix for triton_mrope (#58526) Signed-off-by: Thien Tran <[email protected]> Co-authored-by: Kimi <[email protected]> Co-authored-by: Isotr0py <[email protected]> * [Bugfix][Quantization] Refresh online NVFP4 scales before reload post-processing (#57954) Signed-off-by: S1ro1 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> Co-authored-by: aoshen02 <[email protected]> * [Perf][DSv4.1] Restore the fused query RMSNorm + MXFP8 quantization path (#57679) Signed-off-by: Juntian Liu <[email protected]> * [Bugfix][LogitsProcessor] Validate ':' separator in custom logits processor FQCN (#56020) Signed-off-by: 100milliongold <[email protected]> * [ROCm][CI][AITER Coverage] Harden MoE sorting-backend/dispatch env-var test matrix (#58393) Signed-off-by: Divakar Verma <[email protected]> * [gRPC] Fix ping tolerance so long non-streaming RPCs are not dropped (#55102) Signed-off-by: Wei Gong <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> * [Perf][Rust Frontend] Make histogram observations lock-free (#58574) Co-authored-by: jthomson04 <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Signed-off-by: Bugen Zhao <[email protected]> * [ROCm][Perf] MXFP8 GEMM on native 32x32 block scales for gfx950 (#58510) Signed-off-by: fai <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Bugfix][Qwen4Exp] Keep pinned PLE prefetch ids out of the CUDA graph pool (#58489) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Perf][Engram] Serialize offloaded lookups and pack host tables into huge pages (#56926) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Kimi Code <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix][Quantization] Fix MXFP8 startup crash on layers below mm_mxfp8 shape limits (#54223) Signed-off-by: samuelkim7 <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]> * [ROCm] Cut 69 wasted contiguous copies per decode step from the skinny GEMM path (#58566) Signed-off-by: lifulu <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [ROCm][Build] Filter crate tags from vLLM version detection (#57744) Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: reidliu41 <[email protected]> * [Perf][Distributed] Add low-SM multimem reduce-scatter for SM100/SM103 (#55072) Signed-off-by: Yongye Zhu <[email protected]> Signed-off-by: Summer Yang <[email protected]> Co-authored-by: Summer Yang <[email protected]> * [PP][XPU]Add the flag to control microbatch feature on MRV2+PP (#55145) Signed-off-by: yisheng <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> * [Feature] Triton kernel dispatcher (#43048) Signed-off-by: wangxiyuan <[email protected]> * [ROCm] Fix CI runtime and tests for MI355 DPX (#58244) Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Mahesh Kunreddi <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [CI][ROCm] Prevent Model Executor apt stalls (#58607) Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> * [Qwen4Exp][ROCm] PLE n-gram table CPU offload (#57497) Signed-off-by: Mathew Odden <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: opencode+deepseek-v4-flash+vllm <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [MM] Add Triton kernel for mm_input_normal. (#56798) Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> Co-authored-by: Isotr0py <[email protected]> * [Structured Outputs] Parse Lark grammars natively in the xgrammar backend (#58321) Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Signed-off-by: Bugen Zhao <[email protected]> * [ROCm] Credit ROCm/aiter for the block32 GEMM's packed kernel and in-launch split-K (#58659) Signed-off-by: Lingpeng Jin <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Frontend] Handle Disable Thinking in /v1/messages (#58613) Signed-off-by: jryberg <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: jryberg <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Bugfix] Stop leaking the internal field name in the max_tokens validation error (#58336) Signed-off-by: shallow10 <[email protected]> * [Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA (#55528) Signed-off-by: lz <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Perf][DSv4] Fuse inverse RoPE + FP8 quant into FlashInfer sparse MLA (#58621) Signed-off-by: Yongye Zhu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [UX][Frontend] Introduce `vllm preload` cli for fast restart (#56680) Signed-off-by: Isotr0py <[email protected]> * [MoE] Defer the TRTLLM-Gen top-k finalize on the modular path (#58635) Signed-off-by: Yongye Zhu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Docs] Add return annotation to `fused_mm_input_norm_triton` (#58687) Signed-off-by: Harry Mellor <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [CI] Shard (H100) Helion Kernels five ways (#58645) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Core] Model console logging as CLI configuration (#57205) Add `--logging-config` CLI argument which can be supplied as JSON or using dotted arguments. The `--log-level` argument is provided for convenience, and `--log-config-file` is deprecated in favor of `--logging-config.pylogging_config_file`. Signed-off-by: Mark McLoughlin <[email protected]> Co-authored-by: AI Assistant <[email protected]> * [ROCm][CI] Pass weight_shape in MXFP8 block32 linear tests (#58698) Signed-off-by: Djordje Ramic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [SpecDecode] Add LiLiCorr drafter (#57934) Signed-off-by: Andrii Skliar <[email protected]> Signed-off-by: Andrii Skliar <[email protected]> Co-authored-by: Andrii Skliar <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Benjamin Chislett <[email protected]> * [MRV2] Minor model_runner.py code cleanup (#58610) Signed-off-by: Nick Hill <[email protected]> * [Bugfix] Fix generative scoring body cancellation (#57729) Signed-off-by: Taneem Ibrahim <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Pooling] Preserve reranker tokenization with document limits (#57666) Signed-off-by: Taneem Ibrahim <[email protected]> Co-authored-by: Wentao Ye <[email protected]> * [Kernel][Perf] Register-resident path for per-token-group 8-bit quant (#55330) Signed-off-by: chao.huan <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [Perf][Kernel] Vectorized flat abs-max for dynamic per-tensor FP8 quantization (#58194) Signed-off-by: Monishver Chandrasekaran <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Misha Goin <[email protected]> * [Core][Logging] Fix JSON logging process decoration (#57957) Signed-off-by: Mark McLoughlin <[email protected]> * [Bugfix] Stop allocator fragmentation from shrinking the KV cache during memory profiling (#58430) Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Bugfix][Frontend] Keep logprobs of parser-suppressed streaming chunks (#58583) Signed-off-by: errmakov <[email protected]> Co-authored-by: Yanxiao Zhao <[email protected]> Co-authored-by: Prakhar Agarwal <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Bugfix][GLM-5.3-Flash] SM90 sparse MLA: index_kpool mismatch leads to corruption via unread query token (#58704) Signed-off-by: Matt Mastracci <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Benchmark] Record model_id in bench latency/throughput --output-json (#58112) Signed-off-by: yashasvi <[email protected]> * [Bugfix][GLM-5.3-Flash] kpool corruption with speculative decoding (#58454) Signed-off-by: Matt Mastracci <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Yifan Qiao <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [GLM5.3 Perf] Optimize glm 5.3 metadata op, 1.6~4.8x kernel level performance improvement (#58450) Signed-off-by: yewentao256 <[email protected]> * [Bugfix][KV Connector] Reap expired NIXL leases behind a heartbeated head (#58292) Signed-off-by: GokayAI <[email protected]> Co-authored-by: GokayAI <[email protected]> * [ROCm][Kimi-K3] Optimize low-concurrency speculative KDA (#58045) Signed-off-by: jiacao-amd <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Cover the AITER MQA logits dispatch on gfx950 (#58724) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Add quantized MoE serving test for gfx950 (#58748) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Test AMD DeepSeek V4 MoE routing against a PyTorch reference (#58740) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Perf] DiffusionGemma: one-pass sampler statistics kernel (#58226) Signed-off-by: Matt Mastracci <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]> * [ROCm][CI] Expand single-GPU coverage on MI355 DPX (#57599) Signed-off-by: Sheral Kumar <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [CI] [MRV2] Restore MRV2 pp dp coverage (#57735) Signed-off-by: Taneem Ibrahim <[email protected]> * [Bugfix][CI] Report subprocess test skips as skips, not passes (#58701) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Run the MLA attention+quant fusion test on ROCm (#58717) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm] Bump torch 2.13, triton 3.8, torchaudio, torchvision (#50605) Signed-off-by: Rohan Potdar <[email protected]> Signed-off-by: Rohan138 <[email protected]> Signed-off-by: jpvillam <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]> Co-authored-by: jpvillam <[email protected]> * [Feature][Frontend] Add granite_thinking_parser reasoning parser for Granite 4.2 (#55957) Signed-off-by: Yousaf shah <[email protected]> Co-authored-by: sfeng33 <[email protected]> * [Bugfix][Frontend] Respect max_output_tokens in the Harmony tool-call loop (#58551) Signed-off-by: errmakov <[email protected]> Co-authored-by: Du Bin <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Flora Feng <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix][Frontend] Document 404 response for `/generative_scoring` (#58788) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Kimi Code <[email protected]> * [Perf][Frontend] Defer reasoning usage recounts for non-continuous chat streams (#56067) Signed-off-by: Cheng Rui <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Bugfix] Default missing detail for Responses API input images (#57241) Signed-off-by: Yifan Zong <[email protected]> Co-authored-by: Ben Browning <[email protected]> * [watermarking] golden tests for backwards compatibility (#56809) Signed-off-by: Raphael Rialland <[email protected]> Signed-off-by: Simon Veitner <[email protected]> Co-authored-by: Simon Veitner <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Bugfix] Don't drop the rest of the allocator config when toggling expandable segments (#57982) Signed-off-by: Oxana Korzh <[email protected]> * [AuxOutput] Only require Model Runner V2 on GPU platform (#58205) Signed-off-by: Linkun Chen <[email protected]> * [Pooling] Preserve BERT-family heads for raw logits (#57664) Signed-off-by: Taneem Ibrahim <[email protected]> Co-authored-by: Wentao Ye <[email protected]> * fix: perf: use startswith(x, i) instead of string slicing to avoid O(N^2) (#52580) Signed-off-by: Ricardo-M-L <[email protected]> Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]> * [Bugfix] V1: fix allowed_token_ids_mask aliasing in InputBatch.swap_states (#48419) Signed-off-by: Rui Zhu <[email protected]> Co-authored-by: Claude <[email protected]> * [Bugfix][Reasoning] Count Kimi K3 reasoning tokens (#58372) Signed-off-by: Elvir Crncevic <[email protected]> Signed-off-by: Flora Feng <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [CI] Split (H200 MIG/MI300) Basic Correctness into named jobs (#57054) Signed-off-by: Thang Nguyen <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> Co-authored-by: Codex <[email protected]> * [Bugfix][Frontend] Fix Inkling tool name leaking into content after reasoning (#58792) Signed-off-by: Baljinder Hothi <[email protected]> Co-authored-by: Flora Feng <[email protected]> * [Core] Bound draft-token RPC waits by the execute-model timeout (#58779) Signed-off-by: VS Chandra Mourya <[email protected]> Co-authored-by: VS Chandra Mourya <[email protected]> * [Perf][DSv4.1] Shard the Engram wkv projection across TP ranks (#58678) Signed-off-by: Shuolei Wang <[email protected]> * [RL] Add sharding-aware NCCL M2N weight transfer (#51520) Signed-off-by: Ke Wen <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: aoshen02 <[email protected]> * [Bugfix][ROCm] AMD-Quark mixed-precision DeepSeek-V4.1 support (#57071) Signed-off-by: Xiao Yu <[email protected]> Co-authored-by: Cursor <[email protected]> * [Bugfix][Frontend][Rust Frontend] Update DeepSeek V4.1 Flash reasoning effort mappings (#58316) Co-authored-by: Claude <[email protected]> Co-authored-by: Bugen Zhao <[email protected]> Signed-off-by: zhec <[email protected]> * [CI] Only isolate the registry tests that need a fresh process (#58764) Signed-off-by: aarushjain29 <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Spec Decode] Enable Gemma4 DSpark adaptive verification with FlashInfer (#57263) Signed-off-by: zixi-qi <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [CI] Split (B200) Miscellaneous Kernels into mHC, FLA Ops and Misc named jobs (#58609) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Kevin H. Luu <[email protected]> * [Security] Gate per-request multimodal processor kwargs (#58830) Signed-off-by: Juan Pérez de Algaba <[email protected]> Co-authored-by: Cursor <[email protected]> * [Elastic EP] Fix EPLB load statistics during scaling (#58473) Signed-off-by: Itay Alroy <[email protected]> Co-authored-by: Tyler Michael Smith <[email protected]> * [Refactor] Remove dead tests utils (#58803) Signed-off-by: yewentao256 <[email protected]> * [Perf][MoE] Index expert mapping lookups in RoutedExperts.load_weights (#58720) Signed-off-by: Willian <[email protected]> Signed-off-by: mgoin <[email protected]> Co-authored-by: mgoin <[email protected]> * [Bugfix] Fix Anthropic Thinking Disabled with P/D (#58786) Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Bugfix][Frontend] Detect Anthropic inline-system merge against the resolved chat template (#58754) Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Mypy] Fix mypy typing for Qwen and Qianfan models (#58046) Signed-off-by: Ashraf Bhuiyan <[email protected]> * [CI] Reduce CUDA graph mode test overhead (#58749) * [GLM5.3 Bug] Fix sparse indexer attn topk backend selection (#58594) Signed-off-by: yewentao256 <[email protected]> * [CI] Stabilize batch submission in full CUDA graph tests (#58810) * [Bugfix][DSV4.1] Avoid host sync in ViT CUDA graph replay metadata (#58499) * [Security] Harden message sanitization (#58832) Signed-off-by: DarkLight1337 <[email protected]> * [Kernel][DSV4.1] Fuse MoE finalize into the TP all-reduce + mHC boundary (#58586) Signed-off-by: Yongye Zhu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [PCP][DCP] Support DCP target model with non-DCP Dspark (#56723) Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: Codex <[email protected]> * [Kernel] Bump FlashKDA to keep the recurrent state in fp32 (#58846) Signed-off-by: Simon Veitner <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Metrics][KV Offload] Add Prometheus metrics for SimpleCPUOffloadConnector (#57251) Signed-off-by: Vincent <[email protected]> * [mooncake] support CUSTOM_MEM_POOL in vllm (#49300) Signed-off-by: bruce.xu <[email protected]> Signed-off-by: Yifan Qiao <[email protected]> Co-authored-by: bruce.xu <[email protected]> Co-authored-by: Yifan Qiao <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Qwen3.8-Flash-Next] Avoid memory fragmentation in QSA indexer logits workspace (#57105) Signed-off-by: Thien Tran <[email protected]> * [Bugfix] Disable sequence parallelism / async TP under batch invariance and add a TP regression test (#56377) Signed-off-by: LioEinaudi <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm][Kimi-K3] Make VLLM_ROCM_USE_AITER_MOE_SITUV2 select a4w4/a8w4/a16w4 (#58201) Signed-off-by: Hongxia Yang <[email protected]> * [Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL (#50047) Signed-off-by: Yannik Hinteregger <[email protected]> Co-authored-by: xijiade.aihemaiti <[email protected]> * [Bugfix] V1: clear stale allowed_token_ids mask in InputBatch.condense (#43931) Signed-off-by: Varshith <[email protected]> Co-authored-by: Claude <[email protected]> * [Perf][DSv4.1] Fuse small-batch WO-A with inverse RoPE and MXFP8 quant on SM100/SM103 (#58634) Signed-off-by: Juntian Liu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [Attention][CPU] Use zentorch SDPA for CPU MLA prefill (#54967) Signed-off-by: Rakul Chauhan <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Cursor <[email protected]> * [Perf][Attention] Remove D2H sync from FlashInfer SM90 sparse MLA plan under async scheduling (#58684) Signed-off-by: NickLucche <[email protected]> * [CI] Split (H200 MIG 35GB) Spec Decode Speculators + MTP into 4 named jobs (#57237) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Thang Nguyen <[email protected]> Co-authored-by: Kimi Code <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [ROCm][Perf] Enable layer-aware CSA2 multi-stream overlap for DeepSeek-V4.1-Flash (#57407) Signed-off-by: Shanshan Shen <[email protected]> * [Perf][Pooling] Avoid blocking seq_lens GPU-to-CPU copy for pooling in FlashInfer metadata builder (#57214) Signed-off-by: frankwang28 <[email protected]> Co-authored-by: Misha Goin <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Vadim Gimpelson <[email protected]> * [Bugfix] Support repsonse_format + tool_choice=auto (#56086) Signed-off-by: Artem Perevedentsev <[email protected]> Co-authored-by: Yufeng He <[email protected]> Co-authored-by: pablopupo <[email protected]> Co-authored-by: hubunt <[email protected]> Co-authored-by: Claude Sonnet 5 <[email protected]> * [Fast Start] Add `/health` endpoint for the weight cache daemon (#58552) Signed-off-by: Xun Sun <[email protected]> Co-authored-by: Isotr0py <[email protected]> * [Perf][MoE] Use fused MiniMax2 routing with non-unit routed scaling (#58880) Signed-off-by: zjy0516 <[email protected]> * [ROCm][Bugfix] Fall back to default GEMM for CPU tensors on ROCm builds (#58923) Signed-off-by: fai <[email protected]> * [Bugfix][KV Connector] Retry Mooncake bootstrap registration on timeout (reopens #55763) (#58919) * [Frontend] Switch Python Harmony dependency to oss-harmony (#55128) Signed-off-by: Anton Peganov <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: OpenAI Codex <[email protected]> * [ROCm] Bump AITER to v0.1.23 (#58867) Signed-off-by: fai <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [Bugfix][Frontend] Count Responses reasoning tokens per tool round (#58927) Signed-off-by: sfeng33 <[email protected]> * [KimiViT][Perf] Fuse per-layer QK RoPE into one in-place kernel (#58651) * [Mypy] Fix mypy typing for Ultravox and Unlimited-OCR models (#58239) Signed-off-by: Ashraf Bhuiyan <[email protected]> * [Bugfix][Frontend] Sample batched chat completions from the adjusted requests (#58929) Signed-off-by: sfeng33 <[email protected]> * [Bugfix][Frontend] Use a fresh parser per choice in non-streaming chat completions (#58939) Signed-off-by: sfeng33 <[email protected]> * [Bugfix][EPD] Skip sampling for encoder-only async steps (#58490) Signed-off-by: Tianyu Guo <[email protected]> * [CPU] Build CPU wheels on Ubuntu 22.04 with AMX-FP8 support (#58515) Signed-off-by: zhejiangxiaomai <[email protected]> Signed-off-by: jiang1.li <[email protected]> Co-authored-by: jiang1.li <[email protected]> * [Test][ROCm] Stabilize the mixed OLMoE LoRA test (#58945) Signed-off-by: Andreas Karatzas <[email protected]> * [Model] Enable LoRA support for RobertaForSequenceClassification (#58884) Signed-off-by: jz-yolo <[email protected]> Signed-off-by: Jane Zhu <[email protected]> Co-authored-by: jz-yolo <[email protected]> * [Bugfix][Frontend] Apply Harmony adjust_request in batched chat completions (#58958) Signed-off-by: sfeng33 <[email protected]> * [Minimax-M3] Add Encoder CUDA graph support (#58673) Signed-off-by: Thien Tran <[email protected]> Co-authored-by: Codex <[email protected]> * [Perf][MRV2] Reuse Mamba/GDN metadata across KV cache groups (#58762) Signed-off-by: Luca Motz <[email protected]> Co-authored-by: Xin Yang <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Codex <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> * [Bugfix] Fix the two multimodal root tests that fail on main (OpenPangu-VL embed merge, MiMo sink test fixture) (#58900) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Benchmark] Add Responses API backend to vllm bench serve (#54628) Signed-off-by: QHarshil <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Chauncey <[email protected]> * [CI] Allowlist-shrink batch 1: wire 17 root-level tests + drop 5 stale watermarking entries into misc.yaml (#58055) Signed-off-by: Turner Jabbour <[email protected]> Co-authored-by: Claude Sonnet 5 <[email protected]> * [CPU] Use accelerator memory API in DiffusionGemma (#58964) Signed-off-by: zhejiangxiaomai <[email protected]> Co-authored-by: Li, Jiang <[email protected]> * [CPU] Add video inferencing via torchcodec on s390x (#58693) Signed-off-by: Rehan Khan <[email protected]> Co-authored-by: Li, Jiang <[email protected]> * [Skills] Update kernel-microbenchmark to include ROCm (#58646) Signed-off-by: Thien Tran <[email protected]> Co-authored-by: Kimi <[email protected]> * [Misc] Name each backend and its kernel block sizes in block-size errors (#58557) Signed-off-by: Eugenio "Jay" Zuccarelli <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [CI] Shard (H200 MIG 35GB / MI355 DPX) Entrypoints Integration (Pooling) into named jobs (#58652) Signed-off-by: Thang Nguyen <[email protected]> Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: khluu <[email protected]> * [CI] Split Dynamic Shapes out of (H200 MIG 35GB) PyTorch Compilation + (MI300) mirror (#58451) Signed-off-by: Thang Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Bugfix][Qwen4Exp] Release the profiling KV cache held by QSA key views (#58961) Signed-off-by: Zihua Wu <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [CPU] Include vLLM Recipes tooling in release image to deploy models using vLLM Recipes (#58796) Signed-off-by: louie-tsai <[email protected]> * Revert "[CI] Shard (H200 MIG 35GB / MI355 DPX) Entrypoints Integration (Pooling) into named jobs (#58652)" (#59011) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [Perf][Qwen4Exp] Fuse HC down projection and SiLU on NVIDIA (#58957) Signed-off-by: Thien Tran <[email protected]> * [Bugfix][Model] Gemma4: register aliased embedding scalars as buffers (#54213) Signed-off-by: Yannick Schnider <[email protected]> * [Bugfix][Spec Decode] Implement get_top_tokens() on the ROCm DeepSeek V4 MTP drafter (#57568) Signed-off-by: BaoYunkai <[email protected]> * [Perf][Qwen3.8] Reduce PLE metadata construction overhead (#58114) Signed-off-by: zjy0516 <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [ROCm][Refactor] Move DeepSeek-V4/V4.1 multi-stream overlap gate to ROCm platform (#58983) Signed-off-by: Shanshan Shen <[email protected]> * [Multimodal] Avoid extra d2d for encoder cudagraph with fused input norm (#56711) Signed-off-by: Isotr0py <[email protected]> * [Bugfix][Logging] Preserve application log record factories (#58747) Signed-off-by: mgoin <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Bugfix] Use the correct repository revision for secondary artifact loaders (#57461) Signed-off-by: Clinton Thomas <[email protected]> Co-authored-by: Lucas Bourtoule <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [ROCm][Perf] Kimi-K3 Enable sharded latent MoE up-projection under EP (#54956) Signed-off-by: Xavier Aguilar <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Feature] Add fixed-token prefill scoring (#54335) Signed-off-by: aoshen02 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Co-authored-by: aoshen02 <[email protected]> Co-authored-by: Claude Opus 5 <[email protected]> * [LoRA] Support variable num_labels for sequence classification (#57766) Signed-off-by: linitra24 <[email protected]> Co-authored-by: Jee Jee Li <[email protected]> * [ROCm][Perf] Replace torch.topk in DSA candidate block selection (#58208) Signed-off-by: fai <[email protected]> Co-authored-by: TJian <[email protected]> Co-authored-by: Cursor Agent <[email protected]> * [ROCm]Keep LMCache OpenTelemetry on the image's 1.40 stack (#59056) Signed-off-by: Micah Williamson <[email protected]> * [Bugfix][Kimi-K3] Refresh DSpark context KV cache pointers after the KV cache is re-bound (#58814) Signed-off-by: Oxana Korzh <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> * [KVConnector][NIXL] Support packed MLA KV layouts in pipeline-parallel push prefill (#50499) Signed-off-by: zixi-qi <[email protected]> * [Quant] Use canonical N-first weight format for CT WNA16 MoE (#52798) Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: HDCharles <{"message":"Not Found","documentation_url":"https://docs.github.com/rest/users/emails#list-email-addresses-for-the-authenticated-user","status":"404"}> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [CI][Bugfix] Relax packed_qk_rope_ correctness test to one ULP (#59008) Signed-off-by: Stefan Koncarevic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][CI] Pin OpenTelemetry to LMCache's cap in the ROCm images (#59051) Signed-off-by: Rohan Potdar <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> * [ROCm][CI] Increase timeout for Entrypoints Unit (#59076) Signed-off-by: Djordje Ramic <[email protected]> Co-authored-by: Andreas Karatzas <[email protected]> * [ROCm][MLA] Add an AITER ASM round-robin decode route for DCP multi-token verify (#56861) Signed-off-by: Xiaohu Guo <[email protected]> Co-authored-by: Claude Opus 5 (1M context) <[email protected]> Co-authored-by: seungrokj <[email protected]> Co-authored-by: Cursor <[email protected]> * [Bugfix] Don't sync-police or retry FlashInfer all-reduce workspace creation in eager mode (#58498) Signed-off-by: khluu <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [Bugfix] Fix resumable request + async scheduling handoff race (#58259) Signed-off-by: Yifan Zong <[email protected]> * [Model Runner V2][Spec Decode] Support spec decode with draft model (#43091) Signed-off-by: Icey <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> * [Perf][MRV2] Allow FULL decode graphs for one-token prompt tails (#58400) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Nicolò Lucchesi <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Yongye Zhu <[email protected]> * [Bugfix][Scheduler] Preserve logprobs across streaming continuations (#57447) Signed-off-by: 0xsensei <[email protected]> * [Mypy] Fix mypy typing for Voxtral and vision models (#58251) Signed-off-by: Ashraf Bhuiyan <[email protected]> Co-authored-by: Wentao Ye <[email protected]> * [Core] Rework scheduler `skipped_waiting` queue (#58947) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Robert Shaw <[email protected]> * [Bugfix][MRV2][Spec Decode] Reject draft slots that were never proposed (#58784) Signed-off-by: zixi-qi <[email protected]> Co-authored-by: Claude <[email protected]> Co-authored-by: Nick Hill <[email protected]> * [ROCm][CI] Add missing test coverage for upstream parity (#50519) Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Stefan Koncarevic <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Co-authored-by: Stefan Koncarevic <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Bugfix][Scheduler] Refresh max tokens for streaming continuations (#57676) Signed-off-by: 0xsensei <[email protected]> Co-authored-by: OpenAI Codex <[email protected]> * [Perf][Spec Decode] Avoid triton recompiles in the acceptance estimator (#57107) Signed-off-by: Giancarlo Delfin <[email protected]> Co-authored-by: Woosuk Kwon <[email protected]> * [CI/Build] Skip the snapshot runtime on CUDA 12.x images (#59118) Signed-off-by: khluu <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit aedaba8664f67d8ae1538e5e5cec2b1ce3f258dd) * [XPU][CI]Skip test_abort_timeout_on_prefiller in nightly (#58307) Signed-off-by: zengxian <[email protected]> Co-authored-by: Kunshang Ji <[email protected]> (cherry picked from commit 09c47db1ca080793dc2351144cb39513fd7984ca) * [Bugfix][Engram] Keep THP tables private when resolving shared memory (#59068) Signed-off-by: Ren Yuzhou <[email protected]> Co-authored-by: Claude <[email protected]> (cherry picked from commit ec5e0c352f079c2cb8f46752fd0317a2745050bf) * [ROCm][CI] Expand MI355 mirrors and route MIG-sized jobs to DPX (#59137) Signed-off-by: Andreas Karatzas <[email protected]> (cherry picked from commit 6ebb5bd64f5ef75dbe1471ee88c009003b3d03ec) * [Bugfix][Mamba] Keep the prompt-end prefill checkpoint under sparse retention (#59146) Signed-off-by: Jared Wen <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Jiangyun Zhu <[email protected]> (cherry picked from commit d882bddbeab6b4a0d5861dfcb171bf61ce2109d6) * [Core][BugFix] Tag prefix-cache extra keys by source (#51899) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Lucas Bourtoule <[email protected]> Co-authored-by: Tai An <[email protected]> Co-authored-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 765872e7ed5b6f372f0898a67e48d3365fa28139) * [Bugfix][Frontend] Reject LoRA adapters named after a served model (#59286) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 105a4e097b905127b1d5f09a7e93bd1095e87a6c) * [Dependency] Upgrade FlashInfer to 0.7.0.post1 (#59323) Signed-off-by: wzhao18 <[email protected]> (cherry picked from commit 678baf53724e06cef7f07628ae8fd4cc6c96f11a) * [Bugfix][HiSparse] Resolve MTP verification rows with a union residency kernel (#59235) Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 2798f668608155bc8a8c74cb97e3ddd0d3053085) * [Bugfix][HiSparse] Never allocate GPU pages without host backing (#59036) Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 90e13fc757cffc11b55689d9cc33d844d9606516) * [CPU][Whisper] Support W4A16 quantized Whisper on the CPU WNA16 kernel (#58268) Signed-off-by: Harshal Adhav <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Li, Jiang <[email protected]> (cherry picked from commit 5faf81a4297921c699f9aa58f9c6395e95ede718) * [Core] Bound UniProc EngineCore startup threads to available CPUs (#58946) Signed-off-by: Andreas Karatzas <[email protected]> (cherry picked from commit 866fa130fac1e1252330a680fbed45ad05eba64f) * [KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA (#57652) (cherry picked from commit 5463fe4962785cdc3383477bf3af6533e7647dfd) * [Bugfix][HiSparse] Preserve host prefix publication after request completion (#59007) Signed-off-by: Yueh-Ting Chen <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> (cherry picked from commit ff1b87cca25690fef6bd12667fd0d26690d949af) * [Bugfix][HiSparse] Adopt GPU prefix copies after the hit's allocation (#59282) Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit 3eb6cec22ad9bb098393021b956feaa97081fc85) * [Bugfix][HiSparse] Stop the host pool feeding device KV cache residency metrics (#58725) Signed-off-by: Yueh-Ting Chen <[email protected]> Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 73c7cae4d746f64c22677348d1cc120eea6f7439) * [Bugfix][Core] Fix mamba prefill checkpoint block reservation and prompt-end eviction in align mode (#59175) Signed-off-by: kliuae <[email protected]> Co-authored-by: Nick Hill <[email protected]> (cherry picked from commit 7e583e615c20ee4ff0cd82aac592c7d6310aa7c3) * [Core] Include the LoRA path in prefix-cache block hashes (#59335) Signed-off-by: Nick Hill <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit c055c1e075ed461cff922aa710d387281f651940) * [Perf][PP] Skip sampled-token broadcasts whose requests leave the engine (#58542) Signed-off-by: LostFox11 <[email protected]> Signed-off-by: Nick Hill <[email protected]> Co-authored-by: LostFox11 <[email protected]> Co-authored-by: Nick Hill <[email protected]> (cherry picked from commit 7314360c9eebb3b4b460ec49f6606ef0a08fcae2) * [CPU][Zen] Add DA8W4 (W4A8) int4 support for dense and MoE layers (#54024) Signed-off-by: R <[email protected]> Signed-off-by: Ganesh R <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> (cherry picked from commit e12291d7332db897885fdc0e6ed61b28c969d743) * [Model Runner V2] Support randomized dummy inputs (#58411) Signed-off-by: Robert Shaw <[email protected]> Co-authored-by: Robert Shaw <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com> Co-authored-by: Nick Hill <[email protected]> (cherry picked from commit e5e38ba9b7d18f9746d989e389a51a94b0f96f6f) * [Bugfix][HiSparse] Fix a chunked-prefill preemption livelock (#59494) Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> (cherry picked from commit 3a6963664537ed21172e2ec12e96e3a2dcd3718c) * [Bugfix][HiSparse] Size the KV cache from the groups HiSparse allocates (#59450) Signed-off-by: Matthew Bonanni <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]> Co-authored-by: Lucas Wilkinson <[email protected]> (cherry picked from commit f7999d2e4489126f1c21868699f3890021f01b73) * [Bugfix][HiSparse] Fix MTP acceptance collapse under FULL graphs with a saturated GPU pool (#59309) Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Matthew Bonanni <[email protected]> Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]> Co-authored-by: Matthew Bonanni <[email protected]> (cherry picked from commit 4056c8ac1f8a7e8fb50cf1c56ff96649fcadccc6) * [CI] Drop a test that depends on #57930 from the #59309 backport Resolving the #59309 cherry-pick conflict in tests/v1/kv_connector/unit/test_hisparse_connector.py pulled in test_scheduled_prefix_hit_publishes_adopted_copies from #57930, which is not on this branch; it imports _allocate_scheduled from tests.v1.core.test_prefix_caching and fails on every platform. After this change the file matches the upstream #59309 diff. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: khluu <[email protected]> * [Bugfix] Fix minimax-m3 multimodal processor compatability with Transformers v5.18 (#59613) Signed-off-by: Isotr0py <[email protected]> (cherry picked from commit b558f160a2c0abcb5902acc3c91a14c38a4af173) * [Misc] Add Transformers version upper bound in requirements (#59614) Signed-off-by: Isotr0py <[email protected]> Co-authored-by: Harry Mellor <[email protected]> (cherry picked from commit 58b3298457dde7b4554b3b4e20b238c0ac2c3a65) * Forward-port the tkv/turbo-attn vLLM seam onto upstream v0.31.0 Squashes arbicity/vllm-turbo main @969c125e9 (upstream v0.28.0 + the seam: b0a14f2eb #28, 76a3e6d72, 6d4c75611 #29) into one commit and replays it onto upstream v0.31.0 (db9527a46, the commit vllm/vllm-openai:v0.31.0 is built from) as a 3-way merge against the real base. Upstream's v0.29-v0.31 KV-cache layout refactor (vllm#51718 and follow-ups) replaced the allocator the seam patched: one backing allocation, per-layer [B, H, N, C] views placed by the engine, page geometry read off the spec, and AttentionBackend.customize_spec applied to every layer's spec by both model runners. The seam is re-expressed on that and shrinks from 49 files to 27. Kept (re-applied on upstream's structure): - plugin KV-cache dtype registry; --kv-cache-dtype choices/type - TURBO_ATTN backend slot (now a distinct enum value: two None members made CUSTOM an alias of TURBO_ATTN), turbo-attn spelling, auto-default for plugin dtypes (#29), CUDA candidate once registered - lifecycle hooks on_model_loaded / on_draft_model_loaded / on_kv_cache_initialized / adjust_kv_budget, fail-loud dispatch to every backend in use - MLA wrapping in the selector; MLA chunked-context _get_gather_op - _tq_layer_idx injection for tkv layers - aggregated_layer_count: fused (composite) pages, now one shared page per fused set in upstream's single-allocation planner - per-group KV pool for hybrid models, re-expressed on v0.31's single backing allocation: the O(1) Mamba/GDN state groups get their own BlockPool sized for max_num_seqs, a region of the allocation after the attention pool; their pages are not unified with attention pages; per-pool admission, events and usage; CoW copies and warmup/profiling block ids per pool; VLLM_SAMPLER_RESERVE_MIB headroom Moved to upstream's extension points (turbo-attn plugin side): - get_kv_cache_spec_class (Attention, MLAAttention, hybrid alignment) -> AttentionBackend.customize_spec - KVCacheSpec.get_manager_class -> KVCacheSpecRegistry MRO lookup - get_supported_kv_cache_dtypes -> supported_kv_cache_dtypes ClassVar - get_kv_cache_shape(kv_cache_spec=...) passthrough and the backend-managed-dtype shape coercion -> spec-driven views Dropped: - spec_decode_warmup.py: upstream registers the same kernels with its JIT warmup registry (aed894c19, vllm#56323) - turbo_attn_warmup.py, utils/cutedsl_cache.py: superseded by turbo-attn's own prefill prewarm (on_kv_cache_initialized) and CuTeDSL cache - padded-page block fill (the split pool removes the page unification it worked around), drain hook / on_kv_manager_created, KVBlockZeroer clamp: no turbo-attn consumer - rotary fast-path registry: upstream guards the import (1f60771c7, vllm#42679) - --kv-cache-dtype-skip-layers-dtype, VLLM_KV_CACHE_SKIP_LAYERS_DTYPE, Triton fused fp8 GEMM hook, gsm8k startup waits, notify-turbo-attn workflow, draft-backend inheritance: unused or superseded - FA2 varlen paged split-K patch: never reached the overlay image (the image installs no compiled _vllm_fa2_C) and the FA pin moved PROTOCOL.md rewritten for the v0.31.0 seam. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> --------- Signed-off-by: Lai, Yejing <[email protected]> Signed-off-by: priyansh jain <[email protected]> Signed-off-by: Taneem Ibrahim <[email protected]> Signed-off-by: reidliu41 <[email protected]> Signed-off-by: Bugen Zhao <[email protected]> Signed-off-by: RyanMa29 <[email protected]> Signed-off-by: R <[email protected]> Signed-off-by: Shrey Gajjar <[email protected]> Signed-off-by: Zijing Liu <[email protected]> Signed-off-by: zhenwei-intel <[email protected]> Signed-off-by: fai <[email protected]> Signed-off-by: Harry Mellor <[email protected]> Signed-off-by: Linze-Shi <[email protected]> Signed-off-by: yewentao256 <[email protected]> Signed-off-by: Yueh-Ting Chen <[email protected]> Signed-off-by: Chenglun Hu <[email protected]> Signed-off-by: hclsys <[email protected]> Signed-off-by: Wauplin <[email protected]> Signed-off-by: Thomas Ortner <[email protected]> Signed-off-by: nightcityblade <[email protected]> Signed-off-by: Nick Hill <[email protected]> Signed-off-by: mgoin <[email protected]> Signed-off-by: Felix Marty <[email protected]> Signed-off-by: Vincent Cave <[email protected]> Signed-off-by: Shiksha Patel <[email protected]> Signed-off-by: Matt Mastracci <[email protected]> Signed-off-by: Aaron Kang <[email protected]> Signed-off-by: Stefan Koncarevic <[email protected]> Signed-off-by: Juntian Liu <[email protected]> Signed-off-by: Jellow <[email protected]> Signed-off-by: Jellow <[email protected]> Signed-off-by: Lin, Fanli <[email protected]> Signed-off-by: Fanli Lin <[email protected]> Signed-off-by: Sunita Nadampalli <[email protected]> Signed-off-by: Milosz Grunwald <[email protected]> Signed-off-by: haosenwang1018 <[email protected]> Signed-off-by: Netanel Haber <[email protected]> Signed-off-by: Yifan Qiao <[email protected]> Signed-off-by: khushali9 <[email protected]> Signed-off-by: Samyabrata Maji <[email protected]> Signed-off-by: SIDDARTHA REDDY <[email protected]> Signed-off-by: samuelkim7 <[email protected]> Signed-off-by: khluu <[email protected]> Signed-off-by: Thien Tran <[email protected]> Signed-off-by: S1ro1 <[email protected]> Signed-off-by: aoshen02 <[email protected]> Signed-off-by: 100milliongold <[email protected]> Signed-off-by: Divakar Verma <[email protected]> Signed-off-by: Wei Gong <[email protected]> Signed-off-by: lifulu <[email protected]> Signed-off-by: Yongye Zhu <[email protected]> Signed-off-by: Summer Yang <[email protected]> Signed-off-by: yisheng <[email protected]> Signed-off-by: wangxiyuan <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Mathew Odden <[email protected]> Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: wang.yuqi <[email protected]> Signed-off-by: Isotr0py <[email protected]> Signed-off-by: Lingpeng Jin <[email protected]> Signed-off-by: jryberg <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: shallow10 <[email protected]> Signed-off-by: lz <[email protected]> Signed-off-by: Lucas Wilkinson <[email protected]> Signed-off-by: Thang Nguyen <[email protected]> Signed-off-by: Mark McLoughlin <[email protected]> Signed-off-by: Djordje Ramic <[email protected]> Signed-off-by: Andrii Skliar <[email protected]> Signed-off-by: Andrii Skliar <[email protected]> Signed-off-by: chao.huan <[email protected]> Signed-off-by: Monishver Chandrasekaran <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: errmakov <[email protected]> Signed-off-by: Robert Shaw <[email protected]> Signed-off-by: yashasvi <[email protected]> Signed-off-by: GokayAI <[email protected]> Signed-off-by: jiacao-amd <[email protected]> Signed-off-by: Sheral Kumar <[email protected]> Signed-off-by: Andreas Karatzas <[email protected]> Signed-off-by: Rohan Potdar <[email protected]> Signed-off-by: Rohan138 <[email protected]> Signed-off-by: jpvillam <[email protected]> Signed-off-by: Yousaf shah <[email protected]> Signed-off-by: Cheng Rui <[email protected]> Signed-off-by: Yifan Zong <[email protected]> Signed-off-by: Raphael Rialland <[email protected]> Signed-off-by: Simon Veitner <[email protected]> Signed-off-by: Oxana Korzh <[email protected]> Signed-off-by: Linkun Chen <[email protected]> Signed-off-by: Ricardo-M-L <[email protected]> Signed-off-by: Rui Zhu <[email protected]> Signed-off-by: Elvir Crncevic <[email protected]> Signed-off-by: Flora Feng <[email protected]> Signed-off-by: Baljinder Hothi <[email protected]> Signed-off-by: VS Chandra Mourya <[email protected]> Signed-off-by: Shuolei Wang <[email protected]> Signed-off-by: Ke Wen <[email protected]> Signed-off-by: Xiao Yu <[email protected]> Signed-off-by: zhec <[email protected]> Signed-off-by: aarushjain29 <[email protected]> Signed-off-by: zixi-qi <[email protected]> Signed-off-by: Juan Pérez de Algaba <[email protected]> Signed-off-by: Itay Alroy <[email protected]> Signed-off-by: Willian <[email protected]> Signed-off-by: Ashraf Bhuiyan <[email protected]> Signed-off-by: DarkLight1337 <[email protected]> Signed-off-by: Vincent <[email protected]> Signed-off-by: bruce.xu <[email protected]> Signed-off-by: LioEinaudi <[email protected]> Signed-off-by: Hongxia Yang <[email protected]> Signed-off-by: Yannik Hinteregger <[email protected]> Signed-off-by: Varshith <[email protected]> Signed-off-by: Rakul Chauhan <[email protected]> Signed-off-by: NickLucche <[email protected]> Signed-off-by: Shanshan Shen <[email protected]> Signed-off-by: frankwang28 <[email protected]> Signed-off-by: Artem Perevedentsev <[email protected]> Signed-off-by: Xun Sun <[email protected]> Signed-off-by: zjy0516 <[email protected]> Signed-off-by: Anton Peganov <[email protected]> Signed-off-by: sfeng33 <[email protected]> Signed-off-by: Tianyu Guo <[email protected]> Signed-off-by: zhejiangxiaomai <[email protected]> Signed-off-by: jiang1.li <[email protected]> Signed-off-by: jz-yolo <[email protected]> Signed-off-by: Jane Zhu <[email protected]> Signed-off-by: Luca Motz <[email protected]> Signed-off-by: QHarshil <[email protected]> Signed-off-by: Turner Jabbour <[email protected]> Signed-off-by: Rehan Khan <[email protected]> Signed-off-by: Eugenio "Jay" Zuccarelli <[email protected]> Signed-off-by: Zihua Wu <[email protected]> Signed-off-by: louie-tsai <[email protected]> Signed-off-by: Yannick Schnider <Yannick.Schnider1@ib…

…head (vllm-project#58292) Signed-off-by: GokayAI <[email protected]> Co-authored-by: GokayAI <[email protected]>

mergify Bot added bug Something isn't working kv-connector labels Sep 23, 2026

gokay-ai marked this pull request as ready for review September 23, 2026 07:08

gokay-ai requested review from ApostaC, NickLucche, ivanium, orozery and xuechendi as code owners September 23, 2026 07:08

Copilot AI lite review requested due to automatic review settings September 23, 2026 07:08

claude Bot reviewed Sep 23, 2026

View reviewed changes

Copilot AI reviewed Sep 23, 2026

snadampal suggested changes Sep 23, 2026

View reviewed changes

njhill reviewed Sep 24, 2026

View reviewed changes

Comment thread vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py Resolved

Comment thread vllm/distributed/kv_transfer/kv_connector/v1/nixl/base_worker.py Resolved

[Bugfix][KV Connector] Keep NIXL reaper clock inside the method

9ee47c6

Address njhill review nits on vllm-project#58292: do not pass now as an arg, and early-return when _reqs_to_send is empty before building the expired list. Signed-off-by: GokayAI <[email protected]>

njhill approved these changes Sep 25, 2026

View reviewed changes

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Sep 25, 2026

Merge branch 'main' into cursor/nixl-lease-reaper-58222-1087

c97a25a

njhill enabled auto-merge (squash) September 25, 2026 15:43

njhill merged commit c89b5b4 into vllm-project:main Sep 25, 2026
67 checks passed

TianDi101 mentioned this pull request Oct 13, 2026

[RFC][IO] Framework-side RAS for MoRI-IO: harden vLLM's MoRIIO KV connector ROCm/mori#759

Open

2 of 13 tasks

njhill mentioned this pull request Oct 15, 2026

Fix/nixl lease reaper ordering #50789

Closed

tanmayv25 mentioned this pull request Oct 30, 2026

DEP: Prefill request cancellation in disaggregated serving ai-dynamo/dynamo#17718

Open

1 of 32 tasks

Conversation

gokay-ai commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

github-actions Bot commented Sep 23, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

liuzijing2014 commented Sep 23, 2026

Uh oh!

snadampal Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

gokay-ai Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

snadampal Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

gokay-ai Sep 24, 2026

Choose a reason for hiding this comment

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gokay-ai commented Sep 25, 2026

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

njhill commented Sep 25, 2026

Uh oh!

github-actions Bot commented Sep 25, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

gokay-ai commented Sep 23, 2026 •

edited

Loading