Skip to content

Add NVFP4 MOE support for Deepseek V4. - #42209

Merged
pavanimajety merged 10 commits into
vllm-project:mainfrom
sychen52:nvfp4_dsv4
May 22, 2026
Merged

Add NVFP4 MOE support for Deepseek V4.#42209
pavanimajety merged 10 commits into
vllm-project:mainfrom
sychen52:nvfp4_dsv4

Conversation

@sychen52

@sychen52 sychen52 commented May 10, 2026 •
edited by github-actions Bot
Loading

Copy link
Copy Markdown
Contributor

Purpose

Add NVFP4 MOE support for Deepseek V4
Add support for swiglu limit in NVFP4 MOE.

Test Plan

Added unittest.
Run NVFP4 MOE DSV4 checkpoint.

Test Result

passed.

Essential Elements of an Effective PR Description Checklist
  • The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
  • The test plan, such as providing test command.
  • The test results, such as pasting the results comparison before and after, or e2e results
  • (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

@mergify mergify Bot added deepseek Related to DeepSeek models nvidia labels May 10, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed May 10, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for gemm1_clamp_limit (also referred to as swiglu_limit) in NVFP4 MoE quantization and integrates the NVFP4 quantization algorithm into the DeepSeek V4 model. The review feedback focuses on performance optimizations, specifically recommending the use of torch.full for more efficient tensor initialization and suggesting that the rescaling of the clamp limit be moved from the inference hot path to the weight loading phase to eliminate redundant computations.

Comment thread vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py Resolved
Comment thread vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py Resolved
Comment thread vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py Resolved
Comment thread vllm/model_executor/layers/fused_moe/experts/trtllm_nvfp4_moe.py Resolved
@sychen52
sychen52 force-pushed the nvfp4_dsv4 branch 3 times, most recently from c1b34f2 to 7ea09fb Compare May 11, 2026 22:22
@sychen52
sychen52 marked this pull request as ready for review May 11, 2026 22:22
claude[bot]
claude Bot reviewed May 11, 2026
View reviewed changes

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@xinli-sw

xinli-sw commented May 12, 2026

Copy link
Copy Markdown
Contributor

@vadiklyutiy can you also review?

xinli-sw
xinli-sw reviewed May 12, 2026
View reviewed changes
Comment thread vllm/model_executor/models/deepseek_v4.py Outdated
ModelOptNvFp4Config,
)

self._nvfp4_config = ModelOptNvFp4Config(

@xinli-sw xinli-sw May 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this be read from model config? I assume that we want to support pro and flash eventually

@sychen52 sychen52 May 15, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The model level config is handled by DeepseekV4FP8Config, this self._nvfp4_config = ModelOptNvFp4Config is used to construct a ModelOptNvFp4FusedMoE layer. And group_size and is_checkpoint_nvfp4_serialized are hardcoded and it is the same for pro and flash and any nvfp4 MOE in the near future.

@sychen52
sychen52 requested review from xinli-sw May 15, 2026 22:53
@xinli-sw

xinli-sw commented May 19, 2026

Copy link
Copy Markdown
Contributor

@sychen52 could you fix pre-commit and docs?
@pavanimajety can you also review and if looks good kick off CI? thank you!

@mergify

mergify Bot commented May 19, 2026

Copy link
Copy Markdown
Contributor

Hi @sychen52, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally: 
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA May 19, 2026
@pavanimajety pavanimajety added the ready ONLY add when PR is ready to merge/full CI is needed label May 19, 2026
@mergify

mergify Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

Hi @sychen52, the pre-commit checks have failed. Please run:

uv pip install pre-commit>=4.5.1
pre-commit install
pre-commit run --all-files

Then, commit the changes and push to your branch.

For future commits, pre-commit will run automatically on changed files before each commit.

Tip

Is mypy failing?
mypy is run differently in CI. If the failure is related to this check, please use the following command to run it locally: 
# For mypy (substitute "3.10" with the failing version if needed)
pre-commit run --hook-stage manual mypy-3.10

@pavanimajety
pavanimajety merged commit fb21d8b into vllm-project:main May 22, 2026
91 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA May 22, 2026
This was referenced May 22, 2026
Closed
Open
Open
@pavanimajety

pavanimajety commented May 23, 2026

Copy link
Copy Markdown
Member

@sychen52 Can you please paste your evals results here?
Please add the vllm server command used and the relevant eval results. Thanks!

ZehaoLu98 added a commit to ZehaoLu98/vllm that referenced this pull request May 23, 2026
@ZehaoLu98
Squashed commit of the following:
46fe1bd 
commit 843715739b7b555c61dd6190cafb5ab7a44c41f1
Author: Yongye Zhu <[email protected]>
Date:   Fri May 22 13:06:31 2026 -0400

    [Refactor] Extract DeepSeek V4 sparse MLA impl into model folder (#43149)

commit b21f3d56d4a2ab5504b56504e87e0475c6d84eb2
Author: Dao007forever <[email protected]>
Date:   Fri May 22 09:14:11 2026 -0700

    [KV Connector] MooncakeStore: don't co-queue save with load to avoid double delayed-free (#43371)

    Signed-off-by: Dao Le <[email protected]>
    Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

commit c7624bea5ebba1c688eb4c216bd4ede7a94f2a82
Author: Zhanda Zhu <[email protected]>
Date:   Fri May 22 12:10:03 2026 -0400

    [Bugfix] Source num_qo_heads from Attention layers in Flashinfer/Triton metadata builders (#42650)

    Signed-off-by: zhanda <[email protected]>
    Co-authored-by: Shang Wang <[email protected]>

commit 91f5b92438a568c89e8b9d6c2c55de5a552291f6
Author: Bugen Zhao <[email protected]>
Date:   Fri May 22 23:22:11 2026 +0800

    [Rust Frontend] [Refactor] Extract a newtype for utility call ID (#43405)

    Signed-off-by: Bugen Zhao <[email protected]>

commit f0feb15e7fc521544d23c2d23de0e327a509876b
Author: Isotr0py <[email protected]>
Date:   Fri May 22 22:31:00 2026 +0800

    [Multimodal] Simplify ViT CUDA graph interfaces (#41234)

    Signed-off-by: Isotr0py <[email protected]>

commit fb21d8b4f9027f4642637c7bb0acc08c29dce387
Author: sychen52 <[email protected]>
Date:   Fri May 22 07:21:51 2026 -0700

    Add NVFP4 MOE support for Deepseek V4. (#42209)

    Signed-off-by: Shiyang Chen <[email protected]>

commit a377631d21cc97db678727455d33c4257435f417
Author: haosdent <[email protected]>
Date:   Fri May 22 22:06:24 2026 +0800

    [CI] Fix AMD docker build tests (#43329)

    Signed-off-by: haosdent <[email protected]>

commit d3a563501bcc6134a348f8458b1a797c94336f1f
Author: Ilya Markov <[email protected]>
Date:   Fri May 22 15:43:27 2026 +0200

    [EPLB] Change default EPLB communicator (#43110)

    Signed-off-by: Markov Ilya <[email protected]>
    Co-authored-by: Markov Ilya <[email protected]>

commit 15f7cd33dc8bd4d2270b70ba49d511827d2413ff
Author: Jee Jee Li <[email protected]>
Date:   Fri May 22 21:41:56 2026 +0800

    [LoRA] Reduce memory of 2D weights when EP is set (#42737)

    Signed-off-by: Jee Jee Li <[email protected]>

commit 79ff0ffa98dc8dd14a8651bce36ce6265ff4d35d
Author: Keyi Li <[email protected]>
Date:   Fri May 22 05:26:41 2026 -0700

    [BugFix] wire make_empty_intermediate_tensors on AyaVision and Voxtral (#43118)

    Signed-off-by: Keyi Li <[email protected]>
    Co-authored-by: Keyi Li <[email protected]>

commit 4658bf882b881287fc85797a23037aa91740b7a7
Author: Tobias Wasner <[email protected]>
Date:   Fri May 22 12:54:29 2026 +0200

    [Bugfix] Clear P0 mm sender cache on sleep/pause to fix mm_hash desync (#43001)

    Signed-off-by: Tobias Wasner <[email protected]>

commit b3c7ffcab82c2439726f8cb213800f6f38c023d3
Author: Taneem Ibrahim <[email protected]>
Date:   Fri May 22 05:43:33 2026 -0500

    [Misc] Replace assert with proper exceptions for security and validation in pooling (#43286)

    Signed-off-by: Taneem Ibrahim <[email protected]>
    Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
    Co-authored-by: Nick Hill <[email protected]>

commit d3d1cf6972607c53327b5ce1748e56a95fc41c37
Author: Ma Jian <[email protected]>
Date:   Fri May 22 18:22:45 2026 +0800

    [XPU]feat: add XPU fallback for MoE topk routing and MXFP4 backend (#42951)

    Signed-off-by: Ma Jian <[email protected]>
    Co-authored-by: Kunshang Ji <[email protected]>

commit 7e1b45a09252a5b513cd83116aa7a2f310220c34
Author: wangxiyuan <[email protected]>
Date:   Fri May 22 17:13:12 2026 +0800

    [Attention] Mamba attention module refactor (#41126)

    Signed-off-by: wangxiyuan <[email protected]>

commit 65b7a812a2dabd212d78c7b5b8a320b4efb9750d
Author: Li, Jiang <[email protected]>
Date:   Fri May 22 16:48:17 2026 +0800

    [CPU] Experimentally enable Triton and MRV2 (#43225)

    Signed-off-by: jiang1.li <[email protected]>

commit 2380bfc2104267914eea36015e2a347b9318c6c0
Author: wang.yuqi <[email protected]>
Date:   Fri May 22 16:43:14 2026 +0800

    [Docs] Note image preprocessing difference between qwen_vl_utils and vllm. (#43393)

    Signed-off-by: wang.yuqi <[email protected]>
    Signed-off-by: wang.yuqi <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

commit a7616977176e12ddb14c0daab00cd2a2161ba37c
Author: mrjunwan-lang <[email protected]>
Date:   Fri May 22 01:36:17 2026 -0700

    Fix the docker build failure in tpu-inference (#43360)

    Signed-off-by: mrjunwan-lang <[email protected]>

commit 694d9a81bbb07977e7a72a597acb44f6a848f774
Author: Nick Hill <[email protected]>
Date:   Fri May 22 00:25:10 2026 -0700

    [BugFix] Fix setuptools-rust dep in requirements files (#43377)

    Signed-off-by: Nick Hill <[email protected]>

commit 6bb8753db1076f498c240fffdd88b1ab983b7f40
Author: Weida Hong <[email protected]>
Date:   Fri May 22 15:21:35 2026 +0800

    Correcting the mock classes for MM GC tests (#43321)

    Signed-off-by: Weida Hong <[email protected]>

commit 025d4f5cd2617bb767663f9e7d62354039887757
Author: haosdent <[email protected]>
Date:   Fri May 22 15:13:59 2026 +0800

    [CI] Fix "test_awq_load[gemma4-moe-*]" failure (#43296)

    Signed-off-by: haosdent <[email protected]>

commit 5ea76fa89aa2e307f0d9a2e7fc19d13aed65a82f
Author: haosdent <[email protected]>
Date:   Fri May 22 14:24:18 2026 +0800

    [CI] Fix test_lora_with_spec_decode on V2 model runner (#43314)

    Signed-off-by: haosdent <[email protected]>
    Co-authored-by: Wentao Ye <[email protected]>

commit fa1ff88b3145d1897558408a9001c030c39383b9
Author: tc-mb <[email protected]>
Date:   Fri May 22 13:44:06 2026 +0800

    [Model] Fix MiniCPM-V 4.6 vit_merger qkv weight loading (#43213)

    Signed-off-by: tc-mb <[email protected]>

commit e746a2eebf09b1f99beb6b3c60a5ba9d2f8c4875
Author: Furkan F <[email protected]>
Date:   Fri May 22 07:28:23 2026 +0200

    [Model] Use `AutoWeightsLoader` for Voyage (#42972)

    Signed-off-by: Furkan Fidan <[email protected]>

commit 1fe3303983e1829fae25edfb0b93e8cbcfad96e6
Author: haosdent <[email protected]>
Date:   Fri May 22 12:15:22 2026 +0800

    [CI] De-flake renderers/test_hf.py::test_resolve_content_format_fallbacks[Qwen/Qwen-VL-string] (#43064)

    Signed-off-by: haosdent <[email protected]>

commit 8c8b1825eb26c1ffae776baaab16f2eebf92b7d3
Author: Xiaochang Wu <[email protected]>
Date:   Fri May 22 12:02:51 2026 +0800

    [XPU] Enable multiple key kernels for sparse attention (#37888)

    Signed-off-by: Xiaochang Wu <[email protected]>
    Signed-off-by: Wu, Xiaochang <[email protected]>
    Co-authored-by: Kunshang Ji <[email protected]>

commit 18a27cc9a3641cc1dd3eae5113b75c7ccc029b5f
Author: qizixi <[email protected]>
Date:   Thu May 21 20:36:22 2026 -0700

    [Bugfix] Make CuMemAllocator free callback stream-aware (#43020)

    Signed-off-by: zixi-qi <[email protected]>
    Co-authored-by: Claude <[email protected]>

commit 0ddd7dd6564f5e403a15bd7c973c7d358ec82454
Author: Wentao Ye <[email protected]>
Date:   Thu May 21 23:33:16 2026 -0400

    [Frontend] DP Supervisor (#40841)

    Signed-off-by: yewentao256 <[email protected]>
    Signed-off-by: Robert Shaw <[email protected]>
    Signed-off-by: Robert Shaw <[email protected]>
    Co-authored-by: robertgshaw2-redhat <[email protected]>
    Co-authored-by: Robert Shaw <[email protected]>
    Co-authored-by: Nick Hill <[email protected]>

commit 60af5c16ee64ea3c1c573d67d0773a713c87a22e
Author: ruizhang <[email protected]>
Date:   Thu May 21 20:32:31 2026 -0700

    [Frontend] Add truncation side to OpenAI endpoints (#43260)

    Signed-off-by: Rui Zhang <[email protected]>
    Signed-off-by: Rui Zhang <[email protected]>
    Co-authored-by: Rui Zhang <[email protected]>

commit 35d0141a0b68a188777e277e372f211098419f58
Author: Divakar Verma <[email protected]>
Date:   Thu May 21 23:17:54 2026 -0400

    [ROCm][CI] add warmup to mem_util test before measurement (#43236)

    Signed-off-by: Divakar Verma <[email protected]>

commit 86ccef7d4400a54441057773d8ffb1f61a20af94
Author: Simon Danielsson <[email protected]>
Date:   Fri May 22 05:06:40 2026 +0200

    [ROCm] Add XGMI backend for MoRI Connector (#41753)

    Signed-off-by: simondanielsson <[email protected]>

commit 2998a047aad7d48bf0399f19b36f1a4d749c59c2
Author: Chengze Fan <[email protected]>
Date:   Thu May 21 19:43:01 2026 -0700

    [Bugfix] Fix DSV4 Base model swiglu limit issue in FP8 path  (#42855)

    Signed-off-by: Chengze Fan <[email protected]>
    Signed-off-by: Chengze Fan <[email protected]>
    Co-authored-by: Lu Fang <[email protected]>

commit ba369b7eb5a3c6593b55f2005655d6586997fa07
Author: Isotr0py <[email protected]>
Date:   Fri May 22 10:26:05 2026 +0800

    [CI] Fix dockerfile dependency graph failure for pre-commit (#43378)

    Signed-off-by: Isotr0py <[email protected]>

commit 39910f2b25aacc09f5e7f166cdf0030b19f8b9e8
Author: Bugen Zhao <[email protected]>
Date:   Fri May 22 08:21:48 2026 +0800

    [Rust Frontend] Move code from `vllm-frontend-rs` (#43283)

    Signed-off-by: Bugen Zhao <[email protected]>
    Signed-off-by: Nick Hill <[email protected]>
    Signed-off-by: Eric Curtin <[email protected]>
    Signed-off-by: Dev-X25874 <[email protected]>
    Signed-off-by: Will.hou <[email protected]>
    Signed-off-by: Will.hou <[email protected]>
    Co-authored-by: Nick Hill <[email protected]>
    Co-authored-by: Eric Curtin <[email protected]>
    Co-authored-by: Dev-X25874 <[email protected]>
    Co-authored-by: Will.hou <[email protected]>
    Co-authored-by: Will.hou <[email protected]>

    Please see https://github.com/Inferact/vllm-frontend-rs for full original commit history.

commit 39d5fa96a7c687f9ed7e14a5a52064965356cede
Author: Lanze Liu <[email protected]>
Date:   Thu May 21 15:42:42 2026 -0700

    [Bugfix] Zero stale is_prefilling in padded CUDA graph rows for Mamba (#41873)

    Signed-off-by: Lanze Liu <[email protected]>

commit 565b745ec5d28dafd14585f1b695b159ba336a04
Author: Nick Hill <[email protected]>
Date:   Thu May 21 15:42:20 2026 -0700

    [BugFix] Use correct logprobs for `logprob_token_ids` (#43125)

    Signed-off-by: Nick Hill <[email protected]>

commit e26e1f09280b6c54e1bc1d1fbc0118f7e309cb10
Author: fangyuchu <[email protected]>
Date:   Fri May 22 06:42:07 2026 +0800

    [Feature] Add `--cpu-distributed-timeout-seconds` CLI Option for CPU Process Group Timeout (#42968)

    Signed-off-by: fangyuchu <[email protected]>
    Signed-off-by: zWaNg3 <[email protected]>
    Co-authored-by: zWaNg3 <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit 0f66623b0d739dc94afddb67863c37d6f5816579
Author: Nick Hill <[email protected]>
Date:   Thu May 21 15:36:58 2026 -0700

    [Frontend] Rework fastokens integration (#43168)

    Signed-off-by: Nick Hill <[email protected]>

commit 0b59fc45dd475f96f6f46f2c3e699d7bc13b3b04
Author: ylangtsou <[email protected]>
Date:   Fri May 22 06:00:52 2026 +0800

    Disable build isolation to bypass CUDA related deps for vllm-tpu (#43038)

    Signed-off-by: Ylang Tsou <[email protected]>
    Co-authored-by: Ylang Tsou <[email protected]>
    Co-authored-by: Michael Goin <[email protected]>

commit 17b69828a013acb7af0cd1d16d24ecc8d7582094
Author: Zheng Luo <[email protected]>
Date:   Thu May 21 13:05:01 2026 -0700

    [Core] Add native ModelExpress load format (#43105)

    Signed-off-by: Zheng Luo <[email protected]>
    Co-authored-by: OpenAI Codex <[email protected]>
    Co-authored-by: Robert Shaw <[email protected]>

commit b29cbf06525254693f29d98686e038eaf225be8c
Author: Wentao Ye <[email protected]>
Date:   Thu May 21 16:00:29 2026 -0400

    [Perf] `zeros` -> `empty` to remove additional fill (#42988)

    Signed-off-by: yewentao256 <[email protected]>

commit 9b54e50e2c1c61ea3b7def032fbafc56dd3179c1
Author: Michael Goin <[email protected]>
Date:   Thu May 21 15:51:12 2026 -0400

    [Deprecation] Mark env vars covered by --moe-backend / --linear-backend (#43148)

    Signed-off-by: mgoin <[email protected]>
    Signed-off-by: Michael Goin <[email protected]>

commit 1c78f76c29a642379ad0ec953a77af9bc44376b6
Author: anish <[email protected]>
Date:   Thu May 21 11:07:46 2026 -0400

    [Bugfix] Add early validation to reject incompatible runner types for embedding models (#43079)

    Signed-off-by: anish <[email protected]>
    Signed-off-by: Your Name <[email protected]>
    Signed-off-by: anish <[email protected]>
    Co-authored-by: anish <[email protected]>
    Co-authored-by: Claude Opus 4.6 <[email protected]>
    Co-authored-by: Wentao Ye <[email protected]>

commit 9b9d5dbaab852a1c615fe83a7f92881d353503db
Author: haosdent <[email protected]>
Date:   Thu May 21 22:28:34 2026 +0800

    [CI] Fix CPU tests failing on `tl.exp2` import (#43311)

    Signed-off-by: haosdent <[email protected]>

commit b730c4635288d75da4788bc28d8d26b5e5c3726c
Author: Francesco Fusco <[email protected]>
Date:   Thu May 21 13:50:54 2026 +0200

    [Perf] [Hybrid] Fused Triton kernel for GPU-side Mamba state postprocessing (#40172)

    Signed-off-by: Francesco Fusco <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit c68c55d43e504745dbfc2d46b552e80acb74d4b9
Author: velonica0 <[email protected]>
Date:   Thu May 21 19:50:49 2026 +0800

    [CPU][RISC-V] Add VLEN=256 support to RVV attention kernels (#42943)

    Signed-off-by: velonica0 <[email protected]>
    Signed-off-by: velonica0 <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
    Co-authored-by: Li, Jiang <[email protected]>

commit 5ecd8e9c708821916323d25d5f7beddb7f41d22b
Author: xiangdong <[email protected]>
Date:   Thu May 21 18:41:38 2026 +0800

    [XPU][CI]Fix Docker image pull-to-run race in Intel GPU CI (#43266)

    Signed-off-by: zengxian <[email protected]>
    Co-authored-by: Kunshang Ji <[email protected]>

commit caf69823d61119ac3f4b066f20a910b62078e41c
Author: haosdent <[email protected]>
Date:   Thu May 21 18:38:07 2026 +0800

    [CI] Pin protoc binary in rust-build stages (#43292)

    Signed-off-by: haosdent <[email protected]>

commit 68e07d59161a8d268b773c181fab17994a7c5d0a
Author: Wentao Ye <[email protected]>
Date:   Thu May 21 04:58:09 2026 -0400

    [Bug] Fix ci issue `assert output_size is not None` AssertionError (#43261)

    Signed-off-by: yewentao256 <[email protected]>
    Signed-off-by: Isotr0py <[email protected]>
    Co-authored-by: Isotr0py <[email protected]>

commit ebbfb34e3e058bd539db9e5015d0c18b7ce5a5e0
Author: Kevin H. Luu <[email protected]>
Date:   Thu May 21 01:57:47 2026 -0700

    [Test] Replace zephyr-7b-beta (7B) with SmolLM2-135M in tokenization test (#43085)

    Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

commit edafea35550fab0b185b885711ec048dfd2e1a4d
Author: zhangxin81 <[email protected]>
Date:   Thu May 21 16:17:12 2026 +0800

    Fix FlashInfer TRTLLM NvFP4 monolithic MoE routing (#43223)

    Signed-off-by: zhangxin81 <[email protected]>

commit b719b1635b4899e2372905def0badf96d4dd242a
Author: zexplorerhj <[email protected]>
Date:   Thu May 21 16:16:27 2026 +0800

    Update KDA chunk prefill decay to use exp2 semantics (#43195)

    Signed-off-by: zexplorerhj <[email protected]>
    Co-authored-by: zexplorerhj <[email protected]>

commit 0a54df28471be07b3d668ea21c5e411569d3baea
Author: Kunshang Ji <[email protected]>
Date:   Thu May 21 07:14:13 2026 +0000

    [XPU] add setuptools-rust for xpu dependency (#43287)

    Signed-off-by: Kunshang Ji <[email protected]>

commit a950e9447e38727fc956afdc242bc6e3796ccb77
Author: haosdent <[email protected]>
Date:   Thu May 21 14:30:14 2026 +0800

    [CI] De-flake test_models for bigscience/bloom-560m (#43197)

    Signed-off-by: haosdent <[email protected]>

commit 050611a3dd19271a3c729788ff69b3470ccfb238
Author: Yiyang "Ian" Liu <[email protected]>
Date:   Wed May 20 22:58:59 2026 -0700

    [Bugfix] Fix glm4_moe_tool_parser._is_string_type for /v1/responses FunctionTool format (#39601)

    Signed-off-by: Yiyang Liu <[email protected]>
    Signed-off-by: Chauncey <[email protected]>
    Signed-off-by: sfeng33 <[email protected]>
    Co-authored-by: Copilot <[email protected]>
    Co-authored-by: Chauncey <[email protected]>
    Co-authored-by: sfeng33 <[email protected]>

commit 905b97adfaf7b08f3cc95b328579e5336ed6d3b6
Author: yzong-rh <[email protected]>
Date:   Thu May 21 01:13:15 2026 -0400

    [Benchmark] Add num-warmup to vllm bench throughput (#43245)

    Signed-off-by: Yifan Zong <[email protected]>

commit a6682d1d259cca69a9ae737ea5608fbbe7520031
Author: Daoyuan Li <[email protected]>
Date:   Wed May 20 21:35:08 2026 -0700

    [Bugfix] Warn when renderer_num_workers has no effect on offline LLM (#42905)

    Signed-off-by: Daoyuan Li <[email protected]>

commit f2ace1d57d28df8d4c5e973dd62d87f47d628cb3
Author: Nick Hill <[email protected]>
Date:   Wed May 20 21:24:48 2026 -0700

    [Frontend][RFC] Rust front-end integration (#40848)

    Signed-off-by: Nick Hill <[email protected]>
    Signed-off-by: Bugen Zhao <[email protected]>
    Co-authored-by: Bugen Zhao <[email protected]>

commit d97ba29fdcf2538359fac5c644c0f07e59bc1988
Author: 손세정 <[email protected]>
Date:   Thu May 21 13:24:08 2026 +0900

    [ToolParser][Bugfix] Re-land: Fix anyOf/oneOf/$ref type resolution in Qwen3CoderToolParser (#37831) (#38973)

    Signed-off-by: AAISSJ <[email protected]>
    Signed-off-by: <>
    Signed-off-by: sejung-son <[email protected]>
    Signed-off-by: sfeng33 <[email protected]>
    Co-authored-by: 세덩 <[email protected]>
    Co-authored-by: sejung-son <[email protected]>
    Co-authored-by: sfeng33 <[email protected]>

commit 6441cf4a44856f4eb4dce7d19a51fd69e1b423cf
Author: Flora Feng <[email protected]>
Date:   Thu May 21 00:24:06 2026 -0400

    [Refactor] Use shared coerce_to_schema_type in Seed-OSS tool parser (#43140)

    Signed-off-by: sfeng33 <[email protected]>

commit 346cf163a11b55e069aa3143ae2878967393ddc2
Author: Ben Browning <[email protected]>
Date:   Thu May 21 00:23:47 2026 -0400

    [Frontend] Normalize reasoning_content to reasoning for client compatibility (#42664)

    Signed-off-by: Ben Browning <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit 7e5070934ee5f28103c5b95cb776904a12fc36f5
Author: haosdent <[email protected]>
Date:   Thu May 21 12:22:10 2026 +0800

    [CI] Fix "test_vit_cudagraph_[image|video][step3_vl]" failure (#43082)

    Signed-off-by: haosdent <[email protected]>

commit 2b75a73b8e23f5df6de92d01a191e059424487e3
Author: Luciano Martins <[email protected]>
Date:   Thu May 21 01:22:06 2026 -0300

    [Perf][Gemma4] Batch vision encoder calls for image and video processing (#43169)

    Signed-off-by: Luciano Martins <[email protected]>
    Co-authored-by: Luciano Martins <[email protected]>

commit e45df8c3f77572d03f638feded5b5efbccdbcc05
Author: sonusflow <[email protected]>
Date:   Thu May 21 06:22:01 2026 +0200

    [Bugfix] Fix Qwen3.5 GatedDeltaNet in_proj_ba Marlin failure at TP>=2 (#36329)

    Signed-off-by: Adi McM Sonus Flow <[email protected]>
    Signed-off-by: Isotr0py <[email protected]>
    Co-authored-by: Isotr0py <[email protected]>
    Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

commit ee05e8137ec48b8e7375228a1142b4c5f2e3360c
Author: Jee Jee Li <[email protected]>
Date:   Thu May 21 12:20:57 2026 +0800

    [Minor]  Bigger overlap for FI AR (#43103)

    Signed-off-by: Jee Jee Li <[email protected]>

commit 5d041cc1fe5181daabf39943efc7b678380d57bd
Author: Louie Tsai <[email protected]>
Date:   Wed May 20 20:57:48 2026 -0700

    update GPU json file based on h200 recipes (#43262)

    Signed-off-by: louie-tsai <[email protected]>

commit 9640970de20b15ade9eb3859825637f64e81ed8c
Author: Wentao Ye <[email protected]>
Date:   Wed May 20 21:00:30 2026 -0400

    [Model Runner V2] Fix lora `Triton Error [CUDA]: device-side assert triggered` (#43139)

    Signed-off-by: yewentao256 <[email protected]>
    Signed-off-by: Wentao Ye <[email protected]>
    Co-authored-by: Nick Hill <[email protected]>

commit 63ea11709bd9e9b14669e3973dff92d2dcea3cb1
Author: Ace Eldeib <[email protected]>
Date:   Thu May 21 02:36:16 2026 +0200

    [CI] Add composed-schema regression tests for DeepSeek V3.2/V4 parsers (#43255)

    Signed-off-by: Ace Eldeib <[email protected]>
    Co-authored-by: Flora Feng <[email protected]>

commit bde560ed6e1dc889debf68410ccbcb00b749513b
Author: akii96 <[email protected]>
Date:   Thu May 21 01:46:51 2026 +0300

    [ROCm] Add QuickReduce min-size override and codec threshold (#41675)

    Signed-off-by: <>

commit 6dc0a71843878ef45e29d4732147290b797b70fd
Author: Jiangyun Zhu <[email protected]>
Date:   Thu May 21 05:19:50 2026 +0800

    [Misc] downgrade nvidia-cutlass-dsl to 4.5.0 (#43230)

    Signed-off-by: zjy0516 <[email protected]>

commit 5774aad9c5b67c5bb67bb7d306a9652a035ed0aa
Author: Michael Goin <[email protected]>
Date:   Wed May 20 17:13:12 2026 -0400

    [Perf][gpt-oss] Downgrade triton_kernels to v3.5.1 (#43135)

    Signed-off-by: mgoin <[email protected]>

commit 452baa860b1169787cc8540a1772c4d96f682c40
Author: Douglas Lehr <[email protected]>
Date:   Wed May 20 16:10:44 2026 -0500

    Add dllehr-amd to CODEOWNERS and committers list (#42772)

    Signed-off-by: Douglas Lehr <[email protected]>

commit 2a43b407c5093b1255a172139da6a5151f410b7a
Author: Flora Feng <[email protected]>
Date:   Wed May 20 14:59:12 2026 -0400

    [Bugfix][CI] Add missing import of pad_nvfp4_activation_for_cutlass in flashinfer (#43237)

    Signed-off-by: sfeng33 <[email protected]>

commit 53ff50fcd3d2012a406e5053026ea6a46c88b2b6
Author: Wentao Ye <[email protected]>
Date:   Wed May 20 14:57:42 2026 -0400

    [Perf] Optimize `CutlassFP8ScaledMMLinearKernel` when padding needed by pre-weight processing, 13.5% TTFT improvement (#42651)

    Signed-off-by: yewentao256 <[email protected]>
    Signed-off-by: Wentao Ye <[email protected]>
    Co-authored-by: Matthew Bonanni <[email protected]>

commit 363fc84407f8c966c1cee6786e45e9e6ab289684
Author: meena-at-work <[email protected]>
Date:   Wed May 20 10:21:11 2026 -0700

    Integrate flashinfer b12x MoE and FP4 GEMM kernels for SM120/121 (#40082)

    Signed-off-by: Meenakshi Venkataraman <[email protected]>
    Co-authored-by: Claude Sonnet 4.6 <[email protected]>

commit f2d5e3d3aeac4cb1f6d285e4a567a502ae507777
Author: haosdent <[email protected]>
Date:   Thu May 21 01:00:24 2026 +0800

    [CI] Lower granite-4.0-h-tiny gsm8k threshold for Hybrid SSM NixlConnector PD accuracy tests (4 GPUs) (#43186)

    Signed-off-by: haosdent <[email protected]>
    Signed-off-by: NickLucche <[email protected]>
    Co-authored-by: NickLucche <[email protected]>

commit 2d6b3489b9a325988ad52507236409747d2098a7
Author: Aaron Hao <[email protected]>
Date:   Wed May 20 09:07:59 2026 -0700

    [R3] Add routed experts to openai entrypoint  (#38939)

    Signed-off-by: ahao-anyscale <[email protected]>
    Co-authored-by: Isotr0py <[email protected]>

commit 9c78c99995b70726f9ea929ff2e535d6303383d6
Author: Vadim Gimpelson <[email protected]>
Date:   Wed May 20 19:50:24 2026 +0400

    [MISC] Fix symm_mem cap-equal gate; log AR backend selection (#42993)

    Signed-off-by: Vadim Gimpelson <[email protected]>

commit a10d69116cb25c8137eeb3f320add71d4e04fda9
Author: Flora Feng <[email protected]>
Date:   Wed May 20 10:21:00 2026 -0400

    [Bugfix] Use shared coerce_to_schema_type in DeepSeekV32 tool parser (#43019)

    Signed-off-by: sfeng33 <[email protected]>

commit 644b2a28e7eb3b11191f157416cfedebd2da995b
Author: Joel Smith <[email protected]>
Date:   Wed May 20 15:10:01 2026 +0100

    [Bugfix] Use enable_sm120_family for per-tensor FP8 CUTLASS kernels on SM12.1 (#41215)

    Signed-off-by: j9smith <[email protected]>
    Signed-off-by: Joel Smith <[email protected]>
    Co-authored-by: Shengqi Chen <[email protected]>

commit ded871201a424dd0d28a00aaf74c5786457a18ee
Author: rishitdholakia13 <[email protected]>
Date:   Wed May 20 10:08:58 2026 -0400

    [Bug][Structured Outputs] Fix bug that leads to unconstrained generations with structural tags (#42452)

    Signed-off-by: rishitdholakia13 <[email protected]>
    Co-authored-by: Cursor <[email protected]>

commit df84fb07a6e57969941841c6363d1efbac1ba1e8
Author: Dipika Sikka <[email protected]>
Date:   Wed May 20 10:01:45 2026 -0400

    Remove additional dead code as a follow-up to #42889 (#43144)

    Signed-off-by: Dipika Sikka <[email protected]>

commit 0a508743d42a26786c1432bb7f2e93f8111b6383
Author: Benjamin Chislett <[email protected]>
Date:   Wed May 20 09:15:52 2026 -0400

    [Spec Decode] Support non-MTP speculation for NemotronH (#43130)

    Signed-off-by: Benjamin Chislett <[email protected]>

commit 19cf334207ed81d3ed75a473acd1a95c785d9ed3
Author: Kebe <[email protected]>
Date:   Wed May 20 21:58:30 2026 +0900

    [Feature] Support manually enabling the cumem allocator (#33648)

    Signed-off-by: Kebe <[email protected]>

commit 87e31455b056c6ce59bf5dcb3c622155431851db
Author: Ray Wang <[email protected]>
Date:   Wed May 20 02:32:03 2026 -0700

    [Doc] Sync CLI guide with actual help modes and launch subcommand (#40326)

    Signed-off-by: Rui Wang <[email protected]>
    Co-authored-by: Rui Wang <[email protected]>

commit cb600d1cdbb079ab9432348f128e71c4e2e0a373
Author: hallerite <[email protected]>
Date:   Wed May 20 10:58:46 2026 +0200

    [Frontend] Forward X-data-parallel-rank header on /inference/v1/generate (#42330)

    Signed-off-by: hallerite <[email protected]>
    Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

commit 6f21558da1ec7362d2b4f3d012bce2b612a74459
Author: xiangdong <[email protected]>
Date:   Wed May 20 16:54:58 2026 +0800

    [XPU][CI] Add 2 server model test files in Intel GPU CI (#42499)

    Signed-off-by: zengxian <[email protected]>

commit 1cb224430bea0d037b57e24cf91001f47b69ddf3
Author: Artem Perevedentsev <[email protected]>
Date:   Wed May 20 11:46:55 2026 +0300

    [GDN] Enable FI Blackwell GDN prefill kernel (#40717)

    Signed-off-by: Artem Perevedentsev <[email protected]>

commit 9b343dd4f54a9870f3ba1e41f5a5b3f4a1e25340
Author: Harry Mellor <[email protected]>
Date:   Wed May 20 17:10:00 2026 +0900

    Enable mermaid diagrams in the docs (#43192)

    Signed-off-by: Harry Mellor <[email protected]>

commit 07aeaf9d4df870a76d5a0dc19d6a7e74b4be5d3b
Author: Chris Leonard <[email protected]>
Date:   Wed May 20 03:18:12 2026 -0400

    [6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued) (#42663)

    Signed-off-by: Mikayla Gawarecki <[email protected]>
    Signed-off-by: Chris Leonard <[email protected]>
    Co-authored-by: Mikayla Gawarecki <[email protected]>
    Co-authored-by: Shengqi Chen <[email protected]>

commit 40651c020772b80f9ca80272aebe749fe01cd38a
Author: Nicolò Lucchesi <[email protected]>
Date:   Wed May 20 09:02:36 2026 +0200

    [Docs][PD][NIXL] Bidirectional kv-cache transfer (#43097)

    Signed-off-by: NickLucche <[email protected]>

commit 7e4bc2cecb3a8aede2d10c86a3a1a4bd98e26100
Author: Nicolò Lucchesi <[email protected]>
Date:   Wed May 20 08:58:25 2026 +0200

    [Docs][PD][NIXL] Lease extension mechanism for blocks on P (#43099)

    Signed-off-by: NickLucche <[email protected]>

commit 85959567c3e71a9965616ebebe1853ca48d8d20f
Author: Kevin H. Luu <[email protected]>
Date:   Tue May 19 23:01:41 2026 -0700

    [ci] Revert model executor test back to L4 (#43188)

    Signed-off-by: Kevin H. Luu <[email protected]>

commit 4f940896a32c9e2a0eba7f50d521bf5f6b4de458
Author: Ronen Schaffer <[email protected]>
Date:   Wed May 20 06:32:08 2026 +0300

    [KV Offload] Pass `OffloadingSpec` instead of `VllmConfig` to secondary tiers (#43076)

    Signed-off-by: Ronen Schaffer <[email protected]>

commit cd0ff26e7acf2c691a33d4c44276db6980bab24b
Author: Michael Goin <[email protected]>
Date:   Tue May 19 23:21:01 2026 -0400

    [CI] Add DSV4-Flash to gsm8k moe-refactor/config-b200.txt (#42111)

    Signed-off-by: mgoin <[email protected]>

commit 2ae910ed88121d7c3acdcb9bab14cd968257b6e6
Author: Izik Golan <[email protected]>
Date:   Wed May 20 06:16:07 2026 +0300

    [Perf] Avoid forward scan for async output placeholders (#42938)

commit fadf5d332c6e9bb6e552c1ca529511bce0f79802
Author: pmaybank <[email protected]>
Date:   Tue May 19 23:16:02 2026 -0400

    add enqueue all option to throughput benchmark (#42975)

    Signed-off-by: Philip Maybank <[email protected]>
    Signed-off-by: pmaybank <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit c628a93a64fb4929c3c11d8e2c7244c4826b4f76
Author: Benjamin Chislett <[email protected]>
Date:   Tue May 19 23:15:57 2026 -0400

    [Perf][Bugfix] Update dflash aux layer indexing (#40727)

    Signed-off-by: Benjamin Chislett <[email protected]>

commit 5774aaed0cbeaa74ca7a75d372c1e8bd4aa11cdb
Author: Terrence Zhao <[email protected]>
Date:   Tue May 19 22:32:06 2026 -0400

    [Cohere] Enable Cohere MoE (#43143)

    Signed-off-by: Terrencezzj <[email protected]>

commit 39bba710bed5b6018718af3e0fd7984f6082118e
Author: Nick Hill <[email protected]>
Date:   Tue May 19 19:19:05 2026 -0700

    [MRV2][BugFix] Fix default-stream CG capture in P/W LoRA case (#43160)

    Signed-off-by: Nick Hill <[email protected]>

commit 73dd2f33b7a5a8a237fe7296039cec246e4c68bd
Author: Aaron Hao <[email protected]>
Date:   Tue May 19 18:01:29 2026 -0700

    [bug] fix WeightTransferConfig.backend to allow for all strings (#43121)

    Signed-off-by: ahao-anyscale <[email protected]>

commit be16785998087f80ffac08b980603241e5da16ab
Author: Fadi Arafeh <[email protected]>
Date:   Wed May 20 00:31:15 2026 +0100

    [CPU][DOC] Fix installation commands for Arm CPUs (#43115)

    Signed-off-by: Fadi Arafeh <[email protected]>

commit 117afeea4665367a3066c1df58d4082d07fcc946
Author: Max de Bayser <[email protected]>
Date:   Tue May 19 17:27:54 2026 -0400

    Fix error in Dynamic NTK scaling (#41277)

    Signed-off-by: Max de Bayser <[email protected]>
    Signed-off-by: Max de Bayser <[email protected]>
    Signed-off-by: wang.yuqi <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
    Co-authored-by: wang.yuqi <[email protected]>

commit 12421962955ac28b6f80a0307f554fad939174dd
Author: Doğaç Eldenk <[email protected]>
Date:   Tue May 19 15:39:00 2026 -0500

    [Model] Support post-norm architecture for EAGLE-3 supeculators (#42764)

    Signed-off-by: Doğaç Eldenk <[email protected]>

commit a65093c1a39a8ddd8455365128ecbe259350e22c
Author: Kevin H. Luu <[email protected]>
Date:   Tue May 19 11:51:34 2026 -0700

    [ci] Move language models tests (hybrid) back to L4 (#43129)

    Signed-off-by: Kevin H. Luu <[email protected]>

commit 9aaf83ef502fc37bc647f6e474314d48ba36cd1c
Author: Wei Zhao <[email protected]>
Date:   Tue May 19 14:44:32 2026 -0400

    [CI failure] Temporarily disable using persistent cache for flashinfer autotune (#43119)

    Signed-off-by: wzhao18 <[email protected]>
    Signed-off-by: Wei Zhao <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

commit f54721bcc3e072d71b0e09c0b0bd6d692eb06161
Author: tomeras91 <[email protected]>
Date:   Tue May 19 21:43:04 2026 +0300

    [Bugfix][MoE] FlashInfer one-sided: workspace union across heterogeneous layers (#42976)

    Signed-off-by: Tomer Asida <[email protected]>

commit aed2eb355a9d9136c8e17690b932983b55fb343f
Author: Dao007forever <[email protected]>
Date:   Tue May 19 11:14:43 2026 -0700

    [Docs] Fix MooncakeStoreConnector role in disaggregated example (#42994)

    Signed-off-by: Dao Le <[email protected]>
    Co-authored-by: Claude <[email protected]>

commit d247a931cc25e7253feccbd6260d48216ff5c081
Author: Dom Brown <[email protected]>
Date:   Tue May 19 17:02:05 2026 +0100

    [feat] Add FP8 per-tensor Q scale support to Triton attention backend (#42080)

    Signed-off-by: Dom Brown <[email protected]>

commit 8200fbe1ac73f00a46b1cdd6c4c93bdaf2c33022
Author: Jinzhen Lin <[email protected]>
Date:   Tue May 19 23:36:47 2026 +0800

    [Misc] add humming to dependencies (#42540)

    Signed-off-by: Jinzhen Lin <[email protected]>

commit 42b4f1fdf7269de8aa83755a805555fe78add28b
Author: Flora Feng <[email protected]>
Date:   Tue May 19 11:21:12 2026 -0400

    [Refactor] Extract extract_types_from_schema utility from Minimax M2 tool parser (#43025)

    Signed-off-by: sfeng33 <[email protected]>

commit 1c6158083a6fc3aff408660d2defd7602f78f556
Author: Wang Yiwen <[email protected]>
Date:   Tue May 19 23:17:42 2026 +0800

    [Model] Openvla support (#42654)

    Signed-off-by: Wang Yiwen <[email protected]>

commit d740e2c02919cfba5a86a40d1c12439d03f5ac07
Author: Xinyu Chen <[email protected]>
Date:   Tue May 19 23:09:07 2026 +0800

    [XPU] update xpu graph usage (#43043)

    Signed-off-by: Xinyu Chen <[email protected]>

commit b82e908b4c65a1f162e2d35a8106f09d95d8aa02
Author: Nick Hill <[email protected]>
Date:   Tue May 19 07:35:54 2026 -0700

    [Perf][4/n] Eliminate various GPU<->CPU syncs (#42347)

    Signed-off-by: Nick Hill <[email protected]>

commit a78b842d0e85d287176031334f4721cd96b6e47d
Author: Sage <[email protected]>
Date:   Tue May 19 13:21:49 2026 +0300

    [Bugfix] Fix top logprobs token placeholders in `/inference/v1/generate` (#42887)

    Signed-off-by: Sage Ahrac <[email protected]>

commit 129019f3342f1b7346ed8f4c1ac9fdefd8fe6ef8
Author: zhanqiuhu <[email protected]>
Date:   Tue May 19 05:44:33 2026 -0400

    [CI] Add MTP + PD disagg test for Qwen3.5 (#42677)

    Signed-off-by: ZhanqiuHu <[email protected]>
    Co-authored-by: Nicolò Lucchesi <[email protected]>

commit ef54a4d604ef3725bd52aa2893f71d671bf5329a
Author: Shanshan Shen <[email protected]>
Date:   Tue May 19 16:43:16 2026 +0800

    [Misc][MM] Remove redundant code in CLIPAttention (#43046)

    Signed-off-by: shen-shanshan <[email protected]>

commit 07beaed8422d2df34a20e8ebd22b7924d563a566
Author: Woosuk Kwon <[email protected]>
Date:   Tue May 19 01:12:46 2026 -0700

    [Model Refactoring] Rename deepseek_v4.py to model.py [4/N] (#43077)

    Signed-off-by: Woosuk Kwon <[email protected]>

commit 056bc2e16646599a96ac94e761c953e680e6fba9
Author: Yifan Qiao <[email protected]>
Date:   Tue May 19 01:07:46 2026 -0700

    [KVConnector][DSV4] HMA support for Mooncake store connector (#42828)

    Signed-off-by: Yifan Qiao <[email protected]>

commit f34623bf3cac5b33451a761e802c9531e83d1c68
Author: Aaron Hao <[email protected]>
Date:   Tue May 19 01:06:21 2026 -0700

    [bug] AsyncScheduler drops first post-resume token after pause_generation + clear_cache (#42117)

    Signed-off-by: hao-aaron <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit b14be81c1f63b70668d26d65a377b6383fbca936
Author: Woosuk Kwon <[email protected]>
Date:   Tue May 19 00:52:54 2026 -0700

    [Model Refactoring] Move deepseek_v4_ops to models/deepseek_v4 [3/N] (#43073)

    Signed-off-by: Woosuk Kwon <[email protected]>

commit 301d986473a0ffc1df563422e01eac4a1efd59e0
Author: wang.yuqi <[email protected]>
Date:   Tue May 19 15:37:40 2026 +0800

    [Frontend] Consolidate beam search by BeamSearchMixin. (#42946)

    Signed-off-by: wang.yuqi <[email protected]>

commit 257af77bc2b612d5ebd0aecea777139036543af3
Author: wang.yuqi <[email protected]>
Date:   Tue May 19 14:43:18 2026 +0800

    [Docs] Reorganize online serving docs. (#41907)

    Signed-off-by: wang.yuqi <[email protected]>
    Signed-off-by: wang.yuqi <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
    Co-authored-by: Cyrus Leung <[email protected]>
    Co-authored-by: Harry Mellor <[email protected]>

commit 4a4fdabe28f3e2c8f9d05bcc80c4bf6d656b1ead
Author: Taneem Ibrahim <[email protected]>
Date:   Tue May 19 01:16:42 2026 -0500

    [Misc] Aligning tokwise pooler heads for consistency (#43041)

    Signed-off-by: Taneem Ibrahim <[email protected]>

commit f1e3f0e6d685082bdb313c20914099ac5ede5f14
Author: Chaojun Zhang <[email protected]>
Date:   Tue May 19 14:14:59 2026 +0800

    [XPU] Use custom op collective behavior  (#41354)

    Signed-off-by: Chaojun,Zhang <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
    Co-authored-by: Kunshang Ji <[email protected]>

commit 9fd8487d2f56468aeec8154123641eb7c2eeacdf
Author: Gracie Guo (UX) <[email protected]>
Date:   Tue May 19 13:50:38 2026 +0800

    [Docs] Add SVG images for pooling models. (#42626)

    Signed-off-by: Gracie Guo <[email protected]>
    Signed-off-by: wang.yuqi <[email protected]>
    Co-authored-by: Gracie Guo <[email protected]>
    Co-authored-by: wang.yuqi <[email protected]>

commit 27f4ba94811ef14bd45bcdc0c0b8e288a7cc6bc6
Author: Junyan Xu <[email protected]>
Date:   Mon May 18 22:29:04 2026 -0700

    fix: use keyword arguments for shard_id and expert_id in weight_loade… (#42671)

    Signed-off-by: junyanxu <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit 6e889b582b6a0b11f22b3764be174266faa9ff5e
Author: Kevin H. Luu <[email protected]>
Date:   Mon May 18 21:58:36 2026 -0700

    [ci] Route 28 gpu_1_queue tests to h200_35gb queue (#43030)

    Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

commit fab07e4d0f7f266643c6ac0dc944f9f433ef2140
Author: Qiuyang Yue <[email protected]>
Date:   Mon May 18 21:22:33 2026 -0700

    [Bugfix][KV Connector] Fix SimpleCPUOffloadScheduler TOCTOU between Phase A and Phase B (#42289)

    Signed-off-by: Qiuyang Yue <[email protected]>
    Co-authored-by: Claude Opus 4.7 <[email protected]>
    Co-authored-by: gemini-code-assist <[email protected]>

commit 3ca8db2ef88ec5a6686e62ee3ac899afae85c7af
Author: gnovack <[email protected]>
Date:   Mon May 18 21:17:56 2026 -0700

    add cutedsl dsv4 indexer fp8 kernel (#42899)

    Signed-off-by: george <[email protected]>
    Co-authored-by: george <[email protected]>

commit 87b08c5f6460cf487e47872c5fbc2595c97e74ef
Author: Woosuk Kwon <[email protected]>
Date:   Mon May 18 21:00:58 2026 -0700

    [Model Refactoring] Move DeepSeek V4 layers to `models/deepseek_v4/` [2/N] (#43039)

    Signed-off-by: Woosuk Kwon <[email protected]>

commit fba010dd74e2f94e4f7223b164ec9097d1b8a6af
Author: Nicolò Lucchesi <[email protected]>
Date:   Tue May 19 05:25:41 2026 +0200

    [Bugfix][MRV2] Fix KVCache tensor explicit `kernel_block_size` dim (#42766)

    Signed-off-by: NickLucche <[email protected]>
    Signed-off-by: Nick Hill <[email protected]>
    Co-authored-by: Nick Hill <[email protected]>

commit da03e549b34685c4e63a091e973d907aee48a68c
Author: Mohammad Miadh Angkad <[email protected]>
Date:   Tue May 19 11:25:37 2026 +0800

    [UX] Add a persistent cache for FlashInfer autotuning (#42537)

    Signed-off-by: Mohammad Miadh Angkad <[email protected]>

commit 36dcaf25d8e091ea0f47b9ce7dcfca05de56f16d
Author: Kunshang Ji <[email protected]>
Date:   Tue May 19 03:17:09 2026 +0000

    [XPU] add gptq(int4) support (#37844)

    Signed-off-by: Kunshang Ji <[email protected]>

commit 8f16c4a5c0feb01f106e5981f22ae8808a94a28b
Author: Ofir Zafrir <[email protected]>
Date:   Tue May 19 06:16:07 2026 +0300

    [BugFix][CPU][Spec Decode] Fix Eagle implementation on CPU backend (#42468)

    Signed-off-by: Ofir Zafrir <[email protected]>

commit afd7b1dce94fed484351fafd5bf5ea6601ac621e
Author: Revital Sur <[email protected]>
Date:   Tue May 19 06:12:04 2026 +0300

    [Bugfix] Use platform-agnostic device in example_connector load (#42926)

    Signed-off-by: Revital Sur <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

commit 287471b99442b44c5a16c4d70b0f3e178dd52732
Author: Woosuk Kwon <[email protected]>
Date:   Mon May 18 19:50:02 2026 -0700

    [Model Refactoring] Migrate DeepSeek V4 to vllm/models/ [1/N]  (#43004)

    Signed-off-by: Woosuk Kwon <[email protected]>

commit 239b5ff30cf46f9196149c888a20be2096fdff03
Author: Michael Goin <[email protected]>
Date:   Mon May 18 20:22:27 2026 -0400

    [Frontend] Add --spec-method/--spec-model/--spec-tokens CLI aliases (#42476)

    Signed-off-by: mgoin <[email protected]>
    Co-authored-by: Claude <[email protected]>

commit f85c76d701fc049a722c17b3affd9401380be1bf
Author: Artem Perevedentsev <[email protected]>
Date:   Tue May 19 02:58:15 2026 +0300

    [CI/Build] Bump nvidia-cutlass-dsl to 4.5.1 (#42991)

    Signed-off-by: Artem Perevedentsev <[email protected]>

commit a171e6b52dff47dc567657e7d51f641bdcb22774
Author: shanjiaz <[email protected]>
Date:   Mon May 18 19:39:09 2026 -0400

    Add parallel drafting to v2 model runner unsupported features (#43010)

    Signed-off-by: shanjiaz <[email protected]>

commit 37ece593c105b5bb818aa94885617b863d390d7f
Author: Wentao Ye <[email protected]>
Date:   Mon May 18 19:38:12 2026 -0400

    [Perf] Padded nvfp4 quant kernel to remove additional copy, 2.4%~5.7% e2e performance improvement (#42774)

    Signed-off-by: yewentao256 <[email protected]>

commit 57fef4e0bf0bfaddf117dfdc9367e1fb957b423f
Author: Flora Feng <[email protected]>
Date:   Mon May 18 17:55:39 2026 -0400

    [Refactor] Extract shared coerce_to_schema_type utility from Minimax M2 tool parser (#43006)

    Signed-off-by: sfeng33 <[email protected]>

commit 0191354827560fe38f68b4e7207f8824d6152ca3
Author: haosdent <[email protected]>
Date:   Tue May 19 05:29:10 2026 +0800

    [Perf][MLA] Enable FULL cudagraph capture for TRITON_MLA decode (#42885)

    Signed-off-by: haosdent <[email protected]>

commit cd49a05d5aa3cc296912297b3c2b577efe4183c8
Author: Wentao Ye <[email protected]>
Date:   Mon May 18 16:41:22 2026 -0400

    [Refactor] Remove dead code (#42889)

    Signed-off-by: yewentao256 <[email protected]>

commit 84747489ded65265ee7d43815bfa3373b0d42279
Author: Ronen Schaffer <[email protected]>
Date:   Mon May 18 22:41:58 2026 +0300

    Tier offload followup (#42529)

    Signed-off-by: Ronen Schaffer <[email protected]>

commit 8fc1c284b94668b60c30737e178cb7e6cd651e89
Author: Tuukka Sarvi <[email protected]>
Date:   Mon May 18 21:56:22 2026 +0300

    [ROCm] Guard AITER GDN decode fast path by layout (#42880)

    Signed-off-by: Tuukka Sarvi <[email protected]>

commit ce88f01c9ac4fcde9dd43a983074d4e893cde65d
Author: Amit Portnoy <[email protected]>
Date:   Mon May 18 21:22:56 2026 +0300

    [Docs] update attribution to reflect EDEN foundation (#41666)

    Signed-off-by: amitport <[email protected]>

commit 00e20e76f775b88f47469ae9fcb0f1ecd7580bb9
Author: Wentao Ye <[email protected]>
Date:   Mon May 18 14:14:21 2026 -0400

    [Refactor] Remove dead cuda kernels (#42767)

    Signed-off-by: yewentao256 <[email protected]>

commit 9758a6e5c5a556275c030db456d5d434ee999d58
Author: czhu-cohere <[email protected]>
Date:   Mon May 18 11:12:06 2026 -0700

    [BugFix] support PP for Cohere vision model (#42819)

    Signed-off-by: <[email protected]>
    Signed-off-by: root <[email protected]>

commit a2c8fc66573664395f491a94da1882fdf92e034b
Author: Bowen Bao <[email protected]>
Date:   Mon May 18 10:46:13 2026 -0700

    [ROCm][Quantization][3/N] Refactor quark_moe w4a4 w/ oracle (#41436)

    Signed-off-by: Bowen Bao <[email protected]>

commit 6859ca76159fdd403b687c0c296e5a12850ba24e
Author: Jinzhen Lin <[email protected]>
Date:   Tue May 19 01:32:26 2026 +0800

    [Bugfix] fix swiglu limit issue for humming backend + deepseek v4 (#42541)

    Signed-off-by: Jinzhen Lin <[email protected]>
    Co-authored-by: Michael Goin <[email protected]>

commit 67f58ce23f469e118688a50687ef0fbb14a1c028
Author: Mohammad Miadh Angkad <[email protected]>
Date:   Tue May 19 01:02:01 2026 +0800

    [Bugfix] Fix DSV4 MTP after ROCm mHC integration (#42930)

    Signed-off-by: Mohammad Miadh Angkad <[email protected]>

commit 8c296de63b47664fc5979831e1ae2d2a14a05b1a
Author: Wei Zhao <[email protected]>
Date:   Mon May 18 12:12:27 2026 -0400

    [Perf] Re-enable flashinfer autotune by default and cleanup (#42857)

    Signed-off-by: wzhao18 <[email protected]>

commit b12745e4f31ffacf401cc20a97c592d6a49f3269
Author: Harry Mellor <[email protected]>
Date:   Tue May 19 00:56:09 2026 +0900

    Fix `--convert` passed without `--runner` on causal models (#42935)

    Signed-off-by: Harry Mellor <[email protected]>

commit e26736973a1981dbb4054dc1ac430e78d8006ef2
Author: Wentao Ye <[email protected]>
Date:   Mon May 18 11:27:21 2026 -0400

    [Model Runner V2] Fix prompt logprobs calculation `Sizes of tensors must match` error (#42778)

    Signed-off-by: yewentao256 <[email protected]>

commit 47829b1159335a010521ea3e5361d51744a36b0a
Author: Netanel Haber <[email protected]>
Date:   Mon May 18 18:26:00 2026 +0300

    [Bugfix] mamba: run single-token extends as decodes (#42430)

    Signed-off-by: Netanel Haber <[email protected]>

commit 4a39b4f55374d48ebaa2ca02312e24639db8e0b8
Author: Blanc Swan <[email protected]>
Date:   Mon May 18 17:20:04 2026 +0200

    [Model] Add Apertus Tool Parser (#41154)

    Signed-off-by: Blanc <[email protected]>

commit 78e7a7b9b0b9c285bf6978c3fc09eeecea3ff230
Author: Siddharth Bedekar <[email protected]>
Date:   Mon May 18 08:02:43 2026 -0700

    Refactor AWQ Marlin MoE onto modular WNA16 oracle (#42483)

    Signed-off-by: Siddharth Bedekar <[email protected]>
    Signed-off-by: Siddharth Bedekar <[email protected]>
    Co-authored-by: Robert Shaw <[email protected]>
    Co-authored-by: OpenAI Codex <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit f5d3dc7115cf77472ba5e274f6becbbeddbf4bd5
Author: Michael Goin <[email protected]>
Date:   Mon May 18 10:26:07 2026 -0400

    [Model Runner v2] Support update_config (#42783)

    Signed-off-by: mgoin <[email protected]>
    Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

commit 1ac10f159a09897baada01b14b6a0dd6442aefd6
Author: vllm-agent <[email protected]>
Date:   Mon May 18 06:02:51 2026 -0700

    Revert "[torch.compile] Add patch for fullgraph compilation" (#42686) (#42913)

    Co-authored-by: Luka Govedič <[email protected]>
    Co-authored-by: Zhewen Li <[email protected]>

commit e5417657e55ec2f42809816e4aa5c9753f390cdd
Author: liranschour <[email protected]>
Date:   Mon May 18 15:59:42 2026 +0300

    [KV Connector][Offloading] Flush all pending jobs on last step (#42611)

    Signed-off-by: Liran Schour <[email protected]>
    Signed-off-by: liranschour <[email protected]>
    Co-authored-by: Or Ozeri <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit 2e40faf08b2cae4ff6e27a255fe10833365de0e8
Author: xiangdong <[email protected]>
Date:   Mon May 18 20:34:48 2026 +0800

    [XPU][CI] Temporarily skip test_moe_lora_align_block_size_mixed_base_and_lora[1] in Intel GPU CI (#42954)

    Signed-off-by: zengxian <[email protected]>

commit 69c91d010a596bb74b553fe157497a1fd6edb47c
Author: Nicolò Lucchesi <[email protected]>
Date:   Mon May 18 14:34:16 2026 +0200

    [MRv2] Default to MRv1 when a connector is present (#42955)

    Signed-off-by: NickLucche <[email protected]>

commit 737bfa3a43ce386bd1894792f3302d9f3f9d73fa
Author: roikoren755 <[email protected]>
Date:   Mon May 18 14:54:00 2026 +0300

    [Bugfix][Hybrid][NemotronH] Fix mamba_cache_mode=all + speculative decoding crash (#41233)

    Signed-off-by: Roi Koren <[email protected]>

commit e414e1f1c020108593526b706efaf89e427c05a2
Author: Kfir Toledo <[email protected]>
Date:   Mon May 18 14:36:02 2026 +0300

    [Bugfix][KV Offload] count appended GPU blocks in store group_sizes (#42945)

    Signed-off-by: Kfir Toledo <[email protected]>

commit df852ed503ac1a79e568271cd6f136a7b2698f5e
Author: inisis <[email protected]>
Date:   Mon May 18 18:33:29 2026 +0800

    fix: remove unused norm for dpskv4 (#41710)

    Signed-off-by: inisis <[email protected]>
    Co-authored-by: Yongye Zhu <[email protected]>

commit 88a860d7545aad69661daad7a1c2b04f59c76144
Author: Yuwen Zhou <[email protected]>
Date:   Mon May 18 18:04:45 2026 +0800

    [CPU] Add MXFP4 W4A16 MoE support (#41922)

    Signed-off-by: yuwenzho <[email protected]>
    Signed-off-by: Yuwen Zhou <[email protected]>

commit cac81b6eda418fb5ca86b81197914dd02666353e
Author: Tianmu Li <[email protected]>
Date:   Mon May 18 03:04:41 2026 -0700

    [CPU Backend] Improve cpu thread utilization (#42666)

    Signed-off-by: Li, Tianmu <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit b4601ad43ff7ff2b9e2f52379144481e45bcf6c5
Author: Li, Jiang <[email protected]>
Date:   Mon May 18 18:04:36 2026 +0800

    [CPU] Add fused GDN support for AMX CPU platform (#42707)

    Signed-off-by: jiang1.li <[email protected]>

commit 2267f70070bdee8057b4afae69cba9b847add587
Author: Jee Jee Li <[email protected]>
Date:   Mon May 18 18:04:31 2026 +0800

    [Kernel] Pack topk id/weights triton kernel (#42527)

    Signed-off-by: Jee Jee Li <[email protected]>

commit 965d076148326f4511b6b832cbe7d974db74dbe9
Author: Tony Lin <[email protected]>
Date:   Mon May 18 17:38:54 2026 +0800

    [CPU] Specify required KV cache layout for CPU attention backend (#42740)

    Signed-off-by: Tony Lin <[email protected]>
    Co-authored-by: Li, Jiang <[email protected]>

commit c38bed4248e97e5ed981569777d035d31ace5368
Author: wenjun liu <[email protected]>
Date:   Mon May 18 16:36:45 2026 +0800

    delete xpu ci (#42582)

    Signed-off-by: wenjun.liu <[email protected]>
    Co-authored-by: Kunshang Ji <[email protected]>
    Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

commit 998714b21b413c78db8eb7af7f384dc90c0b10dc
Author: Xin Yang <[email protected]>
Date:   Mon May 18 01:32:46 2026 -0700

    [Perf] Add do_not_specialize in fused FP8 RoPE kernel (#42849)

    Signed-off-by: Xin Yang <[email protected]>

commit 9537542537728af9fac418ecf1604ad8e8d9ff93
Author: Harry Mellor <[email protected]>
Date:   Mon May 18 17:31:06 2026 +0900

    Revert checkpoint specific workaround in Transformers modelling backend (#42923)

    Signed-off-by: Harry Mellor <[email protected]>

commit 5ab6d1b3fd407404cd78488bf6f4cbcde6d912b7
Author: Rishapveer Singh <[email protected]>
Date:   Mon May 18 10:14:36 2026 +0200

    [Model] [Perf] Use flatten for Qwen3.5's GDN output projection (#42311)

    Signed-off-by: Rishapveer Singh <[email protected]>

commit 7d5b033782681acee274f4f379c9fadc557fd7e8
Author: Jee Jee Li <[email protected]>
Date:   Mon May 18 15:22:26 2026 +0800

    [LoRA] Support 2D and 3D MoE LoRA adapter  at the same time (#42242)

    Signed-off-by: Jee Jee Li <[email protected]>
    Signed-off-by: Jee Jee Li <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
    Co-authored-by: Roger Wang <[email protected]>

commit e3aeee5ff8bf7e89fea231d2a965701248eb43c0
Author: Nguyễn Thế Duy <[email protected]>
Date:   Mon May 18 14:17:53 2026 +0700

    [Bugfix] moe lora align kernel grid (#40131)

    Signed-off-by: TheDuyIT <[email protected]>
    Signed-off-by: Jee Jee Li <[email protected]>
    Signed-off-by: dtnguyen <[email protected]>
    Co-authored-by: Jee Jee Li <[email protected]>
    Co-authored-by: Jee Jee Li <[email protected]>

commit c1f7854342d1e80f7f2406524d242b8ee5476d6d
Author: Harry Mellor <[email protected]>
Date:   Mon May 18 15:33:32 2026 +0900

    Improve logging when docs build is skipped (#42929)

    Signed-off-by: Harry Mellor <[email protected]>

commit 23c15acd770cf16ed36c6d3fed8e7d78db7d5282
Author: gaozihao-shy <[email protected]>
Date:   Mon May 18 13:07:16 2026 +0800

    [BugFix] Kimi-K2.5: skip vision tower dtype conversion when using quantization (#42869)

    Signed-off-by: gaozihao-shy <[email protected]>
    Signed-off-by: gaozihao <[email protected]>

commit b50646e5effd7cb5884cd96fdff4c53c18521198
Author: Andreas Karatzas <[email protected]>
Date:   Sun May 17 22:57:59 2026 -0500

    [ROCm][CI] Stabilize ROCm pooling and multimodal CI (#42909)

    Signed-off-by: Andreas Karatzas <[email protected]>

commit 990f49bdcb8ff51c0ceb1d784c3ca16e6c276927
Author: Soyaazz <[email protected]>
Date:   Mon May 18 11:19:13 2026 +0800

    [MM][CG] Enable encoder Cudagraph for Step3VL (#42224)

    Signed-off-by: JisoLya <[email protected]>
    Signed-off-by: Soyaazz <[email protected]>

commit 107210442da1bc6985bfa615b55e1e5c2dd98958
Author: Alec <[email protected]>
Date:   Sun May 17 19:11:46 2026 -0700

    [CI] Add NIXL EP import canary (#42567)

    Signed-off-by: Alec Flowers <[email protected]>
    Co-authored-by: OpenAI Codex <[email protected]>

commit 03ddc1c9bc5e448e0da6236268a611d7d001dbae
Author: Yiliu Dong <[email protected]>
Date:   Mon May 18 09:57:04 2026 +0800

    [Perf] Wire silu_and_mul_per_block_quant into TritonFP8MoE (MiniMax-M2)  (#42497)

    Signed-off-by: qianlihuang <[email protected]>
    Signed-off-by: Yiliu Dong <[email protected]>
    Co-authored-by: qianlihuang <[email protected]>

commit 966903eb93a053a908fbf8b931fcebfb28c4741a
Author: Luka Govedič <[email protected]>
Date:   Sun May 17 15:49:16 2026 -0400

    [torch.compile] Add patch for fullgraph compilation (#42686)

    Signed-off-by: Luka Govedič <[email protected]>

commit 599e75f432e5fd7c77e65dc95587f3441201bdbc
Author: TJian <[email protected]>
Date:   Mon May 18 00:18:50 2026 +0800

    [ROCm] [Bugfix] Fix DeepSeek V4 Functionality and Accuracy (#42810)

    Signed-off-by: tjtanaa <[email protected]>

commit 1c8e9c0399f6a6a98f406dce5947a2ad318e195a
Author: Taneem Ibrahim <[email protected]>
Date:   Sun May 17 09:40:21 2026 -0500

    Refactor: Pass num_labels explicitly to PoolerClassify instead of reading from global config (#42851)

    Signed-off-by: Taneem Ibrahim <[email protected]>

commit 0fa888465e5a30b797bdf2cdcd0f57fc77541cef
Author: zofia <[email protected]>
Date:   Sun May 17 16:55:10 2026 +0800

    [XPU] fix weight scale shape (#42725)

    Signed-off-by: Zhu, Zufang <[email protected]>
    Co-authored-by: Kunshang Ji <[email protected]>

commit ff712f6447093d07747c88680b9d006b119f5890
Author: liuzhenwei <[email protected]>
Date:   Sun May 17 12:15:50 2026 +0800

    [MRV2][XPU] add Model Runner V2 log (#42710)

    Signed-off-by: zhenwei-intel <[email protected]>

commit 504a26ce2be2415118b73966480b4fc04d9b7bf8
Author: Qi Zhou <[email protected]>
Date:   Sat May 16 17:54:58 2026 -0700

    Support bf16 for mamba ssm cache (#41680)

    Signed-off-by: Qi Zhou <[email protected]>

commit a94189295b8b9c1d952be438b49ed5793db59159
Author: weizhoublue <[email protected]>
Date:   Sun May 17 08:54:27 2026 +0800

    Fix Weight loading for  Qwen3.5-MTP and Qwen3-VL using runai_streamer (#42716)

    Signed-off-by: weizhoublue <[email protected]>

commit 0867497368f390212a3f9684e2e05f698f8d1149
Author: Artem Perevedentsev <[email protected]>
Date:   Sun May 17 00:55:12 2026 +0300

    [CI/Build] Bump flashinfer to v0.6.11.post2 (#41711)

    Signed-off-by: Artem Perevedentsev <[email protected]>
    Co-authored-by: Vadim Gimpelson <[email protected]>

commit 36e74c9ea4feb5ade38ffa1ea96f24dd73316e02
Author: Zhewen Li <[email protected]>
Date:   Sat May 16 13:34:15 2026 -0700

    [KV Connector] Support disk offloading in MooncakeStoreConnector (#42689)

    Signed-off-by: Zhewen Li <[email protected]>
    Co-authored-by: Zhewen Li <[email protected]>
    Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

commit 787bc0d0313840c16e403dfa2d135781d41d3614
Author: Taneem Ibrahim <[email protected]>
Date:   Sat May 16 14:58:16 2026 -0400

    Add unit tests for pooler activation functions (#42824)

    Signed-off-by: Taneem Ibrahim <[email protected]>

commit d1586e1a1242754d2f6ac51f4f16680f7d4b129b
Author: weizhoublue <[email protected]>
Date:   Sun May 17 01:02:54 2026 +0800

    Fix: Propagate pinned model revisions into Ultravox secondary weight loading (#42830)

commit 8a56da3845270837424ef4b7ee83ca97a7883025
Author: Jiangyun Zhu <[email protected]>
Date:   Sat May 16 22:04:12 2026 +0800

    [Experimental] Breakable CUDA graph (#42304)

    Signed-off-by: zjy0516 <[email protected]>

commit 4db300e95fd29f5b1a4a7c34f4fbe91b7e9abb24
Author: Andreas Karatzas <[email protected]>
Date:   Sat May 16 04:35:05 2026 -0500

    [ROCm][CI] Removed problematic command override mechanism (#42807)

    Signed-off-by: Andreas Karatzas <[email protected]>

commit 657b42b5922d21fef00529144ef5bb5633ad04b1
Author: Zhewen Li <[email protected]>
Date:   Sat May 16 00:26:25 2026 -0700

    [Docker][KVConnector] Build mooncake-transfer-engine from source (#42114)

    Signed-off-by: Zhewen Li <[email protected]>
    Signed-off-by: khluu <[email protected]>
    Co-authored-by: Zhewen Li <[email protected]>
    Co-authored-by: khluu <[email protected]>

commit 32b7177909d1c9928bcedd81de7de5a1fa21d2b3
Author: Jee Jee Li <[email protected]>
Date:   Sat May 16 11:22:35 2026 +0800

    [LoRA][Bugfix] Dedup LoRA wrapping for modules referenced from multiple attribute paths (MoE gate) (#42757)

    Signed-off-by: Jee Jee Li <[email protected]>

commit 39c67d714ef091df1533181bdc3df82dc9ac3e07
Author: DustHunter <[email protected]>
Date:   Sat May 16 09:29:27 2026 +0800

    fix: add API key authorization to /v2 endpoints (#42594)

    Signed-off-by: DustHunter <[email protected]>
    Signed-off-by: wang.yuqi <[email protected]>
    Co-authored-by: Qwen-Coder <[email protected]>
    Co-authored-by: wang.yuqi <[email protected]>

commit 87a2adcb43513ead1434aff03a535d86f56f768b
Author: Viktor Pus <[email protected]>
Date:   Sat May 16 02:44:48 2026 +0200

    [Misc] Add common random prefix option to structured-output serving benchmark (#41632)

    Signed-off-by: Viktor Pus <[email protected]>
    Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

commit 852f567444cf8c206219edb7b2c42aec55fc41cf
Author: Michael Goin <[email protected]>
Date:   Fri May 15 20:15:52 2026 -0400

    [Bugfix] Respect explicit --kv-cache-dtype over checkpoint kv_cache_scheme (#42782)

    Signed-off-by: mgoin <[email protected]>

commit b2a27b82d970efa0203c06be6dc0d94526edaab0
Author: Michael Goin <[email protected]>
Date:   Fri May 15 20:07:39 2026 -0400

    [Kernel][UX] Add `--linear-backend` arg for linear kernel selection (#39538)

    Signed-off-by: mgoin <[email protected]>

commit d0921bafeff9bbe7a7b4efef6371700e69224702
Author: Keyi Li <[email protected]>
Date:   Fri May 15 16:20:33 2026 -0700

    [Bugfix] Unwrap VLM wrappers for EPLB on Model Runner V2 (#42706)

commit 1ccdf87507407cb02460ec2e7a3e1a4cac9b0a4a
Author: rasdani <[email protected]>
Date:   Fri May 15 15:20:53 2026 -0700

    [Bugfix] Fix layerwise reload alias-buffer corruption (#42481)

    Signed-off-by: rasdani <[email protected]>
    Co-authored-by: OpenAI Codex <[email protected]>
    Co-authored-by: Roger Wang <[email protected]>

commit bd9dbe60601c986b50260f299fe279d057d7d89f
Author: Rita Brugarolas <[email protected]>
Date:   Fri May 15 13:50:03 2026 -0700

    [ROCm][Bugfix] Fix fused_mla_dual_rms_norm for AITER API rename _fused_qk_rmsnorm (#42606)

    Signed-off-by: Rita Brugarolas Brufau <[email protected]>

commit de2d76f35239c58202e49469dc5524b6f6fc4ffb
Author: Michael Goin <[email protected]>
Date:   Fri May 15 16:46:16 2026 -0400

    [Build] Switch CUDA 12.9 wheel builds to PyTorch manylinux_2_28 base (#41668)

    Signed-off-by: mgoin <[email protected]>
    Co-authored-by: Claude <[email protected]>

commit 9a7a273dfe6a89bbe00639fe99b0d61095fbc40a
Author: Sergei Skvortsov <[email protected]>
Date:   Fri May 15 21:01:21 2026 +0100

    Add HumanEval and GSM8K benchmarks to datasets (#42648)

    Signed-off-by: southfreebird <[email protected]>
    Co-authored-by: Michael Goin <[email protected]>

commit b2c58ee9427f15563210e184c57a6e530f37e464
Author: Lanze Liu <[email protected]>
Date:   Fri May 15 12:34:59 2026 -0700

    [FlashAttn] Fix supports_kv_cache_dtype() accepting unhandled fp8 kv-cache dtype variants (#42685)

    Signed-off-by: Lanze Liu <[email protected]>
    Signed-off-by: Matthew Bonanni <[email protected]>
    Co-authored-by: Matthew Bonanni <[email protected]>

commit 4d67d3bde25f94b6199ce16c7ef239ae4412bb8f
Author: frida-andersson <fanders…
@pasta-paul pasta-paul mentioned this pull request May 23, 2026
Closed
@sychen52

sychen52 commented May 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author
  • within trtllm-gen: NVFP4 mostly beats MXFP4.
  • trtllm-gen vs MegaMoE — Prefill: MegaMoE wins TTFT in most concurrencies across all DEPs.
  • trtllm-gen vs MegaMoE — Decode + overall (DEP-dependent): trtllm-gen dominates at DEP=1 and DEP=4 for most concurrency; MegaMoE overtakes overall at DEP=8, 16
  • this results suggests there is value to support NVFP4 in MegaMoE.

DEP=1 TTFT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 177.57 169.42 131.80 22.21%
4 539.44 332.55 255.69 23.11%
16 644.17 544.24 423.29 22.22%
64 851.61 809.37 522.39 35.46%
256 848.90 741.65 589.12 20.57%
         
DEP=1 TPOT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 6.84 6.89 10.01 0.73%
4 8.18 8.16 11.59 0.24%
16 12.81 12.81 17.22 0.0%
64 22.78 22.63 28.58 0.66%
256 39.06 38.37 45.28 1.77%
         
DEP=1 tok_s        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 142.88 141.95 98.75 0.66%
4 419.38 425.03 303.39 1.35%
16 1172.71 1186.20 899.21 1.15%
64 2553.30 2568.98 2094.59 0.61%
256 6286.40 6370.97 5384.10 1.35%
         
DEP=4 TTFT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 183.86 183.85 131.06 28.71%
4 341.91 192.84 138.37 28.25%
16 501.27 337.87 320.87 5.03%
64 673.61 561.10 491.07 12.48%
256 691.49 567.74 578.83 1.92%
         
DEP=4 TPOT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 8.58 8.49 9.07 1.05%
4 9.31 9.17 10.28 1.50%
16 11.81 11.07 11.77 5.95%
64 15.69 13.63 14.20 4.01%
256 18.83 23.09 17.57 6.69%
         
DEP=4 tok_s        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 114.48 115.55 108.79 0.93%
4 364.98 374.27 341.36 2.55%
16 1067.11 1210.47 1158.56 4.48%
64 3707.67 4234.34 4146.16 2.13%
256 12181.72 10854.49 13977.54 14.74%
         
DEP=8 TTFT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 189.56 189.34 131.69 30.45%
4 223.20 274.43 132.99 40.42%
16 391.20 353.94 631.76 9.52%
64 574.25 481.93 577.96 16.08%
256 802.88 870.11 758.23 5.56%
         
DEP=8 TPOT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 9.01 8.82 8.14 7.71%
4 9.64 9.43 8.60 8.80%
16 10.89 10.49 9.75 7.05%
64 13.49 13.90 11.32 16.09%
256 16.31 15.37 13.16 14.38%
         
DEP=8 tok_s        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 108.95 111.27 121.01 8.75%
4 404.35 413.13 456.31 10.45%
16 1100.28 1332.90 1025.69 21.14%
64 4316.67 4098.72 4993.96 15.69%
256 13269.14 12558.79 15736.57 18.60%
         
DEP=16 TTFT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 197.87 200.56 129.97 34.32%
4 288.98 206.01 142.96 30.61%
16 385.07 382.14 259.10 32.20%
64 418.13 583.34 478.39 12.60%
256 693.27 950.99 901.50 23.10%
         
DEP=16 TPOT_ms        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 10.40 10.19 8.12 20.31%
4 11.31 10.64 8.53 19.83%
16 12.21 12.30 9.13 25.23%
64 13.17 13.42 9.75 25.97%
256 15.28 15.37 10.97 28.21%
         
DEP=16 tok_s        
concurrency mxfp4_trtllm nvfp4_trtllm mxfp4_mega_moe best_better_than_second_pct
1 94.68 96.44 121.29 25.77%
4 301.95 367.11 460.46 25.43%
16 1008.42 939.09 1205.67 19.56%
64 3921.25 3930.52 5098.62 29.72%
256 14495.70 14151.91 18665.28 28.76%

example command:

  VLLM_USE_FLASHINFER_MOE_FP4=1 \
  VLLM_FLASHINFER_MOE_BACKEND=latency \
  python -m vllm.entrypoints.cli.main serve \
    /checkpoint/path \
    --trust-remote-code \
    --kv-cache-dtype fp8 \
    --served-model-name nvfp4

h1t35h pushed a commit to h1t35h/vllm that referenced this pull request May 26, 2026
@sychen52 @h1t35h
Add NVFP4 MOE support for Deepseek V4. (vllm-project#42209)
56fb089 
Signed-off-by: Shiyang Chen <[email protected]>
@pasta-paul pasta-paul mentioned this pull request May 26, 2026
Closed
@mxinO mxinO mentioned this pull request May 28, 2026
Merged
Liuweixiong0118 pushed a commit to Liuweixiong0118/vllm that referenced this pull request Jun 1, 2026
@sychen52 @Liuweixiong0118
Add NVFP4 MOE support for Deepseek V4. (vllm-project#42209)
9f53975 
Signed-off-by: Shiyang Chen <[email protected]>
Signed-off-by: Liuweixiong0118 <[email protected]>
amd-callumm added a commit to ROCm/vllm that referenced this pull request Jun 2, 2026
@amd-callumm @jikunshang
@mmangkad @NickLucche @njhill @WoosukKwon @gnovack george @qyYue1389 @claude @khluu @junyanxu @mergify @gracie-guo Gracie Guo @noooop @chaojun-zhang @taneem-ibrahim @gemini-code-assist @DarkLight1337 @hmellor @hao-aaron @ivanium @shen-shanshan @ZhanqiuHu @sagearc @xinyu-intel @yiwen101 @sfeng33 @jinzhen-lin @DomBrown @Dao007forever @tomeras91 @wzhao18 @Dogacel @maxdebayser @fadara01 @Terrencezzj @benchislett @pmaybank @izikgo @mgoin @ronensc @cleonard530 @mikaylagawarecki @Harry-Chen @arpera @zxd1997066 @hallerite @wangrui6 @raygorous @kebe7jun @dsikka @rishitdholakia13 @cursoragent @j9smith @vadiklyutiy @Isotr0py @haosdent @meena-at-work @yewentao256 @MatthewBonanni @dllehr-amd @ZJY0516 @akii96 @alexeldeib @louie-tsai @jeejeelee @adimcm2 @lucianommartins @bbrowning @AAISSJ 세덩 sejung-son @BugenZhao @DaoyuanLi2816 @yzong-rh @ianliuy @chaunceyjiang @zexplorerhj @zhangxin81 @velonica0 @bigPYJ1151 @fuscof-ibm @anishesg @zhengluo-nv @codex @robertgshaw2-redhat @ylangtsou Ylang Tsou @fangyuchu @zWaNg3 @liulanze @zx3xyy @houseroad @simondanielsson @divakar-amd @ruizhang99 Rui Zhang @zixi-qi @xwu-intel @yufufi @tc-mb @wdhongtw @mrjunwan-lang @wangxiyuan @majian4work @wasnertobias @JasonKeyiL Keyi Li @ilmarkov Markov Ilya @sychen52 @zhandaz @wangshangsam @zyongye @russellb @SandishKumarHN @juhi10071998 @itayalroy @djmmoss @SumanthRH @AndreasKaratzas @ashwing @abinggo @mayuyuace @zhougit86 xiaogang.zhou @meenchen @lucifer1004 @popkart-EZ @jzakrzew @ffggs @huanghua1994 @fuergaosi233 @bedeks @tjtanaa @orozery @danisereb @bananighosh @rshavitt @weizhoublue @TheDuyIT @esmeetu @Yihuki @zhewenl Zhewen Li @yma11 @QingZhou-YangHY @gau-nernst @zhejiangxiaomai @CienetStingLin @Jie-Fang @ILikeIneine @yubofredwang @chfeng-cs @ThibaultCastells @linzm1007 @javierdejesusda @bnellnm @markmc Zhewen Li @xyang16 @LucasWilkinson @angelayi @aoshen02 @nholmber @zhangtao2-1 zhangtao @adityasingh2400 @injaeryou @chunyang-wen @jatseng-ai
Merge upstream vLLM code into gfx11 (#983)
cdd11a6 
* [XPU] add gptq(int4) support (#37844)

Signed-off-by: Kunshang Ji <[email protected]>

* [UX] Add a persistent cache for FlashInfer autotuning (#42537)

Signed-off-by: Mohammad Miadh Angkad <[email protected]>

* [Bugfix][MRV2] Fix KVCache tensor explicit `kernel_block_size` dim (#42766)

Signed-off-by: NickLucche <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
Co-authored-by: Nick Hill <[email protected]>

* [Model Refactoring] Move DeepSeek V4 layers to `models/deepseek_v4/` [2/N] (#43039)

Signed-off-by: Woosuk Kwon <[email protected]>

* add cutedsl dsv4 indexer fp8 kernel (#42899)

Signed-off-by: george <[email protected]>
Co-authored-by: george <[email protected]>

* [Bugfix][KV Connector] Fix SimpleCPUOffloadScheduler TOCTOU between Phase A and Phase B (#42289)

Signed-off-by: Qiuyang Yue <[email protected]>
Co-authored-by: Claude Opus 4.7 <[email protected]>
Co-authored-by: gemini-code-assist <[email protected]>

* [ci] Route 28 gpu_1_queue tests to h200_35gb queue (#43030)

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* fix: use keyword arguments for shard_id and expert_id in weight_loade… (#42671)

Signed-off-by: junyanxu <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Docs] Add SVG images for pooling models. (#42626)

Signed-off-by: Gracie Guo <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Co-authored-by: Gracie Guo <[email protected]>
Co-authored-by: wang.yuqi <[email protected]>

* [XPU] Use custom op collective behavior  (#41354)

Signed-off-by: Chaojun,Zhang <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <[email protected]>

* [Misc] Aligning tokwise pooler heads for consistency (#43041)

Signed-off-by: Taneem Ibrahim <[email protected]>

* [Docs] Reorganize online serving docs. (#41907)

Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <[email protected]>
Co-authored-by: Harry Mellor <[email protected]>

* [Frontend] Consolidate beam search by BeamSearchMixin. (#42946)

Signed-off-by: wang.yuqi <[email protected]>

* [Model Refactoring] Move deepseek_v4_ops to models/deepseek_v4 [3/N] (#43073)

Signed-off-by: Woosuk Kwon <[email protected]>

* [bug] AsyncScheduler drops first post-resume token after pause_generation + clear_cache (#42117)

Signed-off-by: hao-aaron <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [KVConnector][DSV4] HMA support for Mooncake store connector (#42828)

Signed-off-by: Yifan Qiao <[email protected]>

* [Model Refactoring] Rename deepseek_v4.py to model.py [4/N] (#43077)

Signed-off-by: Woosuk Kwon <[email protected]>

* [Misc][MM] Remove redundant code in CLIPAttention (#43046)

Signed-off-by: shen-shanshan <[email protected]>

* [CI] Add MTP + PD disagg test for Qwen3.5 (#42677)

Signed-off-by: ZhanqiuHu <[email protected]>
Co-authored-by: Nicolò Lucchesi <[email protected]>

* [Bugfix] Fix top logprobs token placeholders in `/inference/v1/generate` (#42887)

Signed-off-by: Sage Ahrac <[email protected]>

* [Perf][4/n] Eliminate various GPU<->CPU syncs (#42347)

Signed-off-by: Nick Hill <[email protected]>

* [XPU] update xpu graph usage (#43043)

Signed-off-by: Xinyu Chen <[email protected]>

* [Model] Openvla support (#42654)

Signed-off-by: Wang Yiwen <[email protected]>

* [Refactor] Extract extract_types_from_schema utility from Minimax M2 tool parser (#43025)

Signed-off-by: sfeng33 <[email protected]>

* [Misc] add humming to dependencies (#42540)

Signed-off-by: Jinzhen Lin <[email protected]>

* [feat] Add FP8 per-tensor Q scale support to Triton attention backend (#42080)

Signed-off-by: Dom Brown <[email protected]>

* [Docs] Fix MooncakeStoreConnector role in disaggregated example (#42994)

Signed-off-by: Dao Le <[email protected]>
Co-authored-by: Claude <[email protected]>

* [Bugfix][MoE] FlashInfer one-sided: workspace union across heterogeneous layers (#42976)

Signed-off-by: Tomer Asida <[email protected]>

* [CI failure] Temporarily disable using persistent cache for flashinfer autotune (#43119)

Signed-off-by: wzhao18 <[email protected]>
Signed-off-by: Wei Zhao <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [ci] Move language models tests (hybrid) back to L4 (#43129)

Signed-off-by: Kevin H. Luu <[email protected]>

* [Model] Support post-norm architecture for EAGLE-3 supeculators (#42764)

Signed-off-by: Doğaç Eldenk <[email protected]>

* Fix error in Dynamic NTK scaling (#41277)

Signed-off-by: Max de Bayser <[email protected]>
Signed-off-by: Max de Bayser <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: wang.yuqi <[email protected]>

* [CPU][DOC] Fix installation commands for Arm CPUs (#43115)

Signed-off-by: Fadi Arafeh <[email protected]>

* [bug] fix WeightTransferConfig.backend to allow for all strings (#43121)

Signed-off-by: ahao-anyscale <[email protected]>

* [MRV2][BugFix] Fix default-stream CG capture in P/W LoRA case (#43160)

Signed-off-by: Nick Hill <[email protected]>

* [Cohere] Enable Cohere MoE (#43143)

Signed-off-by: Terrencezzj <[email protected]>

* [Perf][Bugfix] Update dflash aux layer indexing (#40727)

Signed-off-by: Benjamin Chislett <[email protected]>

* add enqueue all option to throughput benchmark (#42975)

Signed-off-by: Philip Maybank <[email protected]>
Signed-off-by: pmaybank <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Perf] Avoid forward scan for async output placeholders (#42938)

* [CI] Add DSV4-Flash to gsm8k moe-refactor/config-b200.txt (#42111)

Signed-off-by: mgoin <[email protected]>

* [KV Offload] Pass `OffloadingSpec` instead of `VllmConfig` to secondary tiers (#43076)

Signed-off-by: Ronen Schaffer <[email protected]>

* [ci] Revert model executor test back to L4 (#43188)

Signed-off-by: Kevin H. Luu <[email protected]>

* [Docs][PD][NIXL] Lease extension mechanism for blocks on P (#43099)

Signed-off-by: NickLucche <[email protected]>

* [Docs][PD][NIXL] Bidirectional kv-cache transfer (#43097)

Signed-off-by: NickLucche <[email protected]>

* [6/n] Migrate activation kernels, gptq, gguf, non cutlass w8a8 to libtorch stable ABI (continued) (#42663)

Signed-off-by: Mikayla Gawarecki <[email protected]>
Signed-off-by: Chris Leonard <[email protected]>
Co-authored-by: Mikayla Gawarecki <[email protected]>
Co-authored-by: Shengqi Chen <[email protected]>

* Enable mermaid diagrams in the docs (#43192)

Signed-off-by: Harry Mellor <[email protected]>

* [GDN] Enable FI Blackwell GDN prefill kernel (#40717)

Signed-off-by: Artem Perevedentsev <[email protected]>

* [XPU][CI] Add 2 server model test files in Intel GPU CI (#42499)

Signed-off-by: zengxian <[email protected]>

* [Frontend] Forward X-data-parallel-rank header on /inference/v1/generate (#42330)

Signed-off-by: hallerite <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

* [Doc] Sync CLI guide with actual help modes and launch subcommand (#40326)

Signed-off-by: Rui Wang <[email protected]>
Co-authored-by: Rui Wang <[email protected]>

* [Feature] Support manually enabling the cumem allocator (#33648)

Signed-off-by: Kebe <[email protected]>

* [Spec Decode] Support non-MTP speculation for NemotronH (#43130)

Signed-off-by: Benjamin Chislett <[email protected]>

* Remove additional dead code as a follow-up to #42889 (#43144)

Signed-off-by: Dipika Sikka <[email protected]>

* [Bug][Structured Outputs] Fix bug that leads to unconstrained generations with structural tags (#42452)

Signed-off-by: rishitdholakia13 <[email protected]>
Co-authored-by: Cursor <[email protected]>

* [Bugfix] Use enable_sm120_family for per-tensor FP8 CUTLASS kernels on SM12.1 (#41215)

Signed-off-by: j9smith <[email protected]>
Signed-off-by: Joel Smith <[email protected]>
Co-authored-by: Shengqi Chen <[email protected]>

* [Bugfix] Use shared coerce_to_schema_type in DeepSeekV32 tool parser (#43019)

Signed-off-by: sfeng33 <[email protected]>

* [MISC] Fix symm_mem cap-equal gate; log AR backend selection (#42993)

Signed-off-by: Vadim Gimpelson <[email protected]>

* [R3] Add routed experts to openai entrypoint  (#38939)

Signed-off-by: ahao-anyscale <[email protected]>
Co-authored-by: Isotr0py <[email protected]>

* [CI] Lower granite-4.0-h-tiny gsm8k threshold for Hybrid SSM NixlConnector PD accuracy tests (4 GPUs) (#43186)

Signed-off-by: haosdent <[email protected]>
Signed-off-by: NickLucche <[email protected]>
Co-authored-by: NickLucche <[email protected]>

* Integrate flashinfer b12x MoE and FP4 GEMM kernels for SM120/121 (#40082)

Signed-off-by: Meenakshi Venkataraman <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>

* [Perf] Optimize `CutlassFP8ScaledMMLinearKernel` when padding needed by pre-weight processing, 13.5% TTFT improvement (#42651)

Signed-off-by: yewentao256 <[email protected]>
Signed-off-by: Wentao Ye <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>

* [Bugfix][CI] Add missing import of pad_nvfp4_activation_for_cutlass in flashinfer (#43237)

Signed-off-by: sfeng33 <[email protected]>

* Add dllehr-amd to CODEOWNERS and committers list (#42772)

Signed-off-by: Douglas Lehr <[email protected]>

* [Perf][gpt-oss] Downgrade triton_kernels to v3.5.1 (#43135)

Signed-off-by: mgoin <[email protected]>

* [Misc] downgrade nvidia-cutlass-dsl to 4.5.0 (#43230)

Signed-off-by: zjy0516 <[email protected]>

* [ROCm] Add QuickReduce min-size override and codec threshold (#41675)

Signed-off-by: <>

* [CI] Add composed-schema regression tests for DeepSeek V3.2/V4 parsers (#43255)

Signed-off-by: Ace Eldeib <[email protected]>
Co-authored-by: Flora Feng <[email protected]>

* [Model Runner V2] Fix lora `Triton Error [CUDA]: device-side assert triggered` (#43139)

Signed-off-by: yewentao256 <[email protected]>
Signed-off-by: Wentao Ye <[email protected]>
Co-authored-by: Nick Hill <[email protected]>

* update GPU json file based on h200 recipes (#43262)

Signed-off-by: louie-tsai <[email protected]>

* [Minor]  Bigger overlap for FI AR (#43103)

Signed-off-by: Jee Jee Li <[email protected]>

* [Bugfix] Fix Qwen3.5 GatedDeltaNet in_proj_ba Marlin failure at TP>=2 (#36329)

Signed-off-by: Adi McM Sonus Flow <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

* [Perf][Gemma4] Batch vision encoder calls for image and video processing (#43169)

Signed-off-by: Luciano Martins <[email protected]>
Co-authored-by: Luciano Martins <[email protected]>

* [CI] Fix "test_vit_cudagraph_[image|video][step3_vl]" failure (#43082)

Signed-off-by: haosdent <[email protected]>

* [Frontend] Normalize reasoning_content to reasoning for client compatibility (#42664)

Signed-off-by: Ben Browning <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Refactor] Use shared coerce_to_schema_type in Seed-OSS tool parser (#43140)

Signed-off-by: sfeng33 <[email protected]>

* [ToolParser][Bugfix] Re-land: Fix anyOf/oneOf/$ref type resolution in Qwen3CoderToolParser (#37831) (#38973)

Signed-off-by: AAISSJ <[email protected]>
Signed-off-by: <>
Signed-off-by: sejung-son <[email protected]>
Signed-off-by: sfeng33 <[email protected]>
Co-authored-by: 세덩 <[email protected]>
Co-authored-by: sejung-son <[email protected]>
Co-authored-by: sfeng33 <[email protected]>

* [Frontend][RFC] Rust front-end integration (#40848)

Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Co-authored-by: Bugen Zhao <[email protected]>

* [Bugfix] Warn when renderer_num_workers has no effect on offline LLM (#42905)

Signed-off-by: Daoyuan Li <[email protected]>

* [Benchmark] Add num-warmup to vllm bench throughput (#43245)

Signed-off-by: Yifan Zong <[email protected]>

* [Bugfix] Fix glm4_moe_tool_parser._is_string_type for /v1/responses FunctionTool format (#39601)

Signed-off-by: Yiyang Liu <[email protected]>
Signed-off-by: Chauncey <[email protected]>
Signed-off-by: sfeng33 <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Chauncey <[email protected]>
Co-authored-by: sfeng33 <[email protected]>

* [CI] De-flake test_models for bigscience/bloom-560m (#43197)

Signed-off-by: haosdent <[email protected]>

* [XPU] add setuptools-rust for xpu dependency (#43287)

Signed-off-by: Kunshang Ji <[email protected]>

* Update KDA chunk prefill decay to use exp2 semantics (#43195)

Signed-off-by: zexplorerhj <[email protected]>
Co-authored-by: zexplorerhj <[email protected]>

* Fix FlashInfer TRTLLM NvFP4 monolithic MoE routing (#43223)

Signed-off-by: zhangxin81 <[email protected]>

* [Test] Replace zephyr-7b-beta (7B) with SmolLM2-135M in tokenization test (#43085)

Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* [Bug] Fix ci issue `assert output_size is not None` AssertionError (#43261)

Signed-off-by: yewentao256 <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Co-authored-by: Isotr0py <[email protected]>

* [CI] Pin protoc binary in rust-build stages (#43292)

Signed-off-by: haosdent <[email protected]>

* [XPU][CI]Fix Docker image pull-to-run race in Intel GPU CI (#43266)

Signed-off-by: zengxian <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

* [CPU][RISC-V] Add VLEN=256 support to RVV attention kernels (#42943)

Signed-off-by: velonica0 <[email protected]>
Signed-off-by: velonica0 <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Li, Jiang <[email protected]>

* [Perf] [Hybrid] Fused Triton kernel for GPU-side Mamba state postprocessing (#40172)

Signed-off-by: Francesco Fusco <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [CI] Fix CPU tests failing on `tl.exp2` import (#43311)

Signed-off-by: haosdent <[email protected]>

* [Bugfix] Add early validation to reject incompatible runner types for embedding models (#43079)

Signed-off-by: anish <[email protected]>
Signed-off-by: Your Name <[email protected]>
Signed-off-by: anish <[email protected]>
Co-authored-by: anish <[email protected]>
Co-authored-by: Claude Opus 4.6 <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>

* [Deprecation] Mark env vars covered by --moe-backend / --linear-backend (#43148)

Signed-off-by: mgoin <[email protected]>
Signed-off-by: Michael Goin <[email protected]>

* [Perf] `zeros` -> `empty` to remove additional fill (#42988)

Signed-off-by: yewentao256 <[email protected]>

* [Core] Add native ModelExpress load format (#43105)

Signed-off-by: Zheng Luo <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Robert Shaw <[email protected]>

* Disable build isolation to bypass CUDA related deps for vllm-tpu (#43038)

Signed-off-by: Ylang Tsou <[email protected]>
Co-authored-by: Ylang Tsou <[email protected]>
Co-authored-by: Michael Goin <[email protected]>

* [Frontend] Rework fastokens integration (#43168)

Signed-off-by: Nick Hill <[email protected]>

* [Feature] Add `--cpu-distributed-timeout-seconds` CLI Option for CPU Process Group Timeout (#42968)

Signed-off-by: fangyuchu <[email protected]>
Signed-off-by: zWaNg3 <[email protected]>
Co-authored-by: zWaNg3 <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [BugFix] Use correct logprobs for `logprob_token_ids` (#43125)

Signed-off-by: Nick Hill <[email protected]>

* [Bugfix] Zero stale is_prefilling in padded CUDA graph rows for Mamba (#41873)

Signed-off-by: Lanze Liu <[email protected]>

* [Rust Frontend] Move code from `vllm-frontend-rs` (#43283)

Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Eric Curtin <[email protected]>
Signed-off-by: Dev-X25874 <[email protected]>
Signed-off-by: Will.hou <[email protected]>
Signed-off-by: Will.hou <[email protected]>
Co-authored-by: Nick Hill <[email protected]>
Co-authored-by: Eric Curtin <[email protected]>
Co-authored-by: Dev-X25874 <[email protected]>
Co-authored-by: Will.hou <[email protected]>
Co-authored-by: Will.hou <[email protected]>

Please see https://github.com/Inferact/vllm-frontend-rs for full original commit history.

* [CI] Fix dockerfile dependency graph failure for pre-commit (#43378)

Signed-off-by: Isotr0py <[email protected]>

* [Bugfix] Fix DSV4 Base model swiglu limit issue in FP8 path  (#42855)

Signed-off-by: Chengze Fan <[email protected]>
Signed-off-by: Chengze Fan <[email protected]>
Co-authored-by: Lu Fang <[email protected]>

* [ROCm] Add XGMI backend for MoRI Connector (#41753)

Signed-off-by: simondanielsson <[email protected]>

* [ROCm][CI] add warmup to mem_util test before measurement (#43236)

Signed-off-by: Divakar Verma <[email protected]>

* [Frontend] Add truncation side to OpenAI endpoints (#43260)

Signed-off-by: Rui Zhang <[email protected]>
Signed-off-by: Rui Zhang <[email protected]>
Co-authored-by: Rui Zhang <[email protected]>

* [Frontend] DP Supervisor (#40841)

Signed-off-by: yewentao256 <[email protected]>
Signed-off-by: Robert Shaw <[email protected]>
Signed-off-by: Robert Shaw <[email protected]>
Co-authored-by: robertgshaw2-redhat <[email protected]>
Co-authored-by: Robert Shaw <[email protected]>
Co-authored-by: Nick Hill <[email protected]>

* [Bugfix] Make CuMemAllocator free callback stream-aware (#43020)

Signed-off-by: zixi-qi <[email protected]>
Co-authored-by: Claude <[email protected]>

* [XPU] Enable multiple key kernels for sparse attention (#37888)

Signed-off-by: Xiaochang Wu <[email protected]>
Signed-off-by: Wu, Xiaochang <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

* [CI] De-flake renderers/test_hf.py::test_resolve_content_format_fallbacks[Qwen/Qwen-VL-string] (#43064)

Signed-off-by: haosdent <[email protected]>

* [Model] Use `AutoWeightsLoader` for Voyage (#42972)

Signed-off-by: Furkan Fidan <[email protected]>

* [Model] Fix MiniCPM-V 4.6 vit_merger qkv weight loading (#43213)

Signed-off-by: tc-mb <[email protected]>

* [CI] Fix test_lora_with_spec_decode on V2 model runner (#43314)

Signed-off-by: haosdent <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>

* [CI] Fix "test_awq_load[gemma4-moe-*]" failure (#43296)

Signed-off-by: haosdent <[email protected]>

* Correcting the mock classes for MM GC tests (#43321)

Signed-off-by: Weida Hong <[email protected]>

* [BugFix] Fix setuptools-rust dep in requirements files (#43377)

Signed-off-by: Nick Hill <[email protected]>

* Fix the docker build failure in tpu-inference (#43360)

Signed-off-by: mrjunwan-lang <[email protected]>

* [Docs] Note image preprocessing difference between qwen_vl_utils and vllm. (#43393)

Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [CPU] Experimentally enable Triton and MRV2 (#43225)

Signed-off-by: jiang1.li <[email protected]>

* [Attention] Mamba attention module refactor (#41126)

Signed-off-by: wangxiyuan <[email protected]>

* [XPU]feat: add XPU fallback for MoE topk routing and MXFP4 backend (#42951)

Signed-off-by: Ma Jian <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

* [Misc] Replace assert with proper exceptions for security and validation in pooling (#43286)

Signed-off-by: Taneem Ibrahim <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>
Co-authored-by: Nick Hill <[email protected]>

* [Bugfix] Clear P0 mm sender cache on sleep/pause to fix mm_hash desync (#43001)

Signed-off-by: Tobias Wasner <[email protected]>

* [BugFix] wire make_empty_intermediate_tensors on AyaVision and Voxtral (#43118)

Signed-off-by: Keyi Li <[email protected]>
Co-authored-by: Keyi Li <[email protected]>

* [LoRA] Reduce memory of 2D weights when EP is set (#42737)

Signed-off-by: Jee Jee Li <[email protected]>

* [EPLB] Change default EPLB communicator (#43110)

Signed-off-by: Markov Ilya <[email protected]>
Co-authored-by: Markov Ilya <[email protected]>

* [CI] Fix AMD docker build tests (#43329)

Signed-off-by: haosdent <[email protected]>

* Add NVFP4 MOE support for Deepseek V4. (#42209)

Signed-off-by: Shiyang Chen <[email protected]>

* [Multimodal] Simplify ViT CUDA graph interfaces (#41234)

Signed-off-by: Isotr0py <[email protected]>

* [Rust Frontend] [Refactor] Extract a newtype for utility call ID (#43405)

Signed-off-by: Bugen Zhao <[email protected]>

* [Bugfix] Source num_qo_heads from Attention layers in Flashinfer/Triton metadata builders (#42650)

Signed-off-by: zhanda <[email protected]>
Co-authored-by: Shang Wang <[email protected]>

* [KV Connector] MooncakeStore: don't co-queue save with load to avoid double delayed-free (#43371)

Signed-off-by: Dao Le <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

* [Refactor] Extract DeepSeek V4 sparse MLA impl into model folder (#43149)

* [Frontend] Simplify AuthenticationMiddleware path extraction (#43426)

Signed-off-by: Russell Bryant <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* [RFC][EPLB][#32028] Remove dead torch.accelerator.synchronize() from sync path (#40733)

Signed-off-by: SandishKumarHN <[email protected]>
Co-authored-by: SandishKumarHN <[email protected]>

* [Bugfix] Detect wrong libcute_dsl_runtime.so variant in FlashInfer GDN (#43427)

Signed-off-by: Artem Perevedentsev <[email protected]>

* [Bugfix] Clear error message for FP8 torchao quantization on unsupported GPUs (#36854)

Signed-off-by: haosdent <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* mhc_post - remove sts & add vectorized copies (#43437)

Signed-off-by: george <[email protected]>
Co-authored-by: george <[email protected]>

* [Quantization][ModelOpt] W4A16 NVFP4 fused MoE + mixed-precision dispatch (#42566)

Signed-off-by: Juhi Mittal <[email protected]>

* [Model Runner V2] Support sharing kv cache layers (#35045)

Signed-off-by: Nick Hill <[email protected]>

* DSv4 fused Q-norm kernel grid refactor (#42353)

* [Perf] Optimize hidden state extraction logic (#37374)

Signed-off-by: Benjamin Chislett <[email protected]>
Signed-off-by: Benjamin Chislett <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [XPU]fix: add XPU platform guards to DeepSeek-V4 ops (#42950)

Signed-off-by: Ma Jian <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

* elastic_ep: stage/commit MoE quant method on reconfigure (#40881)

Signed-off-by: Itay Alroy <[email protected]>

* [Attention] Add head_dim=512 support for FlashInfer trtllm attention backend (#38822)

* Add `model` to `WeightTransferEngine.__init__` (#42922)

Signed-off-by: SumanthRH <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [DSV4] More multi-stream enablement for c4a (#42925)

Signed-off-by: Yongye Zhu <[email protected]>

* [ROCm][CI] Stabilize runner teardown between sampler tests (#43023)

Signed-off-by: Andreas Karatzas <[email protected]>

* [ROCm][CI] Stabilize Granite tool-use and test URL construction (#43017)

Signed-off-by: Andreas Karatzas <[email protected]>

* [Bugfix] Auto-raise max_num_batched_tokens for prefix-LM multimodal models (#43051)

Signed-off-by: Ashwin Giridharan <[email protected]>
Co-authored-by: abinggo <[email protected]>

* [ROCm][CI] Fix ROCm LoRA Transformers fallback with full CUDA graphs (#41577)

Signed-off-by: Andreas Karatzas <[email protected]>

* [XPU]feat: enable FP8 block-scaled quantization on XPU (#42952)

Signed-off-by: Ma Jian <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

* [XPU] reudce host overhead of XPU MOE (#42915)

Signed-off-by: mayuyuace <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>

* [7/n] Migrate pos_encoding and norm kernels to libtorch stable ABI (continued) (#43209)

Signed-off-by: Mikayla Gawarecki <[email protected]>
Signed-off-by: Chris Leonard <[email protected]>
Co-authored-by: Mikayla Gawarecki <[email protected]>
Co-authored-by: Shengqi Chen <[email protected]>

* [Misc] Added missing return type annotations to improve mypy and IDE tooling (#43383)

Signed-off-by: Taneem Ibrahim <[email protected]>

* [Bugfix] Fix native Triton top-k/top-p kernel assumes contiguous logi… (#42739)

Signed-off-by: xiaogang.zhou <[email protected]>
Co-authored-by: xiaogang.zhou <[email protected]>

* [ModelOpt] Support Qwen3.5/3.6 VLM quantized prefix mapping (#42546)

Signed-off-by: weimingc <[email protected]>

* Keep scheduler alive for delayed KV connector frees (#43433)

Signed-off-by: Zihua Wu <[email protected]>

* fix(eagle3): read norm_before_fc from eagle_config for NVIDIA checkpoint (#42143)

Signed-off-by: FERRARIZHENG <[email protected]>

* [Kernel] Batch invariant NVFP4 linear using cutlass (#39912)

Signed-off-by: Jakub Zakrzewski <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>
Co-authored-by: Yongye Zhu <[email protected]>

* [ROCm][CI] Remove benchmarks test group and shard long test groups (#41669)

Signed-off-by: Andreas Karatzas <[email protected]>

* [Bugfix][Frontend] Fix input_audio parsing when uuid is present  (#43414)

Signed-off-by: ffggs <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>

* [MM] Enable FlashInfer metadata support for Qwen2.5-VL vision attention (#42787)

Signed-off-by: Hua Huang <[email protected]>
Co-authored-by: Isotr0py <[email protected]>

* [Docs] Fix stale version number in token_embed.md (#43488)

Signed-off-by: holegots <[email protected]>

* [Docs] Fix stale version number in token_classify.md (#43489)

Signed-off-by: holegots <[email protected]>

* [MoE] Migrate W4A8 CT to oracle kernel setup (#42680)

Signed-off-by: Siddharth Bedekar <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>

* [Mooncake] Add metrics for MooncakeStoreConnector operations (#43392)

* [ROCm][Critical] Fix the GDN import bug (#43486)

Signed-off-by: tjtanaa <[email protected]>

* Revert "[Misc] add humming to dependencies" (#43492)

* [Bugfix] Fix reasoning dropped on streaming boundary deltas (#42691)

Signed-off-by: sfeng33 <[email protected]>

* [Model Runner v2] Force v1 runner for tests (#43233)

Signed-off-by: yewentao256 <[email protected]>

* [KV Connector] Keep MooncakeStore full hits block-aligned (#43494)

Signed-off-by: Dao Le <[email protected]>
Signed-off-by: Dao Le <[email protected]>
Co-authored-by: Claude <[email protected]>

* [kv_offload]: Add DSv4 support (#43142)

Signed-off-by: Or Ozeri <[email protected]>

* [ROCm][CI] Stabilize 400 error return code for invalid schema inputs (#43016)

Signed-off-by: Andreas Karatzas <[email protected]>

* [ROCm] [DSv4] [Perf] Support DeepSeek v4 MTP (#43385)

Signed-off-by: tjtanaa <[email protected]>

* Tuning script and configs for Triton Mamba SSU kernel (#43083)

Signed-off-by: Banani Ghosh <[email protected]>
Signed-off-by: Daniel Serebrenik <[email protected]>
Co-authored-by: Banani Ghosh <[email protected]>

* File system secondary tier implemented in python (#41735)

Signed-off-by: Rotem Shavitt <[email protected]>
Signed-off-by: Or Ozeri <[email protected]>
Co-authored-by: Or Ozeri <[email protected]>

* [Kernel] Add mhc_pre_big_fuse_with_norm_tilelang  (#43474)

Signed-off-by: Jee Jee Li <[email protected]>

* fix: MoE model using shared routed experts crashes on AMD GPUs (#42373)

Signed-off-by: [email protected] <[email protected]>

* [Docs] Reorganize offline inference docs.  (#43552)

Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Docker] Non-root support for vllm-openai; add opt-in vllm-openai-nonroot target (#40275)

Signed-off-by: TheDuyIT <[email protected]>
Signed-off-by: dtnguyen <[email protected]>
Co-authored-by: Claude <[email protected]>

* [Feat][KVConnector] Support DSV4 in SimpleCPUOffloadBackend (#42296)

Signed-off-by: Yifan Qiao <[email protected]>

* [Doc] Add section on escalating stalled contributions (#43568)

Signed-off-by: esmeetu <[email protected]>

* Reduce memory usage for granite_speech. (#42933)

Signed-off-by: Yihuki <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [KV Connector] Handle Mooncake finish after preemption (#43281)

Signed-off-by: Zhewen Li <[email protected]>
Co-authored-by: Zhewen Li <[email protected]>

* [Misc] Print accuracy value for PD tests even on success  (#43583)

Signed-off-by: NickLucche <[email protected]>

* [Kernel] Remove NormGateLinear (#43554)

Signed-off-by: Jee Jee Li <[email protected]>

* [XPU] Ensure RNG offset alignment with PyTorch requirements in XPU sampler (#43028)

Signed-off-by: chaojun-zhang <[email protected]>
Signed-off-by: Chaojun Zhang <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [LoRA] Add one shot triton kernel For MoE LoRA (#42290)

Signed-off-by: Jee Jee Li <[email protected]>

* [DeepSeek V4] Move MegaMoE input prep kernel to nvidia/ops (#43632)

Signed-off-by: Woosuk Kwon <[email protected]>

* [KV Connector][Bugfix] MooncakeStore: don't double-apply Eagle prune in load_mask (#43516)

Signed-off-by: Dao Le <[email protected]>
Signed-off-by: Dao Le <[email protected]>
Co-authored-by: Claude <[email protected]>

* [KV Connector] Propagate MooncakeStore load failures (#42788)

Signed-off-by: Dao Le <[email protected]>

* [Bugfix] fix device mismatch in MiniCPM-o-4_5 resampler (#43194)

Signed-off-by: Yan Ma <[email protected]>

* [Frontend] Split the offline inference APIs and utils. (#43553)

Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [Bugfix][Model] Fix GPT2ForSequenceClassification sub-module prefix (#43579)

Signed-off-by: QingZhou-YangHY <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>

* [GDN] GDN Prefill kernel for SM100 (#43273)

Signed-off-by: Thien Tran <[email protected]>

* [CPU] Enable non-divisible GQA for decode workitems in mixed batches (#43032)

Signed-off-by: zhejiangxiaomai <[email protected]>

* Upgrade tpu-inference to v0.20.0 (#43394)

* Add CuTe DSL sparse compressor support (#43584)

Signed-off-by: Yongye Zhu <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Yongye Zhu <[email protected]>

* [chores][log] change registry log from `warning` to `debug` (#43045)

Signed-off-by: Hank <[email protected]>

* [Bugfix] Apply fc_norm in Eagle3DeepseekV2 combine_hidden_states (#43482)

Signed-off-by: Yubo Wang <[email protected]>
Co-authored-by: Claude <[email protected]>

* [KV Transfer] Enable HMA by default for connectors that support it (#41847)

Signed-off-by: Ethan Feng <[email protected]>

* [Misc][Refactor][ROCm] Convert MoRI-related envvars to extra config args (#43303)

Signed-off-by: simondanielsson <[email protected]>
Co-authored-by: TJian <[email protected]>

* [Misc] Support interleaved custom image benchmark datasets (#43636)

Signed-off-by: ThibaultCastells <[email protected]>

* [Reasoning] [Bugfix] Reject invalid thinking_token_budget values (#43402)

Signed-off-by: linzm1007 <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Model] Use AutoWeightsLoader for InternLM2 (#38278)

Signed-off-by: Jesus De Jesus <[email protected]>
Signed-off-by: javierdejesusda <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>

* [XPU] Fix fused MoE LoRA kernel crash on XPU by using platform-agnos num_compute_units (#43646)

Signed-off-by: Chaojun,Zhang <[email protected]>

* Fix CuPy runtime deps and restore humming (#43530)

Signed-off-by: Mohammad Miadh Angkad <[email protected]>

* [Docs][ROCm] MoRI-IO Connector Usage Guide (#43603)

Signed-off-by: simondanielsson <[email protected]>
Signed-off-by: Simon Danielsson <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [ROCm][CI] Extend ROCm quick reduce coverage (#40990)

Signed-off-by: Andreas Karatzas <[email protected]>

* [Feat][DSV4] Fuse q pad into deepseek v4 fused kernel (#43162)

* [MoE Refactor] Migrate ModelOptMxFp8FusedMoE to oracle (#42768)

Signed-off-by: Bill Nell <[email protected]>
Co-authored-by: Robert Shaw <[email protected]>

* [MoE Refactor] W4a8 int8 oracle (#42789)

Signed-off-by: Bill Nell <[email protected]>
Co-authored-by: Robert Shaw <[email protected]>

* [ROCm] Remove MegaMoE integration in deepseek v4 (#43629)

Signed-off-by: Woosuk Kwon <[email protected]>

* Add LM head quantization support for ModelOpt (#42124)

Signed-off-by: weimingc <[email protected]>

* [Doc] Add line limit to AGENTS.md (#43635)

Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: Mark McLoughlin <[email protected]>
Co-authored-by: Mark McLoughlin <[email protected]>

* [DSv4] Drop _get_compressed_kv_buffer in DeepseekCompressor (#43690)

Signed-off-by: Woosuk Kwon <[email protected]>

* [CI] Soft-fail AMD entrypoints mirror tests (#43709)

Signed-off-by: Kevin Luu <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* [Kernel] Porting  fuse_minimax_qk_norm  to manual fusion (#43410)

Signed-off-by: Jee Jee Li <[email protected]>

* [KV Connector] MooncakeStore: drop dead discard_partial_chunks parameter (#43627)

Signed-off-by: Zhewen Li <[email protected]>
Co-authored-by: Zhewen Li <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

* [Bugfix][V1] Fix TOCTOU race causing intermittent `EADDRINUSE` on multi-API-server DP startup (#42585)

Signed-off-by: Vadim Gimpelson <[email protected]>
Signed-off-by: Vadim Gimpelson <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* [ci] Add arm64 ci image (#41303)

Signed-off-by: khluu <[email protected]>
Signed-off-by: Kevin H. Luu <[email protected]>
Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* [Bugfix] Split attention groups by num_heads_q for spec-decode drafts (#43543)

Signed-off-by: Luciano Martins <[email protected]>
Co-authored-by: Luciano Martins <[email protected]>

* [Rust Frontend] Add reasoning/tool parser & renderer roundtrip tests (#43582)

Signed-off-by: Bugen Zhao <[email protected]>

* [ROCm][CI] Fix ROCm multimodal Qwen2.5-VL activation compile and Phi4MM ragged image mask handling (#43647)

Signed-off-by: Andreas Karatzas <[email protected]>

* [Perf] Optimize Fp8BlockScaledMMLinearKernel input_scale tensor using new_empty() (#43677)

Signed-off-by: Xin Yang <[email protected]>

* [Attention] Make FlexAttention and FlashAttention use num-blocks first layouts (#42095)

Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Co-authored-by: Nicolò Lucchesi <[email protected]>

* [MLA][Attention] Add OOT MLA prefill backend registration mechanism (#43325)

Signed-off-by: Matthew Bonanni <[email protected]>

* [Deprecation] Deprecate functions as scheduled for v0.21.0 (#43358)

Signed-off-by: yewentao256 <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [DSv4] Refactor compressor & Fix ROCm compatibility (#43710)

Signed-off-by: Woosuk Kwon <[email protected]>

* Fix test_aot_compile for torch 2.12 (#43695)

Signed-off-by: Angela Yi <[email protected]>

* [KVConnector][Mooncake] Wire reset_cache cascade end-to-end (#42694)

Signed-off-by: aoshen524 <[email protected]>
Signed-off-by: Ao Shen <[email protected]>
Co-authored-by: aoshen524 <[email protected]>
Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

* [ROCm][Perf] Expose AITER MoE sorting dispatch policy via env var (#39177)

Signed-off-by: nholmber <[email protected]>

* [MRV2][BugFix] Fix KV connector handling in spec decode case (#43719)

Signed-off-by: Nick Hill <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>

* [Frontend] Add MiniCPM5 XML tool call parser (#43175)

Signed-off-by: zhangtao <[email protected]>
Signed-off-by: zhangtao2 <[email protected]>
Co-authored-by: zhangtao <[email protected]>
Co-authored-by: Chauncey <[email protected]>

* [ROCm][GPT-OSS] Avoid repeated compile-time `cos_sin_cache.to(bf16)` casts in rotary path (#42833)

Signed-off-by: Aakif Nawaz <[email protected]>

* [Doc] Add Ascend NPU tab to the quickstart installation guide (#43550)

Signed-off-by: Aditya Singh <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* [Rust Frontend] Align tool parser fallback behavior between streaming & non-streaming paths (#43662)

Signed-off-by: Bugen Zhao <[email protected]>

* [Docs] Fix MLA prefill backend default docs (#43697)

Signed-off-by: Mohammad Miadh Angkad <[email protected]>

* [Kernel] Enable TritonW4A16LinearKernel as CUDA fallback for non-Marlin-aligned W4A16 shapes (#43731)

Signed-off-by: Luciano Martins <[email protected]>
Co-authored-by: Luciano Martins <[email protected]>

* [Bugfix] Map reasoning_effort to enable_thinking in chat template kwargs (#43401)

Signed-off-by: Ashwin Giridharan <[email protected]>
Signed-off-by: Chauncey <[email protected]>
Co-authored-by: Chauncey <[email protected]>

* [misc] Bump cutedsl version to 4.5.2 (#43745)

Signed-off-by: Yongye Zhu <[email protected]>

* [BugFix] HFValidationError with cloud storage URIs when HF_HUB_OFFLINE=1 (#39155)

Signed-off-by: Injae Ryou <[email protected]>

* [Docs] Fix the duplicate doc icon issue (#43546)

Signed-off-by: chunyang.wen <[email protected]>

* Fix early CUDA init (#43791)

Signed-off-by: Harry Mellor <[email protected]>

* [ROCm] mori: add InterNodeV1LL inter-node kernel selection via VLLM_MORI_INTERNODE_KERNEL (#41751)

Signed-off-by: jatseng-ai <[email protected]>

* [8/n] Migrate merge_attn_states, mamba, sampler to torch stable ABI (continued) (#43361)

Signed-off-by: Mikayla Gawarecki <[email protected]>
Signed-off-by: Chris Leonard <[email protected]>
Co-authored-by: Mikayla Gawarecki <[email protected]>
Co-authored-by: Shengqi Chen <[email protected]>

* [Quantization] Fix Humming RoutedExperts import (#43540)

Signed-off-by: Minh Vu <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>

* [CI] build-rocm-wheels.yml: reduce MAX_JOBS to prevent OOM

Signed-off-by:  <[email protected]>

---------

Signed-off-by: Kunshang Ji <[email protected]>
Signed-off-by: Mohammad Miadh Angkad <[email protected]>
Signed-off-by: NickLucche <[email protected]>
Signed-off-by: Nick Hill <[email protected]>
Signed-off-by: Woosuk Kwon <[email protected]>
Signed-off-by: george <[email protected]>
Signed-off-by: Qiuyang Yue <[email protected]>
Signed-off-by: junyanxu <[email protected]>
Signed-off-by: Gracie Guo <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: Chaojun,Zhang <[email protected]>
Signed-off-by: Taneem Ibrahim <[email protected]>
Signed-off-by: wang.yuqi <[email protected]>
Signed-off-by: hao-aaron <[email protected]>
Signed-off-by: Yifan Qiao <[email protected]>
Signed-off-by: shen-shanshan <[email protected]>
Signed-off-by: ZhanqiuHu <[email protected]>
Signed-off-by: Sage Ahrac <[email protected]>
Signed-off-by: Xinyu Chen <[email protected]>
Signed-off-by: Wang Yiwen <[email protected]>
Signed-off-by: sfeng33 <[email protected]>
Signed-off-by: Jinzhen Lin <[email protected]>
Signed-off-by: Dom Brown <[email protected]>
Signed-off-by: Dao Le <[email protected]>
Signed-off-by: Tomer Asida <[email protected]>
Signed-off-by: wzhao18 <[email protected]>
Signed-off-by: Wei Zhao <[email protected]>
Signed-off-by: Kevin H. Luu <[email protected]>
Signed-off-by: Doğaç Eldenk <[email protected]>
Signed-off-by: Max de Bayser <[email protected]>
Signed-off-by: Max de Bayser <[email protected]>
Signed-off-by: Fadi Arafeh <[email protected]>
Signed-off-by: ahao-anyscale <[email protected]>
Signed-off-by: Terrencezzj <[email protected]>
Signed-off-by: Benjamin Chislett <[email protected]>
Signed-off-by: Philip Maybank <[email protected]>
Signed-off-by: pmaybank <[email protected]>
Signed-off-by: mgoin <[email protected]>
Signed-off-by: Ronen Schaffer <[email protected]>
Signed-off-by: Mikayla Gawarecki <[email protected]>
Signed-off-by: Chris Leonard <[email protected]>
Signed-off-by: Harry Mellor <[email protected]>
Signed-off-by: Artem Perevedentsev <[email protected]>
Signed-off-by: zengxian <[email protected]>
Signed-off-by: hallerite <[email protected]>
Signed-off-by: Rui Wang <[email protected]>
Signed-off-by: Kebe <[email protected]>
Signed-off-by: Dipika Sikka <[email protected]>
Signed-off-by: rishitdholakia13 <[email protected]>
Signed-off-by: j9smith <[email protected]>
Signed-off-by: Joel Smith <[email protected]>
Signed-off-by: Vadim Gimpelson <[email protected]>
Signed-off-by: haosdent <[email protected]>
Signed-off-by: Meenakshi Venkataraman <[email protected]>
Signed-off-by: yewentao256 <[email protected]>
Signed-off-by: Wentao Ye <[email protected]>
Signed-off-by: Douglas Lehr <[email protected]>
Signed-off-by: zjy0516 <[email protected]>
Signed-off-by: <>
Signed-off-by: Ace Eldeib <[email protected]>
Signed-off-by: louie-tsai <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Adi McM Sonus Flow <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: Luciano Martins <[email protected]>
Signed-off-by: Ben Browning <[email protected]>
Signed-off-by: AAISSJ <[email protected]>
Signed-off-by: sejung-son <[email protected]>
Signed-off-by: Bugen Zhao <[email protected]>
Signed-off-by: Daoyuan Li <[email protected]>
Signed-off-by: Yifan Zong <[email protected]>
Signed-off-by: Yiyang Liu <[email protected]>
Signed-off-by: Chauncey <[email protected]>
Signed-off-by: zexplorerhj <[email protected]>
Signed-off-by: zhangxin81 <[email protected]>
Signed-off-by: Isotr0py <[email protected]>
Signed-off-by: velonica0 <[email protected]>
Signed-off-by: velonica0 <[email protected]>
Signed-off-by: Francesco Fusco <[email protected]>
Signed-off-by: anish <[email protected]>
Signed-off-by: Your Name <[email protected]>
Signed-off-by: anish <[email protected]>
Signed-off-by: Michael Goin <[email protected]>
Signed-off-by: Zheng Luo <[email protected]>
Signed-off-by: Ylang Tsou <[email protected]>
Signed-off-by: fangyuchu <[email protected]>
Signed-off-by: zWaNg3 <[email protected]>
Signed-off-by: Lanze Liu <[email protected]>
Signed-off-by: Chengze Fan <[email protected]>
Signed-off-by: Chengze Fan <[email protected]>
Signed-off-by: simondanielsson <[email protected]>
Signed-off-by: Divakar Verma <[email protected]>
Signed-off-by: Rui Zhang <[email protected]>
Signed-off-by: Rui Zhang <[email protected]>
Signed-off-by: Robert Shaw <[email protected]>
Signed-off-by: Robert Shaw <[email protected]>
Signed-off-by: zixi-qi <[email protected]>
Signed-off-by: Xiaochang Wu <[email protected]>
Signed-off-by: Wu, Xiaochang <[email protected]>
Signed-off-by: Furkan Fidan <[email protected]>
Signed-off-by: tc-mb <[email protected]>
Signed-off-by: Weida Hong <[email protected]>
Signed-off-by: mrjunwan-lang <[email protected]>
Signed-off-by: jiang1.li <[email protected]>
Signed-off-by: wangxiyuan <[email protected]>
Signed-off-by: Ma Jian <[email protected]>
Signed-off-by: Tobias Wasner <[email protected]>
Signed-off-by: Keyi Li <[email protected]>
Signed-off-by: Markov Ilya <[email protected]>
Signed-off-by: Shiyang Chen <[email protected]>
Signed-off-by: zhanda <[email protected]>
Signed-off-by: Russell Bryant <[email protected]>
Signed-off-by: SandishKumarHN <[email protected]>
Signed-off-by: Juhi Mittal <[email protected]>
Signed-off-by: Benjamin Chislett <[email protected]>
Signed-off-by: Itay Alroy <[email protected]>
Signed-off-by: SumanthRH <[email protected]>
Signed-off-by: Yongye Zhu <[email protected]>
Signed-off-by: Andreas Karatzas <[email protected]>
Signed-off-by: Ashwin Giridharan <[email protected]>
Signed-off-by: mayuyuace <[email protected]>
Signed-off-by: xiaogang.zhou <[email protected]>
Signed-off-by: weimingc <[email protected]>
Signed-off-by: Zihua Wu <[email protected]>
Signed-off-by: FERRARIZHENG <[email protected]>
Signed-off-by: Jakub Zakrzewski <[email protected]>
Signed-off-by: ffggs <[email protected]>
Signed-off-by: Hua Huang <[email protected]>
Signed-off-by: holegots <[email protected]>
Signed-off-by: Siddharth Bedekar <[email protected]>
Signed-off-by: tjtanaa <[email protected]>
Signed-off-by: Dao Le <[email protected]>
Signed-off-by: Or Ozeri <[email protected]>
Signed-off-by: Banani Ghosh <[email protected]>
Signed-off-by: Daniel Serebrenik <[email protected]>
Signed-off-by: Rotem Shavitt <[email protected]>
Signed-off-by: [email protected] <[email protected]>
Signed-off-by: TheDuyIT <[email protected]>
Signed-off-by: dtnguyen <[email protected]>
Signed-off-by: esmeetu <[email protected]>
Signed-off-by: Yihuki <[email protected]>
Signed-off-by: Zhewen Li <[email protected]>
Signed-off-by: chaojun-zhang <[email protected]>
Signed-off-by: Chaojun Zhang <[email protected]>
Signed-off-by: Jee Jee Li <[email protected]>
Signed-off-by: Yan Ma <[email protected]>
Signed-off-by: QingZhou-YangHY <[email protected]>
Signed-off-by: Thien Tran <[email protected]>
Signed-off-by: zhejiangxiaomai <[email protected]>
Signed-off-by: Hank <[email protected]>
Signed-off-by: Yubo Wang <[email protected]>
Signed-off-by: Ethan Feng <[email protected]>
Signed-off-by: ThibaultCastells <[email protected]>
Signed-off-by: linzm1007 <[email protected]>
Signed-off-by: Jesus De Jesus <[email protected]>
Signed-off-by: javierdejesusda <[email protected]>
Signed-off-by: Simon Danielsson <[email protected]>
Signed-off-by: Bill Nell <[email protected]>
Signed-off-by: Mark McLoughlin <[email protected]>
Signed-off-by: Kevin Luu <[email protected]>
Signed-off-by: Zhewen Li <[email protected]>
Signed-off-by: Vadim Gimpelson <[email protected]>
Signed-off-by: khluu <[email protected]>
Signed-off-by: Xin Yang <[email protected]>
Signed-off-by: Lucas Wilkinson <[email protected]>
Signed-off-by: Matthew Bonanni <[email protected]>
Signed-off-by: Angela Yi <[email protected]>
Signed-off-by: aoshen524 <[email protected]>
Signed-off-by: Ao Shen <[email protected]>
Signed-off-by: nholmber <[email protected]>
Signed-off-by: zhangtao <[email protected]>
Signed-off-by: zhangtao2 <[email protected]>
Signed-off-by: Aakif Nawaz <[email protected]>
Signed-off-by: Aditya Singh <[email protected]>
Signed-off-by: Injae Ryou <[email protected]>
Signed-off-by: chunyang.wen <[email protected]>
Signed-off-by: jatseng-ai <[email protected]>
Signed-off-by: Minh Vu <[email protected]>
Signed-off-by: <[email protected]>
Co-authored-by: Kunshang Ji <[email protected]>
Co-authored-by: Mohammad Miadh Angkad <[email protected]>
Co-authored-by: Nicolò Lucchesi <[email protected]>
Co-authored-by: Nick Hill <[email protected]>
Co-authored-by: Woosuk Kwon <[email protected]>
Co-authored-by: gnovack <[email protected]>
Co-authored-by: george <[email protected]>
Co-authored-by: Qiuyang Yue <[email protected]>
Co-authored-by: Claude Opus 4.7 <[email protected]>
Co-authored-by: gemini-code-assist <[email protected]>
Co-authored-by: Kevin H. Luu <[email protected]>
Co-authored-by: Junyan Xu <[email protected]>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Gracie Guo (UX) <[email protected]>
Co-authored-by: Gracie Guo <[email protected]>
Co-authored-by: wang.yuqi <[email protected]>
Co-authored-by: Chaojun Zhang <[email protected]>
Co-authored-by: Taneem Ibrahim <[email protected]>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <[email protected]>
Co-authored-by: Harry Mellor <[email protected]>
Co-authored-by: Aaron Hao <[email protected]>
Co-authored-by: Yifan Qiao <[email protected]>
Co-authored-by: Shanshan Shen <[email protected]>
Co-authored-by: zhanqiuhu <[email protected]>
Co-authored-by: Sage <[email protected]>
Co-authored-by: Xinyu Chen <[email protected]>
Co-authored-by: Wang Yiwen <[email protected]>
Co-authored-by: Flora Feng <[email protected]>
Co-authored-by: Jinzhen Lin <[email protected]>
Co-authored-by: Dom Brown <[email protected]>
Co-authored-by: Dao007forever <[email protected]>
Co-authored-by: tomeras91 <[email protected]>
Co-authored-by: Wei Zhao <[email protected]>
Co-authored-by: Doğaç Eldenk <[email protected]>
Co-authored-by: Max de Bayser <[email protected]>
Co-authored-by: Fadi Arafeh <[email protected]>
Co-authored-by: Terrence Zhao <[email protected]>
Co-authored-by: Benjamin Chislett <[email protected]>
Co-authored-by: pmaybank <[email protected]>
Co-authored-by: Izik Golan <[email protected]>
Co-authored-by: Michael Goin <[email protected]>
Co-authored-by: Ronen Schaffer <[email protected]>
Co-authored-by: Chris Leonard <[email protected]>
Co-authored-by: Mikayla Gawarecki <[email protected]>
Co-authored-by: Shengqi Chen <[email protected]>
Co-authored-by: Artem Perevedentsev <[email protected]>
Co-authored-by: xiangdong <[email protected]>
Co-authored-by: hallerite <[email protected]>
Co-authored-by: Ray Wang <[email protected]>
Co-authored-by: Rui Wang <[email protected]>
Co-authored-by: Kebe <[email protected]>
Co-authored-by: Dipika Sikka <[email protected]>
Co-authored-by: rishitdholakia13 <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Joel Smith <[email protected]>
Co-authored-by: Vadim Gimpelson <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Co-authored-by: haosdent <[email protected]>
Co-authored-by: meena-at-work <[email protected]>
Co-authored-by: Wentao Ye <[email protected]>
Co-authored-by: Matthew Bonanni <[email protected]>
Co-authored-by: Douglas Lehr <[email protected]>
Co-authored-by: Jiangyun Zhu <[email protected]>
Co-authored-by: akii96 <[email protected]>
Co-authored-by: Ace Eldeib <[email protected]>
Co-authored-by: Louie Tsai <[email protected]>
Co-authored-by: Jee Jee Li <[email protected]>
Co-authored-by: sonusflow <[email protected]>
Co-authored-by: Luciano Martins <[email protected]>
Co-authored-by: Luciano Martins <[email protected]>
Co-authored-by: Ben Browning <[email protected]>
Co-authored-by: 손세정 <[email protected]>
Co-authored-by: 세덩 <[email protected]>
Co-authored-by: sejung-son <[email protected]>
Co-authored-by: Bugen Zhao <[email protected]>
Co-authored-by: Daoyuan Li <[email protected]>
Co-authored-by: yzong-rh <[email protected]>
Co-authored-by: Yiyang "Ian" Liu <[email protected]>
Co-authored-by: Copilot <[email protected]>
Co-authored-by: Chauncey <[email protected]>
Co-authored-by: zexplorerhj <[email protected]>
Co-authored-by: zexplorerhj <[email protected]>
Co-authored-by: zhangxin81 <[email protected]>
Co-authored-by: Isotr0py <[email protected]>
Co-authored-by: velonica0 <[email protected]>
Co-authored-by: Li, Jiang <[email protected]>
Co-authored-by: Francesco Fusco <[email protected]>
Co-authored-by: anish <[email protected]>
Co-authored-by: anish <[email protected]>
Co-authored-by: Zheng Luo <[email protected]>
Co-authored-by: OpenAI Codex <[email protected]>
Co-authored-by: Robert Shaw <[email protected]>
Co-authored-by: ylangtsou <[email protected]>
Co-authored-by: Ylang Tsou <[email protected]>
Co-authored-by: fangyuchu <[email protected]>
Co-authored-by: zWaNg3 <[email protected]>
Co-authored-by: Lanze Liu <[email protected]>
Co-authored-by: Chengze Fan <[email protected]>
Co-authored-by: Lu Fang <[email protected]>
Co-authored-by: Simon Danielsson <[email protected]>
Co-authored-by: Divakar Verma <[email protected]>
Co-authored-by: ruizhang <[email protected]>
Co-authored-by: Rui Zhang <[email protected]>
Co-authored-by: robertgshaw2-redhat <[email protected]>
Co-authored-by: qizixi <[email protected]>
Co-authored-by: Xiaochang Wu <[email protected]>
Co-authored-by: Furkan F <[email protected]>
Co-authored-by: tc-mb <[email protected]>
Co-authored-by: Weida Hong <[email protected]>
Co-authored-by: mrjunwan-lang <[email protected]>
Co-authored-by: wangxiyuan <[email protected]>
Co-authored-by: Ma Jian <[email protected]>
Co-authored-by: Tobias Wasner <[email protected]>
Co-authored-by: Keyi Li <[email protected]>
Co-authored-by: Keyi Li <[email protected]>
Co-authored-by: Ilya Markov <[email protected]>
Co-authored-by: Markov Ilya <[email protected]>
Co-authored-by: sychen52 <[email protected]>
Co-authored-by: Zhanda Zhu <[email protected]>
Co-authored-by: Shang Wang <[email protected]>
Co-authored-by: Yongye Zhu <[email protected]>
Co-authored-by: Russell Bryant <[email protected]>
Co-authored-by: SandishKumarHN <[email protected]>
Co-authored-by: SandishKumarHN <[email protected]>
Co-authored-by: Juhi Mittal <[email protected]>
Co-authored-by: Itay Alroy <[email protected]>
Co-authored-by: Duncan Moss <[email protected]>
Co-authored-by: Sumanth R Hegde <[email protected]>
Co-authored-by: Andreas Karatzas <[email protected]>
Co-authored-by: Ashwin Giridharan <[email protected]>
Co-authored-by: abinggo <[email protected]>
Co-authored-by: Qiming Zhang <[email protected]>
Co-authored-by: Xiaogang Zhou <[email protected]>
Co-authored-by: xiaogang.zhou <[email protected]>
Co-authored-by: Wei-Ming Chen <[email protected]>
Co-authored-by: Gabriel Wu <[email protected]>
Co-authored-by: GuangYaoZheng <[email protected]>
Co-authored-by: Jakub Zakrzewski <[email protected]>
Co-authored-by: ffggs <[email protected]>
Co-authored-by: Cyrus Leung <[email protected]>
Co-authored-by: Hua Huang <[email protected]>
Co-authored-by: Holegots <[email protected]>
Co-authored-by: Siddharth Bedekar <[email protected]>
Co-authored-by: TJian <[email protected]>
Co-authored-by: Or Ozeri <[email protected]>
Co-authored-by: danisereb <[email protected]>
Co-authored-by: Banani Ghosh <[email protected]>
Co-authored-by: Rotem Shavitt <[email protected]>
Co-authored-by: weizhoublue <[email protected]>
Co-authored-by: Nguyễn Thế Duy <[email protected]>
Co-authored-by: Roy Wang <[email protected]>
Co-authored-by: Yihuki <[email protected]>
Co-authored-by: Zhewen Li <[email protected]>
Co-authored-by: Zhewen Li <[email protected]>
Co-authored-by: Yan Ma <[email protected]>
Co-authored-by: Huanyu Yang <[email protected]>
Co-authored-by: Thien Tran <[email protected]>
Co-authored-by: zhao, zhenhui <[email protected]>
Co-authored-by: Sting Lin <[email protected]>
Co-authored-by: Jie Fang <[email protected]>
Co-authored-by: Hank_ <[email protected]>
Co-authored-by: Yubo Wang <[email protected]>
Co-authored-by: Ethan Feng <[email protected]>
Co-authored-by: Thibault Castells <[email protected]>
Co-authored-by: linzm1007 <[email protected]>
Co-authored-by: Javier De Jesus <[email protected]>
Co-authored-by: bnellnm <[email protected]>
Co-authored-by: Mark McLoughlin <[email protected]>
Co-authored-by: Zhewen Li <[email protected]>
Co-authored-by: Xin Yang <[email protected]>
Co-authored-by: Lucas Wilkinson <[email protected]>
Co-authored-by: Angela Yi <[email protected]>
Co-authored-by: aoshen02 <[email protected]>
Co-authored-by: aoshen524 <[email protected]>
Co-authored-by: Nico Holmberg <[email protected]>
Co-authored-by: zhangtao2-1 <[email protected]>
Co-authored-by: zhangtao <[email protected]>
Co-authored-by: Aditya Singh <[email protected]>
Co-authored-by: Injae Ryou <[email protected]>
Co-authored-by: Chunyang Wen <[email protected]>
Co-authored-by: jatseng-ai <[email protected]>
Co-authored-by: Minh Vu <[email protected]>
mxinO added a commit to NVIDIA/Model-Optimizer that referenced this pull request Jun 4, 2026
@mxinO
DSV4 PTQ example with dequant on the fly (#1341)
ca7eb64 
### What does this PR do?

Type of change: new example <!-- Use one of the following: Bug fix, new
feature, new example, new tests, documentation. -->

Add deepseek v4 official modeling ptq example

### Usage

See readme, and it requires the vllm PR:
vllm-project/vllm#42209

### Testing
Tested with ptq and export of dsv4 flash and served with vllm.

### Before your PR is "*Ready for review*"

Make sure you read and follow [Contributor
guidelines](https://github.com/NVIDIA/Model-Optimizer/blob/main/CONTRIBUTING.md)
and your commits are signed (`git commit -s -S`).

Make sure you read and follow the [Security Best
Practices](https://github.com/NVIDIA/Model-Optimizer/blob/main/SECURITY.md#security-coding-practices-for-contributors)
(e.g. avoiding hardcoded `trust_remote_code=True`, `torch.load(...,
weights_only=False)`, `pickle`, etc.).

- Is this change backward compatible?: ✅ / ❌ / N/A <!--- If ❌, explain
why. -->
- If you copied code from any other sources or added a new PIP
dependency, did you follow guidance in `CONTRIBUTING.md`: ✅ / ❌ / N/A
<!--- Mandatory -->
- Did you write any new necessary tests?: ✅ / ❌ / N/A <!--- Mandatory
for new features or examples. -->
- Did you update
[Changelog](https://github.com/NVIDIA/Model-Optimizer/blob/main/CHANGELOG.rst)?:
✅ / ❌ / N/A <!--- Only for new features, API changes, critical bug fixes
or backward incompatible changes. -->

### Additional Information
<!-- E.g. related issue. -->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added DeepSeek‑V4 routed‑expert post‑training quantization and an
NVFP4 checkpoint conversion utility.

* **Documentation**
* Expanded DeepSeek quantization guide with directory layout, updated
V3/V3.2 workflows, and detailed V4 routed‑expert calibration,
single/multi‑node examples, and export guidance.

* **Chores**
  * Made example quantization scripts location‑independent.
* Updated pre‑commit license hook to skip DeepSeek example quantization
files.

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/NVIDIA/Model-Optimizer/pull/1341?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Meng Xin <[email protected]>
@danielwoz danielwoz mentioned this pull request Jun 30, 2026
Closed
philippesic pushed a commit to philippesic/vllm-semantic-cache that referenced this pull request Jul 19, 2026
@sychen52
Add NVFP4 MOE support for Deepseek V4. (vllm-project#42209)
4ab4b46 
Signed-off-by: Shiyang Chen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@claude claude[bot] claude[bot] left review comments
@pavanimajety pavanimajety pavanimajety approved these changes
@zyongye zyongye zyongye approved these changes
@mgoin mgoin Awaiting requested review from mgoin mgoin is a code owner
@tlrmchlsmth tlrmchlsmth Awaiting requested review from tlrmchlsmth tlrmchlsmth is a code owner
@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon
@yewentao256 yewentao256 Awaiting requested review from yewentao256 yewentao256 is a code owner
@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner
@xinli-sw xinli-sw Awaiting requested review from xinli-sw
@Harry-Chen Harry-Chen Awaiting requested review from Harry-Chen Harry-Chen is a code owner
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ci/build deepseek Related to DeepSeek models nvidia ready ONLY add when PR is ready to merge/full CI is needed

Projects

NVIDIA
Status: Done

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sychen52 @xinli-sw @pavanimajety @zyongye