Skip to content

Commit 4c5d4fd

Browse files
ArthurZuckervasqu
authored andcommitted
Add minimax m3vl (#46600)
* Add MiniMax M3 VL: draft scaffold with modular, tiny fixtures, MXFP8 dequant - Modular implementation reusing minimax_m2 for the M2-shaped pieces (router, decoder layer skeleton, expert layout) and adding M3 deltas: shared experts + sigmoid routing scaling, dense-MLP-for-first-N-layers, swigluoai with alpha/limit clamp, partial RoPE (rotary_dim < head_dim), per-head Gemma-style QK norm, and the sparse-attention lightning-index branch (full dense math fallback for v1; top-k block selection deferred). - CLIP-like vision tower with Conv3d patch embed and a 3D RoPE over (T, H, W) patch grids, plus LLaVA-style projector + spatial patch merger. - Fast image / video processors and a multimodal processor ported from the snapshot's preprocessing code; auto registrations for config / model / processor / image / video. - Tiny model fixtures: build_tiny_model.py (transformers layout) and build_tiny_model_sglang.py (re-keyed for sglang FusedMoE), so parity checks across backends do not require the 100B-param real model. - dequantize_mxfp8.py converts the MXFP8 ([1, 32]-blocked E4M3) checkpoint shipped by MiniMax to bf16 -- transformers has MXFP4 today but no MXFP8, so the first port dequantizes up front. Status: text + vision forward passes work end-to-end on the tiny model; both AutoModelForImageTextToText and AutoModelForCausalLM are wired. Sparse attention is dense-math only and MTP modules are skipped. * Strip dead branches + port lightning indexer from deepseek_v4 * Drop config fields that the real M3 VL checkpoint has only a single value for and inline the constants: - text: use_gemma_norm, use_qk_norm, qk_norm_type, attention_output_gate, hidden_act, scoring_func, partial_rotary_factor - composite: multimodal_projector_bias, projector_hidden_act, vision_feature_layer, vision_feature_select_strategy, process_image_mode, img_token_compression_config, image_grid_pinpoints, image_seq_length - vision: rope_mode, vision_segment_max_frames, hidden_act * Norms: single Gemma-style RMSNorm class (drop the `gemma:` switch and the plain RMSNorm alias). Activation in the dense MLP / experts / projector is always swigluoai / GELU; remove the alternate branches. * Attention: per-head Gemma QK norm and partial RoPE are always on; remove the per_layer / no-norm / full-RoPE branches. * Replace the dense fallback of the sparse-attention layer with a proper lightning indexer modeled on `DeepseekV4Indexer`: - MiniMaxM3VLIndexer scores key blocks via idx_q · idx_k, max-reduces over `sparse_block_size`, takes top-k blocks per query, then forces the first `sparse_init_block` and last `sparse_local_block` blocks visible. - Encodes the selection as an additive `[B, 1, S, S]` block_bias (`-inf` outside the allowed blocks), summed onto attention_mask before SDPA -- same scatter pattern as deepseek_v4. - Index branch still runs its own SDPA over (idx_q, idx_k, idx_v) and adds the result through `o_proj`, matching sglang's parallel index output path. `disable_index_value` layers skip the value side. Tiny VLM / text-only LM forward passes still work end-to-end via AutoModelForImageTextToText / AutoModelForCausalLM. * KV-cache the sparse index branch + add minimax.py generation demo Cache: * Add MiniMaxM3VLSparseCacheLayer(DynamicLayer) — auto-registers under layer_type "minimax_m3_sparse". Stores idx_keys / idx_values alongside the main K/V so the lightning indexer can keep scoring against the full prefix during decode without recomputing it. Mirrors the DeepseekV4CSACache pattern (per-layer cache instance reached via past_key_values.layers[layer_idx]; custom update_index method analogous to DSv4's store_compression_weights / update_compressor_states). * Register the new value in `ALLOWED_LAYER_TYPES` so the strict-dataclass validator accepts mixed ["full_attention", ..., "minimax_m3_sparse"] layer_types on the text config. * Derive layer_types from sparse_attention_freq inside MiniMaxM3VLTextModel.__init__ so old checkpoints without a layer_types field still dispatch the right cache class. Indexer: * MiniMaxM3VLIndexer.forward now takes position_ids + past_key_values + layer_idx, looks up the per-layer cache, calls update_index with the new token's (idx_k, idx_v), and scores idx_q against the full cached idx_keys (not just the new token's idx_k). Block_bias and the index branch's own SDPA both use absolute query positions so causality works for Sq < Sk during decode. Demo: * minimax.py runs the image+text → AutoProcessor → .generate() round-trip on the tiny model. Tiny model config now uses the real 200064 vocab and the snapshot image_token_index (200025), so the snapshot tokenizer + chat_template drop in unchanged. * Drop the no-longer-needed `image_processor_class`/`video_processor_class` attrs on MiniMaxM3VLProcessor (the AutoImageProcessor / AutoVideoProcessor registrations cover the lookup and the explicit strings emit a deprecation warning). Verified: * Cached prefill + decode argmax matches the no-cache full forward. * model.generate() produces tokens end-to-end. * minimax.py runs on the tiny model with the snapshot tokenizer. * updates * updates * Wire video into M3 VL forward + native MXFP8 path - add get_video_features / video placeholder mask and consume pixel_values_videos + video_grid_thw in the model forward, scattering video features into the video-token slots - support the native (non-dequantized) MXFP8 compute path: keep experts in float8_e4m3fn and carry the SwiGLU-OAI gate onto the FP8 experts - add model-level video tests (forward, token mismatch) and a slow end-to-end video generation check; add docs page + toctree entry - realign deepseek_v4 cache_layer annotation in the modular source so the generated file matches * Stop tracking local dev scripts (minimax.py demo, build_tiny_model.py) * update * cleanup * up * up * update * updates * nits * update * nits * more cleanup * add visuals * more cleanups * up * nits * nits * man made * UP * up * fix * revert some modular stuff * fix bug, rename * update * update with kernel on the hub * comments * small update * start adressing comments * ? * nits * update * nits * update * latest changes * up * fix padding * nit to define pos ids once * just a nit * update * nits * date * skip 1 test * nits * extra test * delete pos ids "fix" * last nit * fix * fix * fix
1 parent 5957e6f commit 4c5d4fd

34 files changed

Lines changed: 5117 additions & 21 deletions

‎docs/source/en/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -741,6 +741,8 @@
741741
title: MiniMax
742742
- local: model_doc/minimax_m2
743743
title: MiniMax-M2
744+
- local: model_doc/minimax_m3_vl
745+
title: MiniMax-M3-VL
744746
- local: model_doc/ministral
745747
title: Ministral
746748
- local: model_doc/ministral3
Lines changed: 253 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,253 @@
1+
<!--Copyright 2026 the HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.
14+
15+
16+
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer.
17+
18+
-->
19+
*This model was contributed to Hugging Face Transformers on 2026-06-12.*
20+
21+
22+
# MiniMax-M3-VL
23+
24+
## Overview
25+
26+
MiniMax-M3-VL is the vision-language member of the MiniMax-M3 family. It pairs a CLIP-style vision tower (Conv3d patch embedding with 3D rotary position embeddings) with the MiniMax-M3 text backbone, a mixed dense/sparse Mixture-of-Experts decoder that uses SwiGLU-OAI gated experts and a lightning indexer for block-sparse attention.
27+
28+
## Architecture
29+
### Block-sparse attention (Lightning Indexer)
30+
31+
Every layer is GQA (`num_key_value_heads = 4`) with per-head QK-norm and **partial RoPE** on the first
32+
`rotary_dim`. `config.layer_types[i]` then picks `"full_attention"` (dense causal) or
33+
`"minimax_m3_sparse"`, where a [`MiniMaxM3VLIndexer`] decides, per query, which block of keys the main attention may see.
34+
35+
The indexer scores every key, then **max-poolsthose per-key scores into blocks of `index_block_size` keys**, so selection happens at the granularity of a *block
36+
of keys*: per query it keeps the top-`index_topk_blocks` key blocks plus the always-on `index_local_blocks`
37+
local-window block (under block-level causality), broadcasts the per-block `0`/`-inf` choice back onto every key in
38+
the block. The result is a `[B, 1, S_q, S_k]` additive bias summed onto the causal mask.
39+
Theoretically this means that the attention is only computed over the selected blocks of keys, but `transformers` does not support the kernels that compute this efficiently!
40+
We are adding it to `kernels` asap!
41+
42+
<img alt="MiniMax M3 Lightning Indexer mask" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/minimax_m3_vl_indexer_mask.svg" />
43+
44+
45+
### Vision tower
46+
47+
A [`MiniMaxM3VLVisionModel`]: a `Conv3d` patch embedding over flattened `[N_patches, C·T·P·P]` input, a stack of
48+
CLIP-style encoder layers carrying a **3D rotary** position embedding (time / height / width bands). A [`MiniMaxM3VLPatchMerger`] groups
49+
`spatial_merge_size²` patches into the channel dim before the 2-layer GELU [`MiniMaxM3VLMultiModalProjector`] maps vision features into the text hidden size.
50+
51+
## Usage examples
52+
53+
The example below runs the model on a real image loaded with [`~transformers.image_utils.load_image`].
54+
55+
```python
56+
import torch
57+
from transformers import AutoModelForImageTextToText, AutoProcessor
58+
from transformers.image_utils import load_image
59+
60+
61+
model = AutoModelForImageTextToText.from_pretrained(
62+
"MiniMaxAI/MiniMax-M3-preview", dtype=torch.bfloat16, device_map="auto",
63+
)
64+
processor = AutoProcessor.from_pretrained("MiniMaxAI/MiniMax-M3-preview")
65+
66+
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg")
67+
messages = [
68+
{
69+
"role": "user",
70+
"content": [
71+
{"type": "image"},
72+
{"type": "text", "text": "Describe this image briefly."},
73+
],
74+
}
75+
]
76+
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
77+
inputs = processor(images=[image], text=text, return_tensors="pt").to(model.device)
78+
79+
generated_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
80+
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
81+
```
82+
83+
### Apple example
84+
85+
This example asks the model about an image of apples, again loading a real image with
86+
[`~transformers.image_utils.load_image`].
87+
88+
```python
89+
import torch
90+
from transformers import AutoModelForImageTextToText, AutoProcessor
91+
from transformers.image_utils import load_image
92+
93+
94+
model = AutoModelForImageTextToText.from_pretrained(
95+
"MiniMaxAI/MiniMax-M3-preview", dtype=torch.bfloat16, device_map="auto",
96+
)
97+
processor = AutoProcessor.from_pretrained("MiniMaxAI/MiniMax-M3-preview")
98+
99+
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg")
100+
messages = [
101+
{
102+
"role": "user",
103+
"content": [
104+
{"type": "image"},
105+
{"type": "text", "text": "How many apples are in this image, and what color are they?"},
106+
],
107+
}
108+
]
109+
text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
110+
inputs = processor(images=[image], text=text, return_tensors="pt").to(model.device)
111+
112+
generated_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False)
113+
print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0])
114+
```
115+
116+
## Fastest inference configuration
117+
118+
| ctx | SDPA decode | MSA decode | MSA decode adv. | SDPA prefill | MSA prefill | MSA prefill adv. |
119+
| --: | ----------: | ---------: | --------------: | -----------: | ----------: | ---------------: |
120+
| 2K | 27.8 tok/s | 31.0 | +12% | 303 ms | 257 ms | 1.18× |
121+
| 4K | 23.4 tok/s | 30.5 | +30% | 684 ms | 460 ms | 1.49× |
122+
| 8K | 17.8 tok/s | 29.6 | +66% | 1906 ms | 976 ms | 1.95× |
123+
| 16K | 12.0 tok/s | 27.6 | +130% | 6110 ms | 2344 ms | 2.61× |
124+
125+
The checkpoint ships in native MXFP8. For **decode throughput**, the fastest validated configuration is
126+
**bf16 (dequantized at load) + the MSA block-sparse attention kernel + tensor & expert parallelism + a
127+
`reduce-overhead` cudagraph compile** — roughly **31 tok/s** decode on 8×B200 at a 2048-token prefill.
128+
129+
Keeping the weights in **native FP8 is a memory-footprint option only — it is never faster on this setup**.
130+
The FP8 Triton experts/linear kernels lower as opaque inductor fallback kernels that cudagraph cannot
131+
capture on the hot expert path, so native-FP8 decode measured ~4.2 tok/s (≈7× slower than the bf16 path)
132+
even under `torch.compile(fullgraph=True)`. Use FP8 only when the bf16 weights do not fit.
133+
134+
| config (sdpa baseline, TP+EP, 2048-token prefill, 8×B200) | decode |
135+
|---|---|
136+
| bf16 dequantize-at-load + **MSA** + compile/cudagraph | **~31 tok/s** |
137+
| bf16 dequantize-at-load + sdpa + compile/cudagraph | ~28 tok/s |
138+
| native FP8 + compile/cudagraph | ~4 tok/s (memory-only, not for speed) |
139+
140+
Dequantizing to bf16 only fits with even sharding across GPUs (TP/EP), not with `device_map="auto"`
141+
(pipeline placement OOMs at load). Launch one process per GPU with `torchrun`:
142+
143+
```bash
144+
torchrun --nproc_per_node=8 fastest_m3_vl.py
145+
```
146+
147+
```python
148+
# fastest_m3_vl.py
149+
import os, sys
150+
import torch
151+
import torch.distributed as dist
152+
from transformers import (
153+
AutoModelForImageTextToText,
154+
AutoTokenizer,
155+
CompileConfig,
156+
FineGrainedFP8Config,
157+
)
158+
from transformers.distributed import DistributedConfig
159+
160+
# The indexer feeds SDPA an additive float mask; the cuDNN SDP backend segfaults on it (B200).
161+
torch.backends.cuda.enable_cudnn_sdp(False)
162+
163+
model = AutoModelForImageTextToText.from_pretrained(
164+
"MiniMaxAI/MiniMax-M3-preview",
165+
dtype=torch.bfloat16,
166+
# Dequantize the native MXFP8 weights to bf16 at load (the speed win); needs even TP/EP sharding.
167+
quantization_config=FineGrainedFP8Config(dequantize=True),
168+
tp_plan="auto",
169+
distributed_config=DistributedConfig(enable_expert_parallel=True),
170+
attn_implementation="kernels-staging/msa@v0", # MSA block-sparse attention kernel
171+
)
172+
model.eval()
173+
174+
tokenizer = AutoTokenizer.from_pretrained("MiniMaxAI/MiniMax-M3-preview")
175+
messages = [{"role": "user", "content": "Summarize the history of computing."}]
176+
inputs = tokenizer.apply_chat_template(
177+
messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
178+
).to(f"cuda:{os.environ.get('LOCAL_RANK', '0')}")
179+
180+
generated_ids = model.generate(
181+
**inputs,
182+
max_new_tokens=128,
183+
do_sample=False,
184+
# Static cache + reduce-overhead cudagraph capture is what pushes decode to ~31 tok/s.
185+
cache_implementation="static",
186+
compile_config=CompileConfig(mode="reduce-overhead", fullgraph=True),
187+
)
188+
if int(os.environ.get("RANK", "0")) == 0:
189+
print(tokenizer.decode(generated_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
190+
191+
# cudagraph-captured NCCL collectives deadlock the NCCL/CUDA destructors at teardown; the output is
192+
# already produced, so hard-exit to skip the hanging cleanup.
193+
if dist.is_initialized():
194+
sys.stdout.flush()
195+
os._exit(0)
196+
```
197+
198+
## MiniMaxM3VLConfig
199+
200+
[[autodoc]] MiniMaxM3VLConfig
201+
202+
## MiniMaxM3VLTextConfig
203+
204+
[[autodoc]] MiniMaxM3VLTextConfig
205+
206+
## MiniMaxM3VLVisionConfig
207+
208+
[[autodoc]] MiniMaxM3VLVisionConfig
209+
210+
## MiniMaxM3VLProcessor
211+
212+
[[autodoc]] MiniMaxM3VLProcessor
213+
214+
## MiniMaxM3VLImageProcessor
215+
216+
This is a standalone (non-modular) image processor: it shares the patch-flattening idea of [`Qwen2VLImageProcessor`]
217+
but does not inherit from it because the two diverge in ways that touch most of the class. The resize budget is driven by
218+
a `max_pixels` attribute and a `{"height", "width"}` `size` rather than Qwen's `shortest_edge`/`longest_edge` scheme; the
219+
`smart_resize` helper clamps the initial rounding with `max(factor, ...)`; and `_preprocess` performs real temporal
220+
handling (5D patches, last-frame repeat to fill `temporal_patch_size`, and a `grid_t` dimension) instead of Qwen's
221+
`grid_t = 1` + expand. Mapping to or subclassing Qwen would therefore change behavior or require overriding nearly
222+
everything, so the processor is kept on its own.
223+
224+
[[autodoc]] MiniMaxM3VLImageProcessor
225+
226+
## MiniMaxM3VLVideoProcessor
227+
228+
[[autodoc]] MiniMaxM3VLVideoProcessor
229+
230+
## MiniMaxM3VLVisionModel
231+
232+
[[autodoc]] MiniMaxM3VLVisionModel
233+
- forward
234+
235+
## MiniMaxM3VLTextModel
236+
237+
[[autodoc]] MiniMaxM3VLTextModel
238+
- forward
239+
240+
## MiniMaxM3VLModel
241+
242+
[[autodoc]] MiniMaxM3VLModel
243+
- forward
244+
245+
## MiniMaxM3VLForCausalLM
246+
247+
[[autodoc]] MiniMaxM3VLForCausalLM
248+
- forward
249+
250+
## MiniMaxM3SparseForConditionalGeneration
251+
252+
[[autodoc]] MiniMaxM3SparseForConditionalGeneration
253+
- forward

‎src/transformers/cache_utils.py‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,12 @@
3333
# ``PreTrainedConfig`` (the decoder text config) as the only positional argument.
3434
LAYER_TYPE_CACHE_MAPPING: dict[str, type] = {}
3535

36+
# Parallel registry for the *static* implementation of a custom layer type, consulted by
37+
# ``StaticCache``. A ``StaticLayer`` subclass with a ``layer_type`` registers here instead of in
38+
# ``LAYER_TYPE_CACHE_MAPPING`` (constructed with ``max_cache_len=...``), so a single ``layer_type``
39+
# can have both a dynamic and a static cache layer (e.g. M3's sparse-attention indexer cache).
40+
LAYER_TYPE_STATIC_CACHE_MAPPING: dict[str, type] = {}
41+
3642

3743
class CacheLayerMixin(ABC):
3844
"""Base, abstract class for a single layer's cache."""
@@ -47,7 +53,11 @@ def __init_subclass__(cls, **kwargs):
4753
super().__init_subclass__(**kwargs)
4854
layer_type = cls.__dict__.get("layer_type", None)
4955
if layer_type is not None:
50-
LAYER_TYPE_CACHE_MAPPING[layer_type] = cls
56+
static_base = globals().get("StaticLayer")
57+
if static_base is not None and issubclass(cls, static_base):
58+
LAYER_TYPE_STATIC_CACHE_MAPPING[layer_type] = cls
59+
else:
60+
LAYER_TYPE_CACHE_MAPPING[layer_type] = cls
5161

5262
def __init__(self):
5363
self.keys: torch.Tensor | None = None
@@ -1578,6 +1588,9 @@ def __init__(
15781588
# LinearAttention layers are static by essence - using `"moe"` as well is a trick, see the comment about it on DynamicCache
15791589
elif layer_type in ("mamba", "conv", "linear_attention", "moe"):
15801590
layer = LinearAttentionLayer()
1591+
# Custom layer types (e.g. M3's sparse-attention indexer cache) that registered a static variant.
1592+
elif layer_type in LAYER_TYPE_STATIC_CACHE_MAPPING:
1593+
layer = LAYER_TYPE_STATIC_CACHE_MAPPING[layer_type](max_cache_len=max_cache_len)
15811594
elif layer_type == "deepseek_sparse_attention":
15821595
# Static / compile-friendly indexed layer (preallocated indexer key cache).
15831596
layer = StaticIndexedLayer(max_cache_len=max_cache_len)

‎src/transformers/configuration_utils.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
"chunked_attention",
6666
"compressed_sparse_attention", # CSA, used in deepseek_v4
6767
"heavily_compressed_attention", # HCA, used in deepseek_v4
68+
"minimax_m3_sparse", # lightning-index sparse attention, used in minimax_m3_vl
6869
"linear_attention", # used in minimax
6970
"conv", # used in LFMv2
7071
"mamba",

0 commit comments

Comments
 (0)