|
| 1 | +<!--Copyright 2026 the HuggingFace Team. All rights reserved. |
| 2 | +
|
| 3 | +Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | +you may not use this file except in compliance with the License. |
| 5 | +You may obtain a copy of the License at |
| 6 | +
|
| 7 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +
|
| 9 | +Unless required by applicable law or agreed to in writing, software |
| 10 | +distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | +See the License for the specific language governing permissions and |
| 13 | +limitations under the License. |
| 14 | +
|
| 15 | +
|
| 16 | +⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. |
| 17 | +
|
| 18 | +--> |
| 19 | +*This model was contributed to Hugging Face Transformers on 2026-06-12.* |
| 20 | + |
| 21 | + |
| 22 | +# MiniMax-M3-VL |
| 23 | + |
| 24 | +## Overview |
| 25 | + |
| 26 | +MiniMax-M3-VL is the vision-language member of the MiniMax-M3 family. It pairs a CLIP-style vision tower (Conv3d patch embedding with 3D rotary position embeddings) with the MiniMax-M3 text backbone, a mixed dense/sparse Mixture-of-Experts decoder that uses SwiGLU-OAI gated experts and a lightning indexer for block-sparse attention. |
| 27 | + |
| 28 | +## Architecture |
| 29 | +### Block-sparse attention (Lightning Indexer) |
| 30 | + |
| 31 | +Every layer is GQA (`num_key_value_heads = 4`) with per-head QK-norm and **partial RoPE** on the first |
| 32 | +`rotary_dim`. `config.layer_types[i]` then picks `"full_attention"` (dense causal) or |
| 33 | +`"minimax_m3_sparse"`, where a [`MiniMaxM3VLIndexer`] decides, per query, which block of keys the main attention may see. |
| 34 | + |
| 35 | +The indexer scores every key, then **max-poolsthose per-key scores into blocks of `index_block_size` keys**, so selection happens at the granularity of a *block |
| 36 | +of keys*: per query it keeps the top-`index_topk_blocks` key blocks plus the always-on `index_local_blocks` |
| 37 | +local-window block (under block-level causality), broadcasts the per-block `0`/`-inf` choice back onto every key in |
| 38 | +the block. The result is a `[B, 1, S_q, S_k]` additive bias summed onto the causal mask. |
| 39 | +Theoretically this means that the attention is only computed over the selected blocks of keys, but `transformers` does not support the kernels that compute this efficiently! |
| 40 | +We are adding it to `kernels` asap! |
| 41 | + |
| 42 | +<img alt="MiniMax M3 Lightning Indexer mask" src="https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/model_doc/minimax_m3_vl_indexer_mask.svg" /> |
| 43 | + |
| 44 | + |
| 45 | +### Vision tower |
| 46 | + |
| 47 | +A [`MiniMaxM3VLVisionModel`]: a `Conv3d` patch embedding over flattened `[N_patches, C·T·P·P]` input, a stack of |
| 48 | +CLIP-style encoder layers carrying a **3D rotary** position embedding (time / height / width bands). A [`MiniMaxM3VLPatchMerger`] groups |
| 49 | +`spatial_merge_size²` patches into the channel dim before the 2-layer GELU [`MiniMaxM3VLMultiModalProjector`] maps vision features into the text hidden size. |
| 50 | + |
| 51 | +## Usage examples |
| 52 | + |
| 53 | +The example below runs the model on a real image loaded with [`~transformers.image_utils.load_image`]. |
| 54 | + |
| 55 | +```python |
| 56 | +import torch |
| 57 | +from transformers import AutoModelForImageTextToText, AutoProcessor |
| 58 | +from transformers.image_utils import load_image |
| 59 | + |
| 60 | + |
| 61 | +model = AutoModelForImageTextToText.from_pretrained( |
| 62 | + "MiniMaxAI/MiniMax-M3-preview", dtype=torch.bfloat16, device_map="auto", |
| 63 | +) |
| 64 | +processor = AutoProcessor.from_pretrained("MiniMaxAI/MiniMax-M3-preview") |
| 65 | + |
| 66 | +image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg") |
| 67 | +messages = [ |
| 68 | + { |
| 69 | + "role": "user", |
| 70 | + "content": [ |
| 71 | + {"type": "image"}, |
| 72 | + {"type": "text", "text": "Describe this image briefly."}, |
| 73 | + ], |
| 74 | + } |
| 75 | +] |
| 76 | +text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
| 77 | +inputs = processor(images=[image], text=text, return_tensors="pt").to(model.device) |
| 78 | + |
| 79 | +generated_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False) |
| 80 | +print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0]) |
| 81 | +``` |
| 82 | + |
| 83 | +### Apple example |
| 84 | + |
| 85 | +This example asks the model about an image of apples, again loading a real image with |
| 86 | +[`~transformers.image_utils.load_image`]. |
| 87 | + |
| 88 | +```python |
| 89 | +import torch |
| 90 | +from transformers import AutoModelForImageTextToText, AutoProcessor |
| 91 | +from transformers.image_utils import load_image |
| 92 | + |
| 93 | + |
| 94 | +model = AutoModelForImageTextToText.from_pretrained( |
| 95 | + "MiniMaxAI/MiniMax-M3-preview", dtype=torch.bfloat16, device_map="auto", |
| 96 | +) |
| 97 | +processor = AutoProcessor.from_pretrained("MiniMaxAI/MiniMax-M3-preview") |
| 98 | + |
| 99 | +image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg") |
| 100 | +messages = [ |
| 101 | + { |
| 102 | + "role": "user", |
| 103 | + "content": [ |
| 104 | + {"type": "image"}, |
| 105 | + {"type": "text", "text": "How many apples are in this image, and what color are they?"}, |
| 106 | + ], |
| 107 | + } |
| 108 | +] |
| 109 | +text = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=False) |
| 110 | +inputs = processor(images=[image], text=text, return_tensors="pt").to(model.device) |
| 111 | + |
| 112 | +generated_ids = model.generate(**inputs, max_new_tokens=32, do_sample=False) |
| 113 | +print(processor.batch_decode(generated_ids, skip_special_tokens=True)[0]) |
| 114 | +``` |
| 115 | + |
| 116 | +## Fastest inference configuration |
| 117 | + |
| 118 | +| ctx | SDPA decode | MSA decode | MSA decode adv. | SDPA prefill | MSA prefill | MSA prefill adv. | |
| 119 | +| --: | ----------: | ---------: | --------------: | -----------: | ----------: | ---------------: | |
| 120 | +| 2K | 27.8 tok/s | 31.0 | +12% | 303 ms | 257 ms | 1.18× | |
| 121 | +| 4K | 23.4 tok/s | 30.5 | +30% | 684 ms | 460 ms | 1.49× | |
| 122 | +| 8K | 17.8 tok/s | 29.6 | +66% | 1906 ms | 976 ms | 1.95× | |
| 123 | +| 16K | 12.0 tok/s | 27.6 | +130% | 6110 ms | 2344 ms | 2.61× | |
| 124 | + |
| 125 | +The checkpoint ships in native MXFP8. For **decode throughput**, the fastest validated configuration is |
| 126 | +**bf16 (dequantized at load) + the MSA block-sparse attention kernel + tensor & expert parallelism + a |
| 127 | +`reduce-overhead` cudagraph compile** — roughly **31 tok/s** decode on 8×B200 at a 2048-token prefill. |
| 128 | + |
| 129 | +Keeping the weights in **native FP8 is a memory-footprint option only — it is never faster on this setup**. |
| 130 | +The FP8 Triton experts/linear kernels lower as opaque inductor fallback kernels that cudagraph cannot |
| 131 | +capture on the hot expert path, so native-FP8 decode measured ~4.2 tok/s (≈7× slower than the bf16 path) |
| 132 | +even under `torch.compile(fullgraph=True)`. Use FP8 only when the bf16 weights do not fit. |
| 133 | + |
| 134 | +| config (sdpa baseline, TP+EP, 2048-token prefill, 8×B200) | decode | |
| 135 | +|---|---| |
| 136 | +| bf16 dequantize-at-load + **MSA** + compile/cudagraph | **~31 tok/s** | |
| 137 | +| bf16 dequantize-at-load + sdpa + compile/cudagraph | ~28 tok/s | |
| 138 | +| native FP8 + compile/cudagraph | ~4 tok/s (memory-only, not for speed) | |
| 139 | + |
| 140 | +Dequantizing to bf16 only fits with even sharding across GPUs (TP/EP), not with `device_map="auto"` |
| 141 | +(pipeline placement OOMs at load). Launch one process per GPU with `torchrun`: |
| 142 | + |
| 143 | +```bash |
| 144 | +torchrun --nproc_per_node=8 fastest_m3_vl.py |
| 145 | +``` |
| 146 | + |
| 147 | +```python |
| 148 | +# fastest_m3_vl.py |
| 149 | +import os, sys |
| 150 | +import torch |
| 151 | +import torch.distributed as dist |
| 152 | +from transformers import ( |
| 153 | + AutoModelForImageTextToText, |
| 154 | + AutoTokenizer, |
| 155 | + CompileConfig, |
| 156 | + FineGrainedFP8Config, |
| 157 | +) |
| 158 | +from transformers.distributed import DistributedConfig |
| 159 | + |
| 160 | +# The indexer feeds SDPA an additive float mask; the cuDNN SDP backend segfaults on it (B200). |
| 161 | +torch.backends.cuda.enable_cudnn_sdp(False) |
| 162 | + |
| 163 | +model = AutoModelForImageTextToText.from_pretrained( |
| 164 | + "MiniMaxAI/MiniMax-M3-preview", |
| 165 | + dtype=torch.bfloat16, |
| 166 | + # Dequantize the native MXFP8 weights to bf16 at load (the speed win); needs even TP/EP sharding. |
| 167 | + quantization_config=FineGrainedFP8Config(dequantize=True), |
| 168 | + tp_plan="auto", |
| 169 | + distributed_config=DistributedConfig(enable_expert_parallel=True), |
| 170 | + attn_implementation="kernels-staging/msa@v0", # MSA block-sparse attention kernel |
| 171 | +) |
| 172 | +model.eval() |
| 173 | + |
| 174 | +tokenizer = AutoTokenizer.from_pretrained("MiniMaxAI/MiniMax-M3-preview") |
| 175 | +messages = [{"role": "user", "content": "Summarize the history of computing."}] |
| 176 | +inputs = tokenizer.apply_chat_template( |
| 177 | + messages, add_generation_prompt=True, return_tensors="pt", return_dict=True |
| 178 | +).to(f"cuda:{os.environ.get('LOCAL_RANK', '0')}") |
| 179 | + |
| 180 | +generated_ids = model.generate( |
| 181 | + **inputs, |
| 182 | + max_new_tokens=128, |
| 183 | + do_sample=False, |
| 184 | + # Static cache + reduce-overhead cudagraph capture is what pushes decode to ~31 tok/s. |
| 185 | + cache_implementation="static", |
| 186 | + compile_config=CompileConfig(mode="reduce-overhead", fullgraph=True), |
| 187 | +) |
| 188 | +if int(os.environ.get("RANK", "0")) == 0: |
| 189 | + print(tokenizer.decode(generated_ids[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)) |
| 190 | + |
| 191 | +# cudagraph-captured NCCL collectives deadlock the NCCL/CUDA destructors at teardown; the output is |
| 192 | +# already produced, so hard-exit to skip the hanging cleanup. |
| 193 | +if dist.is_initialized(): |
| 194 | + sys.stdout.flush() |
| 195 | + os._exit(0) |
| 196 | +``` |
| 197 | + |
| 198 | +## MiniMaxM3VLConfig |
| 199 | + |
| 200 | +[[autodoc]] MiniMaxM3VLConfig |
| 201 | + |
| 202 | +## MiniMaxM3VLTextConfig |
| 203 | + |
| 204 | +[[autodoc]] MiniMaxM3VLTextConfig |
| 205 | + |
| 206 | +## MiniMaxM3VLVisionConfig |
| 207 | + |
| 208 | +[[autodoc]] MiniMaxM3VLVisionConfig |
| 209 | + |
| 210 | +## MiniMaxM3VLProcessor |
| 211 | + |
| 212 | +[[autodoc]] MiniMaxM3VLProcessor |
| 213 | + |
| 214 | +## MiniMaxM3VLImageProcessor |
| 215 | + |
| 216 | +This is a standalone (non-modular) image processor: it shares the patch-flattening idea of [`Qwen2VLImageProcessor`] |
| 217 | +but does not inherit from it because the two diverge in ways that touch most of the class. The resize budget is driven by |
| 218 | +a `max_pixels` attribute and a `{"height", "width"}` `size` rather than Qwen's `shortest_edge`/`longest_edge` scheme; the |
| 219 | +`smart_resize` helper clamps the initial rounding with `max(factor, ...)`; and `_preprocess` performs real temporal |
| 220 | +handling (5D patches, last-frame repeat to fill `temporal_patch_size`, and a `grid_t` dimension) instead of Qwen's |
| 221 | +`grid_t = 1` + expand. Mapping to or subclassing Qwen would therefore change behavior or require overriding nearly |
| 222 | +everything, so the processor is kept on its own. |
| 223 | + |
| 224 | +[[autodoc]] MiniMaxM3VLImageProcessor |
| 225 | + |
| 226 | +## MiniMaxM3VLVideoProcessor |
| 227 | + |
| 228 | +[[autodoc]] MiniMaxM3VLVideoProcessor |
| 229 | + |
| 230 | +## MiniMaxM3VLVisionModel |
| 231 | + |
| 232 | +[[autodoc]] MiniMaxM3VLVisionModel |
| 233 | + - forward |
| 234 | + |
| 235 | +## MiniMaxM3VLTextModel |
| 236 | + |
| 237 | +[[autodoc]] MiniMaxM3VLTextModel |
| 238 | + - forward |
| 239 | + |
| 240 | +## MiniMaxM3VLModel |
| 241 | + |
| 242 | +[[autodoc]] MiniMaxM3VLModel |
| 243 | + - forward |
| 244 | + |
| 245 | +## MiniMaxM3VLForCausalLM |
| 246 | + |
| 247 | +[[autodoc]] MiniMaxM3VLForCausalLM |
| 248 | + - forward |
| 249 | + |
| 250 | +## MiniMaxM3SparseForConditionalGeneration |
| 251 | + |
| 252 | +[[autodoc]] MiniMaxM3SparseForConditionalGeneration |
| 253 | + - forward |
0 commit comments