|
| 1 | +<!--Copyright 2026 the HuggingFace Team. All rights reserved. |
| 2 | +
|
| 3 | +Licensed under the Apache License, Version 2.0 (the "License"); |
| 4 | +you may not use this file except in compliance with the License. |
| 5 | +You may obtain a copy of the License at |
| 6 | +
|
| 7 | + http://www.apache.org/licenses/LICENSE-2.0 |
| 8 | +
|
| 9 | +Unless required by applicable law or agreed to in writing, software |
| 10 | +distributed under the License is distributed on an "AS IS" BASIS, |
| 11 | +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. |
| 12 | +See the License for the specific language governing permissions and |
| 13 | +limitations under the License. |
| 14 | +
|
| 15 | +
|
| 16 | +⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. |
| 17 | +
|
| 18 | +--> |
| 19 | +*This model was contributed to Hugging Face Transformers on 2026-06-03.* |
| 20 | + |
| 21 | + |
| 22 | +# Gemma4 Unified |
| 23 | + |
| 24 | +## Overview |
| 25 | + |
| 26 | +Gemma 4 12B Unified is an **encoder-free** multimodal model with pretrained and instruction-tuned variants. Unlike [standard Gemma 4](./gemma4), which uses dedicated encoder towers, Gemma 4 12B Unified projects raw inputs directly into the language model's embedding space through lightweight linear pipelines. This results in a simpler architecture while maintaining strong multimodal performance. |
| 27 | + |
| 28 | +Key differences from standard Gemma 4: |
| 29 | +- **No Vision Tower**: Raw pixel patches are projected directly into LM space via a `Dense + LayerNorm` pipeline with factorized 2D positional embeddings, replacing the vision encoder. |
| 30 | +- **No Audio Tower**: Raw 16 kHz waveform samples are chunked into fixed-length frames and projected through a simple `RMSNorm → Linear` pipeline, replacing the mel spectrogram + Conformer encoder. |
| 31 | +- **Shared Multimodal Pipeline**: Both vision and audio use the same `Gemma4UnifiedMultimodalEmbedder` (RMSNorm → Linear) for the final projection to text hidden space. |
| 32 | + |
| 33 | +You can find the original Gemma 4 12B Unified checkpoints under the [Gemma 4](https://huggingface.co/collections/google/gemma-4) release. |
| 34 | + |
| 35 | +### Encoder-Free Vision Pipeline |
| 36 | + |
| 37 | +The key architectural difference from standard Gemma 4 is the removal of the vision encoder tower. Instead, Gemma 4 12B Unified processes images through a lightweight pipeline: |
| 38 | + |
| 39 | +1. **Patchification**: Images are split into `16×16` pixel patches |
| 40 | +2. **Patch Merging**: Adjacent `3×3` patches are merged into `48×48` model patches, each with `48² × 3 = 6,912` raw pixel channels |
| 41 | +3. **Projection**: `LayerNorm → Dense → LayerNorm` projects each merged patch into the LM embedding dimension |
| 42 | +4. **Positional Embedding**: Factorized 2D positional embeddings are added (separate learned embeddings for x and y axes, summed together) |
| 43 | +5. **Final Norm**: A final `LayerNorm` is applied |
| 44 | +6. **Multimodal Embedder**: `RMSNorm → Linear` projects to the text hidden size |
| 45 | + |
| 46 | +Like standard Gemma 4, the model processes **images of different sizes** using a **fixed-budget number of tokens**. The same constraints apply: |
| 47 | +- The total number of pixels must fit within a patch budget |
| 48 | +- Both height and width must be divisible by **48** (= patch size 16 × pooling kernel 3) |
| 49 | + |
| 50 | +> [!IMPORTANT] |
| 51 | +> Gemma 4 12B Unified does **not** apply mean/std normalization. The model's own patch embedding layer handles the final scaling internally. |
| 52 | +
|
| 53 | +The number of soft tokens per image is configurable. The supported options and default (**280 soft tokens**) are: |
| 54 | + |
| 55 | +| Soft Tokens | Patches (before pooling) | Approx. Image Area | |
| 56 | +|:-----------:|:------------------------:|:-------------------:| |
| 57 | +| 70 | 630 | ~161K pixels | |
| 58 | +| 140 | 1,260 | ~323K pixels | |
| 59 | +| **280** | **2,520** | **~645K pixels** | |
| 60 | +| 560 | 5,040 | ~1.3M pixels | |
| 61 | +| 1,120 | 10,080 | ~2.6M pixels | |
| 62 | + |
| 63 | +### Encoder-Free Audio Pipeline |
| 64 | + |
| 65 | +The audio pipeline is similarly simplified. Instead of computing mel spectrograms and processing them through a Conformer encoder, raw 16 kHz waveform samples are: |
| 66 | + |
| 67 | +1. **Chunked** into fixed-length frames of 640 samples each (40ms per frame at 16 kHz) |
| 68 | +2. **Projected** directly through `RMSNorm → Linear` via the shared `Gemma4UnifiedMultimodalEmbedder` |
| 69 | + |
| 70 | +Since there is **no downsampling**, the number of output soft tokens equals the number of input frames: `ceil(num_samples / 640)`. |
| 71 | + |
| 72 | +## Usage examples |
| 73 | + |
| 74 | +The example below demonstrates how to generate text based on an image and an audio sample with [`Pipeline`] or the [`AutoModel`] class. |
| 75 | + |
| 76 | +<hfoptions id="usage"> |
| 77 | +<hfoption id="Pipeline"> |
| 78 | + |
| 79 | +```python |
| 80 | +from transformers import pipeline |
| 81 | + |
| 82 | + |
| 83 | +pipe = pipeline( |
| 84 | + task="any-to-any", |
| 85 | + model="google/gemma-4-12B-it", |
| 86 | +) |
| 87 | + |
| 88 | +image_messages = [ |
| 89 | + { |
| 90 | + "role": "user", |
| 91 | + "content": [ |
| 92 | + { |
| 93 | + "type": "image", |
| 94 | + "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" |
| 95 | + }, |
| 96 | + { |
| 97 | + "type": "text", |
| 98 | + "text": "What is shown in this image?" |
| 99 | + } |
| 100 | + ] |
| 101 | + } |
| 102 | +] |
| 103 | + |
| 104 | +image_output = pipe(image_messages, return_full_text=False) |
| 105 | +print(image_output[0]["generated_text"]) |
| 106 | + |
| 107 | +audio_messages = [ |
| 108 | + { |
| 109 | + "role": "user", |
| 110 | + "content": [ |
| 111 | + {"type": "text", "text": "Please transcribe the following audio:"}, |
| 112 | + { |
| 113 | + "type": "audio", |
| 114 | + "url": "https://huggingface.co/datasets/eustlb/audio-samples/resolve/main/bcn_weather.mp3", |
| 115 | + }, |
| 116 | + ], |
| 117 | + } |
| 118 | +] |
| 119 | + |
| 120 | +audio_output = pipe(audio_messages, return_full_text=False) |
| 121 | +print(audio_output[0]["generated_text"]) |
| 122 | +``` |
| 123 | + |
| 124 | +</hfoption> |
| 125 | +<hfoption id="AutoModel"> |
| 126 | + |
| 127 | +### Image |
| 128 | + |
| 129 | +```python |
| 130 | +from transformers import AutoModelForMultimodalLM, AutoProcessor |
| 131 | + |
| 132 | + |
| 133 | +model = AutoModelForMultimodalLM.from_pretrained( |
| 134 | + "google/gemma-4-12B-it", |
| 135 | + device_map="auto" |
| 136 | +) |
| 137 | +processor = AutoProcessor.from_pretrained( |
| 138 | + "google/gemma-4-12B-it" |
| 139 | +) |
| 140 | + |
| 141 | +messages = [ |
| 142 | + { |
| 143 | + "role": "user", "content": [ |
| 144 | + {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"}, |
| 145 | + {"type": "text", "text": "What is shown in this image?"}, |
| 146 | + ] |
| 147 | + }, |
| 148 | +] |
| 149 | +inputs = processor.apply_chat_template( |
| 150 | + messages, |
| 151 | + tokenize=True, |
| 152 | + return_dict=True, |
| 153 | + return_tensors="pt", |
| 154 | + add_generation_prompt=True, |
| 155 | +).to(model.device) |
| 156 | +input_len = inputs["input_ids"].shape[-1] |
| 157 | + |
| 158 | +output = model.generate(**inputs, max_new_tokens=50, cache_implementation="static") |
| 159 | +print(processor.decode(output[0][input_len:], skip_special_tokens=True)) |
| 160 | +``` |
| 161 | + |
| 162 | +### Audio |
| 163 | + |
| 164 | +```python |
| 165 | +from transformers import AutoModelForMultimodalLM, AutoProcessor |
| 166 | + |
| 167 | + |
| 168 | +messages = [ |
| 169 | + { |
| 170 | + "role": "user", |
| 171 | + "content": [ |
| 172 | + {"type": "text", "text": "Please transcribe the following audio:"}, |
| 173 | + { |
| 174 | + "type": "audio", |
| 175 | + "url": "https://huggingface.co/datasets/eustlb/audio-samples/resolve/main/bcn_weather.mp3", |
| 176 | + }, |
| 177 | + ], |
| 178 | + } |
| 179 | +] |
| 180 | + |
| 181 | +model = AutoModelForMultimodalLM.from_pretrained( |
| 182 | + "google/gemma-4-12B-it", |
| 183 | + device_map="auto" |
| 184 | +) |
| 185 | +processor = AutoProcessor.from_pretrained( |
| 186 | + "google/gemma-4-12B-it" |
| 187 | +) |
| 188 | + |
| 189 | +inputs = processor.apply_chat_template( |
| 190 | + messages, |
| 191 | + tokenize=True, |
| 192 | + add_generation_prompt=True, |
| 193 | + return_dict=True, |
| 194 | + return_tensors="pt" |
| 195 | +).to(model.device, dtype=model.dtype) |
| 196 | + |
| 197 | +input_len = inputs["input_ids"].shape[-1] |
| 198 | + |
| 199 | +outputs = model.generate(**inputs, max_new_tokens=200) |
| 200 | +print(processor.decode(outputs[0][input_len:], skip_special_tokens=False)) |
| 201 | +``` |
| 202 | + |
| 203 | +</hfoption> |
| 204 | +</hfoptions> |
| 205 | + |
| 206 | +## Gemma4UnifiedAudioConfig |
| 207 | + |
| 208 | +[[autodoc]] Gemma4UnifiedAudioConfig |
| 209 | + |
| 210 | +## Gemma4UnifiedConfig |
| 211 | + |
| 212 | +[[autodoc]] Gemma4UnifiedConfig |
| 213 | + |
| 214 | +## Gemma4UnifiedTextConfig |
| 215 | + |
| 216 | +[[autodoc]] Gemma4UnifiedTextConfig |
| 217 | + |
| 218 | +## Gemma4UnifiedVisionConfig |
| 219 | + |
| 220 | +[[autodoc]] Gemma4UnifiedVisionConfig |
| 221 | + |
| 222 | +## Gemma4UnifiedAudioFeatureExtractor |
| 223 | + |
| 224 | +[[autodoc]] Gemma4UnifiedAudioFeatureExtractor |
| 225 | + - __call__ |
| 226 | + |
| 227 | +## Gemma4UnifiedImageProcessor |
| 228 | + |
| 229 | +[[autodoc]] Gemma4UnifiedImageProcessor |
| 230 | + |
| 231 | +## Gemma4UnifiedVideoProcessor |
| 232 | + |
| 233 | +[[autodoc]] Gemma4UnifiedVideoProcessor |
| 234 | + |
| 235 | +## Gemma4UnifiedProcessor |
| 236 | + |
| 237 | +[[autodoc]] Gemma4UnifiedProcessor |
| 238 | + - __call__ |
| 239 | + |
| 240 | +## Gemma4UnifiedPreTrainedModel |
| 241 | + |
| 242 | +[[autodoc]] Gemma4UnifiedPreTrainedModel |
| 243 | + - forward |
| 244 | + |
| 245 | +## Gemma4UnifiedModel |
| 246 | + |
| 247 | +[[autodoc]] Gemma4UnifiedModel |
| 248 | + - forward |
| 249 | + |
| 250 | +## Gemma4UnifiedTextModel |
| 251 | + |
| 252 | +[[autodoc]] Gemma4UnifiedTextModel |
| 253 | + - forward |
| 254 | + |
| 255 | +## Gemma4UnifiedForCausalLM |
| 256 | + |
| 257 | +[[autodoc]] Gemma4UnifiedForCausalLM |
| 258 | + |
| 259 | +## Gemma4UnifiedForConditionalGeneration |
| 260 | + |
| 261 | +[[autodoc]] Gemma4UnifiedForConditionalGeneration |
| 262 | + - forward |
0 commit comments