Skip to content

Commit 1423d22

Browse files
douglas-reidsgerrardArthurZuckervasqumolbap
authored
who needs encoders? (#46385)
Co-authored-by: Douglas Reid <[email protected]> Co-authored-by: Sara Smoot <[email protected]> Co-authored-by: Arthur Zucker <[email protected]> Co-authored-by: Anton Vlasjuk <[email protected]> Co-authored-by: Pablo Montalvo-Leroux <[email protected]>
1 parent 50eb20a commit 1423d22

38 files changed

Lines changed: 8034 additions & 20 deletions

‎docs/source/en/_toctree.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1239,12 +1239,16 @@
12391239
title: Florence2
12401240
- local: model_doc/gemma4_assistant
12411241
title: Gemma 4 Assistant
1242+
- local: model_doc/gemma4_unified_assistant
1243+
title: Gemma 4 Unified Assistant
12421244
- local: model_doc/gemma3
12431245
title: Gemma3
12441246
- local: model_doc/gemma3n
12451247
title: Gemma3n
12461248
- local: model_doc/gemma4
12471249
title: Gemma4
1250+
- local: model_doc/gemma4_unified
1251+
title: Gemma4 Unified
12481252
- local: model_doc/git
12491253
title: GIT
12501254
- local: model_doc/glm46v
Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
<!--Copyright 2026 the HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.
14+
15+
16+
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer.
17+
18+
-->
19+
*This model was contributed to Hugging Face Transformers on 2026-06-03.*
20+
21+
22+
# Gemma4 Unified
23+
24+
## Overview
25+
26+
Gemma 4 12B Unified is an **encoder-free** multimodal model with pretrained and instruction-tuned variants. Unlike [standard Gemma 4](./gemma4), which uses dedicated encoder towers, Gemma 4 12B Unified projects raw inputs directly into the language model's embedding space through lightweight linear pipelines. This results in a simpler architecture while maintaining strong multimodal performance.
27+
28+
Key differences from standard Gemma 4:
29+
- **No Vision Tower**: Raw pixel patches are projected directly into LM space via a `Dense + LayerNorm` pipeline with factorized 2D positional embeddings, replacing the vision encoder.
30+
- **No Audio Tower**: Raw 16 kHz waveform samples are chunked into fixed-length frames and projected through a simple `RMSNorm → Linear` pipeline, replacing the mel spectrogram + Conformer encoder.
31+
- **Shared Multimodal Pipeline**: Both vision and audio use the same `Gemma4UnifiedMultimodalEmbedder` (RMSNorm → Linear) for the final projection to text hidden space.
32+
33+
You can find the original Gemma 4 12B Unified checkpoints under the [Gemma 4](https://huggingface.co/collections/google/gemma-4) release.
34+
35+
### Encoder-Free Vision Pipeline
36+
37+
The key architectural difference from standard Gemma 4 is the removal of the vision encoder tower. Instead, Gemma 4 12B Unified processes images through a lightweight pipeline:
38+
39+
1. **Patchification**: Images are split into `16×16` pixel patches
40+
2. **Patch Merging**: Adjacent `3×3` patches are merged into `48×48` model patches, each with `48² × 3 = 6,912` raw pixel channels
41+
3. **Projection**: `LayerNorm → Dense → LayerNorm` projects each merged patch into the LM embedding dimension
42+
4. **Positional Embedding**: Factorized 2D positional embeddings are added (separate learned embeddings for x and y axes, summed together)
43+
5. **Final Norm**: A final `LayerNorm` is applied
44+
6. **Multimodal Embedder**: `RMSNorm → Linear` projects to the text hidden size
45+
46+
Like standard Gemma 4, the model processes **images of different sizes** using a **fixed-budget number of tokens**. The same constraints apply:
47+
- The total number of pixels must fit within a patch budget
48+
- Both height and width must be divisible by **48** (= patch size 16 × pooling kernel 3)
49+
50+
> [!IMPORTANT]
51+
> Gemma 4 12B Unified does **not** apply mean/std normalization. The model's own patch embedding layer handles the final scaling internally.
52+
53+
The number of soft tokens per image is configurable. The supported options and default (**280 soft tokens**) are:
54+
55+
| Soft Tokens | Patches (before pooling) | Approx. Image Area |
56+
|:-----------:|:------------------------:|:-------------------:|
57+
| 70 | 630 | ~161K pixels |
58+
| 140 | 1,260 | ~323K pixels |
59+
| **280** | **2,520** | **~645K pixels** |
60+
| 560 | 5,040 | ~1.3M pixels |
61+
| 1,120 | 10,080 | ~2.6M pixels |
62+
63+
### Encoder-Free Audio Pipeline
64+
65+
The audio pipeline is similarly simplified. Instead of computing mel spectrograms and processing them through a Conformer encoder, raw 16 kHz waveform samples are:
66+
67+
1. **Chunked** into fixed-length frames of 640 samples each (40ms per frame at 16 kHz)
68+
2. **Projected** directly through `RMSNorm → Linear` via the shared `Gemma4UnifiedMultimodalEmbedder`
69+
70+
Since there is **no downsampling**, the number of output soft tokens equals the number of input frames: `ceil(num_samples / 640)`.
71+
72+
## Usage examples
73+
74+
The example below demonstrates how to generate text based on an image and an audio sample with [`Pipeline`] or the [`AutoModel`] class.
75+
76+
<hfoptions id="usage">
77+
<hfoption id="Pipeline">
78+
79+
```python
80+
from transformers import pipeline
81+
82+
83+
pipe = pipeline(
84+
task="any-to-any",
85+
model="google/gemma-4-12B-it",
86+
)
87+
88+
image_messages = [
89+
{
90+
"role": "user",
91+
"content": [
92+
{
93+
"type": "image",
94+
"url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"
95+
},
96+
{
97+
"type": "text",
98+
"text": "What is shown in this image?"
99+
}
100+
]
101+
}
102+
]
103+
104+
image_output = pipe(image_messages, return_full_text=False)
105+
print(image_output[0]["generated_text"])
106+
107+
audio_messages = [
108+
{
109+
"role": "user",
110+
"content": [
111+
{"type": "text", "text": "Please transcribe the following audio:"},
112+
{
113+
"type": "audio",
114+
"url": "https://huggingface.co/datasets/eustlb/audio-samples/resolve/main/bcn_weather.mp3",
115+
},
116+
],
117+
}
118+
]
119+
120+
audio_output = pipe(audio_messages, return_full_text=False)
121+
print(audio_output[0]["generated_text"])
122+
```
123+
124+
</hfoption>
125+
<hfoption id="AutoModel">
126+
127+
### Image
128+
129+
```python
130+
from transformers import AutoModelForMultimodalLM, AutoProcessor
131+
132+
133+
model = AutoModelForMultimodalLM.from_pretrained(
134+
"google/gemma-4-12B-it",
135+
device_map="auto"
136+
)
137+
processor = AutoProcessor.from_pretrained(
138+
"google/gemma-4-12B-it"
139+
)
140+
141+
messages = [
142+
{
143+
"role": "user", "content": [
144+
{"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"},
145+
{"type": "text", "text": "What is shown in this image?"},
146+
]
147+
},
148+
]
149+
inputs = processor.apply_chat_template(
150+
messages,
151+
tokenize=True,
152+
return_dict=True,
153+
return_tensors="pt",
154+
add_generation_prompt=True,
155+
).to(model.device)
156+
input_len = inputs["input_ids"].shape[-1]
157+
158+
output = model.generate(**inputs, max_new_tokens=50, cache_implementation="static")
159+
print(processor.decode(output[0][input_len:], skip_special_tokens=True))
160+
```
161+
162+
### Audio
163+
164+
```python
165+
from transformers import AutoModelForMultimodalLM, AutoProcessor
166+
167+
168+
messages = [
169+
{
170+
"role": "user",
171+
"content": [
172+
{"type": "text", "text": "Please transcribe the following audio:"},
173+
{
174+
"type": "audio",
175+
"url": "https://huggingface.co/datasets/eustlb/audio-samples/resolve/main/bcn_weather.mp3",
176+
},
177+
],
178+
}
179+
]
180+
181+
model = AutoModelForMultimodalLM.from_pretrained(
182+
"google/gemma-4-12B-it",
183+
device_map="auto"
184+
)
185+
processor = AutoProcessor.from_pretrained(
186+
"google/gemma-4-12B-it"
187+
)
188+
189+
inputs = processor.apply_chat_template(
190+
messages,
191+
tokenize=True,
192+
add_generation_prompt=True,
193+
return_dict=True,
194+
return_tensors="pt"
195+
).to(model.device, dtype=model.dtype)
196+
197+
input_len = inputs["input_ids"].shape[-1]
198+
199+
outputs = model.generate(**inputs, max_new_tokens=200)
200+
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))
201+
```
202+
203+
</hfoption>
204+
</hfoptions>
205+
206+
## Gemma4UnifiedAudioConfig
207+
208+
[[autodoc]] Gemma4UnifiedAudioConfig
209+
210+
## Gemma4UnifiedConfig
211+
212+
[[autodoc]] Gemma4UnifiedConfig
213+
214+
## Gemma4UnifiedTextConfig
215+
216+
[[autodoc]] Gemma4UnifiedTextConfig
217+
218+
## Gemma4UnifiedVisionConfig
219+
220+
[[autodoc]] Gemma4UnifiedVisionConfig
221+
222+
## Gemma4UnifiedAudioFeatureExtractor
223+
224+
[[autodoc]] Gemma4UnifiedAudioFeatureExtractor
225+
- __call__
226+
227+
## Gemma4UnifiedImageProcessor
228+
229+
[[autodoc]] Gemma4UnifiedImageProcessor
230+
231+
## Gemma4UnifiedVideoProcessor
232+
233+
[[autodoc]] Gemma4UnifiedVideoProcessor
234+
235+
## Gemma4UnifiedProcessor
236+
237+
[[autodoc]] Gemma4UnifiedProcessor
238+
- __call__
239+
240+
## Gemma4UnifiedPreTrainedModel
241+
242+
[[autodoc]] Gemma4UnifiedPreTrainedModel
243+
- forward
244+
245+
## Gemma4UnifiedModel
246+
247+
[[autodoc]] Gemma4UnifiedModel
248+
- forward
249+
250+
## Gemma4UnifiedTextModel
251+
252+
[[autodoc]] Gemma4UnifiedTextModel
253+
- forward
254+
255+
## Gemma4UnifiedForCausalLM
256+
257+
[[autodoc]] Gemma4UnifiedForCausalLM
258+
259+
## Gemma4UnifiedForConditionalGeneration
260+
261+
[[autodoc]] Gemma4UnifiedForConditionalGeneration
262+
- forward

0 commit comments

Comments
 (0)