Skip to content

Commit 76613e6

Browse files
molbapzucchini-nlpeustlbCyrilvallez
authored andcommitted
Add Inkling model (#47347)
* processor very rough draft * whatever claude says for now, clean next * mm towers * . * import base model * make it init model from scratch * conversion maybe * vision loads * tiny change * add normalization * more nits * more * oops * typo * copy linear stuff from qwen, seems quite similar * a bit more * relative position bias draft * no act * dtype force fp32 to match as closely as we can * make it run * torchaudio dep doesn't work for me, just skip for now * moe loads kinda * oke now the keys match * moe router fix * . * hmm, I think mlp has to be deinterlaved not chunked * i am defi breaking the model again XD * same for routed experts mlp * dont shard shared gammas ig * well we still need cache for full vs slliding * this is ugly! * ig? * delete * fix norm * fix normer * tied + inits * sync modular * Add TML audio support (feature extractor, processor, audio tower wiring) Ports the dMel audio pipeline onto the synced skeleton-modular base: - feature extractor returns the log-mel spectrogram + mask - processor quantizes it into audio_input_ids (dMel tokens) + mask - TmlAudioModelEmbeddings/TmlAudioModel embed the dMel tokens - get_audio_features + generalized get_placeholder_mask wire audio into the forward - conversion_mapping renames audio.encoder -> embed_audio_tokens, final_norm -> norm * parallelism wars * fix position_bias -> use cache primitives directly * do not slice the mask * position_ids * re add some inheritance * init global scales at 1: super important if we want to train correctly * inherit more * simplify conv * no need to pad before update, the update does it * add accelerate hook decorator * big cache refactor to have native implem * small fix * add static cache support * make it compileable * remove cache param completely * do not transpose every time * oupsi already had conversions * remove the very suspicious dtype casting * support sdpa * fix bad reshape * add flex support * fix * oupsi, pass layer_idx * upstream TP fix * big model fixes + conv1d update diff from qwen * fix_compile * TP transpose conversion is free * Reuse HiggsAudioV2Embeddings for TmlAudioModelEmbeddings The controlled port had inlined the dMel embeddings because the standalone-era skeleton-modular had no cross-model inheritance. Now that skeleton-modular uses inheritance again, restore TmlAudioModelEmbeddings(HiggsAudioV2Embeddings) + the TmlAudioConfig num_codebooks/codebook_size/hidden_size attribute map. Behavior and weight names unchanged (verified: logits + load identical). * add kernels * log scaling + image processing changes * Ink it up * add load_audio_backend * allow to rollback linear cache layers * make repo * record past * test audio utils * audio/ image bos * TmlProcessingIntegrationTest * add TmlIntegrationTest * nit * nit * add requires_backends * can rollback prefill as well * no rescale_image_frac * avoid downcasting for linear attn * add kernels + fix env vars for kernel disablement * format * fix save pretrained for Interleave * better mappings * more automappings * change name * implement mtp cache rollback and correction * attempts at fixing MTP (noisy outputs) * solve all indexing issues between main_model index and ntp layers * small fix * oupsi forgot to apply modular * replace sliding layers for now * allow sliding in mtp layers * fix the masking for mtp layers * fix cropping and a lot of issues by replacing get_seq_length to get offsets * oupsi * replace layers * bucket to hf-internal-testing * add test_apply_chat_template_audio_without_attention_mask * fix mtp * update test path * improve cache invalidation a lot * fix tests 1 / N * update integration test * update bucket path * fix tests 2/N * fix tests 3 / N * fix tests 4/ N: Chunk dim 1 * fix fe edge case * fix tests 5 / N * update all mamba models * upddate token ids * slice them * few missing * [core] force contiguity on chunking * fix tests 6 / N * chat template testing 7 / N * format + processor test fix 8 / N * small fix * skip tiny model offload fix 9 / N * small fix * conflict dropped the crochet * style --------- Co-authored-by: raushan <[email protected]> Co-authored-by: Eustache Le Bihan <[email protected]> Co-authored-by: Cyril Vallez <[email protected]> Co-authored-by: eustlb <[email protected]>
1 parent f5a67a4 commit 76613e6

65 files changed

Lines changed: 5789 additions & 310 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/source/en/_toctree.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1321,6 +1321,8 @@
13211321
title: Idefics2
13221322
- local: model_doc/idefics3
13231323
title: Idefics3
1324+
- local: model_doc/inkling
1325+
title: Inkling
13241326
- local: model_doc/instructblip
13251327
title: InstructBLIP
13261328
- local: model_doc/instructblipvideo
Lines changed: 103 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,103 @@
1+
<!--Copyright 2026 the HuggingFace Team. All rights reserved.
2+
3+
Licensed under the Apache License, Version 2.0 (the "License");
4+
you may not use this file except in compliance with the License.
5+
You may obtain a copy of the License at
6+
7+
http://www.apache.org/licenses/LICENSE-2.0
8+
9+
Unless required by applicable law or agreed to in writing, software
10+
distributed under the License is distributed on an "AS IS" BASIS,
11+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
See the License for the specific language governing permissions and
13+
limitations under the License.
14+
15+
16+
⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer.
17+
18+
-->
19+
*This model was contributed to Hugging Face Transformers on 2026-07-15.*
20+
21+
22+
# Inkling
23+
24+
## Overview
25+
26+
The Inkling model was proposed in [<INSERT PAPER NAME HERE>](<INSERT PAPER LINK HERE>) by <INSERT AUTHORS HERE>.
27+
<INSERT SHORT SUMMARY HERE>
28+
29+
The abstract from the paper is the following:
30+
31+
<INSERT PAPER ABSTRACT HERE>
32+
33+
Tips:
34+
35+
<INSERT TIPS ABOUT MODEL HERE>
36+
37+
This model was contributed by [INSERT YOUR HF USERNAME HERE](https://huggingface.co/<INSERT YOUR HF USERNAME HERE>).
38+
The original code can be found [here](<INSERT LINK TO GITHUB REPO HERE>).
39+
40+
## Usage examples
41+
42+
<INSERT SOME NICE EXAMPLES HERE>
43+
44+
## InklingAudioConfig
45+
46+
[[autodoc]] InklingAudioConfig
47+
48+
## InklingConfig
49+
50+
[[autodoc]] InklingConfig
51+
52+
## InklingTextConfig
53+
54+
[[autodoc]] InklingTextConfig
55+
56+
## InklingVisionConfig
57+
58+
[[autodoc]] InklingVisionConfig
59+
60+
## InklingAudioModel
61+
62+
[[autodoc]] InklingAudioModel
63+
- forward
64+
65+
## InklingForCausalLM
66+
67+
[[autodoc]] InklingForCausalLM
68+
69+
## InklingForConditionalGeneration
70+
71+
[[autodoc]] InklingForConditionalGeneration
72+
73+
## InklingModel
74+
75+
[[autodoc]] InklingModel
76+
- forward
77+
78+
## InklingPreTrainedModel
79+
80+
[[autodoc]] InklingPreTrainedModel
81+
- forward
82+
83+
## InklingTextModel
84+
85+
[[autodoc]] InklingTextModel
86+
- forward
87+
88+
## InklingVisionModel
89+
90+
[[autodoc]] InklingVisionModel
91+
- forward
92+
93+
## InklingImageProcessor
94+
95+
[[autodoc]] InklingImageProcessor
96+
97+
## InklingFeatureExtractor
98+
99+
[[autodoc]] InklingFeatureExtractor
100+
101+
## InklingProcessor
102+
103+
[[autodoc]] InklingProcessor

‎src/transformers/audio_utils.py‎

Lines changed: 24 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@
3535
is_numpy_array,
3636
is_soundfile_available,
3737
is_torch_tensor,
38+
is_torchaudio_available,
3839
is_torchcodec_available,
3940
requires_backends,
4041
)
@@ -53,6 +54,9 @@
5354
# TODO: @eustlb, we actually don't need librosa but soxr is installed with librosa
5455
import soxr
5556

57+
if is_torchaudio_available():
58+
import torchaudio
59+
5660
if is_torchcodec_available():
5761
TORCHCODEC_VERSION = version.parse(importlib.metadata.version("torchcodec"))
5862

@@ -207,7 +211,9 @@ def load_audio(audio: str | np.ndarray, sampling_rate=16000, timeout=None, backe
207211
The timeout value in seconds for the URL request.
208212
backend (`str`, *optional*, defaults to `"auto"`):
209213
Decoding backend: `"auto"` uses torchcodec when available (>=0.3.0) and falls back to
210-
librosa; `"torchcodec"` or `"librosa"` force that backend (and error if it is missing).
214+
librosa; `"torchcodec"`, `"librosa"` or `"torchaudio"` force that backend (and error if it
215+
is missing). `"torchaudio"` decodes with `torchaudio.load` and resamples with
216+
`torchaudio.functional.resample` (matches serving stacks such as sglang bit-for-bit).
211217
212218
Returns:
213219
`np.ndarray`: A numpy array representing the audio.
@@ -222,11 +228,15 @@ def load_audio(audio: str | np.ndarray, sampling_rate=16000, timeout=None, backe
222228

223229
# torchcodec handles audio/video; librosa only plain audio. `backend` lets callers pin one.
224230
if backend == "auto":
225-
use_torchcodec = is_torchcodec_available() and version.parse("0.3.0") <= TORCHCODEC_VERSION
226-
elif backend in ("torchcodec", "librosa"):
227-
use_torchcodec = backend == "torchcodec"
231+
resolved_backend = (
232+
"torchcodec" if is_torchcodec_available() and version.parse("0.3.0") <= TORCHCODEC_VERSION else "librosa"
233+
)
234+
elif backend in ("torchcodec", "librosa", "torchaudio"):
235+
resolved_backend = backend
228236
else:
229-
raise ValueError(f"Unknown backend {backend!r}; expected 'auto', 'torchcodec', or 'librosa'.")
237+
raise ValueError(f"Unknown backend {backend!r}; expected 'auto', 'torchcodec', 'librosa', or 'torchaudio'.")
238+
# soundfile-based backends (librosa / torchaudio) cannot decode the video-ish formats below.
239+
use_torchcodec = resolved_backend == "torchcodec"
230240

231241
# 1. Identify the format from the source string (extension / `data:` media type), without fetching.
232242
filetype = _format_from_source(audio)
@@ -256,6 +266,15 @@ def load_audio(audio: str | np.ndarray, sampling_rate=16000, timeout=None, backe
256266
# `num_channels=1` matches what most models expect and librosa's default.
257267
return AudioDecoder(source, sample_rate=sampling_rate, num_channels=1).get_all_samples().data[0].numpy()
258268

269+
if resolved_backend == "torchaudio":
270+
requires_backends(load_audio, ["torchaudio"])
271+
waveform, src_sampling_rate = torchaudio.load(BytesIO(source) if isinstance(source, bytes) else source)
272+
waveform = waveform.mean(dim=0) # to mono
273+
274+
if src_sampling_rate != sampling_rate:
275+
waveform = torchaudio.functional.resample(waveform, orig_freq=src_sampling_rate, new_freq=sampling_rate)
276+
return waveform.numpy().astype(np.float32)
277+
259278
requires_backends(load_audio, ["librosa"])
260279
return librosa.load(BytesIO(source) if isinstance(source, bytes) else source, sr=sampling_rate)[0]
261280

0 commit comments

Comments
 (0)