Repository navigation
Commit 76613e6
Add Inkling model (#47347)
* processor very rough draft
* whatever claude says for now, clean next
* mm towers
* .
* import base model
* make it init model from scratch
* conversion maybe
* vision loads
* tiny change
* add normalization
* more nits
* more
* oops
* typo
* copy linear stuff from qwen, seems quite similar
* a bit more
* relative position bias draft
* no act
* dtype force fp32 to match as closely as we can
* make it run
* torchaudio dep doesn't work for me, just skip for now
* moe loads kinda
* oke now the keys match
* moe router fix
* .
* hmm, I think mlp has to be deinterlaved not chunked
* i am defi breaking the model again XD
* same for routed experts mlp
* dont shard shared gammas ig
* well we still need cache for full vs slliding
* this is ugly!
* ig?
* delete
* fix norm
* fix normer
* tied + inits
* sync modular
* Add TML audio support (feature extractor, processor, audio tower wiring)
Ports the dMel audio pipeline onto the synced skeleton-modular base:
- feature extractor returns the log-mel spectrogram + mask
- processor quantizes it into audio_input_ids (dMel tokens) + mask
- TmlAudioModelEmbeddings/TmlAudioModel embed the dMel tokens
- get_audio_features + generalized get_placeholder_mask wire audio into the forward
- conversion_mapping renames audio.encoder -> embed_audio_tokens, final_norm -> norm
* parallelism wars
* fix position_bias -> use cache primitives directly
* do not slice the mask
* position_ids
* re add some inheritance
* init global scales at 1: super important if we want to train correctly
* inherit more
* simplify conv
* no need to pad before update, the update does it
* add accelerate hook decorator
* big cache refactor to have native implem
* small fix
* add static cache support
* make it compileable
* remove cache param completely
* do not transpose every time
* oupsi already had conversions
* remove the very suspicious dtype casting
* support sdpa
* fix bad reshape
* add flex support
* fix
* oupsi, pass layer_idx
* upstream TP fix
* big model fixes + conv1d update diff from qwen
* fix_compile
* TP transpose conversion is free
* Reuse HiggsAudioV2Embeddings for TmlAudioModelEmbeddings
The controlled port had inlined the dMel embeddings because the standalone-era
skeleton-modular had no cross-model inheritance. Now that skeleton-modular uses
inheritance again, restore TmlAudioModelEmbeddings(HiggsAudioV2Embeddings) + the
TmlAudioConfig num_codebooks/codebook_size/hidden_size attribute map. Behavior and
weight names unchanged (verified: logits + load identical).
* add kernels
* log scaling + image processing changes
* Ink it up
* add load_audio_backend
* allow to rollback linear cache layers
* make repo
* record past
* test audio utils
* audio/ image bos
* TmlProcessingIntegrationTest
* add TmlIntegrationTest
* nit
* nit
* add requires_backends
* can rollback prefill as well
* no rescale_image_frac
* avoid downcasting for linear attn
* add kernels + fix env vars for kernel disablement
* format
* fix save pretrained for Interleave
* better mappings
* more automappings
* change name
* implement mtp cache rollback and correction
* attempts at fixing MTP (noisy outputs)
* solve all indexing issues between main_model index and ntp layers
* small fix
* oupsi forgot to apply modular
* replace sliding layers for now
* allow sliding in mtp layers
* fix the masking for mtp layers
* fix cropping and a lot of issues by replacing get_seq_length to get offsets
* oupsi
* replace layers
* bucket to hf-internal-testing
* add test_apply_chat_template_audio_without_attention_mask
* fix mtp
* update test path
* improve cache invalidation a lot
* fix tests 1 / N
* update integration test
* update bucket path
* fix tests 2/N
* fix tests 3 / N
* fix tests 4/ N: Chunk dim 1
* fix fe edge case
* fix tests 5 / N
* update all mamba models
* upddate token ids
* slice them
* few missing
* [core] force contiguity on chunking
* fix tests 6 / N
* chat template testing 7 / N
* format + processor test fix 8 / N
* small fix
* skip tiny model offload fix 9 / N
* small fix
* conflict dropped the crochet
* style
---------
Co-authored-by: raushan <[email protected]>
Co-authored-by: Eustache Le Bihan <[email protected]>
Co-authored-by: Cyril Vallez <[email protected]>
Co-authored-by: eustlb <[email protected]>1 parent f5a67a4 commit 76613e6
65 files changed
Lines changed: 5789 additions & 310 deletions
File tree
- docs/source/en
- model_doc
- src/transformers
- generation
- integrations
- models
- auto
- bamba
- falcon_h1
- falcon_mamba
- granitemoehybrid
- inkling
- jamba
- lfm2_moe
- lfm2
- mamba2
- mamba
- nemotron_h
- qwen3_5_moe
- qwen3_5
- qwen3_next
- zamba2
- zamba
- zaya
- utils
- tests
- generation
- models
- falcon_mamba
- inkling
- mamba2
- mamba
- nemotron_h
- zaya
- utils
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1321 | 1321 | | |
1322 | 1322 | | |
1323 | 1323 | | |
| 1324 | + | |
| 1325 | + | |
1324 | 1326 | | |
1325 | 1327 | | |
1326 | 1328 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
35 | 35 | | |
36 | 36 | | |
37 | 37 | | |
| 38 | + | |
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
| |||
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
56 | 60 | | |
57 | 61 | | |
58 | 62 | | |
| |||
207 | 211 | | |
208 | 212 | | |
209 | 213 | | |
210 | | - | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
211 | 217 | | |
212 | 218 | | |
213 | 219 | | |
| |||
222 | 228 | | |
223 | 229 | | |
224 | 230 | | |
225 | | - | |
226 | | - | |
227 | | - | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
228 | 236 | | |
229 | | - | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
230 | 240 | | |
231 | 241 | | |
232 | 242 | | |
| |||
256 | 266 | | |
257 | 267 | | |
258 | 268 | | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
259 | 278 | | |
260 | 279 | | |
261 | 280 | | |
| |||
0 commit comments