Add Inkling model#47347
Merged
Merged
Conversation
…ddition-tm into skeleton-modular
…ddition-tm into skeleton-modular
ArthurZucker
approved these changes
Jul 15, 2026
Contributor
|
[For maintainers] Suggested jobs to run (before merge) run-slow: auto, bamba, falcon_h1, falcon_mamba, granitemoehybrid, inkling, jamba, lfm2, lfm2_moe, mamba, mamba2, nemotron_h, qwen3_5, qwen3_5_moe, qwen3_next, zamba |
Contributor
CI recapDashboard: View test results in Grafana
|
ArthurZucker
pushed a commit
that referenced
this pull request
Jul 15, 2026
* processor very rough draft * whatever claude says for now, clean next * mm towers * . * import base model * make it init model from scratch * conversion maybe * vision loads * tiny change * add normalization * more nits * more * oops * typo * copy linear stuff from qwen, seems quite similar * a bit more * relative position bias draft * no act * dtype force fp32 to match as closely as we can * make it run * torchaudio dep doesn't work for me, just skip for now * moe loads kinda * oke now the keys match * moe router fix * . * hmm, I think mlp has to be deinterlaved not chunked * i am defi breaking the model again XD * same for routed experts mlp * dont shard shared gammas ig * well we still need cache for full vs slliding * this is ugly! * ig? * delete * fix norm * fix normer * tied + inits * sync modular * Add TML audio support (feature extractor, processor, audio tower wiring) Ports the dMel audio pipeline onto the synced skeleton-modular base: - feature extractor returns the log-mel spectrogram + mask - processor quantizes it into audio_input_ids (dMel tokens) + mask - TmlAudioModelEmbeddings/TmlAudioModel embed the dMel tokens - get_audio_features + generalized get_placeholder_mask wire audio into the forward - conversion_mapping renames audio.encoder -> embed_audio_tokens, final_norm -> norm * parallelism wars * fix position_bias -> use cache primitives directly * do not slice the mask * position_ids * re add some inheritance * init global scales at 1: super important if we want to train correctly * inherit more * simplify conv * no need to pad before update, the update does it * add accelerate hook decorator * big cache refactor to have native implem * small fix * add static cache support * make it compileable * remove cache param completely * do not transpose every time * oupsi already had conversions * remove the very suspicious dtype casting * support sdpa * fix bad reshape * add flex support * fix * oupsi, pass layer_idx * upstream TP fix * big model fixes + conv1d update diff from qwen * fix_compile * TP transpose conversion is free * Reuse HiggsAudioV2Embeddings for TmlAudioModelEmbeddings The controlled port had inlined the dMel embeddings because the standalone-era skeleton-modular had no cross-model inheritance. Now that skeleton-modular uses inheritance again, restore TmlAudioModelEmbeddings(HiggsAudioV2Embeddings) + the TmlAudioConfig num_codebooks/codebook_size/hidden_size attribute map. Behavior and weight names unchanged (verified: logits + load identical). * add kernels * log scaling + image processing changes * Ink it up * add load_audio_backend * allow to rollback linear cache layers * make repo * record past * test audio utils * audio/ image bos * TmlProcessingIntegrationTest * add TmlIntegrationTest * nit * nit * add requires_backends * can rollback prefill as well * no rescale_image_frac * avoid downcasting for linear attn * add kernels + fix env vars for kernel disablement * format * fix save pretrained for Interleave * better mappings * more automappings * change name * implement mtp cache rollback and correction * attempts at fixing MTP (noisy outputs) * solve all indexing issues between main_model index and ntp layers * small fix * oupsi forgot to apply modular * replace sliding layers for now * allow sliding in mtp layers * fix the masking for mtp layers * fix cropping and a lot of issues by replacing get_seq_length to get offsets * oupsi * replace layers * bucket to hf-internal-testing * add test_apply_chat_template_audio_without_attention_mask * fix mtp * update test path * improve cache invalidation a lot * fix tests 1 / N * update integration test * update bucket path * fix tests 2/N * fix tests 3 / N * fix tests 4/ N: Chunk dim 1 * fix fe edge case * fix tests 5 / N * update all mamba models * upddate token ids * slice them * few missing * [core] force contiguity on chunking * fix tests 6 / N * chat template testing 7 / N * format + processor test fix 8 / N * small fix * skip tiny model offload fix 9 / N * small fix * conflict dropped the crochet * style --------- Co-authored-by: raushan <[email protected]> Co-authored-by: Eustache Le Bihan <[email protected]> Co-authored-by: Cyril Vallez <[email protected]> Co-authored-by: eustlb <[email protected]>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
nvpohanh
reviewed
Jul 20, 2026
| hidden_states = hidden_states[:, :, -seq_len:] | ||
|
|
||
| hidden_states = hidden_states.transpose(1, 2) | ||
| hidden_states = (hidden_states + residual).to(dtype=input_dtype) |
Contributor
There was a problem hiding this comment.
is this correct? It seems that we add residual once here but another time in InklingDecoderLayer's forward().
Contributor
There was a problem hiding this comment.
@molbap @ArthurZucker could you help us to check if this is intended? Are there really two residual adds per attn-block and per-mlp/moe-block?
nvpohanh
reviewed
Jul 20, 2026
| relative_states = relative_states.view(*input_shape, self.num_heads, -1) | ||
| position_bias = self.rel_logits_proj(relative_states, q_positions, kv_positions) | ||
|
|
||
| # original impl applies log scalnig in f32 |
Contributor
There was a problem hiding this comment.
nit: spelling error: scalnig -> scaling
This was referenced Jul 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Add new model from Thinking Machines!