Exportable kimi by IlyasMoutawwakil · Pull Request #47096 · huggingface/transformers

IlyasMoutawwakil · 2026-07-06T06:55:32Z

What does this PR do?

Fixes # (issue)

Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result,
we ask that new users do not submit pure code agent PRs at this time.
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously.

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result,
this policy is likely to be updated regularly in the near future. For more information, please read CONTRIBUTING.md.

I confirm that this is not a pure code agent PR.

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline and the
Pull Request checks?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes according to the guidelines?
Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

HuggingFaceDocBuilderDev · 2026-07-06T07:07:38Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Exporter support for Kimi-K2.5 (vision/audio attention registered with the reshaped vision-attention patch) plus: - Kimi vision rotary now emits the standard packed (seq, head_dim) cos/sin instead of the batched (1, seq, head_dim) LLM form, matching every other vision encoder (no exporter-side normalization needed). - ExecuTorch: materialise non-contiguous reshapes via a backend-local _patch_reshape rather than an unconditional clone in the shared vision patch, so ONNX/OpenVINO/dynamo graphs don't carry the copy. - is_multimodal short-circuits to False for non-PreTrainedModel inputs. - vision_utils.get_vision_cu_seqlens gains a merge_temporal option. Co-Authored-By: Claude Opus 4.8 <[email protected]>

vasqu

Just have a few smaller comments because the changes are honestly a bit hard to verify without integration tests cc @zucchini-nlp whenever you have time to check

vasqu · 2026-07-07T08:06:18Z

-    # (n_seg, heads, seg_len, dim) → (n_seg, seg_len, heads, dim) → (seq, heads*dim)
-    attn_output = attn_output.transpose(1, 2).reshape(seq_length, -1).contiguous()
+    # (n_seg, heads, seg_len, dim) → (n_seg, seg_len, heads, dim) → (seq, heads*dim).
+    attn_output = attn_output.transpose(1, 2).reshape(seq_length, -1)


hmm why remove?

not needed in the default (dynamo export) path, if a backend needs it it should handle it on its own (see patch_reshape)

vasqu · 2026-07-07T08:07:30Z



+@register_patch("executorch", "torch.reshape", "torch.Tensor.reshape")
+def _patch_reshape(original):


hmm ok so here we have the issue indirectly re contiguous?

the issue was always in executorch, imo this is more direct 😆

vasqu · 2026-07-07T08:16:13Z

+    A non-`PreTrainedModel` (e.g. a bare `nn.Module`) has no canonical `get_encoder`/`get_decoder`
+    accessors and is trivially not multi-modal, so it short-circuits to `False`.
+    """
+    return isinstance(model, PreTrainedModel) and bool(_find_multimodal_submodules(model))


just fmi where do we suddenly call on a normal nn module?

not suddenly tbh, this happens during testing and i didn't catch it before

vasqu · 2026-07-07T08:20:53Z

+        time_table = torch.cat(
+            [self.time_position_embeddings.new_zeros(1, self.dim), self.time_position_embeddings.squeeze(1)]
+        )
+        pos = pos + time_table[get_vision_frame_index(grid_thw, kwargs=kwargs)]


is this not quite expensive?

hmm yes could be, it's part of vectorizing the old per-image grid_thw.tolist() loop (untraceable for export)
the zero-pad table replaced the old if t > 1 branch. you are right we can avoid rebuilding it per forward !
moved the pad into the buffer at init, so forward is now just a gather.

vasqu · 2026-07-07T08:21:42Z

 class Kimi_K25VisionRotaryEmbedding(Gemma4VisionRotaryEmbedding):
    def forward(self, x, position_ids):
-        position_ids_expanded = position_ids.permute(1, 2, 0)[..., None].float()  # shape (bs, positions, 2, 1)
+        position_ids_expanded = position_ids.transpose(0, 1)[..., None].float()  # (positions, 2, 1)


what happened to the batch dim?

the extra (batch) dim was always size 1 in this packed path, so carrying it through rope was pointless and it was squeezed out

zucchini-nlp · 2026-07-07T13:25:38Z

I'm off 9-15 July so will do it after I am back (unless someone wants to add tests before that) 😄

IlyasMoutawwakil · 2026-07-22T20:10:48Z

run-slow: kimi_k25

github-actions · 2026-07-22T20:15:15Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/kimi_k25"]
quantizations: []

github-actions · 2026-07-22T20:25:04Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	785c9ca0	workflow commit (merge commit)
PR	a86222ca	branch commit (from PR)
main	bb3ffb97	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

github-actions · 2026-07-22T21:49:46Z

[For maintainers] Suggested jobs to run (before merge)

run-slow: kimi_k25

github-actions · 2026-07-22T21:57:54Z

CI recap

Dashboard: View test results in Grafana
Latest run: 29960366841:1
Result: success | Jobs: 2 | Tests: 24 | Failures: 0 | Duration: 2m 4s

vasqu

Just some smaller comments but overll looks good

vasqu · 2026-07-23T16:48:13Z

+    inputs["cu_seqlens"] = get_vision_cu_seqlens(grid_thw, merge_temporal=temporal_encoder)
+    inputs["max_seqlen"] = get_max_seqlen(inputs["cu_seqlens"], model.config, kwargs=inputs)


Why not use get_vision_attention_seqlens as before?

vasqu · 2026-07-23T16:52:11Z

        self.final_layernorm = nn.LayerNorm(config.hidden_size, eps=1e-05)
        self.post_init()

-    def temporal_patch_merger(


Technically slightly breaking can we keep BC for a bit at least?

vasqu · 2026-07-23T16:54:01Z

-        )
-
-        cu_seqlens = lengths.cumsum(dim=0, dtype=torch.int32)
+        cu_seqlens = get_vision_cu_seqlens(grid_thw, merge_temporal=True, kwargs=kwargs)


Can apply our fused version for max seq len together

vasqu · 2026-07-23T16:56:25Z

General comment and not on you: But we do introduce some very model specific workarounds. I feel like it would make more and more sense to start moving things into a collator / processor that handles these

vasqu · 2026-07-23T16:57:30Z

Also cc @zucchini-nlp if you want to take a look

IlyasMoutawwakil commented Jul 6, 2026

View reviewed changes

Comment thread src/transformers/exporters/exporter_dynamo.py Outdated

IlyasMoutawwakil force-pushed the exportable-kimi branch from c0cd540 to 7f0d32e Compare July 6, 2026 19:47

IlyasMoutawwakil marked this pull request as ready for review July 6, 2026 19:48

vasqu reviewed Jul 7, 2026

View reviewed changes

fix

bc3524f

zucchini-nlp mentioned this pull request Jul 17, 2026

[Kimi] add integration tests #47383

Merged

IlyasMoutawwakil added 2 commits July 22, 2026 17:37

Merge branch 'main' into exportable-kimi

cf1c752

Merge branch 'main' into exportable-kimi

a86222c

Merge branch 'main' into exportable-kimi

6a675a5

IlyasMoutawwakil requested a review from vasqu July 23, 2026 14:04

vasqu approved these changes Jul 23, 2026

View reviewed changes



		@register_patch("executorch", "torch.reshape", "torch.Tensor.reshape")
		def _patch_reshape(original):

		inputs["cu_seqlens"] = get_vision_cu_seqlens(grid_thw, merge_temporal=temporal_encoder)
		inputs["max_seqlen"] = get_max_seqlen(inputs["cu_seqlens"], model.config, kwargs=inputs)

Uh oh!

Conversation

IlyasMoutawwakil commented Jul 6, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Code Agent Policy

Before submitting

Who can review?

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Jul 6, 2026

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Jul 7, 2026

Uh oh!

IlyasMoutawwakil commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

CI Results

Commit Info

Uh oh!

github-actions Bot commented Jul 22, 2026

Uh oh!

github-actions Bot commented Jul 22, 2026

CI recap

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vasqu commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

IlyasMoutawwakil commented Jul 6, 2026 •

edited by github-actions Bot

Loading