Fix conversion for clip models by zucchini-nlp · Pull Request #46406 · huggingface/transformers

zucchini-nlp · 2026-06-04T09:48:05Z

What does this PR do?

Should be mapping by class name because the model-type is shared by other PreTrainedModels that do not need any conversion. In case of Sam3, it loads a ClipTextModelWithProjection that in the past manually unwrapped text_model.text_model

See:

transformers/src/transformers/models/clip/modeling_clip.py

Lines 951 to 957 in 08810b1

    
           text_model = CLIPTextModel._from_config(config) 
        
           self.text_model = text_model.text_model 
        
           self.text_projection = nn.Linear(config.hidden_size, config.projection_dim, bias=False) 
        
           # Initialize weights and apply final processing

zucchini-nlp · 2026-06-04T09:50:38Z

run-slow: clip, clipseg, altclip, siglip, siglip2, metaclip_2, chinese_clip, x_clip, llava, llava_next, llava_next_video, paligemma, gemma3, aya_vision, got_ocr2, internvl, vipllava, mistral3, pp_chart2table, video_llava, qwen2_vl, colqwen2, colpali

github-actions · 2026-06-04T09:52:03Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/altclip", "models/aya_vision", "models/chinese_clip", "models/clip", "models/clipseg", "models/colpali", "models/colqwen2", "models/gemma3", "models/got_ocr2", "models/internvl", "models/llava", "models/llava_next", "models/llava_next_video", "models/metaclip_2", "models/mistral3", "models/paligemma", "models/pp_chart2table", "models/qwen2_vl", "models/siglip", "models/siglip2", "models/video_llava", "models/vipllava", "models/x_clip"]
quantizations: []

zucchini-nlp · 2026-06-04T10:02:39Z

run-slow: clip, clipseg, altclip, siglip, siglip2, metaclip_2, chinese_clip, x_clip, llava, llava_next, llava_next_video, paligemma, gemma3, aya_vision, got_ocr2, internvl, vipllava, mistral3, pp_chart2table, video_llava, qwen2_vl, colqwen2, colpali

github-actions · 2026-06-04T10:03:29Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	2cbdf1bd	workflow commit (merge commit)
PR	e6b921a7	branch commit (from PR)
main	b07d99be	base commit (on `main`)

⚠️ No test being reported (jobs are skipped or cancelled)!

github-actions · 2026-06-04T10:04:42Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/altclip", "models/aya_vision", "models/chinese_clip", "models/clip", "models/clipseg", "models/colpali", "models/colqwen2", "models/gemma3", "models/got_ocr2", "models/internvl", "models/llava", "models/llava_next", "models/llava_next_video", "models/metaclip_2", "models/mistral3", "models/paligemma", "models/pp_chart2table", "models/qwen2_vl", "models/siglip", "models/siglip2", "models/video_llava", "models/vipllava", "models/x_clip"]
quantizations: []

ArthurZucker

can we add a test please of what was broken? 😢

github-actions · 2026-06-04T11:31:34Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	8e7f0e21	workflow commit (merge commit)
PR	a07e7c8a	branch commit (from PR)
main	b07d99be	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

zucchini-nlp · 2026-06-04T12:31:33Z

The whole Sam3ModelIntegrationTest was broken 😿 I don't know why we didn't see and fix it though, have to track down the line but prob it was that period when CI was down

zucchini-nlp · 2026-06-04T12:32:14Z

run-slow: sam3

github-actions · 2026-06-04T12:32:22Z

Workflow Run ⚙️💔 This comment contains run-slow, but unknown error occurred and the workflow run aborted!

zucchini-nlp · 2026-06-04T12:33:09Z

run-slow: sam3

github-actions · 2026-06-04T12:33:23Z

Workflow Run ⚙️💔 This comment contains run-slow, but unknown error occurred and the workflow run aborted!

github-actions · 2026-06-04T12:34:52Z

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/sam3"]
quantizations: []

HuggingFaceDocBuilderDev · 2026-06-04T12:45:23Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

github-actions · 2026-06-04T12:53:47Z

CI Results

Workflow Run ⚙️

Commit Info

Context	Commit	Description
RUN	eb56f79e	workflow commit (merge commit)
PR	2dd6bfee	branch commit (from PR)
main	03dbff6c	base commit (on `main`)

✅ No failing test specific to this PR 🎉 👏 !

zucchini-nlp · 2026-06-04T13:07:21Z

Integration tests are all green now in workflow (and failing in today's nightly CI):

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_efficient_multi_prompt_single_image PASSED [ 83%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_efficient_multi_prompt_single_image [PASSED] 11.17s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_efficient_single_prompt_multi_images PASSED [ 84%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_efficient_single_prompt_multi_images [PASSED] 5.24s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_batched_images PASSED [ 84%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_batched_images [PASSED] 3.82s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_batched_mixed_prompts PASSED [ 84%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_batched_mixed_prompts [PASSED] 3.88s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_combined_prompts PASSED [ 84%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_combined_prompts [PASSED] 3.14s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_multi_box_prompt PASSED [ 85%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_multi_box_prompt [PASSED] 3.17s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_single_box_prompt PASSED [ 85%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_single_box_prompt [PASSED] 3.12s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_text_prompt_only PASSED [ 85%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_inference_text_prompt_only [PASSED] 3.12s

tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_semantic_segmentation_output PASSED [ 85%]tests/models/sam3/test_modeling_sam3.py::Sam3ModelIntegrationTest::test_semantic_segmentation_output [PASSED] 3.09s

vasqu

I know run-slow didnt say any new failures for the affected models but I think we need to manually check the failures 😢 just in case we missed them similarly to sam. seeing this whole dependency tree makes me think we should be extra careful

This is a careful approval because we need to fix and it seems to work (first glance).

zucchini-nlp · 2026-06-04T13:41:26Z

Future note: we might need a dep-tree of conversion mapping so easily get possible affected model lists. This is getting out-of-hand as we add more models to it

github-actions · 2026-06-04T14:17:22Z

CI Dashboard: View test results in Grafana

* fix it * oops, unrelated diff * wtf * just how?

fix it

e6b921a

zucchini-nlp added the for patch Tag issues / labels that should be included in the next patch label Jun 4, 2026

zucchini-nlp added 3 commits June 4, 2026 11:55

oops, unrelated diff

6ada7c7

wtf

73724f5

just how?

a07e7c8

ArthurZucker reviewed Jun 4, 2026

View reviewed changes

Merge branch 'main' into conversion-fix-clip

2dd6bfe

zucchini-nlp requested a review from ArthurZucker June 4, 2026 13:09

vasqu approved these changes Jun 4, 2026

View reviewed changes

Merge branch 'main' into conversion-fix-clip

5a8a6fc

zucchini-nlp enabled auto-merge June 4, 2026 14:09

zucchini-nlp added this pull request to the merge queue Jun 4, 2026

Merged via the queue into huggingface:main with commit bb1cfda Jun 4, 2026
117 of 118 checks passed

zucchini-nlp deleted the conversion-fix-clip branch June 4, 2026 14:37

vasqu pushed a commit that referenced this pull request Jun 4, 2026

Fix conversion for clip models (#46406)

8a4ffee

* fix it * oops, unrelated diff * wtf * just how?


	text_model = CLIPTextModel._from_config(config)
	self.text_model = text_model.text_model

	self.text_projection = nn.Linear(config.hidden_size, config.projection_dim, bias=False)

	# Initialize weights and apply final processing

Conversation

zucchini-nlp commented Jun 4, 2026

What does this PR do?

Uh oh!

zucchini-nlp commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

zucchini-nlp commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

CI Results

Commit Info

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 4, 2026

CI Results

Commit Info

Uh oh!

zucchini-nlp commented Jun 4, 2026

Uh oh!

zucchini-nlp commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

zucchini-nlp commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

HuggingFaceDocBuilderDev commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

CI Results

Commit Info

Uh oh!

zucchini-nlp commented Jun 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vasqu left a comment

Choose a reason for hiding this comment

Uh oh!

zucchini-nlp commented Jun 4, 2026

Uh oh!

github-actions Bot commented Jun 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

zucchini-nlp commented Jun 4, 2026 •

edited

Loading