Skip to content

Support loading Qwen3 MoE GGUF #39638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
SunMarc merged 7 commits into from
Jul 29, 2025
Merged

Support loading Qwen3 MoE GGUF #39638

SunMarc merged 7 commits into from
Jul 29, 2025

Conversation

@ctcanbol
Copy link
Contributor

@ctcanbol ctcanbol commented Jul 24, 2025 •
edited
Loading

What does this PR do?

Currently, GGUF versions of Qwen3 MoE models raises "GGUF model with architecture qwen3moe is not supported yet" error. This PR resolves this issue.

Fixes #39721

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline,
    Pull Request section?
  • Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @SunMarc @MekkCyber

@ctcanbol
Copy link
Contributor Author

ctcanbol commented Jul 27, 2025

Ready for review

@ctcanbol ctcanbol mentioned this pull request Jul 28, 2025
Closed
@ArthurZucker ArthurZucker requested a review from SunMarc July 28, 2025 11:26
MekkCyber
MekkCyber reviewed Jul 28, 2025
View reviewed changes
Copy link
Contributor

@MekkCyber MekkCyber left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this! Left a question about the integration

@MekkCyber
Copy link
Contributor

MekkCyber commented Jul 29, 2025

run-slow: ggml

@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2025

This comment contains run-slow, running the specified jobs:

models: []
quantizations: ['quantization/ggml'] ...

@github-actions
Copy link
Contributor

github-actions bot commented Jul 29, 2025

[For maintainers] Suggested jobs to run (before merge)

run-slow: ggml

SunMarc
SunMarc approved these changes Jul 29, 2025
View reviewed changes
Copy link
Member

@SunMarc SunMarc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks !

@SunMarc SunMarc enabled auto-merge (squash) July 29, 2025 13:32
@SunMarc SunMarc merged commit fb141e2 into huggingface:main Jul 29, 2025
25 checks passed
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Jul 29, 2025

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ctcanbol ctcanbol mentioned this pull request Jul 29, 2025
Open
1 task
@Readon
Copy link

Readon commented Aug 4, 2025 •
edited
Loading

There are still errors that report

"ValueError: The checkpoint you are trying to load has model type qwen3moe but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date."
@ python3.11/site-packages/transformers/models/auto/configuration_auto.py, line 1271 in from_pretrained function.

When I run the Qwen3-235B-A22B-Thinking-2507-Q4_K_M.gguf file provided by unsloth.
The architecture of this file is qwen3moe which is different from qwen3_moe which is widely used in the configs.

@SunMarc
Copy link
Member

SunMarc commented Aug 5, 2025

There must have been a typo in this PR then. Would you like to fix this ?

@Readon
Copy link

Readon commented Aug 5, 2025

There must have been a typo in this PR then. Would you like to fix this ?

I do not find the evidence that a typo exist.
But in gguf world the architecture name is 'qwen3moe', on otherside, in huggingface world it is 'qwen3_moe'.

@ctcanbol
Copy link
Contributor Author

ctcanbol commented Aug 6, 2025

Any suggestions for the fix? There is a fundamental incompability imo.

@Readon
Copy link

Readon commented Aug 7, 2025

Any suggestions for the fix? There is a fundamental incompability imo.

I think the same problem exist when qwen2moe and qwen2_moe system.
I am not familiar with the architecture of transformers and vllm.

A patch might be in src\transformers\modeling_gguf_pytorch_utils.py file, like 392 line we could add qwen3moe related one.
Also, it could be add at line 447 to replace the model_type only, if I remember right this could work.

@ctcanbol ctcanbol mentioned this pull request Aug 7, 2025
Merged
5 tasks
@ctcanbol
Copy link
Contributor Author

ctcanbol commented Aug 7, 2025

#39976 fixed here. Anyone can test is appreciated.

@jusjinuk jusjinuk mentioned this pull request Aug 10, 2025
Merged
5 tasks
@Gh0u1L5 Gh0u1L5 mentioned this pull request Aug 13, 2025
Merged
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
8142c02 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
d4d4aa0 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
1f66826 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
3fd9148 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
ebdbca8 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
35f0165 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
zaristei pushed a commit to zaristei/transformers that referenced this pull request Sep 9, 2025
@ctcanbol @zaristei
Support loading Qwen3 MoE GGUF (huggingface#39638)
8c901f5 
* support loading qwen3 gguf

* qwen3moe test cases

* fix whitespaces

* fix ggml tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@MekkCyber MekkCyber MekkCyber left review comments

@SunMarc SunMarc SunMarc approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Support loading Qwen3 MoE GGUF

5 participants

@ctcanbol @MekkCyber @HuggingFaceDocBuilderDev @Readon @SunMarc