Skip to content

[QUANTIZATION] Add AWQ support #1703

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Qubitium merged 134 commits into from
Sep 18, 2025
Merged

[QUANTIZATION] Add AWQ support #1703

Qubitium merged 134 commits into from
Sep 18, 2025

Conversation

@ZX-ModelCloud
Copy link
Collaborator

@ZX-ModelCloud ZX-ModelCloud commented Aug 21, 2025

No description provided.

@ZX-ModelCloud ZX-ModelCloud marked this pull request as draft August 21, 2025 05:56
@Qubitium Qubitium changed the title support autoawq [QUANTIZATION[ Add AWQ support Aug 21, 2025
@Qubitium Qubitium changed the title [QUANTIZATION[ Add AWQ support [QUANTIZATION] Add AWQ support Aug 21, 2025
@Qubitium
Copy link
Collaborator

Qubitium commented Aug 21, 2025 •
edited
Loading

TODO:

  1. Use GPTQModel QuantConfig format. Only new zero_point property is unique to awq. version already perfectly maps to our format. Add zero_point as awq_zero_point property. Not AWQ quants will have thie awq_ prefixed peroperty stripped on quant save. So every quant format has a unique {format_name}_ prefix to store unique properties that does not polute or collide with shared properties.
  2. Merge the kernels and QuantLinear into GPTQModel structure.
  3. Add awq processes pipelines.
  4. Add CI tests.
  5. Finally add awq config to GPTQConfig runtime conversion so we can execute autoawq quantized models as well. Future models will use our config.
  6. Upstream changes to hf/sglang/vllm so they can recognize our config format (minimal code change).

@ZX-ModelCloud
config add "zero_point" field
6176f34 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
add awq kernels
6ee83ca 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
Merge remote-tracking branch 'origin/main' into zx_add_autoawq
7f422c4
@ZX-ModelCloud
add AWQuantLinear
9a62027 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
add awq_processor.py
9231b4c 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
loop_processor added pre_quantize(self, module: Module, device: torch…
72df6c6 
….device)

Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
fix init_quant()
558a773 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
cleanup
04236ce 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
Fixed the issue where _module_forward() was too slow to execute
1eebfbc 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
fix OOM
9062075 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
fix save AWQ quantized model
73f9f1f 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
AWQProcessor add log stats
3c35586 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
AWQProcessor add calculate_w_wq_diff
d5a2727 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
cleanup
b354012 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
cleanup
109c813 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
added awq code
d0e1c78 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
added AWQ format
110c7e0 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
select_quant_linear() added "quant_method" argument
ca05154 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
added AWQuantLinear_EXLLAMA
b1f23c5 
Signed-off-by: ZX-ModelCloud <[email protected]>
@benjamin-marie benjamin-marie mentioned this pull request Sep 11, 2025
Closed
@Qubitium Qubitium marked this pull request as ready for review September 17, 2025 08:28
LRL-ModelCloud and others added 24 commits September 17, 2025 16:39
fix
07e64fe
clean
c8698b8
update
b9fd151
update
31f38fc
update
893111f
@ZX-ModelCloud
Adjust the order of q/k/v and gate/up
d395e60 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
fix Wrong quant_method
5870a04 
Signed-off-by: ZX-ModelCloud <[email protected]>
@ZX-ModelCloud
add test_awq_moe.py
7585b36 
Signed-off-by: ZX-ModelCloud <[email protected]>
cleanup
a6c0ef8
cleanup
bd2c289
fix deepseekv2/v3
5eaa351
cleanup
ce47017
fix layer0
1a4937b
@ZX-ModelCloud
Fix FORMAT.GEMV
34ba292 
Signed-off-by: ZX-ModelCloud <[email protected]>
fix norm
9cdbea6
Merge branch 'zx_add_autoawq' of https://github.com/ModelCloud/GPTQModel
0e8f345 
 into zx_add_autoawq
fix norm
838e409
rename
d836de6
format
4ecf990
format
070ad22
rename
4b443f6
@ZX-ModelCloud
Fix FORMAT.GEMV_FAST
b4a0b54 
Signed-off-by: ZX-ModelCloud <[email protected]>
cleanup
03e6fb6
Merge remote-tracking branch 'main/zx_add_autoawq' into zx_add_autoawq
1167e3b
@Qubitium Qubitium merged commit 2d8f35f into main Sep 18, 2025
4 checks passed
@BenjaminBossan BenjaminBossan mentioned this pull request Nov 18, 2025
Open
@Qubitium Qubitium deleted the zx_add_autoawq branch November 18, 2025 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@ZX-ModelCloud @Qubitium @CSY-ModelCloud