Compile time option to use bf16 for quants without MMQ kernels #261
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Awesome! Will re-quant over night and test tomorrow! |
Collaborator
In case you still have the old quants, you can just use those with the new code you don't have to make new quants. |
|
Unfortunately I don’t! My cache drive is limited so I tend to delete pretty soon. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
IQ2_KS, IQ2_K, ..., IQ6_Kquantization types do not have MMQ kernels, so matrix multiplications for model weights quantized with these types are done via dequantization tofp16andcublasGemmExGEMM usingfp16precision. For the DeepSeek series of MoE models this leads to NaNs.Ideally I should add MMQ kernels for these quantization types. But for now, the PR provides a quick fix: dequantize to
bf16and usebf16cuBLAS GEMM. This is added as a compile time option enabled via(or, if you like me prefer using
ccmake, after pulling the PR,cmake .. && ccmake ., and then set theGGML_CUDA_IQK_FORCE_BF16toON).I have tested with DeepSeek-Lite quantized with
IQ4_KSSandIQ4_K. In both cases I get NaNs when runningperplexityon the main branch. Turning on theGGML_CUDA_IQK_FORCE_BF16option provided by this PR results in meaningful PPL values.@davidsyoung This should solve the issues with the
IQ4_KSSDeepSeek-R1 model you created.