Repository navigation
[CUDA] FPxINT quantized matmul for Hopper - #3160
Merged
Merged
Conversation
angeloskath
approved these changes
Feb 24, 2026
angeloskath
left a comment
Member
There was a problem hiding this comment.
Looks good to me.
The performance seems a bit low ~2.1x vs >3x expected. Do you have a guess as to why?
Member
Author
|
Skimming the Machete code which achieve 3x for batch size 16/32, I think the performance difference mainly comes from a better designed weights layout. (Machete is modified from the cutlass kernel this PR uses.) |
jagrit06
approved these changes
Feb 24, 2026
jagrit06
left a comment
Member
There was a problem hiding this comment.
We should merge this! That said, for the sizes of M we have, we should also be reporting the memory bandwidths since flops isn't the metric we will be maxing out
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs #2536.
This PR implements QMM for Hopper only, using CUTLASS.
Performance of FP16xINT4 with group size 64 on H100:
Independent C++ code for profiling the kernel
For small batch size (<=16) the performance data is similar to (though slower than) Machete so I think it is doing good. When M becomes larger than 32 it can no longer surpass cuBLAS, and it can be solved by increasing tile size, which I will fix in following PRs.
Another problem is this kernel behaves badly for small N/K (0.5x for n4096k4096), I'm not sure if it is something TMA kernels are not good at and we might have to fallback to Ampere optimized kernels.
There are also a few things need further optimizations:
--shuffleswitch that can be used to measure the performance difference.