Fix hybrid attention sliding window for Gemma (#320)#338
Merged
HenryNdubuaku merged 5 commits intocactus-compute:mainfrom Feb 12, 2026
Merged
Fix hybrid attention sliding window for Gemma (#320)#338HenryNdubuaku merged 5 commits intocactus-compute:mainfrom
HenryNdubuaku merged 5 commits intocactus-compute:mainfrom
Conversation
Signed-off-by: Jisha Rajala <[email protected]>
Signed-off-by: Jisha Rajala <[email protected]>
Signed-off-by: Jisha Rajala <[email protected]>
jrajala6
force-pushed
the
gemma_hybrid_attention
branch
from
1ac7c56 to
c0a7973
Compare
Signed-off-by: Jisha Rajala <[email protected]>
Signed-off-by: HenryNdubuaku <[email protected]>
Collaborator
|
thanks a million for this @jrajala6 |
ncylich
pushed a commit
that referenced
this pull request
Feb 24, 2026
* Added hybrid attention functionality to gemma Signed-off-by: Jisha Rajala <[email protected]> * Reduced diff in kernel_attention Signed-off-by: Jisha Rajala <[email protected]> * Reduced diffs in kernel_attention.cpp Signed-off-by: Jisha Rajala <[email protected]> * Remove int4_learning_lab.cpp from PR Signed-off-by: Jisha Rajala <[email protected]> * some changes Signed-off-by: HenryNdubuaku <[email protected]> --------- Signed-off-by: Jisha Rajala <[email protected]> Signed-off-by: HenryNdubuaku <[email protected]> Co-authored-by: HenryNdubuaku <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The hybrid INT8 attention kernel (cactus_attention_hybrid_int8_fp16) wasn't using window_size during decode, so local attention layers in Gemma attended to the full context instead of just recent tokens, breaking the hybrid attention mechanism.
This pr threads window_size from model_gemma.cpp through the graph API down to the kernel. In the kernel, positions outside the window get masked to -inf before softmax. The parameter defaults to 0 (global attention) so nothing changes for other models.
Tested with test_kernel (10/10), test_graph (38/38), and end-to-end chat with Gemma 3 1B.