support qk norm for append attn #3145
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
| metadata.kv_signal_data_list[layer.layer_id], | ||
| getattr(layer, "q_norm_weight", None), | ||
| getattr(layer, "k_norm_weight", None), | ||
| getattr(layer, "rms_norm_eps"), |
| const uint32_t elem_nums = | ||
| use_neox_style ? bsz * (num_heads + 2 * kv_num_heads) * dim_head / 2 | ||
| : bsz * (num_heads + 2 * kv_num_heads) * dim_head; | ||
| constexpr int HEAD_DIM = 128; |
Collaborator
There was a problem hiding this comment.
这里是不是需要加判断：不支持dim_head不等于128的
fastdeploy/utils.py
Outdated
| instance_key = (cls, frozenset(kwargs.items())) | ||
| if instance_key not in instances: | ||
| instances[instance_key] = cls(*args, **kwargs) | ||
| return instances[instance_key] |
Collaborator
There was a problem hiding this comment.
这里的修改不属于q k norm的范畴吧，而且目前ep engine的实现不需要修改这个
Collaborator
Author
There was a problem hiding this comment.
这里的修改不属于q k norm的范畴吧，而且目前ep engine的实现不需要修改这个
Done
rsmallblue
force-pushed
the
qk_norm
branch
4 times, most recently
from
b09f64e to
ee996e3
Compare
Collaborator
|
在attention layer那里添加一下注释说明下use_qk_norm做的是qk_norm after rope，开源社区的其他模型看起来都是qk_norm before rope，这个diff需要显式指出来 |
rsmallblue
force-pushed
the
qk_norm
branch
3 times, most recently
from
37b6618 to
ab37724
Compare
Collaborator
Author
done |
RichardWooSJTU
approved these changes
Aug 5, 2025
megemini
added a commit
to megemini/FastDeploy
that referenced
this pull request
Aug 6, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
在append attn 中支持在rope后对q、k进行rms norm的计算