[Executor] Refactor GetBlockShapeAndSplitKVBlock Kernel #2989
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thanks for your contribution! |
gongshaotian
force-pushed
the
refactor_get_block_op
branch
2 times, most recently
from
d589fb7 to
7226338
Compare
gongshaotian
commented
Jul 29, 2025
gongshaotian
commented
Jul 29, 2025
fastdeploy/model_executor/layers/attention/append_attn_backend.py
Outdated
Show resolved
Hide resolved
gongshaotian
commented
Jul 29, 2025
gongshaotian
commented
Jul 29, 2025
Comment on lines
+86
to
+87
| encoder_block_shape_q: int = -1, | ||
| decoder_block_shape_q: int = -1, |
Collaborator
Author
There was a problem hiding this comment.
这俩参数不是所有 backend 共有的，是不是放 init metadata 里比较好？
gongshaotian
changed the title
gongshaotian
changed the title
gongshaotian
changed the title
gongshaotian
commented
Jul 30, 2025
Comment on lines
+194
to
+196
| encoder_block_shape_q = 64 | ||
| decoder_block_shape_q = 16 |
freeliuzc
reviewed
Jul 30, 2025
ming1753
reviewed
Jul 30, 2025
| }) | ||
| .Outputs({ | ||
| paddle::Optional("encoder_batch_ids"), | ||
| paddle::Optional("encoder_tile_ids_per_batch"), |
Collaborator
There was a problem hiding this comment.
这里的Optional如果对性能不大的话，就去掉返回空tensor呢
Collaborator
Author
There was a problem hiding this comment.
这里的Optional如果对性能不大的话，就去掉返回空tensor呢
encoder的这三个tensor 在纯解码阶段返回的是 shape 为 0 的 tensor，Mix 或纯Prefill 阶段返回非空Tensor是符合预期的。是想也固定下Return的shape吗？
yuanlehome
approved these changes
Jul 30, 2025
gongshaotian
force-pushed
the
refactor_get_block_op
branch
from
f076e56 to
4ffa462
Compare
gongshaotian
commented
Jul 31, 2025
K11OntheBoat
reviewed
Aug 1, 2025
| # MLA | ||
| metadata.max_enc_len_this_time = metadata.set_max_lengths[1] | ||
| metadata.max_dec_len_this_time = metadata.set_max_lengths[2] | ||
| forward_meta.max_enc_len_this_time = metadata.set_max_lengths[1] |
Collaborator
There was a problem hiding this comment.
deepseek 组网那会使用 forward_meta.max_dec_len_this_time 来判断prefill和decode. 这里直接删除后，组网那里会有问题
This was referenced Aug 1, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
GetBlockShapeAndSplitKVBlockis an operator in pre-processing that calculates the input of certain types ofAttentionBackend. In previous implementations, these inputs were not directly managed by the model runner。The root cause is the unreasonable boundary division betweenForwardMetaandAttentionMetaData, which resulted in two issues:Memcpyrequired to implement CudaGraph functionality, which leads to low performanceCurrent PR only addresses the second issue. The scope of modification includes:
GetBlockShapeAndSplitKVBlockkernelContrast
When the Batch Size of Erine-21B Model is 64, in some cases a single step can be reduced by 2ms. The larger the Batch Size, the higher the acceleration ratio.