Skip to content

Commit 0eb5e03

Browse files
committed
Fix sdpa prefill with position_bias (#47359)
* slice position_bias * fix
1 parent 2fa8c19 commit 0eb5e03

1 file changed

Lines changed: 3 additions & 0 deletions

File tree

‎src/transformers/integrations/sdpa_attention.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,9 @@ def sdpa_attention_forward(
141141
if is_causal and attention_mask is None and q_length > 1 and kv_length > q_length:
142142
key = key[:, :, :q_length, :]
143143
value = value[:, :, :q_length, :]
144+
# If we have a position_bias, we need to crop it as well (on last dim, which is the kv seq_len dim)
145+
if position_bias is not None:
146+
position_bias = position_bias[:, :, :, :q_length]
144147

145148
# If we have a position_bias, create the correct floating-point mask by combining it with the existing mask, or a causal mask
146149
# if `is_causal=True`

0 commit comments

Comments
 (0)