Repository navigation
[FA2] Keep split-KV block steps and accumulator offsets in 64 bits - #3101
Conversation
The split-KV kernel stepped K, V and the appended K/V back one block with -int(kBlockN * row_stride), so a row stride of 2**31 / kBlockN or more (2**25 elements at head dim 256) wrapped and every block after the first was read from the wrong address. The out_accum offset was an int product widened only after the multiplication, so it wrapped for any split whose offset, about split_idx * b * h * seqlen_q * d_rounded, reached 2**31. The combine kernel's per-split step (b * h * seqlen_q * d_rounded) and its out_accum row offsets wrapped once that product passed 2**31. lse_accum shares the out_accum sub-expression and is widened with it. Widen the first operand of each product, as Dao-AILab#2260 did for dq_accum, and add regression tests for both cases. Fixes Dao-AILab#3096 Signed-off-by: 0z5a <[email protected]>
|
@drisspg @Johnsonms could one of you review this? It's the FA2 split-KV counterpart of int32 fixes you've already landed:
The change only casts the first operand of each product to 64 bits in On merge order: #3004, #3093, #3040, #2851 and #2961 (split-KV early exit, LSE offsets, combine scales) each touch lines this PR also changes. This PR is purely mechanical, so it's the easiest one to rebase: I'll rebase it onto whichever of them goes in first, and I can rerun the GH200 tests on any revision. |
Johnsonms
left a comment
There was a problem hiding this comment.
Reviewed on B200 with sm_100 builds of the base 4616f511 and this PR.
- The issue's repro faults on the base at a 2^25+128 row stride (
cudaErrorInvalidAddressSpace, as on GH200) and matches the dense result (max abs error 0.0012) with this PR, fork_cacheandv_cacheat 2^25±128 and 2^26+128. - The six new tests: 6/6 illegal memory access on the base, 6/6 pass here; 300 random
test_flash_attn_splitkv/test_flash_attn_kvcacheids: 139 passed, 161 skipped, 0 failed. - Every hunk is a first-operand cast or the same product in 64 bits; the
-int(kBlockN * rotary_dim / 2)steps that remain are ≤ 8192. Not for this PR:flash_bwd_kernel.hhas the same-int(kBlockM * *_row_stride)back-steps (8 places), the same class of bug for huge row strides.
…tests Ported from Dao-AILab/flash-attention#3101 Generated by autojump (AI); review before merging.
* [FA2] Keep the backward softmax_lse and softmax_d offsets in 64 bits row_offset_lse and row_offset_dpsum were computed in int and only then widened to index_t. Once b * h * seqlen_q_rounded (or, with cu_seqlens_q, h * (total_q + 128 * b)) reaches 2^31 the offset wraps negative: compute_dot_do_o stores its sums before the start of softmax_d and the backward kernel reads softmax_lse and softmax_d from there. Cast the first operand to index_t, as #2260 did for dq_accum and #3101 for the forward split-KV accumulators. Related: #3162 * [FA2] Keep the forward softmax_lse offsets in 64 bits get_lse_tile built the softmax_lse layout with int strides, so the offset of (bidb, bidh) was computed in int. Once b * h * seqlen_q (or, with cu_seqlens_q, h * total_q) reaches 2^31 it wraps negative and the forward kernel stores LSE before the start of softmax_lse (illegal memory access). The split-KV kernel, which flash_attn_varlen_func runs when given a block_table, did the same with bidh * total_q in row_offset_lseaccum. Use index_t strides in get_lse_tile and cast the first operand in row_offset_lseaccum. Related: #3162 * [FA2] Keep the forward attention-probs offset in 64 bits row_offset_p (the attention probabilities returned with dropout_p > 0 and return_attn_probs=True) was computed in int and only then widened to index_t. Once b * h * seqlen_q_rounded * seqlen_k_rounded reaches 2^31 the offset wraps negative and the kernel writes before the start of the buffer, e.g. b = 9, h = 16, seqlen_q = seqlen_k = 4096.
Fixes #3096.
The FA2 split-KV kernel stepped K, V and the appended K/V back one block with
-int(kBlockN * row_stride). A row stride of2^31 / kBlockNor more wraps, which is 2^25 elements at head dim 256. Every block after the first is then read from the wrong address.Three other products in the same kernel family were
intuntil assignment:out_accumoffset wraps for any split whose offset, aboutsplit_idx * b * h * seqlen_q * d_rounded, reaches 2^31;b * h * seqlen_q * d_rounded) and itsout_accumrow offsets wrap once that product passes 2^31;lse_accumshares theout_accumsub-expression and is widened with it.This PR widens the first operand of each product, as #2260 did for
dq_accum.Testing (GH200, sm_90 build of main 4616f51 and of this PR)
test_flash_attn_kvcache_large_row_stride(k_cache, v_cache, k, v; row stride 2^25 + 128,num_splits=1)test_flash_attn_kvcache_splitkv_large_accum(16 splits, offsets of splits 13–15 past 2^31)test_flash_attn_kvcache_splitkv_large_accum(2 splits,b*h*seqlen_q*d= 2.28e9)test_flash_attn_splitkvandtest_flash_attn_kvcacheThe new tests skip when the GPU has less free memory than they need: 12 GiB for the row-stride test, and 16 or 44 GiB for the accumulator tests.
The non-split LSE offsets would need
b * h * seqlen_q >= 2^31and are not changed.The repro script from #3096 also fails on main on GH200 with
cudaErrorInvalidAddressSpace: K at row strides 2^25 + 128 and 2^26 + 128, and V at 2^25 + 128. With this PR, K at both strides matches the reference (max abs error 0.0012).