Skip to content

[FA2] Keep split-KV block steps and accumulator offsets in 64 bits - #3101

Merged
Johnsonms merged 1 commit into
Dao-AILab:mainfrom
0z5a:fa2-splitkv-int64-offsets
Nov 9, 2026
Merged

Johnsonms merged 1 commit into
Dao-AILab:mainfrom
0z5a:fa2-splitkv-int64-offsets

Conversation

@0z5a

@0z5a 0z5a commented Oct 31, 2026

Copy link
Copy Markdown
Contributor

Fixes #3096.

The FA2 split-KV kernel stepped K, V and the appended K/V back one block with -int(kBlockN * row_stride). A row stride of 2^31 / kBlockN or more wraps, which is 2^25 elements at head dim 256. Every block after the first is then read from the wrong address.

Three other products in the same kernel family were int until assignment:

  • the out_accum offset wraps for any split whose offset, about split_idx * b * h * seqlen_q * d_rounded, reaches 2^31;
  • the combine kernel's per-split step (b * h * seqlen_q * d_rounded) and its out_accum row offsets wrap once that product passes 2^31;
  • lse_accum shares the out_accum sub-expression and is widened with it.

This PR widens the first operand of each product, as #2260 did for dq_accum.

Testing (GH200, sm_90 build of main 4616f51 and of this PR)

Test main this PR
test_flash_attn_kvcache_large_row_stride (k_cache, v_cache, k, v; row stride 2^25 + 128, num_splits=1) 4/4 illegal memory access 4/4 pass
test_flash_attn_kvcache_splitkv_large_accum (16 splits, offsets of splits 13–15 past 2^31) illegal memory access pass
test_flash_attn_kvcache_splitkv_large_accum (2 splits, b*h*seqlen_q*d = 2.28e9) illegal memory access pass
300 random cases of test_flash_attn_splitkv and test_flash_attn_kvcache — 139 passed, 161 skipped

The new tests skip when the GPU has less free memory than they need: 12 GiB for the row-stride test, and 16 or 44 GiB for the accumulator tests.

The non-split LSE offsets would need b * h * seqlen_q >= 2^31 and are not changed.

The repro script from #3096 also fails on main on GH200 with cudaErrorInvalidAddressSpace: K at row strides 2^25 + 128 and 2^26 + 128, and V at 2^25 + 128. With this PR, K at both strides matches the reference (max abs error 0.0012).

The split-KV kernel stepped K, V and the appended K/V back one block with
-int(kBlockN * row_stride), so a row stride of 2**31 / kBlockN or more
(2**25 elements at head dim 256) wrapped and every block after the first
was read from the wrong address.

The out_accum offset was an int product widened only after the
multiplication, so it wrapped for any split whose offset, about
split_idx * b * h * seqlen_q * d_rounded, reached 2**31. The combine
kernel's per-split step (b * h * seqlen_q * d_rounded) and its out_accum
row offsets wrapped once that product passed 2**31. lse_accum shares the
out_accum sub-expression and is widened with it.

Widen the first operand of each product, as Dao-AILab#2260 did for dq_accum, and
add regression tests for both cases.

Fixes Dao-AILab#3096

Signed-off-by: 0z5a <[email protected]>
@0z5a

0z5a commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

@drisspg @Johnsonms could one of you review this? It's the FA2 split-KV counterpart of int32 fixes you've already landed:

The change only casts the first operand of each product to 64 bits in flash_fwd_kernel.h (the eight K/V block steps, the out_accum/lse_accum offsets, the combine kernel's steps). No control flow changes. On GH200 the new tests hit an illegal memory access on main and pass with this PR; 300 random test_flash_attn_splitkv/test_flash_attn_kvcache cases still pass.

On merge order: #3004, #3093, #3040, #2851 and #2961 (split-KV early exit, LSE offsets, combine scales) each touch lines this PR also changes. This PR is purely mechanical, so it's the easiest one to rebase: I'll rebase it onto whichever of them goes in first, and I can rerun the GH200 tests on any revision.

@Johnsonms Johnsonms left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on B200 with sm_100 builds of the base 4616f511 and this PR.

  • The issue's repro faults on the base at a 2^25+128 row stride (cudaErrorInvalidAddressSpace, as on GH200) and matches the dense result (max abs error 0.0012) with this PR, for k_cache and v_cache at 2^25±128 and 2^26+128.
  • The six new tests: 6/6 illegal memory access on the base, 6/6 pass here; 300 random test_flash_attn_splitkv / test_flash_attn_kvcache ids: 139 passed, 161 skipped, 0 failed.
  • Every hunk is a first-operand cast or the same product in 64 bits; the -int(kBlockN * rotary_dim / 2) steps that remain are ≤ 8192. Not for this PR: flash_bwd_kernel.h has the same -int(kBlockM * *_row_stride) back-steps (8 places), the same class of bug for huge row strides.

@Johnsonms
Johnsonms merged commit 478b5b0 into Dao-AILab:main Nov 9, 2026
ll2088 added a commit to ll2088/sglang that referenced this pull request Nov 10, 2026
…tests

Ported from Dao-AILab/flash-attention#3101
Generated by autojump (AI); review before merging.
Johnsonms pushed a commit that referenced this pull request Nov 15, 2026
* [FA2] Keep the backward softmax_lse and softmax_d offsets in 64 bits

row_offset_lse and row_offset_dpsum were computed in int and only then
widened to index_t. Once b * h * seqlen_q_rounded (or, with cu_seqlens_q,
h * (total_q + 128 * b)) reaches 2^31 the offset wraps negative:
compute_dot_do_o stores its sums before the start of softmax_d and the
backward kernel reads softmax_lse and softmax_d from there.

Cast the first operand to index_t, as #2260 did for dq_accum and #3101
for the forward split-KV accumulators.

Related: #3162

* [FA2] Keep the forward softmax_lse offsets in 64 bits

get_lse_tile built the softmax_lse layout with int strides, so the offset
of (bidb, bidh) was computed in int. Once b * h * seqlen_q (or, with
cu_seqlens_q, h * total_q) reaches 2^31 it wraps negative and the forward
kernel stores LSE before the start of softmax_lse (illegal memory access).
The split-KV kernel, which flash_attn_varlen_func runs when given a
block_table, did the same with bidh * total_q in row_offset_lseaccum.

Use index_t strides in get_lse_tile and cast the first operand in
row_offset_lseaccum.

Related: #3162

* [FA2] Keep the forward attention-probs offset in 64 bits

row_offset_p (the attention probabilities returned with dropout_p > 0
and return_attn_probs=True) was computed in int and only then widened
to index_t. Once b * h * seqlen_q_rounded * seqlen_k_rounded reaches
2^31 the offset wraps negative and the kernel writes before the start
of the buffer, e.g. b = 9, h = 16, seqlen_q = seqlen_k = 4096.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] [FA2] 32-bit offsets overflow: the LSE, the split-KV accumulators, the split-KV key/value step, the combine kernel's rows

2 participants