Skip to content

Commit 85e10f5

Browse files
yyihuangaveryhNVclaude
authored
fix(cake): gate NVFP4 context programs on CUDA 13.4 and fix two Cake test expectations (#7632)
## Summary This PR fixes three test failures from the scheduled nightly runs of the test suite. It changes no kernel sources: - On toolkits older than CUDA 13.4, NVFP4 context attention programs now fail closed instead of failing in ptxas. - An SVDQuant tail-split test now checks the actual schedule rule. - The warp-decode PDL test now reads graph edges without a cuda-bindings use-after-free. ## Changes **NVFP4 context programs require CUDA 13.4** (`flashinfer/cake_fmha.py`, `tests/attention/test_cake_fmha_context_programs.py`) - The NVFP4 programs under `csrc/cake_fmha_context_nvfp4/` (Q511 and Q511_SCALE_PTR, SM100A/SM103A) dequantize E2M1 pages with the PTX ISA 9.4 `mul.rn.satfinite.e4m3x4.e2m1x4.e4m3x4`. Only CUDA 13.4+ supports this instruction. With CUDA 12.9 and 13.0, ptxas rejects it (`Illegal modifier '.satfinite' for instruction 'mul'`). This failed `test_cake_context_quantized_programs_match_reference[case2-Q511]`, `[case3-Q511_SCALE_PTR]` and the six `test_cake_context_programs_match_the_jit_route_bitwise[nvfp4_*]` rows on B200, GB200 and GB300. CUDA 13.4 passed. - Add `CONTEXT_NVFP4_PROGRAMS_MIN_CUDA = "13.4"` and `context_nvfp4_programs_unavailable()`, which returns a reason string or `None` via `is_cuda_version_at_least`. - On an older toolkit, `get_cake_fmha_context_module` routes an NVFP4 program route to compat_v1 with a `RuntimeWarning` ("failed closed to compat_v1"), as the NVFP4 JIT route already does. `_load_cake_fmha_context_program` raises a `RuntimeError` naming the program and the reason instead of launching a build that ptxas rejects. `flashinfer/jit/cake_dsv4.py` already requires CUDA 13.4 for the same instruction in the DSv4 NVFP4 decode kernels. - Tests: on such toolkits, `_skip_without_nvfp4_programs()` skips the NVFP4 rows and states the reason. The new `test_cake_context_nvfp4_programs_fail_closed_without_their_toolkit` monkeypatches the check. It asserts that the loader raises and that the route falls back to compat_v1 with the warning. **SVDQuant tail-split routing test** (`tests/gemm/test_cake_nvfp4_svdquant_gemm_backend.py`) - `test_tailsplit_rows_split_on_this_device` required every `pair_tailsplit` row to split its tail (`split == 2 and full < units`) on 148/152-SM devices. Since #7516, the row M 6889, N 12288, K 3072, rank 32, bias dispatches there on sm_100a. - On 148 SMs (74 CTA pairs) the row has 1296 pair tiles (27 × 48): 17 full waves plus 38 tail tiles. - 2 × 38 = 76 half units exceed 74 pairs, so `tailsplit_units` keeps whole tiles (`tail_split = 1`). - B200 therefore failed with `AssertionError: ('M6889-N12288-K3072-r32-bias', 148)` under CUDA 13.0 and 13.4. - The test now asserts the schedule rule. A partial last wave must exist (`tail > 0`). If `2 * tail <= pairs`, the tail becomes `2 * tail` half units; otherwise all tiles stay whole. - `test_tailsplit_units_schedule` adds `units(m=6889, n=12288, sm_count=148) == (1296, 1296, 1296, 1)` and `units(m=6889, n=12288, sm_count=152) == (1296, 1300, 1292, 2)`. - There is no loader or kernel change, and the #7516 dispatch stays as is. **Warp-decode PDL graph edges** (`tests/moe/test_cake_warp_decode_pdl_gpu.py`) - `test_first_kernel_launch_dependency[direct|route_pack|fused_route_pack]` read captured-graph edges with `cuda.bindings.driver.cuGraphGetEdges` (`cuGraphGetEdges_v2` in the CUDA 12 bindings). - cuda-bindings before 12.9.7 and 13.3.0 return `CUgraphEdgeData` wrappers backed by a buffer that is freed when the call returns (NVIDIA/cuda-python#1804, fixed in NVIDIA/cuda-python#2110). - In the CUDA 13.0 environment (cuda-bindings 13.0.3), the test saw edge types 255, 250, 76, 178, 40, 57, ... instead of 1 (programmatic) or 0 (default). - The 12.9.7 and 13.4.2 bindings passed. - The new `_graph_edges(graph_handle)` calls `cuGraphGetEdges_v2` from `libcuda.so.1` via ctypes. It uses a `CUgraphEdgeData` Structure (`from_port`, `to_port`, `type`, `reserved[5]`). - `_kernel_graph` still uses cuda-bindings for node enumeration, node types and kernel names, which the bug does not affect. ## Validation Validation is in flight and will be summarized here once the GPU matrix (B200, GB200, GB300 with the CUDA 12.9, 13.0 and 13.4 environments) on the three test files and the public CI run have completed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Compatibility** * NVFP4 context programs require CUDA 13.4 or newer. On older toolkits, loading these programs reports the version requirement, and module selection warns before falling back to a compatible implementation. This makes the toolkit limitation clearer and allows supported workloads to use the fallback where available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Co-authored-by: Avery Huang <[email protected]> Co-authored-by: Claude Fable 5.1 <[email protected]>
1 parent 9ff11bc commit 85e10f5

4 files changed

Lines changed: 145 additions & 13 deletions

File tree

‎flashinfer/cake_fmha.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5363,6 +5363,29 @@ def __exit__(self, *args: Any) -> None:
53635363
self.device.__exit__(*args)
53645364

53655365

5366+
# The NVFP4 context programs dequantize their E2M1 pages with the PTX ISA 9.4
5367+
# packed multiply ``mul.rn.satfinite.e4m3x4.e2m1x4.e4m3x4`` (CUDA 13.4 or
5368+
# newer); an older nvcc / ptxas rejects the instruction, so on such a toolkit
5369+
# a route to one of them fails closed to compat_v1 instead of a build error.
5370+
CONTEXT_NVFP4_PROGRAMS_MIN_CUDA = "13.4"
5371+
5372+
5373+
def context_nvfp4_programs_unavailable() -> str | None:
5374+
"""Why this CUDA toolkit cannot build the NVFP4 context programs, or ``None``."""
5375+
5376+
from .jit.cpp_ext import get_cuda_version, is_cuda_version_at_least
5377+
5378+
if is_cuda_version_at_least(CONTEXT_NVFP4_PROGRAMS_MIN_CUDA):
5379+
return None
5380+
return (
5381+
"the generated NVFP4 context programs spell the E2M1 x E4M3 page "
5382+
"dequantization as the PTX ISA 9.4 packed multiply "
5383+
"(mul.rn.satfinite.e4m3x4.e2m1x4.e4m3x4), which needs CUDA "
5384+
f"{CONTEXT_NVFP4_PROGRAMS_MIN_CUDA} or newer; the CUDA toolkit found is "
5385+
f"{get_cuda_version()}"
5386+
)
5387+
5388+
53665389
_CONTEXT_PROGRAM_MODULES: dict[tuple[str, str, str], CakeFmhaContextProgramModule] = {}
53675390

53685391

@@ -5391,6 +5414,10 @@ def _load_cake_fmha_context_program(
53915414
raise RuntimeError(
53925415
f"Cake FMHA {route.component} names no generated program {route.program!r}"
53935416
)
5417+
if route.component == "context_nvfp4":
5418+
unavailable = context_nvfp4_programs_unavailable()
5419+
if unavailable is not None:
5420+
raise RuntimeError(f"Cake FMHA cannot build {pinned[0]}: {unavailable}")
53945421
loader_name = f"cake_fmha_{route.component}"
53955422
loader = importlib.import_module(f".jit.{loader_name}", __package__)
53965423
program = getattr(loader, route.program, None)
@@ -5429,6 +5456,16 @@ def get_cake_fmha_context_module(
54295456
if route.target != _cake_fmha_target(device):
54305457
raise RuntimeError("Cake FMHA context route target does not match the device")
54315458
if route.program is not None:
5459+
if route.component == "context_nvfp4":
5460+
unavailable = context_nvfp4_programs_unavailable()
5461+
if unavailable is not None:
5462+
warnings.warn(
5463+
f"Cake FMHA NVFP4 context program {route.program} failed closed "
5464+
f"to compat_v1: {unavailable}",
5465+
RuntimeWarning,
5466+
stacklevel=2,
5467+
)
5468+
return load_cake_fmha_compat_module(route.target)
54325469
return _load_cake_fmha_context_program(route)
54335470
if route.exact_profile is not None:
54345471
raise RuntimeError(

‎tests/attention/test_cake_fmha_context_programs.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1484,6 +1484,42 @@ def test_cake_context_program_refuses_a_strided_page_column(monkeypatch) -> None
14841484
)
14851485

14861486

1487+
def _skip_without_nvfp4_programs(component: str) -> None:
1488+
"""Skip a test of an NVFP4 program where the CUDA toolkit cannot build it
1489+
(their routes fail closed to compat_v1 there, see ``cake_fmha``)."""
1490+
if component != "context_nvfp4":
1491+
return
1492+
unavailable = cake_api.context_nvfp4_programs_unavailable()
1493+
if unavailable is not None:
1494+
pytest.skip(f"the NVFP4 context programs are not built here: {unavailable}")
1495+
1496+
1497+
@_GPU
1498+
def test_cake_context_nvfp4_programs_fail_closed_without_their_toolkit(
1499+
monkeypatch,
1500+
) -> None:
1501+
"""On a CUDA toolkit that cannot assemble the NVFP4 programs (their PTX ISA
1502+
9.4 packed multiply needs CUDA 13.4), a route to one of them loads
1503+
compat_v1 with a warning instead of a build ptxas rejects, and the program
1504+
loader itself refuses the build."""
1505+
1506+
target = cake_api._cake_fmha_target(torch.device("cuda"))
1507+
monkeypatch.setattr(cake_api, "_cake_fmha_target", lambda device: target)
1508+
kwargs = _context_kwargs("nvfp4_b4_q511_k2047", device="meta")
1509+
route = cake_api.select_cake_fmha_context_route(torch.device("meta"), **kwargs)
1510+
assert route is not None and route.component == "context_nvfp4"
1511+
assert route.program is not None
1512+
reason = "a toolkit that cannot assemble the programs"
1513+
monkeypatch.setattr(cake_api, "context_nvfp4_programs_unavailable", lambda: reason)
1514+
monkeypatch.setattr(cake_api, "_CONTEXT_PROGRAM_MODULES", {})
1515+
with pytest.raises(RuntimeError, match=reason):
1516+
cake_api._load_cake_fmha_context_program(route)
1517+
with pytest.warns(RuntimeWarning, match="failed closed to compat_v1"):
1518+
module = cake_api.get_cake_fmha_context_module(torch.device("cuda"), route)
1519+
assert not isinstance(module, cake_api.CakeFmhaContextProgramModule)
1520+
assert module is cake_api.load_cake_fmha_compat_module(target)
1521+
1522+
14871523
@pytest.fixture
14881524
def served_programs(monkeypatch):
14891525
"""The generated programs the context dispatch loads, in call order."""
@@ -1701,6 +1737,8 @@ def test_cake_context_quantized_programs_match_reference(
17011737

17021738
_whole_tile_routes(monkeypatch)
17031739
target = cake_api._cake_fmha_target(torch.device("cuda"))
1740+
if case["kv_dtype"] == "nvfp4":
1741+
_skip_without_nvfp4_programs("context_nvfp4")
17041742
if program.startswith("Q511"):
17051743
program = f"{program}_SM{_TARGET_ARCH[target][3:].upper()}"
17061744
original = prefill.trtllm_batch_context_with_kv_cache
@@ -1886,6 +1924,7 @@ def test_cake_context_programs_match_the_jit_route_bitwise(
18861924
_whole_tile_routes(monkeypatch)
18871925
row, batch, max_q = BITWISE_ROWS[name]
18881926
case = _context_case(row, batch, max_q)
1927+
_skip_without_nvfp4_programs(case.component)
18891928
program = case.programs[cake_api._cake_fmha_target(torch.device("cuda"))]
18901929
out = _run_context(case)
18911930
assert served_programs == [program]

‎tests/gemm/test_cake_nvfp4_svdquant_gemm_backend.py‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -982,6 +982,10 @@ def test_tailsplit_units_schedule() -> None:
982982
assert units(m=4096, n=12288, sm_count=148) == (768, 796, 740, 2)
983983
# 192 pair tiles on 76 pairs: 2 waves + 40 tail tiles, 80 > 76 -> whole tiles.
984984
assert units(m=4096, n=3072, sm_count=152) == (192, 192, 192, 1)
985+
# 1296 pair tiles (27 x 48) on 74 pairs: 17 waves + 38 tail tiles, 76 > 74 ->
986+
# whole tiles; on 76 pairs 17 waves + 4 tail tiles -> 8 half units.
987+
assert units(m=6889, n=12288, sm_count=148) == (1296, 1296, 1296, 1)
988+
assert units(m=6889, n=12288, sm_count=152) == (1296, 1300, 1292, 2)
985989
# A whole number of waves: nothing to split.
986990
assert units(m=4096, n=3072, sm_count=128) == (192, 192, 192, 1)
987991
# Forced schedules and the admitted values.
@@ -992,15 +996,29 @@ def test_tailsplit_units_schedule() -> None:
992996

993997

994998
def test_tailsplit_rows_split_on_this_device() -> None:
995-
"""Every row routed to the tail-split program splits its tail on the
996-
device the suite runs on (148 or 152 SMs)."""
999+
"""Every row routed to the tail-split program has a partial last wave on
1000+
the device the suite runs on (148 or 152 SMs), and splits it unless the
1001+
split would need more pairs than the device has: the tail tiles become half
1002+
units when twice their number fits the CTA pairs, otherwise they stay whole
1003+
tiles. (6889, 12288) is such a row on 148 SMs: 1296 tiles are 17 waves of
1004+
74 pairs and 38 tail tiles, whose 76 half units exceed the 74 pairs; the
1005+
program still won that row on sm_100a by measurement."""
9971006
sms = loader.sm_count(_device().index or 0)
1007+
pairs = sms // 2
9981008
for row in (*PERF_ROWS, *COVERAGE_ROWS):
9991009
m, n, k, rank, bias = row
10001010
if loader.MODULES[_route(row)]["plan"]["grid_rule"] != "pair_tailsplit":
10011011
continue
10021012
tiles, units, full, split = loader.tailsplit_units(m=m, n=n, sm_count=sms)
1003-
assert split == 2 and full < units, (_row_id(row), sms)
1013+
tail = tiles % pairs
1014+
assert tail > 0, (_row_id(row), sms)
1015+
if 2 * tail <= pairs:
1016+
assert (units, full, split) == (tiles + tail, tiles - tail, 2), (
1017+
_row_id(row),
1018+
sms,
1019+
)
1020+
else:
1021+
assert (units, full, split) == (tiles, tiles, 1), (_row_id(row), sms)
10041022

10051023

10061024
def _raster_tile(

‎tests/moe/test_cake_warp_decode_pdl_gpu.py‎

Lines changed: 48 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121

2222
from __future__ import annotations
2323

24+
import ctypes
2425
from dataclasses import replace
2526

2627
import pytest
@@ -263,6 +264,50 @@ def load(stale: tuple[str, ...]) -> None:
263264
module.cake_fused_moe_warp_decode_release_workspace(receipt)
264265

265266

267+
class _EdgeData(ctypes.Structure):
268+
"""``CUgraphEdgeData`` of the driver API (CUDA 12.3 and newer)."""
269+
270+
_fields_ = (
271+
("from_port", ctypes.c_uint8),
272+
("to_port", ctypes.c_uint8),
273+
("type", ctypes.c_uint8),
274+
("reserved", ctypes.c_uint8 * 5),
275+
)
276+
277+
278+
def _graph_edges(graph_handle: int) -> list[tuple[int, int, int]]:
279+
"""``(source node, target node, CUgraphDependencyType)`` of every edge of
280+
the graph, read with ``cuGraphGetEdges_v2`` of the driver library itself.
281+
282+
cuda-bindings before 12.9.7 and 13.3.0 return the edge data of
283+
``cuGraphGetEdges`` from a buffer they have already freed (cuda-python
284+
issue #1804), so the edge types they report are whatever that memory holds
285+
by then; the ctypes call keeps the test independent of the bindings' version.
286+
"""
287+
driver = ctypes.CDLL("libcuda.so.1")
288+
get_edges = driver.cuGraphGetEdges_v2
289+
get_edges.restype = ctypes.c_int
290+
get_edges.argtypes = (
291+
ctypes.c_void_p,
292+
ctypes.POINTER(ctypes.c_void_p),
293+
ctypes.POINTER(ctypes.c_void_p),
294+
ctypes.POINTER(_EdgeData),
295+
ctypes.POINTER(ctypes.c_size_t),
296+
)
297+
count = ctypes.c_size_t(0)
298+
assert get_edges(graph_handle, None, None, None, ctypes.byref(count)) == 0
299+
total = count.value
300+
sources = (ctypes.c_void_p * total)()
301+
targets = (ctypes.c_void_p * total)()
302+
data = (_EdgeData * total)()
303+
assert get_edges(graph_handle, sources, targets, data, ctypes.byref(count)) == 0
304+
assert count.value == total
305+
return [
306+
(int(sources[i] or 0), int(targets[i] or 0), int(data[i].type))
307+
for i in range(total)
308+
]
309+
310+
266311
def _kernel_graph(graph) -> tuple[list[str], list[tuple[int, int, int]]]:
267312
"""Kernel node names and kernel-to-kernel edges (source, target, type) of
268313
a captured graph that has not been instantiated."""
@@ -288,17 +333,10 @@ def checked(result):
288333
func = checked(driver.cuKernelGetFunction(params.kern))
289334
index[int(node)] = len(names)
290335
names.append(checked(driver.cuFuncGetName(func)).decode())
291-
# The edge types come from cuGraphGetEdges_v2 in CUDA 12 bindings and from
292-
# cuGraphGetEdges in CUDA 13 bindings.
293-
get_edges = getattr(driver, "cuGraphGetEdges_v2", None) or driver.cuGraphGetEdges
294-
*_, count = checked(get_edges(raw, 0))
295-
sources, targets, data, count = checked(get_edges(raw, count))
296336
edges = [
297-
(index[int(source)], index[int(target)], int(edge.type))
298-
for source, target, edge in zip(
299-
sources[:count], targets[:count], data[:count], strict=True
300-
)
301-
if int(source) in index and int(target) in index
337+
(index[source], index[target], kind)
338+
for source, target, kind in _graph_edges(int(raw))
339+
if source in index and target in index
302340
]
303341
return names, edges
304342

0 commit comments

Comments
 (0)