Skip to content

Commit ea0950a

Browse files
[TIRX] Split backend.cuda.intrinsics, fold tcgen05 descriptors, and fix two PTX dialect gaps (#20120)
Rebased onto current `main`. Five commits, all confined to the TIRx CUDA backend and its tests. ### `[REFACTOR][TIRX]` split `backend.cuda.intrinsics` into `codegen/` and `cpp/` `backend/cuda/intrinsics/` mixed two unrelated concerns: the machinery that generates intrinsic bindings, and the C++/PTX surface those bindings expose. They now live apart — `codegen/` holds the registry, schema, header, and type plumbing; `cpp/` holds the descriptors, ASM builtins, instrumentation, and NVSHMEM surface. `cp_async`, `math`, `memory`, `sync`, and `wgmma` are folded into their new homes. ### `[FEAT][TIRX]` fold the block-scaled tcgen05 descriptors at compile time The block-scaled tcgen05 instruction descriptors were assembled at runtime from values that are all known while lowering. They are now folded during compilation, which removes the per-call packing work from the generated kernel. ### `[FIX][TIRX]` support widened signed PTX loads The PTX dialect table rejected loads whose destination register is wider than the signed source operand, which is legal and is what a sign-extending load emits. ### `[FIX][TIRX]` support no-hint mbarrier parity waits `mbarrier.try_wait.parity` without a hint operand failed to match a table entry, so a parity wait written without an explicit hint would not lower. ### `[CHORE]` tir-test skill GPU selection The `tir-test` skill refused to run when any candidate GPU showed non-zero utilization or resident memory, and aborted outright if another process appeared on a selected card at baseline. On a shared machine that blocked the suite for reasons unrelated to the code under test. Selection now takes every Blackwell device. ### Testing TIRx suite: 3024 passed, 73 skipped, 0 failed.
1 parent 5137563 commit ea0950a

29 files changed

Lines changed: 1506 additions & 1578 deletions

File tree

‎.agents/skills/tir-test/SKILL.md‎

Lines changed: 17 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -4,29 +4,19 @@ Run the full TIRX test suite.
44

55
Run all commands below in the same Bash shell from the TVM repository root.
66

7-
1. Point Python at this workspace's repos and select every eligible idle Blackwell GPU.
7+
1. Point Python at this workspace's repos and select every Blackwell GPU.
88
`TIR_TEST_GPUS` may explicitly provide physical GPU IDs, and
99
`TIR_TEST_NUM_GPUS` may optionally cap the automatically selected count. Otherwise,
10-
selection uses all eligible GPUs, ordered by least memory in use.
10+
selection uses all Blackwell GPUs, ordered by least memory in use.
1111
```bash
1212
export WORKSPACE=/path/to/workspace
1313
export TIR_TEST_NUM_GPUS="${TIR_TEST_NUM_GPUS:-}"
14-
export TIR_TEST_MIN_FREE_GB="${TIR_TEST_MIN_FREE_GB:-32}"
15-
export TIR_TEST_MAX_USED_MIB="${TIR_TEST_MAX_USED_MIB:-256}"
1614
export TIR_TEST_XDIST_WORKERS="${TIR_TEST_XDIST_WORKERS:-16}"
1715

1816
[[ -z "$TIR_TEST_NUM_GPUS" || "$TIR_TEST_NUM_GPUS" =~ ^[1-9][0-9]*$ ]] || {
1917
echo "TIR_TEST_NUM_GPUS must be empty or a positive integer" >&2
2018
exit 2
2119
}
22-
[[ "$TIR_TEST_MIN_FREE_GB" =~ ^[0-9]+$ ]] || {
23-
echo "TIR_TEST_MIN_FREE_GB must be a non-negative integer" >&2
24-
exit 2
25-
}
26-
[[ "$TIR_TEST_MAX_USED_MIB" =~ ^[0-9]+$ ]] || {
27-
echo "TIR_TEST_MAX_USED_MIB must be a non-negative integer" >&2
28-
exit 2
29-
}
3020
[[ "$TIR_TEST_XDIST_WORKERS" =~ ^[1-9][0-9]*$ ]] || {
3121
echo "TIR_TEST_XDIST_WORKERS must be a positive integer" >&2
3222
exit 2
@@ -37,19 +27,15 @@ Run all commands below in the same Bash shell from the TVM repository root.
3727
else
3828
mapfile -t TIR_TEST_SELECTED_GPUS < <(
3929
nvidia-smi \
40-
--query-gpu=index,memory.total,memory.used,utilization.gpu,compute_cap \
30+
--query-gpu=index,memory.used,compute_cap \
4131
--format=csv,noheader,nounits \
42-
| awk -F, \
43-
-v min_free_mib="$((TIR_TEST_MIN_FREE_GB * 1024))" \
44-
-v max_used_mib="$TIR_TEST_MAX_USED_MIB" '
32+
| awk -F, '
4533
{
46-
for (i = 1; i <= 5; ++i) {
34+
for (i = 1; i <= 3; ++i) {
4735
gsub(/^[[:space:]]+|[[:space:]]+$/, "", $i)
4836
}
49-
free_mib = $2 - $3
50-
if (($5 + 0) >= 10 && free_mib >= min_free_mib &&
51-
($3 + 0) <= max_used_mib && ($4 + 0) == 0) {
52-
print $1, $3
37+
if (($3 + 0) >= 10) {
38+
print $1, $2
5339
}
5440
}' \
5541
| sort -k2,2n -k1,1n \
@@ -64,7 +50,7 @@ Run all commands below in the same Bash shell from the TVM repository root.
6450
fi
6551

6652
(( ${#TIR_TEST_SELECTED_GPUS[@]} > 0 )) || {
67-
echo "no eligible CUDA compute capability >= 10 GPU is available" >&2
53+
echo "no CUDA compute capability >= 10 GPU is available" >&2
6854
exit 1
6955
}
7056
for gpu in "${TIR_TEST_SELECTED_GPUS[@]}"; do
@@ -80,7 +66,9 @@ Run all commands below in the same Bash shell from the TVM repository root.
8066

8167
export CUDA_VISIBLE_DEVICES
8268
CUDA_VISIBLE_DEVICES="$(IFS=,; printf '%s' "${TIR_TEST_SELECTED_GPUS[*]}")"
83-
export TIRX_KERNEL_TEST_MIN_FREE_GB="$TIR_TEST_MIN_FREE_GB"
69+
# Registry correctness tests skip a device with less than this much free
70+
# memory; 0 runs them regardless of what else is on the GPU.
71+
export TIRX_KERNEL_TEST_MIN_FREE_GB=0
8472
export PYTHONPATH="${WORKSPACE}/tirx-kernels:${WORKSPACE}/tvm/python"
8573
export TVM_LIBRARY_PATH="${WORKSPACE}/tvm/build/lib"
8674

@@ -111,64 +99,24 @@ Run all commands below in the same Bash shell from the TVM repository root.
11199
done
112100
```
113101

114-
Automatic selection uses every eligible idle device. Set
102+
Automatic selection uses every Blackwell device. Set
115103
`TIR_TEST_NUM_GPUS=4` to cap the count or `TIR_TEST_GPUS=3,4,5,6` to override
116104
selection with exact physical IDs.
117105

118-
2. Start one GPU monitor per selected physical GPU. A baseline foreign process
119-
is a selection race; stop and select a different GPU before running tests.
120-
```bash
121-
TIR_TEST_MONITOR_PIDS=()
122-
TIR_TEST_GPU_LOGS=()
123-
124-
cleanup_tir_test_monitors() {
125-
local pid
126-
for pid in "${TIR_TEST_MONITOR_PIDS[@]}"; do
127-
kill "$pid" 2>/dev/null || true
128-
done
129-
for pid in "${TIR_TEST_MONITOR_PIDS[@]}"; do
130-
wait "$pid" 2>/dev/null || true
131-
done
132-
}
133-
trap cleanup_tir_test_monitors EXIT
134-
135-
for gpu in "${TIR_TEST_SELECTED_GPUS[@]}"; do
136-
log="/tmp/tir_test_gpu_${gpu}_$$.log"
137-
bash .agents/scripts/monitor_gpu.sh \
138-
--gpu "$gpu" --interval 5 --log "$log" &
139-
TIR_TEST_MONITOR_PIDS+=("$!")
140-
TIR_TEST_GPU_LOGS+=("$log")
141-
done
142-
sleep 1
143-
144-
baseline_conflict=0
145-
for log in "${TIR_TEST_GPU_LOGS[@]}"; do
146-
if grep -q '\[FOREIGN\]' "$log"; then
147-
echo "foreign process present at baseline: $log" >&2
148-
baseline_conflict=1
149-
fi
150-
done
151-
if (( baseline_conflict )); then
152-
cleanup_tir_test_monitors
153-
trap - EXIT
154-
exit 1
155-
fi
156-
```
157-
158-
3. Import gate: bench workloads. Fail fast if any kernel listed in
106+
2. Import gate: bench workloads. Fail fast if any kernel listed in
159107
`workloads.yaml` fails to import.
160108
```bash
161109
"$PYTHON_BIN" -m tirx_kernels.bench_suite --check-imports
162110
```
163111
A non-zero exit means a pinned workload kernel failed to import. Fix it before
164112
proceeding.
165113

166-
4. Full kernel import gate for correctness-suite coverage:
114+
3. Full kernel import gate for correctness-suite coverage:
167115
```bash
168116
"$PYTHON_BIN" -m tirx_kernels.registry --cc 10 --strict
169117
```
170118

171-
5. Run the full test suite with xdist parallelism:
119+
4. Run the full test suite with xdist parallelism:
172120
```bash
173121
"$PYTHON_BIN" -m pytest tests/python/tirx/ -n "$TIR_TEST_XDIST_WORKERS"
174122
```
@@ -180,29 +128,8 @@ Run all commands below in the same Bash shell from the TVM repository root.
180128
first selected physical GPU. MegaMoE remains skipped because it requires its
181129
dedicated multi-process scheduler, not ordinary xdist device assignment.
182130

183-
6. Stop every monitor and check every selected GPU for foreign usage:
184-
```bash
185-
cleanup_tir_test_monitors
186-
trap - EXIT
187-
188-
foreign_usage=0
189-
for i in "${!TIR_TEST_GPU_LOGS[@]}"; do
190-
gpu="${TIR_TEST_SELECTED_GPUS[$i]}"
191-
log="${TIR_TEST_GPU_LOGS[$i]}"
192-
if grep -qE 'FOREIGN USER|\[FOREIGN\]' "$log"; then
193-
echo "GPU $gpu foreign-process events:"
194-
grep -E 'FOREIGN USER|\[FOREIGN\]' "$log"
195-
foreign_usage=1
196-
else
197-
echo "GPU $gpu: no foreign GPU usage observed"
198-
fi
199-
done
200-
```
201-
202-
7. Report results: selected physical GPUs, elapsed time, total passed, failed,
203-
skipped, and errors, plus both import-gate results. If any foreign-user events
204-
are present in step 6, mention the affected physical GPUs. Flaky failures
205-
should be re-evaluated on clean GPUs before being attributed to code changes.
131+
5. Report results: selected physical GPUs, elapsed time, total passed, failed,
132+
skipped, and errors, plus both import-gate results.
206133

207134
## Failure triage rules
208135

‎docs/tirx/api/backend.rst‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,9 +51,17 @@ tvm.backend.cuda.script
5151
:imported-members:
5252
:exclude-members: PrimExpr, Op, Call
5353

54-
tvm.backend.cuda.intrinsics
55-
***************************
56-
.. automodule:: tvm.backend.cuda.intrinsics
54+
tvm.backend.cuda.codegen
55+
************************
56+
.. automodule:: tvm.backend.cuda.codegen
57+
:members:
58+
:imported-members:
59+
:no-index:
60+
:exclude-members: PrimExpr, Op, Call
61+
62+
tvm.backend.cuda.cpp
63+
********************
64+
.. automodule:: tvm.backend.cuda.cpp
5765
:members:
5866
:imported-members:
5967
:no-index:

‎python/tvm/backend/cuda/__init__.py‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,8 +24,9 @@
2424
from tvm.base import _LOADED_LIBS
2525

2626
_LAZY_SUBMODULES = {
27+
"codegen",
28+
"cpp",
2729
"iket",
28-
"intrinsics",
2930
"lang",
3031
"op",
3132
"ptx",
@@ -70,7 +71,10 @@ def register_backend():
7071
for name, namespace in script_namespaces().items():
7172
builder_ir.register_script_namespace(name, namespace)
7273

73-
import_module(f"{__name__}.intrinsics")
74+
# script_namespaces() above pulls in ptx, which only imports the shared
75+
# codegen layer -- not the device-helper modules. This import is the sole
76+
# trigger that registers their codegens for codegen_cuda.cc to find.
77+
import_module(f"{__name__}.cpp")
7478
import_module(f"{__name__}.tile_primitive")
7579
import_module(f"{__name__}.target_tags")
7680

@@ -106,8 +110,9 @@ def __getattr__(name: str):
106110

107111

108112
__all__ = [
113+
"codegen",
114+
"cpp",
109115
"iket",
110-
"intrinsics",
111116
"lang",
112117
"op",
113118
"ptx",

python/tvm/backend/cuda/intrinsics/__init__.py renamed to python/tvm/backend/cuda/codegen/__init__.py

Lines changed: 17 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -15,35 +15,36 @@
1515
# specific language governing permissions and limitations
1616
# under the License.
1717
# pylint: disable=unused-import
18-
"""CUDA HW intrinsic codegens, grouped by feature domain.
18+
"""Shared machinery behind every CUDA device-intrinsic codegen.
1919
20-
- ``mma`` / ``wgmma`` / ``tcgen05`` — matrix-multiply hardware (Volta+/Hopper/Blackwell).
21-
- ``cp_async`` — cp.async + cp.async.bulk + cp.async.bulk.tensor (TMA), incl. TMA address helpers.
22-
- ``sync`` — barriers, fences, mbarrier, cluster.barrier, warp vote, elect, sync helpers.
23-
- ``math`` — packed-f32x2 arithmetic, exp2/rcp/reduce3, warp/CTA reductions.
24-
- ``cvt`` — PTX data movement and conversion instruction forms.
25-
- ``memory`` — typed copies, ldg, ld.global.acquire, atomics, type conversions, address casts.
26-
- ``nvshmem`` — NVSHMEM RMA / signal / collective.
27-
- ``misc`` — register-allocation control, profiler timer, debug helpers (printf / trap).
20+
Both dialects build on this layer: :mod:`tvm.backend.cuda.ptx` renders PTX
21+
instructions from a table, and :mod:`tvm.backend.cuda.cpp` registers
22+
hand-written CUDA C++ device helpers.
2823
29-
Plus the support modules:
30-
31-
- ``header`` — CUDA header generator and helper-tag table.
32-
- ``registry`` — codegen registry.
33-
- ``types`` — PTX dtype enum.
24+
- ``registry`` — the op-name → codegen map C++ queries during codegen.
25+
- ``schema`` — :func:`device_intrinsic`, the declarative helper registration.
26+
- ``header`` — CUDA header generator and its helper-tag table.
27+
- ``types`` — PTX dtype enum mirroring ``src/backend/cuda/codegen/ptx.cc``.
3428
- ``utils`` — small parsing / validation helpers.
29+
30+
Importing ``registry`` and ``header`` is load-bearing: their module-level
31+
``register_global_func`` decorators publish ``tirx.intrinsics.cuda.get_codegen``
32+
and ``tirx.intrinsics.cuda.header_generator``, which ``codegen_cuda.cc`` looks
33+
up when it emits a kernel. Nothing else imports them for their side effects, so
34+
dropping them here fails at kernel-build time, not at import time.
3535
"""
3636

37-
# Import op modules to register their codegen functions.
38-
from . import cp_async, math, memory, misc, nvshmem, sync, tcgen05, wgmma
37+
from . import header, registry, schema, types, utils
3938
from .header import TAGS, header_generator
4039
from .registry import CODEGEN_REGISTRY, get_codegen, register_codegen
40+
from .schema import device_intrinsic
4141
from .types import PTXDataType
4242

4343
__all__ = [
4444
"CODEGEN_REGISTRY",
4545
"TAGS",
4646
"PTXDataType",
47+
"device_intrinsic",
4748
"get_codegen",
4849
"header_generator",
4950
"register_codegen",
File renamed without changes.
File renamed without changes.

python/tvm/backend/cuda/intrinsics/_schema.py renamed to python/tvm/backend/cuda/codegen/schema.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,9 +43,10 @@
4343

4444
from collections.abc import Callable
4545

46-
from tvm.backend.cuda.intrinsics.registry import register_codegen
4746
from tvm.backend.cuda.op import cuda_func_call
4847

48+
from .registry import register_codegen
49+
4950
# C primitive type → TVM dtype string. Used when the caller specifies a
5051
# non-void ``return_type`` but no explicit ``tvm_return_type`` — the helper
5152
# knows the TVM-side dtype from the C return type.
File renamed without changes.
File renamed without changes.
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
# pylint: disable=unused-import
18+
"""Hand-written CUDA C++ device helpers.
19+
20+
The CUDA backend emits device intrinsics two ways. A single PTX instruction is
21+
a row in the :mod:`tvm.backend.cuda.ptx` table, rendered by a generic engine.
22+
Everything else -- anything needing a hand-written ``__device__`` function body
23+
-- is registered here, grouped by why it needs one:
24+
25+
- ``builtins`` — a CUDA builtin or library call wrapped as an op, no asm.
26+
- ``asm`` — a body that must be hand-written asm: spin-wait loops, barrier
27+
pairs, empty compiler-barrier asm, special-register reads.
28+
- ``descriptors`` — wgmma / tcgen05 descriptor bitfield encoding, plus the
29+
tcgen05 MMA dtype-kind and shape validation those encoders enforce.
30+
- ``instrument`` — profiler timers, IKET events, ``printf`` / ``trap``.
31+
- ``nvshmem`` — NVSHMEM RMA, signal, and collective bindings.
32+
33+
Importing this package is what registers those codegens; the shared registry,
34+
``device_intrinsic`` schema, and header generator live in
35+
:mod:`tvm.backend.cuda.codegen`.
36+
"""
37+
38+
# Import op modules to register their codegen functions.
39+
from . import asm, builtins, descriptors, instrument, nvshmem
40+
41+
__all__ = []

0 commit comments

Comments
 (0)