Repository navigation
[BUG]: Graph adjacency / nodes / edges silently truncate past fixed buffer sizes (16 / 128) #2529
Description
Is this a duplicate?
- I confirmed there appear to be no duplicate issues for this bug and that I agree to the Code of Conduct.
Type of Bug
Incorrect Behavior
Component
cuda.core
Describe the bug
cuda.core graph APIs use a fixed-size buffer for a single driver call, then assume the out-count is the true total when deciding whether to re-query. CUDA driver graph query APIs clamp the out-count to the buffer capacity when the real size is larger, so the grow-and-retry path is unreachable and results are silently truncated.
Tracked as internal bug 6572657. Same root cause in four places (introduced with #1859):
| API | File | Threshold |
|---|---|---|
node.pred / node.succ iteration (query) |
_adjacency_set_proxy.pyx |
16 |
x in node.pred/succ (contains) |
_adjacency_set_proxy.pyx |
16 |
GraphDefinition.nodes() |
_graph_definition.pyx |
128 |
GraphDefinition.edges() |
_graph_definition.pyx |
128 |
clear() and pred/succ setters iterate then delete one-by-one. Iteration stops at 16, so each call only removes 16 edges; remaining edges stay in the graph with no error. Users can build a CUDA graph whose execution order does not match the topology they intended.
Observed with 20 successors:
len(hub.succ) = 20 # correct (NULL count path)
len(list(hub.succ)) = 16 # truncated
kids[19] in hub.succ = False
hub.succ.clear() → len(g.edges()) = 4 # should be 0
Also: len(list(GraphDefinition with 130 empty nodes).nodes()) == 128 (not 130).
Driver APIs such as cuGraphNodeGetDependentNodes / cuGraphGetNodes / cuGraphGetEdges fill the buffer and return success with out-count equal to capacity (copied count) when capacity is too small — not the true total. See cuGraphNodeGetDependencies. cuda.core treated that out-count as the true total.
How to Reproduce
from cuda.core import Device
from cuda.core.graph import GraphDefinition
Device().set_current()
g = GraphDefinition()
hub = g.empty()
kids = [g.empty() for _ in range(20)]
hub.succ.update(kids)
assert len(hub.succ) == 20
assert len(list(hub.succ)) == 20 # FAILS: 16
assert kids[-1] in hub.succ # FAILS: False
hub.succ.clear()
assert len(g.edges()) == 0 # FAILS: 4Expected behavior
Adjacency iteration, membership, clear(), GraphDefinition.nodes(), and GraphDefinition.edges() should report the full topology. When the initial buffer is too small, cuda.core should re-query with a correctly sized buffer (or an equivalent correct pattern) rather than treating the clamped out-count as complete.
Activity
rparolin commentedon Aug 7, 2026
A small-buffer-optimized vector would eliminate this class of bugs. It lets you keep the correct two-call pattern while still avoiding heap allocation in the common small case:
No “grow and retry” logic needed — the container handles stack vs heap internally based on the true count. This removes the temptation to optimize away the NULL first-call, which is what caused this bug.
Established implementations: boost::container::small_vector, llvm::SmallVector, folly::small_vector, eastl::fixed_vector<T, N, /overflow=/true>.
mdboom commentedon Aug 7, 2026
Regular C static arrays are natively supported in Cython. What is the advantage of these other things? I suppose it's bounds checking but I'm not sure Cython would end up exercising whatever bounds checking they provide.