Skip to content

Commit 2373757

Browse files
feat(cuda.core): add HOST_COPY_FN and pack_host_copy_args for GIL-free host copies (#3269)
* feat(cuda.core): add HOST_COPY_FN and pack_host_copy_args HOST_COPY_FN is a CUhostFn that memcpys without the GIL. pack_host_copy_args packs (dst, src, nbytes) and checks bounds, host accessibility of Buffer memory resources, null addresses, and overlap without calling the driver. The host callback resolver rejects HOST_COPY_FN without a packed record. * docs(cuda.core): document HOST_COPY_FN Covers the shared per-context host function thread and chunk sizing. * examples(cuda.core): add a chunked device-to-NumPy copy with HOST_COPY_FN * fix(cuda.core): reject bool and scalar addresses in pack_host_copy_args True packed as address 1, and a NumPy integer scalar or a ctypes scalar exports the buffer protocol, so it packed the address of its own storage instead of the address it holds. Raise TypeError and point to int(addr). * docs(cuda.core): annotate pack_host_copy_args and complete its docs Annotate dst and src as Buffer.fill does. List OverflowError, say a bytearray must not be resized before the copy runs, and replace the Notes claim that a host callback can enqueue work, which host_launch forbids. Document the HOST_COPY_FN record error on the host callback update methods. * test(cuda.core): extend pack_host_copy_args coverage Pin the GPU-free path for a Buffer without a memory resource, cover the src > dst overlap and its adjacent boundary, and check OverflowError for out-of-range int addresses as well as nbytes.
1 parent 2513b54 commit 2373757

16 files changed

Lines changed: 526 additions & 14 deletions

‎cuda_core/cuda/core/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,8 @@ class _PatchedProperty(metaclass=_PatchedPropMeta):
152152
from cuda.core._graphics import __all__ as _graphics_all
153153
from cuda.core._host import *
154154
from cuda.core._host import __all__ as _host_all
155+
from cuda.core._host_callback import *
156+
from cuda.core._host_callback import __all__ as _host_callback_all
155157
from cuda.core._launch_config import *
156158
from cuda.core._launch_config import __all__ as _launch_config_all
157159
from cuda.core._launcher import *
@@ -181,6 +183,7 @@ class _PatchedProperty(metaclass=_PatchedPropMeta):
181183
*_event_all,
182184
*_graphics_all,
183185
*_host_all,
186+
*_host_callback_all,
184187
*_launch_config_all,
185188
*_launcher_all,
186189
*_linker_all,

‎cuda_core/cuda/core/_host_callback.pyi‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,20 @@
11
# This file was generated by stubgen-pyx v0.2.23 from cuda_core/cuda/core/_host_callback.pyx
22

33
import sys
4+
from _ctypes import CFuncPtr
5+
from typing import TypedDict
46

7+
from cuda.core._memory._buffer import Buffer
8+
from cuda.core._utils.pycompat import BufferProtocol
9+
10+
__all__ = ['HOST_COPY_FN', 'pack_host_copy_args']
511
_CUHOSTFN_HINT = 'ctypes.CFUNCTYPE(None, ctypes.c_void_p)' if sys.platform != 'win32' else 'ctypes.CFUNCTYPE(None, ctypes.c_void_p) or ctypes.WINFUNCTYPE(None, ctypes.c_void_p)'
12+
HOST_COPY_FN: CFuncPtr = ...
13+
14+
class _HostCopyArgs(TypedDict):
15+
dst: int
16+
src: int
17+
nbytes: int
618

719
def _cuhostfn_type_error(detail):
820
"""Build the rejection message for a non-conforming ctypes callback."""
@@ -14,3 +26,59 @@ def _validate_ctypes_host_callback(fn):
1426
keeps ctypes' defaults -- a ``c_int`` result and unspecified arguments --
1527
until the caller declares otherwise, so it must be declared to be accepted.
1628
"""
29+
def pack_host_copy_args(dst: int | Buffer | BufferProtocol, src: int | Buffer | BufferProtocol, nbytes: int) -> bytes:
30+
"""Pack the arguments of :data:`HOST_COPY_FN` into ``user_data`` bytes.
31+
32+
The result is a record of three native-width fields in this order:
33+
``uintptr_t dst``, ``uintptr_t src``, ``size_t nbytes``, with no padding.
34+
``struct.pack("@PPN", dst, src, nbytes)`` builds the same record, and
35+
``struct.unpack("@PPN", data)`` reads one back. A caller that keeps such a
36+
record in memory of its own can pass its address as an ``int``
37+
``user_data`` instead.
38+
39+
``dst`` and ``src`` are not retained. The caller must keep both alive,
40+
and must not resize them (a ``bytearray``, for example), until the copy
41+
has run.
42+
43+
Parameters
44+
----------
45+
dst : int | :class:`Buffer` | :obj:`collections.abc.Buffer`
46+
The destination. Only an ``int`` is a host address. A buffer-protocol
47+
object must be contiguous and writable.
48+
src : int | :class:`Buffer` | :obj:`collections.abc.Buffer`
49+
The source. Only an ``int`` is a host address. A buffer-protocol
50+
object must be contiguous.
51+
nbytes : int
52+
The number of bytes to copy. Zero packs a copy that does nothing.
53+
54+
Returns
55+
-------
56+
bytes
57+
The packed record, to pass as ``user_data``.
58+
59+
Raises
60+
------
61+
TypeError
62+
If ``dst`` or ``src`` is not an ``int``, a :class:`Buffer`, or a
63+
buffer-protocol object, or is a ``bool``, a NumPy integer scalar, or a
64+
ctypes scalar such as ``ctypes.c_void_p``. A buffer-protocol object
65+
that cannot export a contiguous view, or a writable one for ``dst``,
66+
raises its own error.
67+
ValueError
68+
If ``nbytes`` exceeds the size of ``dst`` or ``src``, if a
69+
:class:`Buffer` comes from a memory resource that is not
70+
host-accessible, if an address is null, or if the ranges overlap.
71+
OverflowError
72+
If ``nbytes`` or an ``int`` address is negative or does not fit in a
73+
native-width field.
74+
RuntimeError
75+
If ``dst`` or ``src`` is a closed :class:`Buffer`.
76+
77+
Notes
78+
-----
79+
This function makes no CUDA calls, so a Python host callback can call it,
80+
for example to fill a record that a later :data:`HOST_COPY_FN` reads
81+
through an ``int`` ``user_data``. It checks host accessibility only for a
82+
:class:`Buffer` with a memory resource. For an ``int`` or a :class:`Buffer`
83+
without one, the caller must ensure that the memory is host-accessible.
84+
"""

‎cuda_core/cuda/core/_host_callback.pyx‎

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,12 +2,21 @@
22
#
33
# SPDX-License-Identifier: Apache-2.0
44

5+
from cpython.buffer cimport (
6+
PyBUF_SIMPLE,
7+
PyBUF_WRITABLE,
8+
PyBuffer_Release,
9+
PyObject_CheckBuffer,
10+
PyObject_GetBuffer,
11+
Py_buffer,
12+
)
513
from libc.stdint cimport uintptr_t
614
from libc.stdlib cimport malloc
715
from libc.string cimport memcpy as c_memcpy
816

917
from cuda.bindings cimport cydriver
1018

19+
from cuda.core._memory._buffer cimport Buffer, Buffer_check_open
1120
from cuda.core._rt cimport (
1221
OpaqueHandle,
1322
make_opaque_malloc,
@@ -16,6 +25,13 @@ from cuda.core._rt cimport (
1625

1726
import sys
1827
import ctypes as ct
28+
from _ctypes import CFuncPtr
29+
30+
import numpy
31+
32+
from cuda.core._utils.pycompat import BufferProtocol
33+
34+
__all__ = ['HOST_COPY_FN', 'pack_host_copy_args']
1935

2036

2137
# CUhostFn is `void (CUDA_CB *)(void*)`. CUDA_CB is __stdcall on Windows and
@@ -63,6 +79,126 @@ cdef bint _is_py_host_trampoline(cydriver.CUhostFn fn) noexcept nogil:
6379
return fn == <cydriver.CUhostFn>_py_host_trampoline
6480

6581

82+
ctypedef struct _HostCopyArgs:
83+
uintptr_t dst
84+
uintptr_t src
85+
size_t nbytes
86+
87+
88+
cdef void _host_copy(void* data) noexcept nogil:
89+
cdef _HostCopyArgs* args = <_HostCopyArgs*>data
90+
if args.nbytes:
91+
c_memcpy(<void*>args.dst, <const void*>args.src, args.nbytes)
92+
93+
94+
cdef object _host_copy_fn_pointer():
95+
return ct.CFUNCTYPE(None, ct.c_void_p)(<uintptr_t><cydriver.CUhostFn>_host_copy)
96+
97+
98+
HOST_COPY_FN: CFuncPtr = _host_copy_fn_pointer()
99+
100+
101+
cdef uintptr_t _host_copy_address(object obj, size_t nbytes, bint writable, str name) except? 0:
102+
cdef Buffer buf
103+
cdef Py_buffer view
104+
# These export the buffer protocol over their own storage, so a value
105+
# meant as an address would otherwise pack the address of that storage.
106+
if isinstance(obj, (bool, numpy.integer, ct._SimpleCData)):
107+
raise TypeError(f"{name} must not be a {type(obj).__name__}; pass int(addr) for an address")
108+
if isinstance(obj, int):
109+
return <uintptr_t>obj
110+
if isinstance(obj, Buffer):
111+
buf = <Buffer>obj
112+
Buffer_check_open(buf)
113+
# A Buffer without a memory resource could only be checked with a
114+
# driver query, which is not allowed on a host callback thread.
115+
if buf._memory_resource is not None and not buf._memory_resource.is_host_accessible:
116+
raise ValueError(f"{name} is a Buffer that is not host-accessible")
117+
if nbytes > buf._size:
118+
raise ValueError(f"nbytes ({nbytes}) exceeds the size of {name} ({buf._size})")
119+
return <uintptr_t>buf.handle
120+
if not PyObject_CheckBuffer(obj):
121+
raise TypeError(
122+
f"{name} must be an int, a Buffer, or a buffer-protocol object, got {type(obj).__name__}")
123+
PyObject_GetBuffer(obj, &view, PyBUF_SIMPLE | PyBUF_WRITABLE if writable else PyBUF_SIMPLE)
124+
try:
125+
if nbytes > <size_t>view.len:
126+
raise ValueError(f"nbytes ({nbytes}) exceeds the size of {name} ({view.len})")
127+
return <uintptr_t>view.buf
128+
finally:
129+
PyBuffer_Release(&view)
130+
131+
132+
def pack_host_copy_args(
133+
dst: int | Buffer | BufferProtocol, src: int | Buffer | BufferProtocol, nbytes: int) -> bytes:
134+
"""Pack the arguments of :data:`HOST_COPY_FN` into ``user_data`` bytes.
135+
136+
The result is a record of three native-width fields in this order:
137+
``uintptr_t dst``, ``uintptr_t src``, ``size_t nbytes``, with no padding.
138+
``struct.pack("@PPN", dst, src, nbytes)`` builds the same record, and
139+
``struct.unpack("@PPN", data)`` reads one back. A caller that keeps such a
140+
record in memory of its own can pass its address as an ``int``
141+
``user_data`` instead.
142+
143+
``dst`` and ``src`` are not retained. The caller must keep both alive,
144+
and must not resize them (a ``bytearray``, for example), until the copy
145+
has run.
146+
147+
Parameters
148+
----------
149+
dst : int | :class:`Buffer` | :obj:`collections.abc.Buffer`
150+
The destination. Only an ``int`` is a host address. A buffer-protocol
151+
object must be contiguous and writable.
152+
src : int | :class:`Buffer` | :obj:`collections.abc.Buffer`
153+
The source. Only an ``int`` is a host address. A buffer-protocol
154+
object must be contiguous.
155+
nbytes : int
156+
The number of bytes to copy. Zero packs a copy that does nothing.
157+
158+
Returns
159+
-------
160+
bytes
161+
The packed record, to pass as ``user_data``.
162+
163+
Raises
164+
------
165+
TypeError
166+
If ``dst`` or ``src`` is not an ``int``, a :class:`Buffer`, or a
167+
buffer-protocol object, or is a ``bool``, a NumPy integer scalar, or a
168+
ctypes scalar such as ``ctypes.c_void_p``. A buffer-protocol object
169+
that cannot export a contiguous view, or a writable one for ``dst``,
170+
raises its own error.
171+
ValueError
172+
If ``nbytes`` exceeds the size of ``dst`` or ``src``, if a
173+
:class:`Buffer` comes from a memory resource that is not
174+
host-accessible, if an address is null, or if the ranges overlap.
175+
OverflowError
176+
If ``nbytes`` or an ``int`` address is negative or does not fit in a
177+
native-width field.
178+
RuntimeError
179+
If ``dst`` or ``src`` is a closed :class:`Buffer`.
180+
181+
Notes
182+
-----
183+
This function makes no CUDA calls, so a Python host callback can call it,
184+
for example to fill a record that a later :data:`HOST_COPY_FN` reads
185+
through an ``int`` ``user_data``. It checks host accessibility only for a
186+
:class:`Buffer` with a memory resource. For an ``int`` or a :class:`Buffer`
187+
without one, the caller must ensure that the memory is host-accessible.
188+
"""
189+
cdef size_t n = nbytes
190+
cdef _HostCopyArgs args
191+
args.dst = _host_copy_address(dst, n, True, "dst")
192+
args.src = _host_copy_address(src, n, False, "src")
193+
args.nbytes = n
194+
if n:
195+
if args.dst == 0 or args.src == 0:
196+
raise ValueError("dst and src must not be null")
197+
if (args.dst - args.src if args.dst > args.src else args.src - args.dst) < n:
198+
raise ValueError("dst and src overlap")
199+
return (<char*>&args)[:sizeof(args)]
200+
201+
66202
cdef void _resolve_host_callback(
67203
object fn, object user_data,
68204
cydriver.CUhostFn* out_fn, void** out_user_data,
@@ -78,15 +214,21 @@ cdef void _resolve_host_callback(
78214
ctypes callbacks are validated against the ``CUhostFn`` ABI before their
79215
address is passed to CUDA.
80216
"""
217+
cdef bint is_host_copy
81218
if isinstance(fn, ct._CFuncPtr):
82219
_validate_ctypes_host_callback(fn)
83220
out_fn[0] = <cydriver.CUhostFn><uintptr_t>ct.cast(fn, ct.c_void_p).value
221+
is_host_copy = out_fn[0] == <cydriver.CUhostFn>_host_copy
84222
if user_data is None:
85223
out_user_data[0] = NULL
86224
elif isinstance(user_data, int):
87225
out_user_data[0] = <void*><uintptr_t>user_data
88226
else:
89227
buf = bytes(user_data)
228+
if is_host_copy and len(buf) != sizeof(_HostCopyArgs):
229+
raise ValueError(
230+
f"HOST_COPY_FN user_data must be {sizeof(_HostCopyArgs)} bytes "
231+
f"from pack_host_copy_args, got {len(buf)}")
90232
if len(buf):
91233
out_user_data[0] = malloc(len(buf))
92234
if out_user_data[0] == NULL:
@@ -95,6 +237,8 @@ cdef void _resolve_host_callback(
95237
out_data_owner[0] = make_opaque_malloc(out_user_data[0])
96238
else:
97239
out_user_data[0] = NULL
240+
if is_host_copy and out_user_data[0] == NULL:
241+
raise ValueError("HOST_COPY_FN requires user_data from pack_host_copy_args or a nonzero address")
98242
else:
99243
if not callable(fn):
100244
raise TypeError(

‎cuda_core/cuda/core/_launcher.pyi‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -51,8 +51,9 @@ def host_launch(stream: Stream | GraphBuilder | IsStreamType, fn, *, user_data=N
5151
passes to the driver as is. A pointer to compiled code runs without
5252
the GIL; a ctypes callback that wraps a Python function acquires the
5353
GIL through ctypes and is subject to the same deadlock rule as a
54-
Python callable. ``cuda.core`` keeps ``fn`` alive until the function
55-
has run and releases it afterwards on the main thread.
54+
Python callable. :data:`HOST_COPY_FN` is such a pointer, for a host
55+
copy that runs without the GIL. ``cuda.core`` keeps ``fn`` alive until
56+
the function has run and releases it afterwards on the main thread.
5657
user_data : int or bytes-like, optional
5758
Only for ctypes function pointers. An ``int`` is a raw pointer that
5859
the caller must keep valid until the function has run. Bytes-like data
@@ -65,7 +66,8 @@ def host_launch(stream: Stream | GraphBuilder | IsStreamType, fn, *, user_data=N
6566
If ``stream`` is ``None`` or not a stream, or if ``fn`` is not
6667
callable or its ctypes prototype is not ``CUhostFn``.
6768
ValueError
68-
If ``user_data`` is given for a Python callable.
69+
If ``user_data`` is given for a Python callable, or if ``fn`` is
70+
:data:`HOST_COPY_FN` and ``user_data`` is not a packed record.
6971
RuntimeError
7072
If ``stream`` is a :class:`Stream` that is capturing a graph, or a
7173
:class:`~graph.GraphBuilder` that is closed or not building.

‎cuda_core/cuda/core/_launcher.pyx‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -94,8 +94,9 @@ def host_launch(stream: Stream | GraphBuilder | IsStreamType, fn, *, user_data=N
9494
passes to the driver as is. A pointer to compiled code runs without
9595
the GIL; a ctypes callback that wraps a Python function acquires the
9696
GIL through ctypes and is subject to the same deadlock rule as a
97-
Python callable. ``cuda.core`` keeps ``fn`` alive until the function
98-
has run and releases it afterwards on the main thread.
97+
Python callable. :data:`HOST_COPY_FN` is such a pointer, for a host
98+
copy that runs without the GIL. ``cuda.core`` keeps ``fn`` alive until
99+
the function has run and releases it afterwards on the main thread.
99100
user_data : int or bytes-like, optional
100101
Only for ctypes function pointers. An ``int`` is a raw pointer that
101102
the caller must keep valid until the function has run. Bytes-like data
@@ -108,7 +109,8 @@ def host_launch(stream: Stream | GraphBuilder | IsStreamType, fn, *, user_data=N
108109
If ``stream`` is ``None`` or not a stream, or if ``fn`` is not
109110
callable or its ctypes prototype is not ``CUhostFn``.
110111
ValueError
111-
If ``user_data`` is given for a Python callable.
112+
If ``user_data`` is given for a Python callable, or if ``fn`` is
113+
:data:`HOST_COPY_FN` and ``user_data`` is not a packed record.
112114
RuntimeError
113115
If ``stream`` is a :class:`Stream` that is capturing a graph, or a
114116
:class:`~graph.GraphBuilder` that is closed or not building.

‎cuda_core/cuda/core/graph/_graph_builder.pyi‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -449,7 +449,8 @@ class GraphBuilder:
449449
If ``fn`` is a ctypes function pointer whose declared prototype
450450
does not match ``CUhostFn``.
451451
ValueError
452-
If ``user_data`` is given for a Python callable.
452+
If ``user_data`` is given for a Python callable, or if ``fn`` is
453+
:data:`HOST_COPY_FN` and ``user_data`` is not a packed record.
453454
"""
454455

455456
class Graph:

‎cuda_core/cuda/core/graph/_graph_builder.pyx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -988,7 +988,8 @@ cdef class GraphBuilder:
988988
If ``fn`` is a ctypes function pointer whose declared prototype
989989
does not match ``CUhostFn``.
990990
ValueError
991-
If ``user_data`` is given for a Python callable.
991+
If ``user_data`` is given for a Python callable, or if ``fn`` is
992+
:data:`HOST_COPY_FN` and ``user_data`` is not a packed record.
992993
"""
993994
GB_callback(self, fn, user_data, False)
994995

‎cuda_core/cuda/core/graph/_graph_node.pyi‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -343,7 +343,8 @@ class GraphNode:
343343
If ``fn`` is a ctypes function pointer whose declared prototype
344344
does not match ``CUhostFn``.
345345
ValueError
346-
If ``user_data`` is given for a Python callable.
346+
If ``user_data`` is given for a Python callable, or if ``fn`` is
347+
:data:`HOST_COPY_FN` and ``user_data`` is not a packed record.
347348
"""
348349
def if_then(self, condition: GraphCondition) -> IfNode:
349350
"""Add an if-conditional node depending on this node.

‎cuda_core/cuda/core/graph/_graph_node.pyx‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -541,7 +541,8 @@ cdef class GraphNode:
541541
If ``fn`` is a ctypes function pointer whose declared prototype
542542
does not match ``CUhostFn``.
543543
ValueError
544-
If ``user_data`` is given for a Python callable.
544+
If ``user_data`` is given for a Python callable, or if ``fn`` is
545+
:data:`HOST_COPY_FN` and ``user_data`` is not a packed record.
545546
"""
546547
return GN_callback(self, fn, user_data)
547548

‎cuda_core/cuda/core/graph/_subclasses.pyi‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -274,7 +274,9 @@ class HostCallbackNode(GraphNode):
274274
``fn`` accepts the same forms as :meth:`~graph.GraphNode.callback`: a
275275
Python callable, or a ctypes function pointer whose declared prototype
276276
matches ``CUhostFn`` (``void (*)(void*)``). A mismatched ctypes
277-
prototype raises ``TypeError``.
277+
prototype raises ``TypeError``. ``user_data`` with a Python callable,
278+
or :data:`HOST_COPY_FN` without a packed record, raises
279+
``ValueError``.
278280
279281
.. warning::
280282
@@ -431,7 +433,9 @@ class ExecutableHostCallbackNode(ExecutableGraphNode):
431433
432434
``fn`` may be a Python callable, or a ctypes function pointer whose
433435
declared prototype matches ``CUhostFn`` (``void (*)(void*)``); a
434-
mismatched prototype raises ``TypeError``.
436+
mismatched prototype raises ``TypeError``. ``user_data`` with a Python
437+
callable, or :data:`HOST_COPY_FN` without a packed record, raises
438+
``ValueError``.
435439
436440
.. warning::
437441

0 commit comments

Comments
 (0)