Skip to content

Commit 2cf07d9

Browse files
Andy-Jostclaude
andcommitted
cuda.core: VMM IPC review fixes; no sharing of host-located virtual memory
Take ownership of each exported or duplicated file descriptor before appending its handle, so a failed append cannot close the descriptor twice. Validate imported chunk sizes against the minimum granularity that cuMemMap requires rather than the resource's granularity option. Name both causes in the note on a failed import: a handle that is not a CUDA allocation of the expected type, or an owning device that this process cannot use. Report is_ipc_enabled False for host-located resources: host NUMA virtual memory has no verified allocation path, so it is not shared yet; the docs and the design note say so. Co-Authored-By: Claude Fable 5.1 <[email protected]>
1 parent 2fb30c3 commit 2cf07d9

6 files changed

Lines changed: 53 additions & 30 deletions

File tree

‎cuda_core/cuda/core/_cpp/rt/VMM_DESIGN.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -144,8 +144,8 @@ what makes that safe: the allocation is released exactly once, when its last map
144144
`location_type="host"` with a handle type other than `None`, which the driver rejects, and a
145145
request for GPUDirect RDMA on a device without support. `__init__` runs it after the VMM-support
146146
check; `modify_allocation` runs it on a per-call configuration, which must also name the
147-
resource's location. The resource reports `is_ipc_enabled = False`, which
148-
`Buffer.ipc_descriptor` reads.
147+
resource's location. The resource reports `is_ipc_enabled` (True for `posix_fd` on Linux with
148+
device-located memory), which `Buffer.ipc_descriptor` and `Buffer.from_ipc_descriptor` check.
149149
- `cdef class VirtualMemoryBuffer(Buffer)` carries no extra state. It is created with
150150
`Buffer_from_deviceptr_handle(h_ptr, size, self, cls=VirtualMemoryBuffer)` and documented in
151151
`api.rst` like `ManagedBuffer`. It overrides `close(stream=None)` to reject a capturing stream
@@ -203,8 +203,9 @@ what makes that safe: the allocation is released exactly once, when its last map
203203

204204
A `VirtualMemoryBuffer` is shared the way pool-backed buffers are, through
205205
`Buffer.ipc_descriptor` and `Buffer.from_ipc_descriptor`, when the resource's
206-
`handle_type` can travel between processes (`posix_fd` on Linux;
207-
`is_ipc_enabled` reports it). The exporter calls `cuMemExportToShareableHandle`
206+
`handle_type` can travel between processes (`posix_fd` on Linux) and the memory
207+
lives on a device (`is_ipc_enabled` reports both; host-located memory is not
208+
shared yet). The exporter calls `cuMemExportToShareableHandle`
208209
once per mapping that covers the buffer's size, in address order, and the
209210
descriptor carries the handles with each allocation's size. Each file
210211
descriptor is owned by an `IPCAllocationHandle` and closed when the descriptor

‎cuda_core/cuda/core/_memory/_ipc.pyx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -293,10 +293,12 @@ cdef class VirtualMemoryIPCBufferDescriptor(IPCBufferDescriptor):
293293
continue
294294
dup = os.dup(operator.index(item))
295295
try:
296-
handles.append(IPCAllocationHandle._init(dup, None))
296+
handle = IPCAllocationHandle._init(dup, None)
297297
except: # noqa: E722 rollback-then-raise: the duplicate is not owned yet
298298
os.close(dup)
299299
raise
300+
# The handle owns the duplicate now; if append fails, the handle closes it once.
301+
handles.append(handle)
300302
return VirtualMemoryIPCBufferDescriptor._from_exports(driver_type, sizes_t, tuple(handles), size)
301303

302304
@property

‎cuda_core/cuda/core/_memory/_virtual_memory_resource.pyi‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -329,18 +329,20 @@ class VirtualMemoryResource(MemoryResource):
329329
def is_ipc_enabled(self) -> bool:
330330
"""Whether buffers of this resource can be shared with other processes.
331331
332-
True when ``config.handle_type`` is ``"posix_fd"`` on Linux;
333-
``handle_type`` is the only switch, there is no separate option. With
334-
``handle_type=None`` the allocations cannot be exported, Win32 KMT
335-
handles are not transported by cuda.core, and fabric handles are not
336-
supported yet.
332+
True when ``config.handle_type`` is ``"posix_fd"`` on Linux and the
333+
memory lives on a device; ``handle_type`` is the only switch, there is
334+
no separate option. With ``handle_type=None`` the allocations cannot
335+
be exported, Win32 KMT handles are not transported by cuda.core, and
336+
fabric handles are not supported yet. Host-located virtual memory
337+
(``location_type`` ``"host"``, ``"host_numa"``, or
338+
``"host_numa_current"``) is not shared yet.
337339
"""
338340
def _import_ipc_buffer(self, desc: VirtualMemoryIPCBufferDescriptor, stream) -> VirtualMemoryBuffer:
339341
"""Import a buffer that another process exported; see :meth:`Buffer.from_ipc_descriptor`.
340342
341343
Every exported allocation is imported with this resource's handle
342-
type and checked to live where this resource allocates (its device,
343-
or its host location). The allocations are then mapped in order into
344+
type and checked to live on this resource's device. The allocations
345+
are then mapped in order into
344346
one new address reservation with the access descriptors this
345347
resource's options produce for its device and peers, and the
346348
deallocation stream is recorded as :meth:`allocate` does. The

‎cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx‎

Lines changed: 29 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -885,20 +885,25 @@ cdef class VirtualMemoryResource(MemoryResource):
885885
def is_ipc_enabled(self) -> bool:
886886
"""Whether buffers of this resource can be shared with other processes.
887887

888-
True when ``config.handle_type`` is ``"posix_fd"`` on Linux;
889-
``handle_type`` is the only switch, there is no separate option. With
890-
``handle_type=None`` the allocations cannot be exported, Win32 KMT
891-
handles are not transported by cuda.core, and fabric handles are not
892-
supported yet.
888+
True when ``config.handle_type`` is ``"posix_fd"`` on Linux and the
889+
memory lives on a device; ``handle_type`` is the only switch, there is
890+
no separate option. With ``handle_type=None`` the allocations cannot
891+
be exported, Win32 KMT handles are not transported by cuda.core, and
892+
fabric handles are not supported yet. Host-located virtual memory
893+
(``location_type`` ``"host"``, ``"host_numa"``, or
894+
``"host_numa_current"``) is not shared yet.
893895
"""
894-
return self.config.handle_type in _IPC_HANDLE_TYPES
896+
return (
897+
self.config.handle_type in _IPC_HANDLE_TYPES
898+
and self.config.location_type not in _HOST_LOCATION_TYPES
899+
)
895900
896901
def _import_ipc_buffer(self, VirtualMemoryIPCBufferDescriptor desc not None, stream) -> VirtualMemoryBuffer:
897902
"""Import a buffer that another process exported; see :meth:`Buffer.from_ipc_descriptor`.
898903

899904
Every exported allocation is imported with this resource's handle
900-
type and checked to live where this resource allocates (its device,
901-
or its host location). The allocations are then mapped in order into
905+
type and checked to live on this resource's device. The allocations
906+
are then mapped in order into
902907
one new address reservation with the access descriptors this
903908
resource's options produce for its device and peers, and the
904909
deallocation stream is recorded as :meth:`allocate` does. The
@@ -919,7 +924,7 @@ cdef class VirtualMemoryResource(MemoryResource):
919924
cdef VmmRangeHandle rng
920925
cdef DevicePtrHandle h_ptr
921926
cdef Buffer buf
922-
cdef size_t gran, addr_align, chunk, total = 0, offset = 0
927+
cdef size_t gran, gran_min = 0, addr_align, chunk, total = 0, offset = 0
923928
cdef Py_ssize_t n, i
924929
cdef cydriver.CUdeviceptr hint
925930
cdef int handle_type, fd
@@ -955,11 +960,17 @@ cdef class VirtualMemoryResource(MemoryResource):
955960
self._fill_prop(cfg, &prop)
956961
self._fill_access(cfg, &prop, descs)
957962
gran = self._granularity(cfg, &prop)
963+
# cuMemMap requires the minimum granularity, whatever this resource's
964+
# granularity option says; the exporter may have used either.
965+
with nogil:
966+
HANDLE_RETURN(cydriver.cuMemGetAllocationGranularity(
967+
&gran_min, &prop, cydriver.CU_MEM_ALLOC_GRANULARITY_MINIMUM))
958968
for chunk_obj in desc._chunk_sizes:
959969
chunk = chunk_obj
960-
if chunk == 0 or chunk % gran != 0:
970+
if chunk == 0 or chunk % gran_min != 0:
961971
raise ValueError(
962-
f"descriptor chunk size {chunk} is not a positive multiple of the {gran}-byte granularity"
972+
f"descriptor chunk size {chunk} is not a positive multiple of the {gran_min}-byte minimum "
973+
"granularity"
963974
)
964975
if chunk > <size_t>-1 - total:
965976
raise OverflowError("the descriptor's chunk sizes do not fit in size_t")
@@ -988,9 +999,10 @@ cdef class VirtualMemoryResource(MemoryResource):
988999
os_handle, <cydriver.CUmemAllocationHandleType>handle_type, chunk, descs.data(), descs.size())
9891000
if not h_alloc:
9901001
_raise_last_error_noting(
991-
f"while importing chunk {i + 1} of {n} of a VirtualMemoryResource buffer descriptor; the "
992-
"shared handle may not refer to a CUDA allocation exported with "
993-
f"handle_type={str(cfg.handle_type)!r}"
1002+
f"while importing chunk {i + 1} of {n} of a VirtualMemoryResource buffer descriptor; either "
1003+
"the shared handle does not refer to a CUDA allocation exported with "
1004+
f"handle_type={str(cfg.handle_type)!r}, or the device that owns the memory is not usable in "
1005+
"this process (for example, CUDA_VISIBLE_DEVICES hides it)"
9941006
)
9951007
# The exporter chose where the memory lives. This resource must map
9961008
# memory of that kind on that device, or its access descriptors
@@ -1106,9 +1118,11 @@ cdef VirtualMemoryIPCBufferDescriptor _export_ipc_descriptor(Buffer buf):
11061118
"allocation takes one file descriptor, so a process near its file descriptor limit fails here",
11071119
)
11081120
try:
1109-
handles.append(IPCAllocationHandle._init(fd, None))
1121+
handle = IPCAllocationHandle._init(fd, None)
11101122
except: # noqa: E722 rollback-then-raise: the fd is not owned yet
11111123
os.close(fd)
11121124
raise
1125+
# The handle owns the fd now; if append fails, the handle closes it once.
1126+
handles.append(handle)
11131127
sizes.append(chunk)
11141128
return VirtualMemoryIPCBufferDescriptor._from_exports(handle_type, tuple(sizes), tuple(handles), buf._size)

‎cuda_core/docs/source/release/1.3.0-notes.rst‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -86,7 +86,8 @@ New features
8686
device that owns the memory, and a buffer or its descriptor can be sent
8787
through ``multiprocessing`` directly (plain ``pickle`` cannot carry file
8888
descriptors). :attr:`VirtualMemoryResource.is_ipc_enabled` is ``True`` when
89-
the resource's ``handle_type`` is ``"posix_fd"`` on Linux. A grown buffer
89+
the resource's ``handle_type`` is ``"posix_fd"`` on Linux and the memory
90+
lives on a device; host-located virtual memory is not shared yet. A grown buffer
9091
imports as one contiguous range. A descriptor holds one file descriptor per
9192
physical allocation and pins the memory while it exists; nothing else does.
9293
A buffer that contains imported memory cannot be exported again. For a

‎cuda_core/tests/memory_ipc/test_vmm_ipc.py‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -181,11 +181,14 @@ def _on_queue_feeder_error(e, _obj):
181181
pytest.param({"handle_type": "fabric"}, False, id="fabric"),
182182
pytest.param({"handle_type": "win32_kmt"}, False, id="win32_kmt"),
183183
pytest.param({"location_type": "host", "handle_type": None}, False, id="host"),
184-
pytest.param({"location_type": "host_numa", "handle_type": "posix_fd"}, True, id="host_numa-posix_fd"),
184+
pytest.param({"location_type": "host_numa", "handle_type": "posix_fd"}, False, id="host_numa-posix_fd"),
185+
pytest.param(
186+
{"location_type": "host_numa_current", "handle_type": "posix_fd"}, False, id="host_numa_current-posix_fd"
187+
),
185188
],
186189
)
187190
def test_is_ipc_enabled(vmm_ipc_device, options, expected):
188-
"""Only POSIX file descriptors have a transport; the location does not matter."""
191+
"""Only POSIX file descriptors on device memory have a transport; host-located memory is not shared yet."""
189192
mr = VirtualMemoryResource(vmm_ipc_device, config=VirtualMemoryResourceOptions(**options))
190193
assert mr.is_ipc_enabled is expected
191194

0 commit comments

Comments
 (0)