Skip to content

Commit 01037b1

Browse files
authored
cuda.core: use torch 2.14+ stable PyObject->Tensor shim, drop upper version cap (#3033)
* cuda.core: bump torch tensor-bridge upper cap to 2.14 * cuda.core: use torch 2.14+ stable PyObject->Tensor shim as primary path PyTorch >= 2.14 exposes torch_tensor_from_pyobject / aoti_torch_delete_tensor_object as part of the stable C ABI (landed via pytorch/pytorch#183323). These let us obtain an AtenTensorHandle from a Python tensor object without peering at THPVariable's internal layout. Dispatch is driven by *symbol presence* via dlsym / GetProcAddress on first call to view_as_torch_tensor: - resolved -> primary path (owned handle, deleted after metadata reads) - NULL -> fall back to the existing pyobj_to_aten_handle pointer-arithmetic trick, which stays correct for torch 2.3-2.13 (the only versions where this matters). The outer version gate in _memoryview.pyx (``2.3 <= torch <= 2.14``) is unchanged -- it still bounds the fallback. Once we drop support for torch < 2.14 we can remove the upper bound entirely because the shim itself is part of torch's stable ABI contract. Diagnostics kept module-private: - _get_pyobj_path_counts() / _reset_pyobj_path_counts() - _get_shim_available() - _set_use_shim(bool) to force the fallback for A/B benchmarking. * cuda.core: drop benchmark toggle and path counters from tensor bridge * cuda.core: drop upper torch version cap, shim covers 2.14+ Keeps the 2.3 lower bound (required for the AOTI metadata functions). The dispatch in _tensor_bridge.view_as_torch_tensor now picks between: * torch >= 2.14 -- stable torch_tensor_from_pyobject shim (pytorch/pytorch#183323; part of torch's stable C ABI contract) * torch 2.3 - 2.13 -- pyobj_to_aten_handle pointer-arithmetic, validated against the two known THPVariable layouts Future torch releases >= 2.14 are automatically covered by the shim, removing the per-release cap bump treadmill. * cuda.core: regenerate _tensor_bridge.pyi for the new ctypedefs
1 parent 80b3359 commit 01037b1

3 files changed

Lines changed: 155 additions & 95 deletions

File tree

‎cuda_core/cuda/core/_memoryview.pyx‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -51,16 +51,20 @@ cdef dict _torch_type_cache = {}
5151
cdef object _torch_version_ok = None
5252

5353
cdef inline bint _torch_version_check():
54-
"""Return True if 2.3 <= torch <= 2.12 (known AOTI ABI range). Memoized.
54+
"""Return True if torch >= 2.3. Memoized.
5555
5656
Lower bound: AOTI functions we use were introduced in PyTorch 2.3.
57-
Upper bound: the ``pyobj_to_aten_handle`` trick relies on the
58-
THPVariable struct layout (PyObject_HEAD followed by at::Tensor cdata)
59-
and the identity ``AtenTensorHandle == at::Tensor*``. Both are
60-
undocumented internals that could change in a future PyTorch version.
61-
We cap at the latest version we have tested against; unknown versions
62-
fall back to the standard DLPack/CAI paths. Bump the upper bound
63-
after verifying a new PyTorch release.
57+
58+
No upper bound. The dispatch in ``_tensor_bridge.view_as_torch_tensor``
59+
selects between two safe paths per torch version:
60+
61+
* torch >= 2.14 -- stable PyObject->Tensor C ABI shim
62+
(``torch_tensor_from_pyobject`` + ``aoti_torch_delete_tensor_object``);
63+
part of torch's stable contract, so any 2.14+ release is covered.
64+
* torch 2.3-2.13 -- ``pyobj_to_aten_handle`` pointer-arithmetic trick,
65+
validated against the two known THPVariable layouts
66+
(``c10::MaybeOwned<at::Tensor>`` for 2.3-2.9; ``at::Tensor`` directly
67+
for 2.10-2.13; see ``_get_cdata_extra_offset`` in ``_tensor_bridge.pyx``).
6468
"""
6569
global _torch_version_ok
6670
if _torch_version_ok is not None:
@@ -72,7 +76,7 @@ cdef inline bint _torch_version_check():
7276
try:
7377
major, minor = int(torch.__version__.split(".")[0]), \
7478
int(torch.__version__.split(".")[1])
75-
_torch_version_ok = (2, 3) <= (major, minor) <= (2, 12)
79+
_torch_version_ok = (major, minor) >= (2, 3)
7680
except (ValueError, IndexError):
7781
_torch_version_ok = False
7882
return <bint>_torch_version_ok

‎cuda_core/cuda/core/_tensor_bridge.pyi‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ from typing_extensions import TypeAlias
5555
AOTITorchError: TypeAlias = int
5656
AtenTensorHandle: TypeAlias = AtenTensorOpaque
5757
_get_cuda_stream_fn_t: TypeAlias = Callable[[int, Any], AOTITorchError]
58+
_tensor_from_pyobj_fn_t: TypeAlias = Callable[[Any, AtenTensorHandle], AOTITorchError]
59+
_delete_tensor_fn_t: TypeAlias = Callable[[AtenTensorHandle], AOTITorchError]
5860

5961
class PyObject(TypedDict): ...
6062

‎cuda_core/cuda/core/_tensor_bridge.pyx‎

Lines changed: 140 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -108,11 +108,18 @@ cdef extern from "_include/aoti_shim.h":
108108
# it lives in torch_cuda.dll (not torch_cpu.dll). It is resolved lazily
109109
# at runtime via dlsym / GetProcAddress — see _resolve_cuda_stream_fn().
110110

111-
# Runtime resolution for aoti_torch_get_current_cuda_stream.
112-
# This symbol lives in torch_cuda.dll (Windows) / libtorch_cuda.so (Linux),
113-
# NOT in torch_cpu. We resolve it lazily on first use so that the module
114-
# can be imported even with CPU-only PyTorch.
111+
# Runtime resolution for symbols not statically bound.
112+
# aoti_torch_get_current_cuda_stream lives in torch_cuda.dll (Windows) /
113+
# libtorch_cuda.so (Linux), NOT in torch_cpu. We resolve it lazily so
114+
# the module imports with CPU-only PyTorch.
115+
#
116+
# torch_tensor_from_pyobject / aoti_torch_delete_tensor_object are only
117+
# present in PyTorch >= 2.14 (stable PyObject<->Tensor C ABI, pytorch/pytorch#183323).
118+
# When present, we use them as the primary PyObject -> AtenTensorHandle path;
119+
# otherwise we fall back to the pyobj_to_aten_handle pointer-arithmetic trick.
115120
ctypedef AOTITorchError (*_get_cuda_stream_fn_t)(int32_t, void**) nogil
121+
ctypedef AOTITorchError (*_tensor_from_pyobj_fn_t)(void*, AtenTensorHandle*) nogil
122+
ctypedef AOTITorchError (*_delete_tensor_fn_t)(AtenTensorHandle) nogil
116123

117124
cdef extern from *:
118125
"""
@@ -123,6 +130,18 @@ cdef extern from *:
123130
if (!h) return NULL;
124131
return (void*)GetProcAddress(h, "aoti_torch_get_current_cuda_stream");
125132
}
133+
static void* _resolve_tensor_from_pyobj_fn(void) {
134+
HMODULE h = GetModuleHandleA("torch_cpu.dll");
135+
if (!h) h = LoadLibraryA("torch_cpu.dll");
136+
if (!h) return NULL;
137+
return (void*)GetProcAddress(h, "torch_tensor_from_pyobject");
138+
}
139+
static void* _resolve_delete_tensor_fn(void) {
140+
HMODULE h = GetModuleHandleA("torch_cpu.dll");
141+
if (!h) h = LoadLibraryA("torch_cpu.dll");
142+
if (!h) return NULL;
143+
return (void*)GetProcAddress(h, "aoti_torch_delete_tensor_object");
144+
}
126145
#else
127146
#include <dlfcn.h>
128147
#ifndef RTLD_DEFAULT
@@ -131,11 +150,22 @@ cdef extern from *:
131150
static void* _resolve_cuda_stream_fn(void) {
132151
return dlsym(RTLD_DEFAULT, "aoti_torch_get_current_cuda_stream");
133152
}
153+
static void* _resolve_tensor_from_pyobj_fn(void) {
154+
return dlsym(RTLD_DEFAULT, "torch_tensor_from_pyobject");
155+
}
156+
static void* _resolve_delete_tensor_fn(void) {
157+
return dlsym(RTLD_DEFAULT, "aoti_torch_delete_tensor_object");
158+
}
134159
#endif
135160
"""
136161
void* _resolve_cuda_stream_fn() nogil
162+
void* _resolve_tensor_from_pyobj_fn() nogil
163+
void* _resolve_delete_tensor_fn() nogil
137164

138165
cdef _get_cuda_stream_fn_t _cached_get_cuda_stream = NULL
166+
cdef _tensor_from_pyobj_fn_t _cached_tensor_from_pyobj = NULL
167+
cdef _delete_tensor_fn_t _cached_delete_tensor = NULL
168+
cdef bint _tried_resolve_pyobj_shim = False
139169

140170
import numpy
141171
import sys
@@ -358,7 +388,7 @@ def view_as_torch_tensor(
358388
If provided, populate this existing view in-place. Otherwise a
359389
new instance is created.
360390
"""
361-
cdef AtenTensorHandle handle = pyobj_to_aten_handle(obj)
391+
cdef AtenTensorHandle handle
362392
cdef void* data_ptr
363393
cdef int64_t ndim
364394
cdef int64_t* sizes_ptr
@@ -367,86 +397,110 @@ def view_as_torch_tensor(
367397
cdef int32_t device_type, device_index
368398
cdef StridedMemoryView buf
369399
cdef intptr_t _stream_ptr_int
370-
371-
# Note: we intentionally skip PyTorch's Python-level __dlpack__ guards
372-
# (requires_grad, is_conj, is_neg, non-strided layout, wrong-device)
373-
# for the same reason PyTorch's own __dlpack_c_exchange_api__ C path
374-
# skips them — the C-level exchange path is designed for performance-
375-
# critical consumers. See DLTensorFromPyObjectNoSync in
376-
# torch/csrc/Module.cpp which calls toDLPackNonOwning with zero checks.
377-
378-
check_aoti(aoti_torch_get_data_ptr(handle, &data_ptr),
379-
b"aoti_torch_get_data_ptr")
380-
check_aoti(aoti_torch_get_dim(handle, &ndim),
381-
b"aoti_torch_get_dim")
382-
check_aoti(aoti_torch_get_sizes(handle, &sizes_ptr),
383-
b"aoti_torch_get_sizes")
384-
check_aoti(aoti_torch_get_strides(handle, &strides_ptr),
385-
b"aoti_torch_get_strides")
386-
check_aoti(aoti_torch_get_dtype(handle, &dtype_code),
387-
b"aoti_torch_get_dtype")
388-
check_aoti(aoti_torch_get_device_type(handle, &device_type),
389-
b"aoti_torch_get_device_type")
390-
check_aoti(aoti_torch_get_device_index(handle, &device_index),
391-
b"aoti_torch_get_device_index")
392-
393-
# -- populate StridedMemoryView --
394-
if view is not None:
395-
buf = view
400+
cdef int itemsize
401+
cdef _StridedLayout layout
402+
cdef bint owned = False
403+
global _cached_tensor_from_pyobj, _cached_delete_tensor
404+
global _tried_resolve_pyobj_shim
405+
406+
# -- Resolve the pytorch stable PyObject<->Tensor shims (torch>=2.14) on
407+
# first call. If present, use them for an owned handle. Otherwise
408+
# fall back to the pyobj_to_aten_handle pointer-arithmetic trick.
409+
if not _tried_resolve_pyobj_shim:
410+
_cached_tensor_from_pyobj = <_tensor_from_pyobj_fn_t>_resolve_tensor_from_pyobj_fn()
411+
_cached_delete_tensor = <_delete_tensor_fn_t>_resolve_delete_tensor_fn()
412+
_tried_resolve_pyobj_shim = True
413+
414+
if _cached_tensor_from_pyobj != NULL and _cached_delete_tensor != NULL:
415+
check_aoti(_cached_tensor_from_pyobj(<void*>obj, &handle),
416+
b"torch_tensor_from_pyobject")
417+
owned = True
396418
else:
397-
buf = StridedMemoryView.__new__(StridedMemoryView)
398-
399-
buf.ptr = <intptr_t>data_ptr
400-
buf._dtype = None # clear cached dtype (view may be reused)
401-
# PyTorch always reports tensors as writable via both DLPack
402-
# (flags=0, no DLPACK_FLAG_BITMASK_READ_ONLY) and CAI
403-
# (__cuda_array_interface__["data"] = (ptr, False)). Tensors that
404-
# cannot be safely exported (requires_grad, conjugate, non-strided)
405-
# are rejected with BufferError rather than marked read-only.
406-
# The AOTI C ABI has no readonly query either, so False is correct.
407-
buf.readonly = False
408-
buf.exporting_obj = obj
409-
buf.dl_tensor = NULL
410-
buf.metadata = None
411-
buf._buffer = None
412-
413-
if device_type == _DEVICE_TYPE_CPU:
414-
buf.device_id = -1
415-
buf.is_device_accessible = False
416-
elif device_type == _DEVICE_TYPE_CUDA:
417-
buf.device_id = <int>device_index
418-
buf.is_device_accessible = True
419-
420-
# -- stream ordering (matches the DLPack contract) --
421-
# stream_ptr=None is ambiguous for CUDA tensors — the caller must
422-
# explicitly choose -1 (no sync) or a valid stream pointer.
423-
if stream_ptr is None:
419+
handle = pyobj_to_aten_handle(obj)
420+
421+
try:
422+
# Note: we intentionally skip PyTorch's Python-level __dlpack__ guards
423+
# (requires_grad, is_conj, is_neg, non-strided layout, wrong-device)
424+
# for the same reason PyTorch's own __dlpack_c_exchange_api__ C path
425+
# skips them — the C-level exchange path is designed for performance-
426+
# critical consumers. See DLTensorFromPyObjectNoSync in
427+
# torch/csrc/Module.cpp which calls toDLPackNonOwning with zero checks.
428+
429+
check_aoti(aoti_torch_get_data_ptr(handle, &data_ptr),
430+
b"aoti_torch_get_data_ptr")
431+
check_aoti(aoti_torch_get_dim(handle, &ndim),
432+
b"aoti_torch_get_dim")
433+
check_aoti(aoti_torch_get_sizes(handle, &sizes_ptr),
434+
b"aoti_torch_get_sizes")
435+
check_aoti(aoti_torch_get_strides(handle, &strides_ptr),
436+
b"aoti_torch_get_strides")
437+
check_aoti(aoti_torch_get_dtype(handle, &dtype_code),
438+
b"aoti_torch_get_dtype")
439+
check_aoti(aoti_torch_get_device_type(handle, &device_type),
440+
b"aoti_torch_get_device_type")
441+
check_aoti(aoti_torch_get_device_index(handle, &device_index),
442+
b"aoti_torch_get_device_index")
443+
444+
# -- populate StridedMemoryView --
445+
if view is not None:
446+
buf = view
447+
else:
448+
buf = StridedMemoryView.__new__(StridedMemoryView)
449+
450+
buf.ptr = <intptr_t>data_ptr
451+
buf._dtype = None # clear cached dtype (view may be reused)
452+
# PyTorch always reports tensors as writable via both DLPack
453+
# (flags=0, no DLPACK_FLAG_BITMASK_READ_ONLY) and CAI
454+
# (__cuda_array_interface__["data"] = (ptr, False)). Tensors that
455+
# cannot be safely exported (requires_grad, conjugate, non-strided)
456+
# are rejected with BufferError rather than marked read-only.
457+
# The AOTI C ABI has no readonly query either, so False is correct.
458+
buf.readonly = False
459+
buf.exporting_obj = obj
460+
buf.dl_tensor = NULL
461+
buf.metadata = None
462+
buf._buffer = None
463+
464+
if device_type == _DEVICE_TYPE_CPU:
465+
buf.device_id = -1
466+
buf.is_device_accessible = False
467+
elif device_type == _DEVICE_TYPE_CUDA:
468+
buf.device_id = <int>device_index
469+
buf.is_device_accessible = True
470+
471+
# -- stream ordering (matches the DLPack contract) --
472+
# stream_ptr=None is ambiguous for CUDA tensors — the caller must
473+
# explicitly choose -1 (no sync) or a valid stream pointer.
474+
if stream_ptr is None:
475+
raise BufferError(
476+
"stream_ptr=None is ambiguous for CUDA tensors; "
477+
"pass stream_ptr=-1 to opt out of synchronization, "
478+
"or pass a valid stream pointer")
479+
_stream_ptr_int = int(stream_ptr)
480+
if _stream_ptr_int != -1:
481+
sync_torch_stream(device_index, _stream_ptr_int)
482+
else:
424483
raise BufferError(
425-
"stream_ptr=None is ambiguous for CUDA tensors; "
426-
"pass stream_ptr=-1 to opt out of synchronization, "
427-
"or pass a valid stream pointer")
428-
_stream_ptr_int = int(stream_ptr)
429-
if _stream_ptr_int != -1:
430-
sync_torch_stream(device_index, _stream_ptr_int)
431-
else:
432-
raise BufferError(
433-
f"Unsupported device type from torch tensor "
434-
f"(AOTI device type id: {device_type})")
435-
436-
# Defer full numpy dtype resolution until first .dtype access.
437-
# Store the raw AOTI dtype code in metadata for lazy lookup.
438-
buf.metadata = <int>dtype_code
439-
440-
# Build _StridedLayout. init_from_ptr copies shape/strides so we are
441-
# safe even though they are borrowed pointers.
442-
cdef int itemsize = _get_aoti_itemsize(dtype_code)
443-
cdef _StridedLayout layout = _StridedLayout.__new__(_StridedLayout)
444-
layout.init_from_ptr(
445-
<int>ndim,
446-
sizes_ptr,
447-
strides_ptr,
448-
itemsize,
449-
)
450-
buf._layout = layout
451-
452-
return buf
484+
f"Unsupported device type from torch tensor "
485+
f"(AOTI device type id: {device_type})")
486+
487+
# Defer full numpy dtype resolution until first .dtype access.
488+
# Store the raw AOTI dtype code in metadata for lazy lookup.
489+
buf.metadata = <int>dtype_code
490+
491+
# Build _StridedLayout. init_from_ptr copies shape/strides so we are
492+
# safe even though they are borrowed pointers (we immediately copy).
493+
itemsize = _get_aoti_itemsize(dtype_code)
494+
layout = _StridedLayout.__new__(_StridedLayout)
495+
layout.init_from_ptr(
496+
<int>ndim,
497+
sizes_ptr,
498+
strides_ptr,
499+
itemsize,
500+
)
501+
buf._layout = layout
502+
503+
return buf
504+
finally:
505+
if owned:
506+
_cached_delete_tensor(handle)

0 commit comments

Comments
 (0)