Skip to content

Commit b69f783

Browse files
pggPLclaude
authored andcommitted
Adding conversion from PyObject to torch::stable::tensor (#183323)
Hi, I would like to propose a PR adding conversions from a Python object to `torch::stable::Tensor` and vice versa. **Motivation** I work on the [Transformer Engine](https://github.com/NVIDIA/TransformerEngine) project. We want to use the torch stable ABI, but we cannot do it in the recommended way of registering a custom op in libtorch. There are 2 issues: - we want to pass arbitrary Python objects with metadata into C++ (it's the core of our design), - something that is logically one op sometimes needs to call multiple C++ functions (especially with complex fusions). So we would like to do something like this: - register into the dispatcher via Python custom ops with opaque objects, - inside these ops, call C++ functions with Python objects as arguments, - if some arguments are tensors, use the functions introduced by this PR to convert them into stable tensors. **Approach** The two shims (`torch_tensor_from_pyobject` / `torch_tensor_to_pyobject`) live in libtorch, but the actual `PyObject <-> at::Tensor` conversion needs `THPVariable_*`, which lives in libtorch_python. To bridge that without libtorch depending on libtorch_python (or on `Python.h`), the PR mirrors the existing `c10::impl::PyInterpreterVTable` indirection: - an interface (`PyObjectConversionInterface`) plus a no-op default in libtorch, reachable through a global pointer; - a concrete implementation in libtorch_python that uses `THPVariable_Unpack` / `THPVariable_Wrap`, registered into libtorch at load time via static init (and reset to the no-op on teardown); - the C shims cross the ABI boundary as `AtenTensorHandle` and forward to whatever implementation is currently installed. `AtenTensorHandle` is confined to the shim layer; the vtable itself works in `at::Tensor`. If libtorch_python is not loaded (e.g. a libtorch-only process), the no-op is still installed and both shims return `AOTI_TORCH_FAILURE` with a clear message instead of crashing (covered by a libtorch-only gtest). **A new class of shims** Existing stable-ABI shims are self-contained in libtorch. These are the first shims that reach *up* from libtorch into libtorch_python for functionality that only exists there (the CPython-facing tensor wrapper), using the vtable indirection to keep the layering intact. The same mechanism can host future Python-interop shims. --- _This PR was prepared with the help of Claude Opus._ Pull Request resolved: #183323 Approved by: https://github.com/janeyx99 Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 862d9ef commit b69f783

15 files changed

Lines changed: 489 additions & 23 deletions

‎build_variables.bzl‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -496,6 +496,7 @@ inductor_core_resources = [
496496
"torch/csrc/inductor/aoti_torch/oss_proxy_executor.cpp",
497497
"torch/csrc/inductor/inductor_ops.cpp",
498498
"torch/csrc/jit/serialization/pickle.cpp",
499+
"torch/csrc/PyObjectConversion.cpp",
499500
"torch/csrc/shim_common.cpp",
500501
"torch/csrc/shim_exception_state.cpp",
501502
]
@@ -967,6 +968,7 @@ libtorch_python_core_sources = [
967968
"torch/csrc/autograd/python_torch_functions_manual.cpp",
968969
"torch/csrc/autograd/python_variable.cpp",
969970
"torch/csrc/autograd/python_variable_indexing.cpp",
971+
"torch/csrc/PyObjectConversionPythonImpl.cpp",
970972
"torch/csrc/distributed/python_placement.cpp",
971973
"torch/csrc/dynamo/python_compiled_autograd.cpp",
972974
"torch/csrc/dynamo/cache_entry.cpp",

‎docs/source/notes/libtorch_stable_abi.md‎

Lines changed: 14 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ It consists of
3030
- torch/csrc/stable/ops.h: Provides a stable interface for calling ATen ops from `native_functions.yaml`.
3131
- torch/csrc/stable/accelerator.h: Provides a stable interface for device-generic objects and APIs
3232
(e.g. `getCurrentStream`, `DeviceGuard`).
33+
- torch/csrc/stable/pyobject.h: Provides conversions between Python objects and their `torch::stable` equivalents, such as `tensor_from_pyobject` / `tensor_to_pyobject` for a Python `torch.Tensor`. To access these APIs, you still need to only link `libtorch`, but `libtorch_python` must be loaded at runtime; see the Python interop shims section below.
3334

3435
We are continuing to improve coverage in our `torch/csrc/stable` APIs. Please file an issue if you'd like to see support for particular APIs in your custom extension.
3536

@@ -42,13 +43,25 @@ The stable C headers started by AOTInductor form the foundation of the stable AB
4243
- [torch/csrc/inductor/aoti_torch/c/shim.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/inductor/aoti_torch/c/shim.h): Includes C-style shim APIs for commonly used regarding Tensors, dtypes, CUDA, and the like.
4344
- [torch/csrc/inductor/aoti_torch/generated/c_shim_aten.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/inductor/aoti_torch/generated/c_shim_aten.h): Includes C-style shim APIs for ATen ops from `native_functions.yaml` (e.g. `aoti_torch_aten_new_empty`).
4445
- [torch/csrc/inductor/aoti_torch/generated/c_shim_*.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/inductor/aoti_torch/generated): Includes C-style shim APIs for specific backend kernels dispatched from `native_functions.yaml` (e.g. `aoti_torch_cuda_pad`). These APIs should only be used for the specific backend they are named after (e.g. `aoti_torch_cuda_pad` should only be used within CUDA kernels), as they opt out of the dispatcher.
45-
- [torch/csrc/stable/c/shim.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/stable/c/shim.h): We are building out more ABIs to logically live in `torch/csrc/stable/c` instead of continuing the AOTI naming that no longer makes sense for our general use case.
46+
- [torch/csrc/stable/c/shim.h](https://github.com/pytorch/pytorch/blob/main/torch/csrc/stable/c/shim.h): We are building out more ABIs to logically live in `torch/csrc/stable/c` instead of continuing the AOTI naming that no longer makes sense for our general use case. This consolidated shim collection also holds the C entry points backing `tensor_from_pyobject`/`tensor_to_pyobject` (see the Python interop shims section below).
4647

4748
These headers are promised to be ABI stable across releases and adhere to a stronger backwards compatibility policy than LibTorch. Specifically, we promise not to modify them for at least 2 years after they are released. However, this is **use at your own risk**. For example, users must handle the memory lifecycle of objects returned by certain APIs. Further, the stack-based APIs discussed below which allow the user to call into the PyTorch dispatcher do not provide strong guarantees on forward and backward compatibility of the underlying op that is called.
4849

4950
Unless absolutely necessary, we recommend the high-level C++ API in `torch/csrc/stable`
5051
which will handle all the rough edges of the C API for the user.
5152

53+
### Python interop shims
54+
55+
`torch/csrc/stable/pyobject.h` converts between Python objects and their
56+
`torch::stable` equivalents (for example `tensor_from_pyobject` / `tensor_to_pyobject` for
57+
converting between `torch::stable::Tensor` in C++ and `torch.Tensor` in Python). Like the
58+
rest of the stable ABI, an extension using them can just link `libtorch`.
59+
60+
Unlike the rest of the stable ABI, they require `libtorch_python` to be loaded at
61+
runtime and must be called with the GIL held. If you call them from a context that
62+
may run without the GIL (for example a boxed `STABLE_TORCH_LIBRARY` kernel),
63+
acquire it first.
64+
5265
## Migrating your kernel to the LibTorch stable ABI
5366

5467
If you'd like your kernel to be ABI stable with LibTorch, meaning you'd the ability to build for one version and run on another, your kernel must only use the limited stable ABI. This following section goes through some steps of migrating an existing kernel and APIs we imagine you would need to swap over.

‎test/cpp/shim/CMakeLists.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ set(TORCH_SHIM_TEST_ROOT ${TORCH_ROOT}/test/cpp/shim)
99
set(TORCH_SHIM_TEST_SRCS
1010
${TORCH_SHIM_TEST_ROOT}/main.cpp
1111
${TORCH_SHIM_TEST_ROOT}/test_stable_ivalue.cpp
12+
${TORCH_SHIM_TEST_ROOT}/test_pyobject_conversion.cpp
1213
)
1314

1415
add_executable(test_shim
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
// Checks that the PyObject<->Tensor conversion shims fall back to a clean error
2+
// (rather than crashing) when libtorch_python is not loaded. This binary links
3+
// only libtorch, so the conversion vtable is the default no-op.
4+
5+
#include <gtest/gtest.h>
6+
7+
#include <ATen/ATen.h>
8+
#include <torch/csrc/inductor/aoti_torch/utils.h>
9+
#include <torch/csrc/stable/c/shim.h>
10+
11+
TEST(TorchPyObjectConversion, NoopErrorsWithoutLibtorchPython) {
12+
// from_pyobject: the no-op errors before touching py_obj, so a dummy non-null
13+
// pointer is enough to reach it.
14+
int dummy = 0;
15+
AtenTensorHandle ath_out = nullptr;
16+
EXPECT_EQ(torch_tensor_from_pyobject(&dummy, &ath_out), AOTI_TORCH_FAILURE);
17+
EXPECT_EQ(ath_out, nullptr);
18+
19+
// to_pyobject: needs a real tensor handle (the shim dereferences it before
20+
// reaching the no-op).
21+
AtenTensorHandle ath = torch::aot_inductor::new_tensor_handle(at::zeros({1}));
22+
void* py_out = nullptr;
23+
EXPECT_EQ(
24+
torch_tensor_to_pyobject(ath, /*py_type=*/nullptr, &py_out),
25+
AOTI_TORCH_FAILURE);
26+
EXPECT_EQ(py_out, nullptr);
27+
aoti_torch_delete_tensor_object(ath);
28+
}
Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,91 @@
1+
#include <Python.h>
2+
3+
#include <torch/csrc/stable/ops.h>
4+
#include <torch/csrc/stable/pyobject.h>
5+
#include <torch/csrc/stable/tensor.h>
6+
7+
// Importable (abi3) module functions that exercise the use case for the
8+
// PyObject<->Tensor stable shims: a raw PyObject arrives straight from Python
9+
// (no dispatcher boxing), so the GIL is naturally held. This is how a consumer
10+
// (e.g. a python-registered custom op) would actually call tensor_from_pyobject
11+
// / tensor_to_pyobject. The module links only libtorch; the conversion is
12+
// serviced by
13+
// the vtable that libtorch_python registers.
14+
15+
using torch::stable::Tensor;
16+
17+
namespace {
18+
19+
// PyObject -> stable Tensor -> PyObject; the result shares storage with input.
20+
PyObject* pyobject_roundtrip(PyObject* /*self*/, PyObject* obj) {
21+
try {
22+
Tensor t = torch::stable::tensor_from_pyobject(obj);
23+
return static_cast<PyObject*>(torch::stable::tensor_to_pyobject(t));
24+
} catch (const std::exception& e) {
25+
if (!PyErr_Occurred()) {
26+
PyErr_SetString(PyExc_RuntimeError, e.what());
27+
}
28+
return nullptr;
29+
}
30+
}
31+
32+
// args = (tensor, py_type); forces the result's exact Python type via
33+
// tensor_to_pyobject's py_type argument (e.g. torch.nn.Parameter).
34+
PyObject* pyobject_to_type(PyObject* /*self*/, PyObject* args) {
35+
PyObject* obj = nullptr;
36+
PyObject* py_type = nullptr;
37+
if (!PyArg_ParseTuple(args, "OO", &obj, &py_type)) {
38+
return nullptr;
39+
}
40+
try {
41+
// Clone so the result wraps a fresh TensorImpl: tensor_to_pyobject with an explicit
42+
// py_type fails if the TensorImpl already has a Python object of a different
43+
// (non-subclass) type, which is the case for `obj` itself.
44+
Tensor t = torch::stable::clone(torch::stable::tensor_from_pyobject(obj));
45+
return static_cast<PyObject*>(torch::stable::tensor_to_pyobject(t, py_type));
46+
} catch (const std::exception& e) {
47+
if (!PyErr_Occurred()) {
48+
PyErr_SetString(PyExc_RuntimeError, e.what());
49+
}
50+
return nullptr;
51+
}
52+
}
53+
54+
// Real work through the stable ABI on a tensor obtained from tensor_from_pyobject.
55+
PyObject* pyobject_sum(PyObject* /*self*/, PyObject* obj) {
56+
try {
57+
Tensor t = torch::stable::tensor_from_pyobject(obj);
58+
return static_cast<PyObject*>(
59+
torch::stable::tensor_to_pyobject(torch::stable::sum(t)));
60+
} catch (const std::exception& e) {
61+
if (!PyErr_Occurred()) {
62+
PyErr_SetString(PyExc_RuntimeError, e.what());
63+
}
64+
return nullptr;
65+
}
66+
}
67+
68+
PyMethodDef methods[] = {
69+
{"pyobject_roundtrip", pyobject_roundtrip, METH_O, nullptr},
70+
{"pyobject_to_type", pyobject_to_type, METH_VARARGS, nullptr},
71+
{"pyobject_sum", pyobject_sum, METH_O, nullptr},
72+
{nullptr, nullptr, 0, nullptr}};
73+
74+
PyModuleDef moduledef = {
75+
PyModuleDef_HEAD_INIT,
76+
"_interop",
77+
nullptr,
78+
-1,
79+
methods,
80+
nullptr,
81+
nullptr,
82+
nullptr,
83+
nullptr};
84+
85+
} // namespace
86+
87+
// PyMODINIT_FUNC (not a bare extern "C") so the init symbol is dllexported on
88+
// Windows; without it the .pyd omits PyInit__interop and the import fails.
89+
PyMODINIT_FUNC PyInit__interop(void) {
90+
return PyModule_Create(&moduledef);
91+
}

‎test/cpp_extensions/libtorch_agn_2_14_extension/libtorch_agn_2_14/__init__.py‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,17 +4,22 @@
44
import torch
55

66

7+
# The op extension (_C) registers this version's and the inherited 2.9-2.13
8+
# STABLE_TORCH_LIBRARY ops; load it via load_library like the other libtorch_agn
9+
# extensions.
710
so_files = list(
811
Path(__file__).parent.glob("_C*" + (".pyd" if sys.platform == "win32" else ".so"))
912
)
1013
if len(so_files) != 1:
1114
raise AssertionError(f"Expected one _C*.{{so,pyd}} file, found {len(so_files)}")
12-
1315
torch.ops.load_library(str(so_files[0]))
1416

15-
from . import ops
17+
# _interop is a separate importable module exposing the PyObject<->Tensor helpers
18+
# used by the interop tests.
19+
from . import _interop, ops
1620

1721

1822
__all__ = [
23+
"_interop",
1924
"ops",
2025
]

‎test/cpp_extensions/libtorch_agn_2_14_extension/setup.py‎

Lines changed: 34 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -46,43 +46,57 @@ def run(self):
4646

4747

4848
def get_extension():
49-
extra_compile_args = {
50-
"cxx": [
51-
"-DTORCH_TARGET_VERSION=0x020e000000000000",
52-
"-DSTABLE_LIB_NAME=libtorch_agn_2_14",
53-
],
54-
}
49+
common_cxx = ["-DTORCH_TARGET_VERSION=0x020e000000000000"]
5550
if not IS_WINDOWS:
56-
extra_compile_args["cxx"].append("-fdiagnostics-color=always")
51+
common_cxx.append("-fdiagnostics-color=always")
5752

58-
# Collect sources from this version and all previous versions
59-
sources = list(CSRC_DIR.glob("**/*.cpp"))
53+
# Op extension (_C): this version's ops + inherited 2.9-2.13 csrc, minus the
54+
# interop module. The op extension is loaded via torch.ops.load_library like
55+
# the other libtorch_agn extensions; the interop module must instead be an
56+
# importable Python module (its PyMethodDef helpers hold the GIL), so it is
57+
# built as a separate extension below.
58+
op_sources = list(CSRC_DIR.glob("**/*.cpp"))
6059
for prev_dir in PREV_CSRC_DIRS:
61-
sources.extend(prev_dir.glob("**/*.cpp"))
60+
op_sources.extend(prev_dir.glob("**/*.cpp"))
61+
op_sources = [s for s in op_sources if s.name != "pyobject_interop_module.cpp"]
6262

63-
extension = CppExtension
63+
op_cxx = common_cxx + ["-DSTABLE_LIB_NAME=libtorch_agn_2_14"]
64+
op_extra = {"cxx": op_cxx}
65+
op_extension = CppExtension
6466
# allow including <cuda_runtime.h>
6567
if torch.cuda.is_available():
66-
extra_compile_args["cxx"].append("-DLAE_USE_CUDA")
67-
extra_compile_args["nvcc"] = [
68+
op_cxx.append("-DLAE_USE_CUDA")
69+
op_extra["nvcc"] = [
6870
"-O2",
6971
"-DUSE_CUDA",
7072
"-DTORCH_TARGET_VERSION=0x020e000000000000",
7173
"-DSTABLE_LIB_NAME=libtorch_agn_2_14",
7274
]
73-
extension = CUDAExtension
74-
sources.extend(CSRC_DIR.glob("**/*.cu"))
75+
op_extension = CUDAExtension
76+
op_sources.extend(CSRC_DIR.glob("**/*.cu"))
7577
for prev_dir in PREV_CSRC_DIRS:
76-
sources.extend(prev_dir.glob("**/*.cu"))
78+
op_sources.extend(prev_dir.glob("**/*.cu"))
7779

7880
return [
79-
extension(
81+
op_extension(
8082
"libtorch_agn_2_14._C",
81-
sources=sorted(str(s) for s in sources),
83+
sources=sorted(str(s) for s in op_sources),
8284
py_limited_api=True,
83-
extra_compile_args=extra_compile_args,
85+
extra_compile_args=op_extra,
8486
extra_link_args=[],
85-
)
87+
),
88+
# Interop extension (_interop): an importable abi3 module. Its
89+
# PyObject<->Tensor helpers are plain module functions, so it is imported
90+
# (which runs PyInit and exposes them) rather than loaded via
91+
# torch.ops.load_library like the op extension.
92+
CppExtension(
93+
"libtorch_agn_2_14._interop",
94+
sources=[str(CSRC_DIR / "pyobject_interop_module.cpp")],
95+
py_limited_api=True,
96+
extra_compile_args={
97+
"cxx": common_cxx + ["-DSTABLE_LIB_NAME=libtorch_agn_2_14_interop"]
98+
},
99+
),
86100
]
87101

88102

‎test/cpp_extensions/test_libtorch_agnostic.py‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -268,6 +268,50 @@ def test_has_storage(self, device):
268268
# Sparse tensors do not own a contiguous storage.
269269
self.assertFalse(libtorch_agnostic.ops.my_has_storage(t.to_sparse()))
270270

271+
# These exercise the use case: a raw PyObject passed straight from Python
272+
# (GIL held, no dispatcher boxing) into from_pyobject / to_pyobject, via the
273+
# extension's importable PyMethodDef module (_interop).
274+
@onlyCPU
275+
@skipIfTorchVersionLessThan(2, 14)
276+
def test_pyobject_roundtrip(self, device):
277+
import libtorch_agn_2_14 as libtorch_agnostic
278+
279+
x = torch.randn(3, 4, device=device)
280+
y = libtorch_agnostic._interop.pyobject_roundtrip(x)
281+
self.assertIsInstance(y, torch.Tensor)
282+
self.assertEqual(y, x)
283+
# from_pyobject / to_pyobject share the underlying TensorImpl.
284+
self.assertEqual(y.data_ptr(), x.data_ptr())
285+
x.add_(1)
286+
self.assertEqual(y, x)
287+
288+
@onlyCPU
289+
@skipIfTorchVersionLessThan(2, 14)
290+
def test_pyobject_sum(self, device):
291+
import libtorch_agn_2_14 as libtorch_agnostic
292+
293+
x = torch.randn(3, 4, device=device)
294+
s = libtorch_agnostic._interop.pyobject_sum(x)
295+
self.assertEqual(s, x.sum())
296+
297+
@onlyCPU
298+
@skipIfTorchVersionLessThan(2, 14)
299+
def test_pyobject_to_parameter_type(self, device):
300+
import libtorch_agn_2_14 as libtorch_agnostic
301+
302+
x = torch.randn(2, 2, device=device)
303+
p = libtorch_agnostic._interop.pyobject_to_type(x, torch.nn.Parameter)
304+
self.assertIsInstance(p, torch.nn.Parameter)
305+
self.assertEqual(p.detach(), x)
306+
307+
@onlyCPU
308+
@skipIfTorchVersionLessThan(2, 14)
309+
def test_pyobject_non_tensor_raises(self, device):
310+
import libtorch_agn_2_14 as libtorch_agnostic
311+
312+
with self.assertRaisesRegex(RuntimeError, "expected torch.Tensor"):
313+
libtorch_agnostic._interop.pyobject_roundtrip("not a tensor")
314+
271315
# TODO: Debug this:
272316
# torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors:
273317
# call_function libtorch_agnostic.my_ones_like.default(*(FakeTensor(..., size=(3, 1)), 'cpu'),

‎torch/csrc/PyObjectConversion.cpp‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
#include <torch/csrc/PyObjectConversion.h>
2+
3+
#include <c10/util/Exception.h>
4+
5+
namespace torch::detail {
6+
7+
namespace {
8+
9+
// Error message shared by both no-op entry points.
10+
constexpr const char* kNoImplMsg =
11+
"Converting between a Python object and a torch::stable::Tensor requires "
12+
"libtorch_python to be loaded (e.g. `import torch` in the running "
13+
"process). This process linked only libtorch.";
14+
15+
// Default implementation used until libtorch_python registers the real one.
16+
// Mirrors NoopPyInterpreterVTable: calling a method is a hard error rather than
17+
// silent misbehavior.
18+
struct NoopPyObjectConversion final : PyObjectConversionInterface {
19+
at::Tensor tensor_from_pyobject(PyObject* /*obj*/) const override {
20+
TORCH_CHECK(false, kNoImplMsg);
21+
}
22+
PyObject* tensor_to_pyobject(const at::Tensor& /*t*/, PyObject* /*py_type*/)
23+
const override {
24+
TORCH_CHECK(false, kNoImplMsg);
25+
}
26+
};
27+
28+
const NoopPyObjectConversion noop_impl;
29+
const PyObjectConversionInterface* g_impl = &noop_impl;
30+
31+
} // namespace
32+
33+
void setPyObjectConversionImpl(const PyObjectConversionInterface* impl) {
34+
g_impl = (impl != nullptr) ? impl : &noop_impl;
35+
}
36+
37+
const PyObjectConversionInterface& getPyObjectConversionImpl() {
38+
return *g_impl;
39+
}
40+
41+
} // namespace torch::detail

0 commit comments

Comments
 (0)