Repository navigation
[Feature Request] Abstract interface for data copy in Tensor. #328
Description
We are currently refactoring our runtime using tvm-ffi.
Instead of building only tvm-ffi, we build the full tvm-runtime, which enables additional device APIs and global functions, allowing us to execute real LLM models.
While enabling the device API, we encountered an issue and wanted to confirm our understanding.
In tvm-ffi, the Tensor API provides an NDAllocator, which abstracts device tensor memory allocation and free.
However, we cannot find any abstract API for data copy, either host → device or device → host.
We are planning to add this in our ffi, but it would be more natural if Tensor exposed a dedicated data-copy interface, such as a Rust trait or a C++ mixin, rather than relying on raw void* buffer copy.
We are wondering is there any roadmap in tvm-ffi providing an abstract interface for data-copy in Tensor.
Activity
tqchen commentedon Dec 12, 2025
Thanks a lot for the note, scope-wise we are explicitly avoiding DeviceAPI for now in the tvm-ffi layer, knowing that at the FFI ABI layer libraries may choose to implement device functions by explicitly using interface like cudaMemcpy.
Agree this is also quite useful in some settings, as of now tvm-runtime comes with such device API features, and one thing we can do is to streamline packaging bring up some support via tvm-runtime
jhlee525 commentedon Dec 12, 2025
Thank you for clarification!