fabricsim is a single-file libcuda.so.1 interposer that makes CU_MEM_HANDLE_TYPE_FABRIC work on an ordinary Linux box with a CUDA GPU but no NVLink fabric and no IMEX channel. It does not simulate GPU memory — every allocation is a real cuMemCreate with a POSIX file-descriptor handle type — it simulates the handle namespace: a fabric handle is 64 bytes of inert data that names an allocation without pinning it, and a file descriptor names the same allocation and can also cross a process boundary, so a 64-byte token plus an out-of-band way to fetch the descriptor behind it reproduces fabric's observable semantics.
On a 4×H100 box with no IMEX channel, cuda_core/tests/memory_ipc/test_vmm_fabric.py goes from 1 passed / 18 skipped to 17 passed / 1 skipped / 1 failed.
The 17 passes are real: cross-process export and import through Queue, Pipe, Process arguments and Pool, with actual GPU memory written in one process and read back in another; descriptor pickling and validation; stale-handle rejection after the exporting buffer closes; refusal to re-export imported memory; the "a queued buffer does not pin the allocation" semantic; and second-GPU owner-UUID reporting. The remaining skip is genuine — it needs a device reporting no fabric support. The remaining failure is a real defect in the code under test, not a simulator artifact.
A green run means cuda.core's descriptor, pickling, ownership and lifetime logic are correct. It is not evidence that the feature works on fabric hardware. It says nothing about the driver's fabric implementation (no NVLink or IMEX code path is reached), nothing about IMEX channel membership or cross-container authorization, nothing about multi-node behaviour (the transport is an abstract unix socket, which does not leave the host), and nothing about the driver's real status codes for stale or forged handles — it returns CUDA_ERROR_INVALID_VALUE because that is what it was told to return.
One measured divergence: a simulated fabric allocation reports requestedHandleTypes = POSIX_FILE_DESCRIPTOR where real fabric would report FABRIC. That is invisible to code comparing only location.type and location.id, but any check on the handle type would pass here and fail on hardware.
Label the CI job a simulator and keep a hardware run as the authority.
cuda-bindings resolves every driver entry point through exactly one dlsym, for cuGetProcAddress_v2 (cuda/bindings/_internal/driver_linux.pyx:607). Overriding that single symbol therefore captures both the Cython bindings and cuda.core's C++ _rt layer — which matters, because _rt.so has no DT_NEEDED on libcuda and no undefined cuMem* symbols, so LD_PRELOAD of cuMemCreate would intercept nothing.
The interposer forwards every query to the real driver and substitutes four functions:
| Function | Behaviour when handleType == FABRIC |
|---|---|
cuMemCreate |
rewrites the request to POSIX_FILE_DESCRIPTOR and calls through |
cuMemExportToShareableHandle |
returns a 64-byte token carrying {magic, pid, slot, generation} |
cuMemImportFromShareableHandle |
fetches the real fd from the exporter and calls through |
cuMemRelease |
retires the slot, so later imports see a stale handle |
Each exporting process runs one listener on an abstract unix socket fabricsim.<pid>. The importer connects, sends its slot and generation, and receives a freshly minted descriptor over SCM_RIGHTS. The fd is minted on demand and never retained, so a fabric export costs the exporter no file descriptor — which the test suite checks by counting /proc/self/fd. Slot liveness is handle liveness, which is how non-pinning is modelled.
cuMemImportFromShareableHandle deliberately dereferences osHandle without validating it, because the real driver does too; a guard there would hide exactly the class of caller bug that passes a non-pointer.
gcc -shared -fPIC -O2 -Wall -Wextra -o lib/libcuda.so.1 fabricsim.c \
-I"$CUDA_HOME/include" -ldl -lpthread
export LD_LIBRARY_PATH=$PWD/lib
pytest cuda_core/tests/memory_ipc/test_vmm_fabric.py -qEnvironment variables: FABRICSIM_VERBOSE=1 traces every hooked call to stderr; FABRICSIM_REAL_LIBCUDA=/path/to/libcuda.so.1 overrides the default /lib/x86_64-linux-gnu/libcuda.so.1.
Always run the transparency control first. Run the whole IPC suite with and without LD_LIBRARY_PATH and compare the non-fabric outcomes; they must be identical, or the interposer is perturbing the thing you are trying to measure. Measured on the reference box: 157 passed and 9 skipped either way, with only the fabric file's numbers moving.
Linux and a CUDA GPU required; x86-64 only as written, purely because of the default driver path.
/* fabricsim -- a libcuda.so.1 impersonator that makes CU_MEM_HANDLE_TYPE_FABRIC work
* on a machine with no NVLink fabric and no IMEX channel.
*
* It does NOT simulate GPU memory. Every allocation is a real cuMemCreate with a
* POSIX file descriptor handle type. What it simulates is the *handle namespace*:
* fabric's defining property is that the handle is 64 bytes of plain data that
* names an allocation and holds no reference to it. A POSIX fd names the same
* allocation and can also cross a process boundary, so a 64-byte token plus a
* way to fetch the fd behind it reproduces fabric's observable semantics.
*
* cuda_bindings resolves every driver entry point through exactly one dlsym,
* 'cuGetProcAddress_v2' (driver_linux.pyx:607), so overriding that single symbol
* captures both the bindings and cuda.core's C++ _rt layer.
*
* Transport: each process that exports runs one listener on an abstract unix
* socket "fabricsim.<pid>". A token carries the pid, a slot index and a slot
* generation; the importer connects, sends index+generation, and the exporter
* mints a fresh POSIX fd for the allocation and passes it back via SCM_RIGHTS.
* The fd is minted on demand and never held: a fabric handle costs the exporter
* no file descriptor, which the tests check by counting /proc/self/fd.
* The slot's liveness IS the handle's liveness, which is how non-pinning is
* modelled: cuMemRelease retires the slot, and a later import then fails
* exactly as a stale fabric handle does.
*
* Limits, stated so results are not over-read: this exercises cuda.core, not the
* driver's fabric implementation. It says nothing about IMEX permissions,
* multi-node behaviour, or the real driver's status codes for forged handles.
*/
#define _GNU_SOURCE
#include <cuda.h>
/* cuda.h does '#define cuGetProcAddress cuGetProcAddress_v2'; we define both. */
#undef cuGetProcAddress
#include <dlfcn.h>
#include <errno.h>
#include <pthread.h>
#include <stddef.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/uio.h>
#include <sys/un.h>
#include <unistd.h>
/* The real driver, by absolute path so we never dlopen ourselves. Override with
* FABRICSIM_REAL_LIBCUDA if your distribution puts it elsewhere. */
#define DEFAULT_REAL_LIBCUDA "/lib/x86_64-linux-gnu/libcuda.so.1"
static const char *real_libcuda_path(void)
{
const char *p = getenv("FABRICSIM_REAL_LIBCUDA");
return (p && *p) ? p : DEFAULT_REAL_LIBCUDA;
}
#define MAGIC "FABSIMv1"
#define MAX_SLOTS 4096
static int verbose;
#define LOG(...) do { if (verbose) { fprintf(stderr, "[fabricsim] " __VA_ARGS__); fputc('\n', stderr); } } while (0)
#define WARN(...) do { fprintf(stderr, "[fabricsim] " __VA_ARGS__); fputc('\n', stderr); } while (0)
typedef CUresult (*gpa_t)(const char *, void **, int, cuuint64_t, CUdriverProcAddressQueryResult *);
static gpa_t real_gpa;
static CUresult (*real_create)(CUmemGenericAllocationHandle *, size_t, const CUmemAllocationProp *, unsigned long long);
static CUresult (*real_export)(void *, CUmemGenericAllocationHandle, CUmemAllocationHandleType, unsigned long long);
static CUresult (*real_import)(CUmemGenericAllocationHandle *, void *, CUmemAllocationHandleType);
static CUresult (*real_release)(CUmemGenericAllocationHandle);
/* Resolved on the first fabric export so the listener thread can re-enter the
* driver on the exporter's context rather than on no context at all. */
static CUresult (*real_ctx_get)(CUcontext *);
static CUresult (*real_ctx_set)(CUcontext);
/* slot table: one entry per exported allocation. No fd is held; the exporter
* mints one per request, so an outstanding fabric handle costs nothing. */
static struct {
CUmemGenericAllocationHandle handle;
CUcontext ctx; /* the context current when the export was made */
unsigned generation; /* bumped when a retired slot is reused */
int live; /* cleared by cuMemRelease -> stale handle */
} slots[MAX_SLOTS];
static int nslots;
static pthread_mutex_t lock = PTHREAD_MUTEX_INITIALIZER;
static int listener_started;
static int srv_fd = -1;
struct fetch_request {
unsigned idx;
unsigned generation;
};
/* ---------------------------------------------------------------- transport */
static void sock_addr(struct sockaddr_un *a, pid_t pid)
{
memset(a, 0, sizeof(*a));
a->sun_family = AF_UNIX;
/* leading NUL => abstract namespace, no filesystem litter */
snprintf(a->sun_path + 1, sizeof(a->sun_path) - 1, "fabricsim.%d", (int)pid);
}
static socklen_t sock_len(struct sockaddr_un *a)
{
return (socklen_t)(offsetof(struct sockaddr_un, sun_path) + 1 + strlen(a->sun_path + 1));
}
static int read_all(int fd, void *buf, size_t n)
{
unsigned char *p = buf;
while (n) {
ssize_t got = read(fd, p, n);
if (got <= 0) return -1;
p += got;
n -= (size_t)got;
}
return 0;
}
static int write_all(int fd, const void *buf, size_t n)
{
const unsigned char *p = buf;
while (n) {
ssize_t put = write(fd, p, n);
if (put <= 0) return -1;
p += put;
n -= (size_t)put;
}
return 0;
}
static int send_fd(int conn, int fd)
{
char dummy = 'x';
char cbuf[CMSG_SPACE(sizeof(int))];
struct iovec iov = {.iov_base = &dummy, .iov_len = 1};
struct msghdr m = {.msg_iov = &iov, .msg_iovlen = 1, .msg_control = cbuf, .msg_controllen = sizeof(cbuf)};
memset(cbuf, 0, sizeof(cbuf));
struct cmsghdr *c = CMSG_FIRSTHDR(&m);
c->cmsg_level = SOL_SOCKET;
c->cmsg_type = SCM_RIGHTS;
c->cmsg_len = CMSG_LEN(sizeof(int));
memcpy(CMSG_DATA(c), &fd, sizeof(int));
return sendmsg(conn, &m, 0) < 0 ? -1 : 0;
}
static int recv_fd(int conn)
{
char dummy;
char cbuf[CMSG_SPACE(sizeof(int))];
struct iovec iov = {.iov_base = &dummy, .iov_len = 1};
struct msghdr m = {.msg_iov = &iov, .msg_iovlen = 1, .msg_control = cbuf, .msg_controllen = sizeof(cbuf)};
memset(cbuf, 0, sizeof(cbuf));
if (recvmsg(conn, &m, 0) <= 0) return -1;
struct cmsghdr *c = CMSG_FIRSTHDR(&m);
if (!c || c->cmsg_level != SOL_SOCKET || c->cmsg_type != SCM_RIGHTS) return -1;
int fd;
memcpy(&fd, CMSG_DATA(c), sizeof(int));
return fd;
}
static void serve(int conn)
{
struct fetch_request req;
if (read_all(conn, &req, sizeof(req)) < 0) return;
CUmemGenericAllocationHandle handle = 0;
CUcontext ctx = NULL;
int found = 0;
pthread_mutex_lock(&lock);
if (req.idx < (unsigned)nslots && slots[req.idx].live && slots[req.idx].generation == req.generation) {
handle = slots[req.idx].handle;
ctx = slots[req.idx].ctx;
found = 1;
}
pthread_mutex_unlock(&lock);
if (!found) {
LOG("serve: slot %u gen %u is retired or unknown -> stale handle", req.idx, req.generation);
return;
}
/* Mint the fd now, on the exporter's context, and drop it once it is sent. */
if (ctx && real_ctx_set) real_ctx_set(ctx);
int fd = -1;
CUresult rc = real_export(&fd, handle, CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, 0);
if (rc != CUDA_SUCCESS || fd < 0) {
LOG("serve: re-export of slot %u failed rc=%d", req.idx, (int)rc);
return;
}
if (send_fd(conn, fd) < 0) LOG("serve: sendmsg failed: %s", strerror(errno));
close(fd);
}
static void *listener_main(void *unused)
{
(void)unused;
for (;;) {
int conn = accept(srv_fd, NULL, NULL);
if (conn < 0) {
if (errno == EINTR) continue;
return NULL;
}
serve(conn);
close(conn);
}
}
/* Called before the first fabric handle can exist, so that the listener's one
* socket is already in /proc/self/fd when a test samples the count. */
static void ensure_listener(void)
{
pthread_mutex_lock(&lock);
if (listener_started) {
pthread_mutex_unlock(&lock);
return;
}
struct sockaddr_un a;
sock_addr(&a, getpid());
srv_fd = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (srv_fd < 0 || bind(srv_fd, (struct sockaddr *)&a, sock_len(&a)) < 0 || listen(srv_fd, 64) < 0) {
WARN("listener failed to bind @%s: %s", a.sun_path + 1, strerror(errno));
if (srv_fd >= 0) { close(srv_fd); srv_fd = -1; }
pthread_mutex_unlock(&lock);
return;
}
pthread_t t;
if (pthread_create(&t, NULL, listener_main, NULL) != 0) {
WARN("listener thread failed to start: %s", strerror(errno));
close(srv_fd);
srv_fd = -1;
pthread_mutex_unlock(&lock);
return;
}
pthread_detach(t);
listener_started = 1;
LOG("listening on @%s", a.sun_path + 1);
pthread_mutex_unlock(&lock);
}
/* A forked child has no listener thread and none of the parent's allocations,
* and the mutex may have been held by the listener at the moment of the fork. */
static void after_fork_child(void)
{
pthread_mutex_init(&lock, NULL);
if (srv_fd >= 0) close(srv_fd);
srv_fd = -1;
listener_started = 0;
nslots = 0;
memset(slots, 0, sizeof(slots));
}
/* ------------------------------------------------------------- interposers */
static CUresult my_create(CUmemGenericAllocationHandle *h, size_t size,
const CUmemAllocationProp *prop, unsigned long long flags)
{
if (prop && prop->requestedHandleTypes == CU_MEM_HANDLE_TYPE_FABRIC) {
CUmemAllocationProp p = *prop;
p.requestedHandleTypes = CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR;
CUresult rc = real_create(h, size, &p, flags);
if (rc == CUDA_SUCCESS) ensure_listener();
LOG("cuMemCreate(FABRIC, %zu) -> rc=%d (served as POSIX_FD)", size, (int)rc);
return rc;
}
return real_create(h, size, prop, flags);
}
static void resolve_ctx_fns(void)
{
if (real_ctx_get) return;
void *get = NULL, *set = NULL;
if (real_gpa("cuCtxGetCurrent", &get, 4000, CU_GET_PROC_ADDRESS_DEFAULT, NULL) == CUDA_SUCCESS &&
real_gpa("cuCtxSetCurrent", &set, 4000, CU_GET_PROC_ADDRESS_DEFAULT, NULL) == CUDA_SUCCESS) {
*(void **)&real_ctx_set = set;
*(void **)&real_ctx_get = get; /* set last: it is the published flag */
}
}
static CUresult my_export(void *out, CUmemGenericAllocationHandle h,
CUmemAllocationHandleType type, unsigned long long flags)
{
if (type != CU_MEM_HANDLE_TYPE_FABRIC) return real_export(out, h, type, flags);
ensure_listener();
if (srv_fd < 0) return CUDA_ERROR_NOT_PERMITTED; /* no transport, no fabric */
resolve_ctx_fns();
/* Ask the driver whether this allocation really is exportable, and report
* its answer; then throw the fd away, because a fabric handle holds none. */
int probe = -1;
CUresult rc = real_export(&probe, h, CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR, 0);
if (rc != CUDA_SUCCESS) {
LOG("export FABRIC: handle=%llu is not exportable, rc=%d", (unsigned long long)h, (int)rc);
return rc;
}
if (probe >= 0) close(probe);
CUcontext ctx = NULL;
if (real_ctx_get) real_ctx_get(&ctx);
pthread_mutex_lock(&lock);
/* Repeated exports of one allocation must return identical bytes. */
int idx = -1;
for (int i = 0; i < nslots; i++)
if (slots[i].live && slots[i].handle == h) { idx = i; break; }
if (idx < 0) {
for (int i = 0; i < nslots; i++)
if (!slots[i].live) { idx = i; break; }
if (idx >= 0) {
/* A new tenant in a retired slot: old tokens must not name it. */
slots[idx].generation++;
} else {
if (nslots >= MAX_SLOTS) { pthread_mutex_unlock(&lock); return CUDA_ERROR_OUT_OF_MEMORY; }
idx = nslots++;
}
slots[idx].handle = h;
slots[idx].ctx = ctx;
slots[idx].live = 1;
}
unsigned generation = slots[idx].generation;
pthread_mutex_unlock(&lock);
/* The driver fills its fields and leaves the padding; zero it, as the real one does. */
unsigned char *tok = (unsigned char *)out;
memset(tok, 0, 64);
memcpy(tok, MAGIC, 8);
int pid = (int)getpid();
unsigned u = (unsigned)idx;
memcpy(tok + 8, &pid, sizeof(pid));
memcpy(tok + 12, &u, sizeof(u));
memcpy(tok + 16, &generation, sizeof(generation));
LOG("export FABRIC: handle=%llu -> slot %d gen %u of pid %d", (unsigned long long)h, idx, generation, pid);
return CUDA_SUCCESS;
}
static CUresult my_import(CUmemGenericAllocationHandle *h, void *osHandle, CUmemAllocationHandleType type)
{
if (type != CU_MEM_HANDLE_TYPE_FABRIC) return real_import(h, osHandle, type);
/* No guard on the pointer: the real driver dereferences it too, and a
* simulator that validated here would hide the forged-descriptor bug. */
const unsigned char *tok = (const unsigned char *)osHandle;
if (memcmp(tok, MAGIC, 8) != 0) {
LOG("import FABRIC: not one of our tokens -> forged");
return CUDA_ERROR_INVALID_VALUE;
}
int pid;
struct fetch_request req;
memcpy(&pid, tok + 8, sizeof(pid));
memcpy(&req.idx, tok + 12, sizeof(req.idx));
memcpy(&req.generation, tok + 16, sizeof(req.generation));
struct sockaddr_un a;
sock_addr(&a, (pid_t)pid);
int c = socket(AF_UNIX, SOCK_STREAM | SOCK_CLOEXEC, 0);
if (c < 0) return CUDA_ERROR_INVALID_VALUE;
if (connect(c, (struct sockaddr *)&a, sock_len(&a)) < 0) {
close(c);
LOG("import FABRIC: exporter pid %d is gone -> stale", pid);
return CUDA_ERROR_INVALID_VALUE; /* exporter exited: a stale handle */
}
if (write_all(c, &req, sizeof(req)) < 0) { close(c); return CUDA_ERROR_INVALID_VALUE; }
int fd = recv_fd(c);
close(c);
if (fd < 0) {
LOG("import FABRIC: slot %u gen %u retired -> stale", req.idx, req.generation);
return CUDA_ERROR_INVALID_VALUE; /* exporting buffer was closed */
}
CUresult rc = real_import(h, (void *)(uintptr_t)fd, CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR);
close(fd); /* the driver keeps no reference to the fd */
LOG("import FABRIC: slot %u of pid %d -> rc=%d", req.idx, pid, (int)rc);
return rc;
}
static CUresult my_release(CUmemGenericAllocationHandle h)
{
pthread_mutex_lock(&lock);
for (int i = 0; i < nslots; i++)
if (slots[i].live && slots[i].handle == h) {
slots[i].live = 0; /* fabric pins nothing: the name goes stale */
LOG("release: slot %d retired", i);
}
pthread_mutex_unlock(&lock);
return real_release(h);
}
/* --------------------------------------------------------------- the hook */
__attribute__((constructor)) static void init(void)
{
verbose = getenv("FABRICSIM_VERBOSE") != NULL;
const char *path = real_libcuda_path();
void *real = dlopen(path, RTLD_NOW | RTLD_LOCAL);
if (!real) { WARN("cannot load %s: %s", path, dlerror()); abort(); }
real_gpa = (gpa_t)dlsym(real, "cuGetProcAddress_v2");
if (!real_gpa) { WARN("no cuGetProcAddress_v2 in %s", path); abort(); }
pthread_atfork(NULL, NULL, after_fork_child);
LOG("armed against %s", path);
}
CUresult cuGetProcAddress_v2(const char *name, void **pfn, int version,
cuuint64_t flags, CUdriverProcAddressQueryResult *res)
{
CUresult rc = real_gpa(name, pfn, version, flags, res);
if (rc != CUDA_SUCCESS || !pfn || !*pfn || !name) return rc;
#define HOOK(sym, slot, impl) \
if (strcmp(name, sym) == 0) { \
*(void **)&(slot) = *pfn; \
*pfn = (void *)(impl); \
LOG("hooked %s", sym); \
}
HOOK("cuMemCreate", real_create, my_create)
else HOOK("cuMemExportToShareableHandle", real_export, my_export)
else HOOK("cuMemImportFromShareableHandle", real_import, my_import)
else HOOK("cuMemRelease", real_release, my_release)
#undef HOOK
return rc;
}
/* Some callers still ask for the v1 name; forward it so nothing regresses. */
CUresult cuGetProcAddress(const char *name, void **pfn, int version, cuuint64_t flags)
{
return cuGetProcAddress_v2(name, pfn, version, flags, NULL);
}