Skip to content

Commit 5fa410c

Browse files
committed
feat(driver-native): add containerd-backed native compute driver
Adds openshell-driver-native, a fifth compute driver that constructs sandboxes from Linux kernel primitives (namespaces, cgroups v2) on top of a system-provided containerd, without a container engine daemon, a hypervisor, or a bundled low-level runtime. This implements two amendments to the originally filed issue NVIDIA#2255 proposal, agreed with the issue author outside the issue thread: - The low-level OCI runtime is never bundled. It is configurable (default `runc`, override to `crun` or any other OCI-runtime-spec binary already installed on the host). - **This driver -- not containerd -- spawns the low-level runtime.** containerd is used only for image pull/unpack and snapshot management (Transfer/Images/Content/Snapshots services); it never creates a containerd Container or Task object, and its io.containerd.runc.v2 shim is never involved. The driver builds a standard OCI bundle (config.json + a mounted rootfs/) from the snapshot containerd prepared and drives the configured runtime directly through its own create/start/state/kill/delete CLI contract -- the same integration pattern containerd's shim, CRI-O, and Podman use, just invoked by this driver's process instead of containerd's. Key pieces: - openshell-driver-native: image pull + chain-ID resolution + snapshot management against containerd, protected from containerd's background GC via a lease per sandbox since no Container/Task references it (image.rs); OCI runtime-spec generation from OpenShell policy (spec.rs); direct create/start/state/kill/delete invocation of the configured low-level runtime as a child process, with its own bundle-directory bookkeeping used to track which sandboxes exist instead of querying containerd (runtime.rs); per-sandbox network namespace + veth + nftables networking (network.rs); GPU kernel device-node passthrough (gpu.rs); and the ComputeDriver gRPC service (driver.rs/grpc.rs), running in-process in the gateway like the Docker/Podman/Kubernetes drivers. - openshell-nft-ruleset: the VM driver's nftables ruleset generator extracted into a shared crate (table-prefix-parameterized) so the native driver reuses the same NAT + default-deny shape without a behavior change to the VM driver's existing tables. - ComputeDriverKind::Native wired through openshell-core config, telemetry, and every openshell-server compute-driver selection path (driver_config.rs, compute/mod.rs, lib.rs, cli.rs, config_file.rs). Verified end to end against a real containerd 2.x + runc/crun install during development: pull -> chain-ID resolve -> lease-protected snapshot prepare -> bundle mount -> create/start/state/delete directly against both runc and crun -> stop -> delete (see tests/containerd_integration.rs, #[ignore]d by default since CI has no containerd), plus confirming a snapshot without a lease is reaped by containerd's GC while one with a lease survives, and that the full openshell-gateway binary connects and selects this driver correctly. Known gaps, documented in the driver README and user-facing docs: rootless mode (Linux user namespace) is implemented but not yet functional (containerd's prepared snapshot isn't remapped to match, so it defaults off); the supervisor binary is bind-mounted from a host path rather than sourced from an image like the other drivers; GPU support is kernel-device-node passthrough only, not full CDI injection; WatchSandboxes polls rather than subscribing to containerd's event stream. Related: NVIDIA#2255 Signed-off-by: Eric Curtin <[email protected]>
1 parent 8029321 commit 5fa410c

30 files changed

Lines changed: 4104 additions & 65 deletions

File tree

‎Cargo.lock‎

Lines changed: 68 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎architecture/build.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ collect telemetry: `openshell-server` (gateway), `openshell-sandbox`
2929
(supervisor), and `openshell-driver-vm`. Every crate depends on
3030
`openshell-core` with `default-features = false`, so the binary crate's feature
3131
is the single switch that enables `openshell-core/telemetry` for its build
32-
graph. In-process drivers (`docker`, `kubernetes`, `podman`) inherit the
33-
gateway's setting through feature unification and carry no passthrough.
32+
graph. In-process drivers (`docker`, `kubernetes`, `podman`, `native`) inherit
33+
the gateway's setting through feature unification and carry no passthrough.
3434

3535
Building a binary with `--no-default-features` compiles out telemetry entirely:
3636
no endpoint, no telemetry HTTP client, and no emission code. With telemetry

‎architecture/compute-runtimes.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,12 +34,14 @@ when a sandbox create request asks for GPU resources.
3434
| Podman | Rootless or single-machine deployments. | Container plus nested sandbox namespace. | Uses the Podman REST API, OCI image volumes, and CDI GPU devices when available. |
3535
| Kubernetes | Cluster deployment through Helm. | Pod plus nested sandbox namespace. | Uses Kubernetes API objects, service accounts, secrets, PVC-backed workspace storage, and GPU resources. |
3636
| VM | Experimental microVM isolation. | Per-sandbox libkrun VM. | Managed endpoint-backed driver. The gateway spawns `openshell-driver-vm`, waits for its Unix socket, and then consumes it through the same remote `compute_driver.proto` path used by unmanaged endpoint drivers. The VM driver boots a cached bootstrap `rootfs.ext4`, prepares requested OCI images inside a bootstrap VM with `umoci`, attaches the prepared image disk read-only, and gives each sandbox a writable `overlay.ext4` for merged-root changes and runtime material. The driver persists each accepted launch request beside the overlay and restarts those VMs on driver startup without recreating the overlay. |
37-
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, and `kubernetes` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
37+
| Native | Early/opt-in kernel-primitive isolation against an existing system `containerd`. | Container built from Linux namespaces + cgroups v2. | In-process driver, like Docker/Podman/Kubernetes. The driver itself — not containerd — spawns the configured `runtime_binary` (`runc` by default, `crun`, or another OCI-runtime-spec-compatible binary) directly through its `create`/`start`/`state`/`delete` CLI contract; containerd is used only for image pull/unpack and snapshot management (protected from containerd's GC via a lease, since no containerd `Container`/`Task` is ever created) and never bundled. Sandboxes are tracked by scanning the driver's own state directory rather than querying containerd or the runtime for a global list. Per-sandbox network namespace + veth pair + nftables ruleset (shared `openshell-nft-ruleset` crate, also used by the VM driver). GPU support is kernel-device-node passthrough only. Rootless mode is implemented but not yet functional (defaults off) — see `crates/openshell-driver-native/README.md`. |
38+
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a non-reserved custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. Reserved built-in names such as `vm`, `docker`, `podman`, `kubernetes`, and `native` cannot be used as unmanaged socket endpoints. The gateway connects to a UDS the operator already provisioned, runs `GetCapabilities`, logs the advertised `driver_name`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
3839

3940
Per-sandbox CPU and memory values currently enter the driver layer through
4041
template resource limits. Docker and Podman apply them as runtime limits.
4142
Kubernetes mirrors each limit into the matching request. VM accepts the fields
42-
but currently ignores them.
43+
but currently ignores them. Native applies them as cgroup v2 CPU quota/period
44+
and memory limits in the generated OCI spec.
4345

4446
Docker and Podman also accept per-sandbox driver-config mounts for existing
4547
runtime-managed named volumes and tmpfs mounts. Podman additionally accepts
@@ -72,6 +74,7 @@ Runtime-specific implementation notes belong in the driver crate README:
7274
- `crates/openshell-driver-podman/README.md`
7375
- `crates/openshell-driver-kubernetes/README.md`
7476
- `crates/openshell-driver-vm/README.md`
77+
- `crates/openshell-driver-native/README.md`
7578

7679
## Supervisor Delivery
7780

@@ -83,6 +86,7 @@ The supervisor must be available inside each sandbox workload:
8386
| Podman | Read-only OCI image volume containing the supervisor binary. |
8487
| Kubernetes | Supervisor image side-loaded into the sandbox pod by image volume or init container. |
8588
| VM | Embedded in the guest rootfs bundle. |
89+
| Native | Bind-mounted from a host path (`supervisor_binary_path`), not sourced from an image like the other in-tree drivers — see the driver README's Known Gaps. |
8690
| Extension | Defined by the out-of-tree driver. |
8791

8892
Driver-controlled environment variables must override sandbox image or template

‎crates/openshell-core/src/config.rs‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ pub enum ComputeDriverKind {
8888
Vm,
8989
Docker,
9090
Podman,
91+
Native,
9192
}
9293

9394
impl ComputeDriverKind {
@@ -98,6 +99,7 @@ impl ComputeDriverKind {
9899
Self::Vm => "vm",
99100
Self::Docker => "docker",
100101
Self::Podman => "podman",
102+
Self::Native => "native",
101103
}
102104
}
103105
}
@@ -138,8 +140,9 @@ impl FromStr for ComputeDriverKind {
138140
"vm" => Ok(Self::Vm),
139141
"docker" => Ok(Self::Docker),
140142
"podman" => Ok(Self::Podman),
143+
"native" => Ok(Self::Native),
141144
other => Err(format!(
142-
"unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman"
145+
"unsupported compute driver '{other}'. expected one of: kubernetes, vm, docker, podman, native"
143146
)),
144147
}
145148
}
@@ -990,6 +993,10 @@ mod tests {
990993
"docker".parse::<ComputeDriverKind>().unwrap(),
991994
ComputeDriverKind::Docker
992995
);
996+
assert_eq!(
997+
"native".parse::<ComputeDriverKind>().unwrap(),
998+
ComputeDriverKind::Native
999+
);
9931000
}
9941001

9951002
#[test]

‎crates/openshell-core/src/telemetry.rs‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -161,6 +161,7 @@ pub enum TelemetryComputeDriver {
161161
Kubernetes,
162162
Podman,
163163
Vm,
164+
Native,
164165
Unknown,
165166
}
166167

@@ -172,6 +173,7 @@ impl TelemetryComputeDriver {
172173
Self::Kubernetes => "kubernetes",
173174
Self::Podman => "podman",
174175
Self::Vm => "vm",
176+
Self::Native => "native",
175177
Self::Unknown => "unknown",
176178
}
177179
}
@@ -183,6 +185,7 @@ impl TelemetryComputeDriver {
183185
"k8s" | "kubernetes" => Self::Kubernetes,
184186
"podman" => Self::Podman,
185187
"vm" => Self::Vm,
188+
"native" => Self::Native,
186189
_ => Self::Unknown,
187190
}
188191
}
@@ -194,6 +197,7 @@ impl TelemetryComputeDriver {
194197
Some(crate::ComputeDriverKind::Kubernetes) => Self::Kubernetes,
195198
Some(crate::ComputeDriverKind::Podman) => Self::Podman,
196199
Some(crate::ComputeDriverKind::Vm) => Self::Vm,
200+
Some(crate::ComputeDriverKind::Native) => Self::Native,
197201
None => Self::Unknown,
198202
}
199203
}
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
[package]
5+
name = "openshell-driver-native"
6+
description = "Linux-native compute driver for OpenShell, backed by a system-provided containerd"
7+
version.workspace = true
8+
edition.workspace = true
9+
rust-version.workspace = true
10+
license.workspace = true
11+
repository.workspace = true
12+
13+
[lib]
14+
name = "openshell_driver_native"
15+
path = "src/lib.rs"
16+
17+
[[bin]]
18+
name = "openshell-driver-native"
19+
path = "src/main.rs"
20+
21+
[dependencies]
22+
openshell-core = { path = "../openshell-core", default-features = false }
23+
openshell-nft-ruleset = { path = "../openshell-nft-ruleset" }
24+
openshell-policy = { path = "../openshell-policy" }
25+
26+
# containerd's own generated gRPC client. This is the "system-provided
27+
# containerd" integration point, used only for image pull/unpack and
28+
# snapshot management (see src/image.rs) -- never bundled, installed, or
29+
# managed by this driver. The driver itself, not containerd, spawns the
30+
# configured low-level OCI runtime (runc/crun/etc.) directly (see
31+
# src/runtime.rs); containerd's container/task services and shim are
32+
# never used.
33+
containerd-client = "0.9"
34+
# OCI runtime-spec and image-spec types (namespaces, cgroups v2 resources,
35+
# seccomp, image manifest/config parsing for chain-ID computation).
36+
oci-spec = { version = "0.10", default-features = false, features = [
37+
"runtime",
38+
"image",
39+
] }
40+
41+
tokio = { workspace = true }
42+
tokio-stream = { workspace = true }
43+
tonic = { workspace = true, features = ["transport"] }
44+
prost = { workspace = true }
45+
prost-types = { workspace = true }
46+
futures = { workspace = true }
47+
clap = { workspace = true }
48+
tracing = { workspace = true }
49+
tracing-subscriber = { workspace = true }
50+
miette = { workspace = true }
51+
serde = { workspace = true }
52+
serde_json = { workspace = true }
53+
nix = { workspace = true }
54+
rand = { workspace = true }
55+
thiserror = { workspace = true }
56+
sha2 = "0.10"
57+
58+
[dev-dependencies]
59+
tempfile = "3"
60+
61+
[lints]
62+
workspace = true

0 commit comments

Comments
 (0)