Skip to content

Commit c26ca9c

Browse files
committed
docs: explain unified portable resource requirements
Document the shared inline and reusable-template resource model, driver behavior, migration from legacy template resource sections, and updated SDK examples. Part of #2838 Signed-off-by: Evan Lezar <[email protected]>
1 parent d919c3d commit c26ca9c

8 files changed

Lines changed: 135 additions & 43 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 29 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -280,20 +280,26 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
280280
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
281281
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
282282

283-
Per-sandbox CPU and memory values currently enter the driver layer through
284-
template resource limits. Docker and Podman apply them as runtime limits.
285-
Kubernetes mirrors each limit into the matching request. VM accepts the fields
286-
but currently ignores them.
283+
Per-sandbox CPU, memory, and GPU requirements enter the driver layer through
284+
the typed `ResourceRequirements` message. Docker and Podman apply CPU and
285+
memory quantities as runtime limits. Kubernetes applies each quantity as a
286+
limit and mirrors it into the matching request unless driver configuration
287+
provides that request. VM and MXC reject unsupported CPU and memory requirements instead of
288+
silently ignoring them.
287289

288290
Reusable sandbox workload templates are resolved before the compute-driver
289291
boundary. Drivers do not receive a separate template resource; the gateway
290292
lowers the selected `SandboxWorkloadTemplate` into the existing sandbox spec
291293
and validates that spec before calling `ValidateSandboxCreate` or
292-
`CreateSandbox`. Template CPU and memory become the same typed resource limits
293-
described above. Template GPU settings become `ResourceRequirements`, preserving
294-
the driver's default GPU assignment when the count is omitted. Template
295-
`driver_config` remains a driver-keyed envelope until the compute layer selects
296-
the active driver block and forwards only that block to the driver.
294+
`CreateSandbox`. `SandboxWorkloadConfig.resources` reuses the public
295+
`ResourceRequirements` message, so the gateway clones the typed requirements
296+
directly into the resolved sandbox spec. Template `driver_config` remains a
297+
driver-keyed envelope until the compute layer selects the active driver block
298+
and forwards only that block to the driver. The gateway retains the former
299+
workload-resource wire field solely at the storage boundary: persisted
300+
templates decode through that legacy shape and are normalized to current
301+
`ResourceRequirements` before Get, List, or sandbox creation. New API writes
302+
cannot populate the legacy field.
297303

298304
Docker and Podman also accept per-sandbox driver-config mounts for existing
299305
runtime-managed named volumes and tmpfs mounts. Podman additionally accepts
@@ -323,9 +329,20 @@ disjoint lifecycle ownership. A shared-mode gateway can target one external
323329
namespace, while operator mode maps workspace names to multiple
324330
platform-provisioned namespaces.
325331

326-
Resource requirements enter the driver layer through `SandboxSpec.resource_requirements`. This includes a set of GPU requirements, where a user
327-
can request a specific number of GPUs or the driver-specific default behaviour.
328-
For all in-tree drivers, this is equivalent to selecting a single GPU.
332+
Resource requirements enter the driver layer through `SandboxSpec.resource_requirements`,
333+
which carries typed GPU, CPU, and memory requirements as portable sandbox-sizing intent.
334+
GPU requests let a user ask for a specific number of GPUs or the driver-specific default
335+
behaviour; for all in-tree drivers, an unspecified count is equivalent to selecting a
336+
single GPU. CPU and memory requirements each carry one optional portable `quantity`
337+
(e.g. `"500m"`, `"4Gi"`). Docker and Podman apply quantities as limits. Kubernetes
338+
applies them as limits and uses driver configuration for an independently tuned or
339+
request-only value, falling back from the quantity when no driver-config request is set.
340+
The VM and MXC drivers reject typed CPU/memory
341+
requirements with a clear error until sizing support lands there, rather than silently
342+
ignoring them.
343+
`SandboxTemplate.resources` remains a platform-native escape hatch for non-portable fields
344+
only — CPU/memory keys under it are rejected in favor of
345+
`resource_requirements.cpu`/`resource_requirements.memory`.
329346

330347
VM runtime state paths are derived only from driver-validated sandbox IDs
331348
matching `[A-Za-z0-9._-]{1,128}`. The gateway-owned VM driver socket uses a

‎crates/openshell-sdk/README.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,8 @@ while let Some(page) = pages.next_page().await? {
9494

9595
```rust
9696
use openshell_sdk::{
97-
ClientConfig, OpenShellClient, SandboxTemplateCreateSpec,
97+
ClientConfig, CpuResourceRequirements, MemoryResourceRequirements, OpenShellClient,
98+
ResourceRequirements, SandboxTemplateCreateSpec,
9899
SandboxWorkloadConfig, SandboxWorkloadTemplate, SandboxWorkloadTemplateSpec,
99100
};
100101

@@ -109,6 +110,11 @@ client
109110
spec: Some(SandboxWorkloadTemplateSpec {
110111
workload: Some(SandboxWorkloadConfig {
111112
image: "ghcr.io/nvidia/openshell-community/sandboxes/python:latest".to_string(),
113+
resources: Some(ResourceRequirements {
114+
cpu: Some(CpuResourceRequirements { quantity: Some("1".to_string()) }),
115+
memory: Some(MemoryResourceRequirements { quantity: Some("512Mi".to_string()) }),
116+
..Default::default()
117+
}),
112118
..Default::default()
113119
}),
114120
..Default::default()

‎docs/reference/sandbox-compute-drivers.mdx‎

Lines changed: 39 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -89,11 +89,6 @@ The gateway connects to the operator-provided endpoint; it does not provision
8989
or supervise the remote driver. The operator must protect the socket so only
9090
the gateway uid can access it.
9191

92-
Sandbox create supports `--cpu` and `--memory` for per-sandbox compute sizing.
93-
Docker and Podman apply them as runtime limits. Kubernetes applies them as both
94-
container requests and limits. The VM driver accepts the fields but currently
95-
ignores them.
96-
9792
Sandbox create also accepts experimental driver-owned config through
9893
`--driver-config-json`. The value is a JSON object keyed by driver name. The
9994
gateway forwards only the block for the active driver, so a Kubernetes gateway
@@ -133,19 +128,48 @@ request. The VM driver accepts `gpu_device_ids`, for example
133128
accepts at most one entry and allows either `--gpu` or `--gpu 1` when
134129
`gpu_device_ids` is set.
135130

131+
## Portable Resource Requirements
132+
133+
CPU, memory, and GPU requests use the typed
134+
`SandboxSpec.resource_requirements` API:
135+
136+
- `cpu.quantity` and `memory.quantity` are optional portable quantities.
137+
- `gpu` indicates a GPU request and can include an optional count.
138+
139+
Reusable workload templates use this same message at
140+
`SandboxWorkloadConfig.resources`. The gateway copies those typed requirements
141+
into the resolved sandbox spec when a sandbox is created from a template.
142+
143+
| Driver | CPU and memory behavior |
144+
|---|---|
145+
| Docker | Converts typed quantities to native container CPU and memory limits. |
146+
| Podman | Converts typed quantities to native container CPU and memory limits. |
147+
| Kubernetes | Applies typed quantities as limits. A quantity also becomes the request unless a Kubernetes driver-config request is set; driver config can also specify request-only resources. |
148+
| VM | Rejects typed CPU and memory requirements. Configure gateway-wide `vcpus` and `mem_mib` instead. |
149+
| MXC | Rejects typed CPU and memory requirements. |
150+
151+
The historical inline `SandboxTemplate.resources` remains available for platform-native resource
152+
fields that do not have a portable representation. Do not place CPU or memory
153+
under its `limits` or `requests` sections. The gateway rejects `limits.cpu`,
154+
`limits.memory`, `requests.cpu`, and `requests.memory` and directs the caller to
155+
the corresponding typed quantity. Kubernetes driver configuration may set
156+
explicit CPU and memory requests. These configured requests replace the request
157+
that Kubernetes would otherwise mirror from the portable quantity. The
158+
portable quantity remains the limit.
159+
136160
## Resource Capability Reporting
137161

138-
API clients can read each configured driver's static resource request support
162+
API clients can read each configured driver's static portable resource support
139163
from the protected `GetGatewayInfo` response. The capability response describes
140-
which request forms a driver implements. It does not report live resource
141-
inventory, availability, or scheduling capacity, so a supported request can
142-
still fail when the selected runtime cannot provide the resource.
143-
144-
CPU and memory capabilities report whether the driver enforces a resource
145-
limit. GPU capabilities report whether the driver accepts a default GPU request
146-
and an explicit GPU count. An omitted `resource_capabilities` message, or an
147-
omitted CPU, memory, or GPU capability, means the driver did not report that
148-
capability. A reported `false` value means that request form is unsupported.
164+
which portable resource inputs a driver accepts. It does not report live
165+
resource inventory, availability, or scheduling capacity, so a supported
166+
request can still fail when the selected runtime cannot provide the resource.
167+
168+
CPU and memory capabilities report whether the driver accepts a portable
169+
quantity. GPU capabilities report whether the driver accepts a default GPU
170+
request and an explicit GPU count. An omitted `resource_capabilities` message,
171+
or an omitted CPU, memory, or GPU capability, means the driver did not report
172+
that capability. A reported `false` value means that input is unsupported.
149173

150174
For Kubernetes, `pod.runtime_class_name` maps to PodSpec `runtimeClassName`.
151175
It overrides the gateway's configured default runtime class for that sandbox,

‎docs/sandboxes/manage-sandboxes.mdx‎

Lines changed: 29 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ openshell gateway select local
7777
7878
### CPU and Memory
7979
80-
Set per-sandbox CPU and memory amounts with `--cpu` and `--memory`:
80+
Set portable CPU and memory quantities with `--cpu` and `--memory`:
8181
8282
```shell
8383
openshell sandbox create --cpu 2 --memory 4Gi -- claude
@@ -86,10 +86,25 @@ openshell sandbox create --cpu 2 --memory 4Gi -- claude
8686
CPU values use Kubernetes-style quantities such as `500m`, `1`, or `2.5`.
8787
Memory values use byte quantities such as `512Mi`, `4Gi`, or `8G`.
8888
89-
Docker and Podman apply these values as runtime limits. Kubernetes applies each
90-
value as both the request and the limit so the scheduler reserves the same
91-
amount the sandbox can use. The VM driver currently accepts these flags but
92-
does not change VM allocation.
89+
OpenShell stores these values as typed portable resource requirements. Docker
90+
and Podman apply them as runtime limits. Kubernetes uses each quantity as its
91+
limit and, unless its driver configuration provides a request, as its request.
92+
The VM and MXC drivers reject
93+
CPU or memory requirements because they do not currently support per-sandbox
94+
sizing. For VM gateways, configure `vcpus` and `mem_mib` on the driver instead.
95+
96+
API and SDK callers set `cpu.quantity` and `memory.quantity`. Workloads that
97+
need Kubernetes requests below those portable quantities can set requests in
98+
the Kubernetes driver configuration. A driver-config request takes precedence
99+
over the quantity-to-request fallback.
100+
101+
<Warning>
102+
CPU and memory values under `SandboxTemplate.resources.limits` or
103+
`SandboxTemplate.resources.requests` are no longer supported. API clients must
104+
use `SandboxSpec.resource_requirements.cpu.quantity` and
105+
`SandboxSpec.resource_requirements.memory.quantity`. The gateway
106+
rejects legacy requests with a migration error.
107+
</Warning>
93108
94109
### Driver-Specific Configuration
95110
@@ -256,6 +271,15 @@ Create a sandbox from a template:
256271
openshell sandbox create --template gpu-kata --provider github -- claude
257272
```
258273
274+
When the gateway creates a sandbox from a template, it resolves the template's
275+
CPU, memory, and GPU settings into the sandbox's typed `resource_requirements`.
276+
The reusable template field `SandboxWorkloadConfig.resources` uses the same
277+
`ResourceRequirements` message as `SandboxSpec.resource_requirements`, so API
278+
and SDK callers use `cpu.quantity`, `memory.quantity`, and `gpu` in both places.
279+
When loading a reusable template stored with the earlier `cpu`, `memory`, and
280+
`gpu` shape, the gateway converts it to the current typed requirements before
281+
returning or using the template.
282+
259283
The `--template` flag cannot be combined with inline workload flags such as `--from`, `--cpu`, `--memory`, `--gpu`, `--env`, or `--driver-config-json`. Put those values on the template instead. Create-time policy and provider attachments remain part of the sandbox request, so each sandbox can keep its own access boundary.
260284
261285
Inspect and manage templates:

‎sdk/go/docs/src/api/sandbox-templates.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,10 @@ template, err := client.SandboxTemplates().Create(ctx, "default", &v1.SandboxWor
2929
Environment: map[string]string{
3030
"NVIDIA_VISIBLE_DEVICES": "all",
3131
},
32-
Resources: &v1.SandboxResources{
33-
CPU: "2",
34-
Memory: "8Gi",
35-
GPU: &v1.SandboxGPURequirements{Count: &gpuCount},
32+
Resources: &v1.ResourceRequirements{
33+
CPU: &v1.CPUResourceRequirements{Quantity: "2"},
34+
Memory: &v1.MemoryResourceRequirements{Quantity: "8Gi"},
35+
GPU: &v1.GPUResourceRequirements{Count: &gpuCount},
3636
},
3737
},
3838
DriverConfig: map[string]any{
@@ -52,7 +52,7 @@ template, err := client.SandboxTemplates().Create(ctx, "default", &v1.SandboxWor
5252
})
5353
```
5454

55-
Set `GPU: &v1.SandboxGPURequirements{}` to request the active driver's default
55+
Set `GPU: &v1.GPUResourceRequirements{}` to request the active driver's default
5656
GPU assignment without specifying a count.
5757

5858
## Create a Sandbox From a Template

‎sdk/go/docs/src/api/sandboxes.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,10 @@ sb, err := client.Sandboxes().Create(ctx, "default", "my-sandbox", &v1.SandboxSp
2020
})
2121
```
2222

23-
Set `GPU: true` to request the active driver's default GPU assignment. Set
24-
`GPUCount` when the sandbox needs a specific GPU count; a non-nil `GPUCount`
25-
also implies `GPU`.
23+
Set `ResourceRequirements.GPU` to request a GPU. Leave its `Count` nil to use
24+
the active driver's default GPU assignment, or set `Count` to request a
25+
specific number. `ResourceRequirements.CPU` and `.Memory` accept portable
26+
quantity limits such as `500m` and `2Gi`.
2627

2728
## Create From Template
2829

‎sdk/typescript/README.md‎

Lines changed: 21 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ const client = await OpenShellClient.connect({
3535

3636
const sandbox = await client.sandbox.create({
3737
image: 'ghcr.io/nvidia/openshell-community/sandboxes/python:latest',
38+
resourceRequirements: {
39+
cpu: { quantity: '2' },
40+
memory: { quantity: '4Gi' },
41+
},
3842
})
3943
await client.sandbox.waitReady(sandbox.name, 120)
4044

@@ -87,6 +91,22 @@ in memory, and renews before expiry. The root client has no explicit close
8791
method because Connect does not retain a dedicated session. Close
8892
operation-scoped streams and forward handles instead.
8993

94+
Set portable compute requirements with `resourceRequirements`. CPU and memory
95+
limits use Kubernetes-style quantities. Set `gpu: {}` to request the active
96+
driver's default GPU assignment, or set `gpu.count` to request a specific
97+
number:
98+
99+
```ts
100+
await client.sandbox.create({
101+
image,
102+
resourceRequirements: {
103+
cpu: { quantity: '500m' },
104+
memory: { quantity: '2Gi' },
105+
gpu: { count: 1 },
106+
},
107+
})
108+
```
109+
90110
Express the create-time safety boundary with `policy`. Sandbox-scoped `setPolicy`
91111
cannot introduce static policy fields later, so set filesystem, landlock,
92112
process, and initial network policy at creation. For proto spec fields the
@@ -188,7 +208,7 @@ const template: SandboxWorkloadTemplate = await client.sandboxTemplates.create(
188208
workload: {
189209
image: 'ghcr.io/nvidia/openshell-community/sandboxes/python:latest',
190210
environment: { FEATURE_FLAG: 'on' },
191-
resources: { cpu: '1', memory: '512Mi' },
211+
resources: { cpu: { quantity: '1' }, memory: { quantity: '512Mi' } },
192212
},
193213
driverConfig: { kubernetes: { pod: { runtime_class_name: 'kata-containers' } } },
194214
},

‎skills/openshell-cli/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,7 @@ Key flags:
255255
- `--provider`: Attach configured credential providers for API keys, tokens, and other secrets (repeatable)
256256
- `--policy`: Custom policy YAML (otherwise uses built-in default or `OPENSHELL_SANDBOX_POLICY` env var)
257257
- `--gpu [COUNT]`: Request the driver's default GPU selection or a specific GPU count
258-
- `--cpu`, `--memory`: Set per-sandbox compute sizing. Docker/Podman apply limits; Kubernetes applies matching requests and limits.
258+
- `--cpu`, `--memory`: Set portable per-sandbox quantities. Docker/Podman apply them as limits. Kubernetes applies them as limits and uses them as matching requests unless Kubernetes driver config sets an explicit request.
259259
- `--driver-config-json`: Pass experimental driver-specific sandbox configuration
260260
- `--template NAME`: Create from a named sandbox workload template. Conflicts with inline workload flags such as `--from`, `--gpu`, `--cpu`, `--memory`, `--env`, and `--driver-config-json`.
261261
- `--label KEY=VALUE`: Add labels for later selection (repeatable)

0 commit comments

Comments
 (0)