You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c26ca9c
Browse filesBrowse the repository at this point in the historyBrowse files
Document the shared inline and reusable-template resource model, driver behavior, migration from legacy template resource sections, and updated SDK examples.
Part of #2838
Signed-off-by: Evan Lezar <[email protected]>
Copy file name to clipboardExpand all lines: architecture/compute-runtimes.md
+29-12Lines changed: 29 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -280,20 +280,26 @@ delete, reconciliation removes the row; otherwise it can remain `Deleting`.
280
280
| VM | Experimental microVM isolation. | Per-sandbox libkrun or QEMU VM. | The NIC-less guest runs `openshell-sandbox` as PID 1; host `openshell-supervisor` owns gateway networking and reaches the guest over vsock. |
281
281
| Extension | Out-of-tree drivers operated alongside the gateway. | Whatever boundary the driver implements. | Selected by a custom `compute_drivers = ["<name>"]` entry with `[openshell.drivers.<name>].socket_path`, or at launch time by pairing `--drivers <name>` with `--compute-driver-socket=<path>`. A launch-time endpoint may use a canonical built-in name to preserve its driver-config key while replacing in-process construction. The gateway connects to an operator-provisioned UDS, snapshots `GetCapabilities`, and dispatches all sandbox lifecycle calls through `compute_driver.proto`. The driver process and socket lifecycle are operator-owned; the gateway does not spawn, supervise, or remove unmanaged extension drivers. The trust boundary is the socket's filesystem permissions: the operator must ensure only the gateway uid can read/write it. |
282
282
283
-
Per-sandbox CPU and memory values currently enter the driver layer through
284
-
template resource limits. Docker and Podman apply them as runtime limits.
285
-
Kubernetes mirrors each limit into the matching request. VM accepts the fields
286
-
but currently ignores them.
283
+
Per-sandbox CPU, memory, and GPU requirements enter the driver layer through
284
+
the typed `ResourceRequirements` message. Docker and Podman apply CPU and
285
+
memory quantities as runtime limits. Kubernetes applies each quantity as a
286
+
limit and mirrors it into the matching request unless driver configuration
287
+
provides that request. VM and MXC reject unsupported CPU and memory requirements instead of
288
+
silently ignoring them.
287
289
288
290
Reusable sandbox workload templates are resolved before the compute-driver
289
291
boundary. Drivers do not receive a separate template resource; the gateway
290
292
lowers the selected `SandboxWorkloadTemplate` into the existing sandbox spec
291
293
and validates that spec before calling `ValidateSandboxCreate` or
292
-
`CreateSandbox`. Template CPU and memory become the same typed resource limits
293
-
described above. Template GPU settings become `ResourceRequirements`, preserving
294
-
the driver's default GPU assignment when the count is omitted. Template
295
-
`driver_config` remains a driver-keyed envelope until the compute layer selects
296
-
the active driver block and forwards only that block to the driver.
294
+
`CreateSandbox`. `SandboxWorkloadConfig.resources` reuses the public
295
+
`ResourceRequirements` message, so the gateway clones the typed requirements
296
+
directly into the resolved sandbox spec. Template `driver_config` remains a
297
+
driver-keyed envelope until the compute layer selects the active driver block
298
+
and forwards only that block to the driver. The gateway retains the former
299
+
workload-resource wire field solely at the storage boundary: persisted
300
+
templates decode through that legacy shape and are normalized to current
301
+
`ResourceRequirements` before Get, List, or sandbox creation. New API writes
302
+
cannot populate the legacy field.
297
303
298
304
Docker and Podman also accept per-sandbox driver-config mounts for existing
299
305
runtime-managed named volumes and tmpfs mounts. Podman additionally accepts
@@ -323,9 +329,20 @@ disjoint lifecycle ownership. A shared-mode gateway can target one external
323
329
namespace, while operator mode maps workspace names to multiple
324
330
platform-provisioned namespaces.
325
331
326
-
Resource requirements enter the driver layer through `SandboxSpec.resource_requirements`. This includes a set of GPU requirements, where a user
327
-
can request a specific number of GPUs or the driver-specific default behaviour.
328
-
For all in-tree drivers, this is equivalent to selecting a single GPU.
332
+
Resource requirements enter the driver layer through `SandboxSpec.resource_requirements`,
333
+
which carries typed GPU, CPU, and memory requirements as portable sandbox-sizing intent.
334
+
GPU requests let a user ask for a specific number of GPUs or the driver-specific default
335
+
behaviour; for all in-tree drivers, an unspecified count is equivalent to selecting a
336
+
single GPU. CPU and memory requirements each carry one optional portable `quantity`
337
+
(e.g. `"500m"`, `"4Gi"`). Docker and Podman apply quantities as limits. Kubernetes
338
+
applies them as limits and uses driver configuration for an independently tuned or
339
+
request-only value, falling back from the quantity when no driver-config request is set.
340
+
The VM and MXC drivers reject typed CPU/memory
341
+
requirements with a clear error until sizing support lands there, rather than silently
342
+
ignoring them.
343
+
`SandboxTemplate.resources` remains a platform-native escape hatch for non-portable fields
344
+
only — CPU/memory keys under it are rejected in favor of
Copy file name to clipboardExpand all lines: docs/reference/sandbox-compute-drivers.mdx
+39-15Lines changed: 39 additions & 15 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -89,11 +89,6 @@ The gateway connects to the operator-provided endpoint; it does not provision
89
89
or supervise the remote driver. The operator must protect the socket so only
90
90
the gateway uid can access it.
91
91
92
-
Sandbox create supports `--cpu` and `--memory` for per-sandbox compute sizing.
93
-
Docker and Podman apply them as runtime limits. Kubernetes applies them as both
94
-
container requests and limits. The VM driver accepts the fields but currently
95
-
ignores them.
96
-
97
92
Sandbox create also accepts experimental driver-owned config through
98
93
`--driver-config-json`. The value is a JSON object keyed by driver name. The
99
94
gateway forwards only the block for the active driver, so a Kubernetes gateway
@@ -133,19 +128,48 @@ request. The VM driver accepts `gpu_device_ids`, for example
133
128
accepts at most one entry and allows either `--gpu` or `--gpu 1` when
134
129
`gpu_device_ids` is set.
135
130
131
+
## Portable Resource Requirements
132
+
133
+
CPU, memory, and GPU requests use the typed
134
+
`SandboxSpec.resource_requirements` API:
135
+
136
+
-`cpu.quantity` and `memory.quantity` are optional portable quantities.
137
+
-`gpu` indicates a GPU request and can include an optional count.
138
+
139
+
Reusable workload templates use this same message at
140
+
`SandboxWorkloadConfig.resources`. The gateway copies those typed requirements
141
+
into the resolved sandbox spec when a sandbox is created from a template.
142
+
143
+
| Driver | CPU and memory behavior |
144
+
|---|---|
145
+
| Docker | Converts typed quantities to native container CPU and memory limits. |
146
+
| Podman | Converts typed quantities to native container CPU and memory limits. |
147
+
| Kubernetes | Applies typed quantities as limits. A quantity also becomes the request unless a Kubernetes driver-config request is set; driver config can also specify request-only resources. |
148
+
| VM | Rejects typed CPU and memory requirements. Configure gateway-wide `vcpus` and `mem_mib` instead. |
149
+
| MXC | Rejects typed CPU and memory requirements. |
150
+
151
+
The historical inline `SandboxTemplate.resources` remains available for platform-native resource
152
+
fields that do not have a portable representation. Do not place CPU or memory
153
+
under its `limits` or `requests` sections. The gateway rejects `limits.cpu`,
154
+
`limits.memory`, `requests.cpu`, and `requests.memory` and directs the caller to
155
+
the corresponding typed quantity. Kubernetes driver configuration may set
156
+
explicit CPU and memory requests. These configured requests replace the request
157
+
that Kubernetes would otherwise mirror from the portable quantity. The
158
+
portable quantity remains the limit.
159
+
136
160
## Resource Capability Reporting
137
161
138
-
API clients can read each configured driver's static resource request support
162
+
API clients can read each configured driver's static portable resource support
139
163
from the protected `GetGatewayInfo` response. The capability response describes
140
-
which request forms a driver implements. It does not report live resource
141
-
inventory, availability, or scheduling capacity, so a supported request can
142
-
still fail when the selected runtime cannot provide the resource.
143
-
144
-
CPU and memory capabilities report whether the driver enforces a resource
145
-
limit. GPU capabilities report whether the driver accepts a default GPU request
146
-
and an explicit GPU count. An omitted `resource_capabilities` message, or an
147
-
omitted CPU, memory, or GPU capability, means the driver did not report that
148
-
capability. A reported `false` value means that request form is unsupported.
164
+
which portable resource inputs a driver accepts. It does not report live
165
+
resource inventory, availability, or scheduling capacity, so a supported
166
+
request can still fail when the selected runtime cannot provide the resource.
167
+
168
+
CPU and memory capabilities report whether the driver accepts a portable
169
+
quantity. GPU capabilities report whether the driver accepts a default GPU
170
+
request and an explicit GPU count. An omitted `resource_capabilities` message,
171
+
or an omitted CPU, memory, or GPU capability, means the driver did not report
172
+
that capability. A reported `false` value means that input is unsupported.
149
173
150
174
For Kubernetes, `pod.runtime_class_name` maps to PodSpec `runtimeClassName`.
151
175
It overrides the gateway's configured default runtime class for that sandbox,
CPU values use Kubernetes-style quantities such as `500m`, `1`, or `2.5`.
87
87
Memory values use byte quantities such as `512Mi`, `4Gi`, or `8G`.
88
88
89
-
Docker and Podman apply these values as runtime limits. Kubernetes applies each
90
-
value as both the request and the limit so the scheduler reserves the same
91
-
amount the sandbox can use. The VM driver currently accepts these flags but
92
-
does not change VM allocation.
89
+
OpenShell stores these values as typed portable resource requirements. Docker
90
+
and Podman apply them as runtime limits. Kubernetes uses each quantity as its
91
+
limit and, unless its driver configuration provides a request, as its request.
92
+
The VM and MXC drivers reject
93
+
CPU or memory requirements because they do not currently support per-sandbox
94
+
sizing. For VM gateways, configure `vcpus` and `mem_mib` on the driver instead.
95
+
96
+
API and SDK callers set`cpu.quantity` and `memory.quantity`. Workloads that
97
+
need Kubernetes requests below those portable quantities can set requests in
98
+
the Kubernetes driver configuration. A driver-config request takes precedence
99
+
over the quantity-to-request fallback.
100
+
101
+
<Warning>
102
+
CPU and memory values under `SandboxTemplate.resources.limits` or
103
+
`SandboxTemplate.resources.requests` are no longer supported. API clients must
104
+
use `SandboxSpec.resource_requirements.cpu.quantity` and
105
+
`SandboxSpec.resource_requirements.memory.quantity`. The gateway
106
+
rejects legacy requests with a migration error.
107
+
</Warning>
93
108
94
109
### Driver-Specific Configuration
95
110
@@ -256,6 +271,15 @@ Create a sandbox from a template:
256
271
openshell sandbox create --template gpu-kata --provider github -- claude
257
272
```
258
273
274
+
When the gateway creates a sandbox from a template, it resolves the template's
275
+
CPU, memory, and GPU settings into the sandbox's typed `resource_requirements`.
276
+
The reusable template field `SandboxWorkloadConfig.resources` uses the same
277
+
`ResourceRequirements` message as `SandboxSpec.resource_requirements`, so API
278
+
and SDK callers use `cpu.quantity`, `memory.quantity`, and `gpu` in both places.
279
+
When loading a reusable template stored with the earlier `cpu`, `memory`, and
280
+
`gpu` shape, the gateway converts it to the current typed requirements before
281
+
returning or using the template.
282
+
259
283
The `--template` flag cannot be combined with inline workload flags such as `--from`, `--cpu`, `--memory`, `--gpu`, `--env`, or `--driver-config-json`. Put those values on the template instead. Create-time policy and provider attachments remain part of the sandbox request, so each sandbox can keep its own access boundary.
-`--gpu [COUNT]`: Request the driver's default GPU selection or a specific GPU count
258
-
-`--cpu`, `--memory`: Set per-sandbox compute sizing. Docker/Podman apply limits; Kubernetes applies matching requests and limits.
258
+
-`--cpu`, `--memory`: Set portable per-sandbox quantities. Docker/Podman apply them as limits. Kubernetes applies them as limits and uses them as matching requests unless Kubernetes driver config sets an explicit request.
-`--template NAME`: Create from a named sandbox workload template. Conflicts with inline workload flags such as `--from`, `--gpu`, `--cpu`, `--memory`, `--env`, and `--driver-config-json`.
261
261
-`--label KEY=VALUE`: Add labels for later selection (repeatable)
0 commit comments