Skip to content

Commit 9301387

Browse files
feat(sandbox): use policy-first OCI image identity
Closes #2331 Preserve per-field policy omission, derive Docker and Podman fallbacks from the inspected immutable image, and resolve the final numeric identity before starting agent children. Signed-off-by: Matthew Grossman <[email protected]>
1 parent 2b7f04f commit 9301387

29 files changed

Lines changed: 1187 additions & 236 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,7 @@ Common findings:
148148
- Docker daemon unavailable: start Docker Desktop or Docker Engine.
149149
- Gateway process stopped: inspect exit status and logs.
150150
- Sandbox image missing or pull denied: verify image reference and registry credentials.
151+
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected.
151152
- Docker driver cannot initialize because it cannot find `openshell-sandbox`: verify `OPENSHELL_DOCKER_SUPERVISOR_BIN`, the sibling binary next to `openshell-gateway`, or the configured supervisor image contains `/openshell-sandbox`.
152153
- Sandbox never registers: check gateway logs and supervisor callback endpoint.
153154
- Supervisor image exits before printing `openshell-sandbox --version`: the image should be the scratch supervisor image from `deploy/docker/Dockerfile.supervisor` and must contain a static executable at `/openshell-sandbox`.
@@ -173,6 +174,7 @@ Common findings:
173174
- Podman socket unavailable: start or expose the user socket.
174175
- Rootless networking unavailable: inspect Podman network configuration.
175176
- Sandbox image missing or pull denied: verify image reference and registry credentials.
177+
- Sandbox fails before readiness with an identity-resolution error: inspect the image's OCI `USER` and matching `/etc/passwd` and `/etc/group` entries, or explicitly set both process identity fields in policy. Root and missing identities are rejected.
176178
- Supervisor cannot call back: check callback endpoint and gateway logs.
177179

178180
### Step 6: Check Kubernetes Helm Gateways

‎.agents/skills/generate-sandbox-policy/SKILL.md‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -473,15 +473,17 @@ filesystem_policy:
473473
landlock:
474474
compatibility: best_effort
475475

476-
process:
477-
run_as_user: sandbox
478-
run_as_group: sandbox
479-
480476
network_policies:
481477
# <generated policies go here>
482478
```
483479

484-
The `filesystem_policy`, `landlock`, and `process` sections above are sensible defaults. Tell the user these are defaults and may need adjustment for their environment. Gateway inference is configured separately through `openshell inference set/get`. The generated `network_policies` block is the primary output.
480+
The `filesystem_policy` and `landlock` sections above are sensible defaults.
481+
Process identity is omitted so the selected compute driver can choose it. For
482+
Docker and Podman, each omitted identity field falls back to the image's OCI
483+
`USER`. Tell the user these are defaults and may need adjustment for their
484+
environment. Gateway inference is configured separately through `openshell
485+
inference set/get`. The generated `network_policies` block is the primary
486+
output.
485487

486488
If the user provides a file path, write to it. Otherwise, ask where to place it. A common convention is a project-local policy file (e.g., `sandbox-policy.yaml`) passed to `openshell sandbox create --policy <path>` or set via the `OPENSHELL_SANDBOX_POLICY` env var.
487489

‎.agents/skills/generate-sandbox-policy/examples.md‎

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -830,10 +830,6 @@ filesystem_policy:
830830
landlock:
831831
compatibility: best_effort
832832
833-
process:
834-
run_as_user: sandbox
835-
run_as_group: sandbox
836-
837833
network_policies:
838834
github_readonly:
839835
name: github_readonly
@@ -858,7 +854,10 @@ network_policies:
858854
- { path: /usr/local/bin/claude }
859855
```
860856

861-
The agent notes that `filesystem_policy`, `landlock`, and `process` are sensible defaults that may need adjustment, and that gateway inference is configured separately via `openshell inference set/get` rather than an `inference` policy block.
857+
The agent notes that `filesystem_policy` and `landlock` are sensible defaults
858+
that may need adjustment. Process identity is omitted so the compute driver can
859+
select it. Gateway inference is configured separately via `openshell inference
860+
set/get` rather than an `inference` policy block.
862861

863862
---
864863

‎.agents/skills/openshell-cli/SKILL.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -422,6 +422,12 @@ The `--from` flag accepts a Dockerfile path, a directory containing a Dockerfile
422422

423423
Local Dockerfile and directory builds require a local gateway because the CLI builds through the local Docker daemon. Use a registry image reference for remote gateways. Bare community names resolve under `ghcr.io/nvidia/openshell-community/sandboxes` unless `OPENSHELL_COMMUNITY_REGISTRY` overrides the prefix.
424424

425+
For Docker and Podman gateways, custom images should declare a non-root OCI
426+
`USER`. Each explicit `process.run_as_user` or `process.run_as_group` policy
427+
field wins independently; omitted fields fall back to the image declaration.
428+
An image with no `USER` fails before readiness unless policy supplies both
429+
fields.
430+
425431
### Forward ports
426432

427433
```bash

‎architecture/compute-runtimes.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,6 +125,29 @@ Driver-controlled environment variables must override sandbox image or template
125125
values for sandbox ID, sandbox name, gateway endpoint, relay socket path, TLS
126126
paths, and command metadata.
127127

128+
## Process Identity
129+
130+
The gateway preserves whether each policy process field was omitted. The active
131+
driver then supplies one authoritative identity input to the supervisor:
132+
133+
- Docker and Podman inspect the final sandbox image, pin container creation to
134+
its immutable image ID, and pass its raw OCI `Config.User`.
135+
- Kubernetes passes its platform-resolved numeric UID/GID, including OpenShift
136+
SCC-derived values.
137+
- VM keeps its existing guest identity behavior.
138+
139+
For Docker and Podman, policy values take precedence independently. An omitted
140+
`run_as_user` or `run_as_group` falls back to the corresponding identity from
141+
the image. The supervisor resolves names from the image's `/etc/passwd` and
142+
`/etc/group`, converts the result to a numeric UID/GID, and uses the same
143+
privilege-drop path for direct and SSH children. It does not rewrite the account
144+
files.
145+
146+
Sandbox creation fails before the workload becomes ready when a required image
147+
identity is absent, malformed, unknown, ambiguous, or resolves to UID/GID 0.
148+
The supervisor itself remains root so it can establish isolation before
149+
starting unprivileged children.
150+
128151
Kubernetes can run the supervisor in the default combined topology or in a
129152
sidecar topology. Combined mode keeps network and process supervision in the
130153
agent container. Sidecar mode runs network enforcement, the proxy, and gateway

‎architecture/sandbox.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ only when the set is already empty; any other outcome fails the spawn.
3232
4. It starts the policy proxy and local SSH server.
3333
5. It opens a supervisor session back to the gateway for connect, exec, file
3434
sync, config polling, and log push.
35-
6. It launches the agent command as the restricted sandbox user.
35+
6. It launches the agent command as the resolved restricted identity.
3636

3737
## Isolation Layers
3838

‎crates/openshell-core/src/sandbox_env.rs‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,15 @@ pub const SANDBOX_UID: &str = "OPENSHELL_SANDBOX_UID";
115115
/// supervisor drops privileges to a group other than the UID's primary group.
116116
pub const SANDBOX_GID: &str = "OPENSHELL_SANDBOX_GID";
117117

118+
/// Raw OCI `Config.User` declaration from the immutable image selected by a
119+
/// local container driver.
120+
///
121+
/// Docker and Podman always overwrite this value, including with an empty
122+
/// string when the image has no `USER`, and clear [`SANDBOX_UID`] and
123+
/// [`SANDBOX_GID`]. The supervisor resolves omitted policy identity fields
124+
/// from this declaration inside the launched image filesystem.
125+
pub const OCI_IMAGE_USER: &str = "OPENSHELL_OCI_IMAGE_USER";
126+
118127
// The corporate upstream-proxy configuration deliberately has no reserved
119128
// environment variables: it travels on the supervisor's argv
120129
// (`--upstream-proxy` and friends), which a sandbox image cannot forge the

‎crates/openshell-driver-docker/README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,14 @@ The gateway runs as a host process. The Docker driver creates one container per
1818
sandbox and starts the `openshell-sandbox` supervisor inside that container. The
1919
supervisor then creates the nested sandbox namespace for the agent process.
2020

21+
Before creating the container, the driver inspects the final sandbox image and
22+
captures its immutable image ID and raw OCI `Config.User`. Container creation
23+
uses that image ID, preventing a mutable tag from changing between inspection
24+
and launch. The supervisor runs as root, resolves omitted policy identity fields
25+
from the image declaration, and drops only agent children to the resulting
26+
numeric UID/GID. Explicit `process.run_as_user` and `process.run_as_group`
27+
values take precedence independently.
28+
2129
Docker containers join an OpenShell-managed bridge network. The driver injects
2230
`host.openshell.internal` and `host.docker.internal` so supervisors have stable
2331
names for reaching the gateway host. On Docker Desktop, Colima, Rancher

‎crates/openshell-driver-docker/src/lib.rs‎

Lines changed: 103 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -227,6 +227,12 @@ struct DockerProvisioningFailure {
227227
message: String,
228228
}
229229

230+
#[derive(Debug, Clone, PartialEq, Eq)]
231+
struct DockerImageMetadata {
232+
id: String,
233+
user: String,
234+
}
235+
230236
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
231237
struct DockerResourceLimits {
232238
nano_cpus: Option<i64>,
@@ -710,7 +716,8 @@ impl DockerComputeDriver {
710716
DockerProvisioningFailure::new("ContainerCreateFailed", status.message())
711717
})?;
712718
let template = validated.template;
713-
self.ensure_image_available(&sandbox.id, &template.image)
719+
let image = self
720+
.ensure_image_available(&sandbox.id, &template.image)
714721
.await
715722
.map_err(|status| {
716723
DockerProvisioningFailure::new("ImagePullFailed", status.message())
@@ -735,11 +742,12 @@ impl DockerComputeDriver {
735742
}
736743
DockerProvisioningFailure::new("ContainerCreateFailed", status.message())
737744
})?;
738-
let create_body = build_container_create_body_with_gpu_devices(
745+
let create_body = build_container_create_body_for_image(
739746
sandbox,
740747
&self.config,
741748
&validated.driver_config,
742749
gpu_devices.as_deref(),
750+
&image,
743751
)
744752
.map_err(|status| {
745753
if token_file_created {
@@ -1280,41 +1288,71 @@ impl DockerComputeDriver {
12801288
}))
12811289
}
12821290

1283-
async fn ensure_image_available(&self, sandbox_id: &str, image: &str) -> Result<(), Status> {
1291+
async fn ensure_image_available(
1292+
&self,
1293+
sandbox_id: &str,
1294+
image: &str,
1295+
) -> Result<DockerImageMetadata, Status> {
12841296
let policy = self.config.image_pull_policy.trim().to_ascii_lowercase();
1285-
match policy.as_str() {
1297+
let inspect = match policy.as_str() {
12861298
"" | "ifnotpresent" => {
1287-
if self.docker.inspect_image(image).await.is_ok() {
1299+
if let Ok(inspect) = self.docker.inspect_image(image).await {
12881300
self.publish_docker_progress(
12891301
sandbox_id,
12901302
"ImagePresent",
12911303
format!("Docker image \"{image}\" is already present"),
12921304
HashMap::from([("image_ref".to_string(), image.to_string())]),
12931305
);
1294-
return Ok(());
1306+
inspect
1307+
} else {
1308+
self.pull_image(sandbox_id, image).await?;
1309+
self.docker
1310+
.inspect_image(image)
1311+
.await
1312+
.map_err(|err| internal_status("inspect Docker image after pull", err))?
12951313
}
1296-
self.pull_image(sandbox_id, image).await
12971314
}
1298-
"always" => self.pull_image(sandbox_id, image).await,
1315+
"always" => {
1316+
self.pull_image(sandbox_id, image).await?;
1317+
self.docker
1318+
.inspect_image(image)
1319+
.await
1320+
.map_err(|err| internal_status("inspect Docker image after pull", err))?
1321+
}
12991322
"never" => match self.docker.inspect_image(image).await {
1300-
Ok(_) => {
1323+
Ok(inspect) => {
13011324
self.publish_docker_progress(
13021325
sandbox_id,
13031326
"ImagePresent",
13041327
format!("Docker image \"{image}\" is already present"),
13051328
HashMap::from([("image_ref".to_string(), image.to_string())]),
13061329
);
1307-
Ok(())
1330+
inspect
1331+
}
1332+
Err(err) if is_not_found_error(&err) => {
1333+
return Err(Status::failed_precondition(format!(
1334+
"docker image '{image}' is not present locally and image_pull_policy=Never"
1335+
)));
13081336
}
1309-
Err(err) if is_not_found_error(&err) => Err(Status::failed_precondition(format!(
1310-
"docker image '{image}' is not present locally and image_pull_policy=Never"
1311-
))),
1312-
Err(err) => Err(internal_status("inspect Docker image", err)),
1337+
Err(err) => return Err(internal_status("inspect Docker image", err)),
13131338
},
1314-
other => Err(Status::failed_precondition(format!(
1315-
"unsupported docker image_pull_policy '{other}'; expected Always, IfNotPresent, or Never",
1316-
))),
1317-
}
1339+
other => {
1340+
return Err(Status::failed_precondition(format!(
1341+
"unsupported docker image_pull_policy '{other}'; expected Always, IfNotPresent, or Never",
1342+
)));
1343+
}
1344+
};
1345+
1346+
let id = inspect.id.ok_or_else(|| {
1347+
Status::failed_precondition(format!(
1348+
"docker image '{image}' inspection did not return an immutable image ID"
1349+
))
1350+
})?;
1351+
let user = inspect
1352+
.config
1353+
.and_then(|config| config.user)
1354+
.unwrap_or_default();
1355+
Ok(DockerImageMetadata { id, user })
13181356
}
13191357

13201358
async fn pull_image(&self, sandbox_id: &str, image: &str) -> Result<(), Status> {
@@ -2132,7 +2170,16 @@ fn cleanup_sandbox_token_file_by_id(sandbox_id: &str, config: &DockerDriverRunti
21322170
}
21332171
}
21342172

2173+
#[cfg(test)]
21352174
fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig) -> Vec<String> {
2175+
build_environment_for_oci_user(sandbox, config, "")
2176+
}
2177+
2178+
fn build_environment_for_oci_user(
2179+
sandbox: &DriverSandbox,
2180+
config: &DockerDriverRuntimeConfig,
2181+
oci_user: &str,
2182+
) -> Vec<String> {
21362183
let mut environment = HashMap::from([
21372184
("HOME".to_string(), "/root".to_string()),
21382185
("PATH".to_string(), SUPERVISOR_PATH.to_string()),
@@ -2204,6 +2251,18 @@ fn build_environment(sandbox: &DriverSandbox, config: &DockerDriverRuntimeConfig
22042251

22052252
environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN);
22062253
environment.remove(openshell_core::sandbox_env::SANDBOX_TOKEN_FILE);
2254+
environment.insert(
2255+
openshell_core::sandbox_env::OCI_IMAGE_USER.to_string(),
2256+
oci_user.to_string(),
2257+
);
2258+
environment.insert(
2259+
openshell_core::sandbox_env::SANDBOX_UID.to_string(),
2260+
String::new(),
2261+
);
2262+
environment.insert(
2263+
openshell_core::sandbox_env::SANDBOX_GID.to_string(),
2264+
String::new(),
2265+
);
22072266

22082267
// Gateway-minted sandbox JWT. Keep the raw bearer out of container
22092268
// metadata; the supervisor reads it from this driver-owned bind mount.
@@ -2288,6 +2347,30 @@ fn build_container_create_body_with_gpu_devices(
22882347
config: &DockerDriverRuntimeConfig,
22892348
driver_config: &DockerSandboxDriverConfig,
22902349
gpu_device_ids: Option<&[String]>,
2350+
) -> Result<ContainerCreateBody, Status> {
2351+
let template = sandbox
2352+
.spec
2353+
.as_ref()
2354+
.and_then(|spec| spec.template.as_ref())
2355+
.ok_or_else(|| Status::invalid_argument("sandbox.spec.template is required"))?;
2356+
build_container_create_body_for_image(
2357+
sandbox,
2358+
config,
2359+
driver_config,
2360+
gpu_device_ids,
2361+
&DockerImageMetadata {
2362+
id: template.image.clone(),
2363+
user: String::new(),
2364+
},
2365+
)
2366+
}
2367+
2368+
fn build_container_create_body_for_image(
2369+
sandbox: &DriverSandbox,
2370+
config: &DockerDriverRuntimeConfig,
2371+
driver_config: &DockerSandboxDriverConfig,
2372+
gpu_device_ids: Option<&[String]>,
2373+
image: &DockerImageMetadata,
22912374
) -> Result<ContainerCreateBody, Status> {
22922375
let spec = sandbox
22932376
.spec
@@ -2328,9 +2411,9 @@ fn build_container_create_body_with_gpu_devices(
23282411
);
23292412

23302413
Ok(ContainerCreateBody {
2331-
image: Some(template.image.clone()),
2414+
image: Some(image.id.clone()),
23322415
user: Some("0".to_string()),
2333-
env: Some(build_environment(sandbox, config)),
2416+
env: Some(build_environment_for_oci_user(sandbox, config, &image.user)),
23342417
entrypoint: Some(vec![SUPERVISOR_MOUNT_PATH.to_string()]),
23352418
// Clear the image CMD so Docker does not append inherited args to the
23362419
// supervisor entrypoint.

0 commit comments

Comments
 (0)