Skip to content

Commit 811bcb8

Browse files
committed
fix(vm): make sandbox process startup fork-safe
Prepare executable paths, named identities, supplementary groups, Landlock handles, and runtime seccomp programs in the supervisor before fork. Entrypoint and SSH pre-exec hooks now consume prepared numeric state and keep their successful path to the required namespace, credential, hardening, Landlock, seccomp-install, and exec syscalls. Direct Linux setgroups/setresgid/setresuid calls avoid libc setxid coordination, while static error markers avoid tracing and formatting in the child. This removes NSS, initgroups, PATH search, filter compilation, and allocator-backed seccomp cleanup from the multithreaded post-fork child. It fixes the supervisor spawn deadlock that left VM clients waiting indefinitely before a process PID could be published. Also mirror the selected lower root filesystem owner and mode onto the VM overlay upper root before mounting it. The overlay upper directory had inherited the host umask and UID, exposing the merged guest root as mode 0700 and causing the named sandbox identity to fail exec with EACCES after the original deadlock was removed. Add regressions for parent-side executable and identity resolution, supplementary-group preservation, SSH hook preparation, and overlay root metadata ordering. Update the sandbox architecture and security enforcement-order documentation. Validated with workspace pre-commit, complete process-supervisor and VM-driver unit suites, VM smoke/host-gateway/resume E2E, and a named-identity interactive PTY relay held open for more than 45 seconds. This was debugged and tested on a Ubuntu 26.04 AMD64 box. Config: ``` $ cat ~/.config/openshell/gateway.toml [openshell.gateway] compute_drivers = ["vm"] log_level = "debug" [openshell.drivers.vm] mem_mib = 4096 vcpus = 2 ``` Before: ``` $ openshell sandbox create --name demo-fixed ✓ Sandbox allocated (0s) ✓ Sandbox allocated (0s) ✓ Sandbox allocated (0s) ✓ Sandbox allocated (0s) ✓ Sandbox allocated (0s) ⠚ Starting sandbox... Waiting for VM supervisor (2m) # gets stuck Error: ... 'The system is not in a state required for the operation's execution' ``` Fixes NVIDIA#2587 Signed-off-by: Dhiru Kholia <[email protected]>
1 parent 4cb77a9 commit 811bcb8

8 files changed

Lines changed: 629 additions & 183 deletions

File tree

‎architecture/sandbox.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,20 @@ container-granted capabilities. This is fail-closed: the supervisor retains
2121
aborts unless the bounding set ends up empty. A `setpcap` `EPERM` is tolerated
2222
only when the set is already empty; any other outcome fails the spawn.
2323

24+
Before it forks, the supervisor resolves the executable path, named users,
25+
groups, supplementary memberships, Landlock path handles, and runtime seccomp
26+
programs. Entrypoint and SSH child hooks receive only prepared values. The
27+
post-fork child performs the unavoidable namespace, credential, hardening,
28+
Landlock, seccomp-install, and exec syscalls; it does not perform name-service
29+
lookups, PATH search, filter compilation, tracing, or successful-path heap
30+
cleanup. Both launch paths therefore have the same identity and enforcement
31+
semantics without invoking process-global runtime services after fork.
32+
33+
The VM driver also mirrors the selected lower root filesystem's owner and mode
34+
onto the overlay upper root before mounting it. This prevents the host process
35+
umask or host UID from making the merged guest root untraversable by the sandbox
36+
identity.
37+
2438
## Startup Flow
2539

2640
1. The compute runtime starts the workload with sandbox identity, callback

‎crates/openshell-driver-vm/scripts/openshell-vm-sandbox-init.sh‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,14 @@ setup_overlay_root() {
270270
fi
271271
fi
272272

273+
# Overlayfs exposes the upperdir root inode as the merged mount's root.
274+
# The host creates the overlay template under a restrictive umask, so its
275+
# upperdir may otherwise be 0700 and owned by the host UID. Mirror the
276+
# selected lower root before mounting so named sandbox users can traverse
277+
# `/` and the workload cannot own the sandbox filesystem root.
278+
chown --reference="$lower_root" /overlay/upper
279+
chmod --reference="$lower_root" /overlay/upper
280+
273281
mount -t overlay overlay \
274282
-o lowerdir="$lower_root",upperdir=/overlay/upper,workdir=/overlay/work \
275283
/newroot

‎crates/openshell-driver-vm/src/rootfs.rs‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ const SUPERVISOR: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/openshell-sa
1414
const UMOCI: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/umoci.zst"));
1515
const ROOTFS_VARIANT_MARKER: &str = ".openshell-rootfs-variant";
1616
const SANDBOX_GUEST_INIT_PATH: &str = "/srv/openshell-vm-sandbox-init.sh";
17+
const SANDBOX_GUEST_INIT_SCRIPT: &str = include_str!("../scripts/openshell-vm-sandbox-init.sh");
1718
const SANDBOX_SUPERVISOR_PATH: &str = openshell_core::driver_utils::SUPERVISOR_CONTAINER_BINARY;
1819
const SANDBOX_UMOCI_PATH: &str = openshell_core::container_paths::VM_UMOCI_PATH;
1920
const SANDBOX_OWNER_NORMALIZED_MARKER: &str =
@@ -362,11 +363,8 @@ fn prepare_sandbox_rootfs(rootfs: &Path, sandbox_uid: u32, sandbox_gid: u32) ->
362363
if let Some(parent) = init_path.parent() {
363364
fs::create_dir_all(parent).map_err(|e| format!("create {}: {e}", parent.display()))?;
364365
}
365-
fs::write(
366-
&init_path,
367-
include_str!("../scripts/openshell-vm-sandbox-init.sh"),
368-
)
369-
.map_err(|e| format!("write {}: {e}", init_path.display()))?;
366+
fs::write(&init_path, SANDBOX_GUEST_INIT_SCRIPT)
367+
.map_err(|e| format!("write {}: {e}", init_path.display()))?;
370368
#[cfg(unix)]
371369
{
372370
use std::os::unix::fs::PermissionsExt as _;
@@ -948,6 +946,25 @@ mod tests {
948946
use std::sync::atomic::{AtomicU64, Ordering};
949947
use std::time::{SystemTime, UNIX_EPOCH};
950948

949+
#[test]
950+
fn guest_init_preserves_lower_root_metadata_before_overlay_mount() {
951+
let chown = SANDBOX_GUEST_INIT_SCRIPT
952+
.find("chown --reference=\"$lower_root\" /overlay/upper")
953+
.expect("guest init must preserve the lower root owner");
954+
let chmod = SANDBOX_GUEST_INIT_SCRIPT
955+
.find("chmod --reference=\"$lower_root\" /overlay/upper")
956+
.expect("guest init must preserve the lower root mode");
957+
let mount = SANDBOX_GUEST_INIT_SCRIPT
958+
.find("mount -t overlay overlay")
959+
.expect("guest init must mount the overlay");
960+
961+
assert!(
962+
chown < mount,
963+
"upper root ownership must be fixed before mount"
964+
);
965+
assert!(chmod < mount, "upper root mode must be fixed before mount");
966+
}
967+
951968
#[test]
952969
fn prepare_sandbox_rootfs_rewrites_guest_layout() {
953970
let dir = unique_temp_dir();

0 commit comments

Comments
 (0)