Skip to content

Commit 9a4c43a

Browse files
authored
Merge pull request NVIDIA#21 from LobsterTrap/merge/align-midstream-with-main
fix(bootstrap): align midstream with main and fix rootful Podman cgroupns
2 parents 8592ab0 + 9ac1ab8 commit 9a4c43a

39 files changed

Lines changed: 1811 additions & 1995 deletions

File tree

‎Cargo.lock‎

Lines changed: 28 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ nix = { version = "0.29", features = ["signal", "process", "user", "fs", "term"]
6464
# Serialization
6565
serde = { version = "1", features = ["derive"] }
6666
serde_json = "1"
67-
serde_yaml = "0.9"
67+
serde_yml = "0.0.12"
6868

6969
# HTTP client
7070
reqwest = { version = "0.12", default-features = false, features = ["json", "rustls-tls"] }

‎architecture/gateway-single-node.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -245,7 +245,7 @@ For the target daemon (local or remote):
245245

246246
After the container starts:
247247

248-
1. **Clean stale nodes**: `clean_stale_nodes()` finds `NotReady` nodes via `kubectl get nodes` and deletes them. This is needed when a container is recreated but reuses the persistent volume -- k3s registers a new node (using the container ID as hostname) while old node entries persist in etcd. Non-fatal on error; returns the count of removed nodes.
248+
1. **Clean stale nodes**: `clean_stale_nodes()` finds nodes whose name does not match the deterministic k3s `--node-name` and deletes them. That node name is derived from the gateway name but normalized to a Kubernetes-safe lowercase form so existing gateway names that contain `_`, `.`, or uppercase characters still produce a valid node identity. This cleanup is needed when a container is recreated but reuses the persistent volume -- old node entries can persist in etcd. Non-fatal on error; returns the count of removed nodes.
249249
2. **Push local images** (optional, local deploy only): If `OPENSHELL_PUSH_IMAGES` is set, the comma-separated image refs are exported from the local Docker daemon as a single tar, uploaded into the container via `docker put_archive`, and imported into containerd via `ctr images import` in the `k8s.io` namespace. After import, `kubectl rollout restart deployment/openshell openshell` is run, followed by `kubectl rollout status --timeout=180s` to wait for completion. See `crates/openshell-bootstrap/src/push.rs`.
250250
3. **Wait for gateway health**: `wait_for_gateway_ready()` polls the Docker HEALTHCHECK status up to 180 times, 2 seconds apart (6 min total). A background task streams container logs during this wait. Failure modes:
251251
- Container exits during polling: error includes recent log lines.

‎architecture/gateway.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -501,7 +501,7 @@ The Helm chart template is at `deploy/helm/openshell/templates/statefulset.yaml`
501501

502502
`SandboxClient` (`crates/openshell-server/src/sandbox/mod.rs`) manages `agents.x-k8s.io/v1alpha1/Sandbox` CRDs.
503503

504-
- **Create**: Translates a `Sandbox` proto into a Kubernetes `DynamicObject` with labels (`openshell.ai/sandbox-id`, `openshell.ai/managed-by: openshell`) and a spec that includes the pod template, environment variables, and gateway-required env vars (`OPENSHELL_SANDBOX_ID`, `OPENSHELL_ENDPOINT`, `OPENSHELL_SSH_LISTEN_ADDR`, etc.).
504+
- **Create**: Translates a `Sandbox` proto into a Kubernetes `DynamicObject` with labels (`openshell.ai/sandbox-id`, `openshell.ai/managed-by: openshell`) and a spec that includes the pod template, environment variables, and gateway-required env vars (`OPENSHELL_SANDBOX_ID`, `OPENSHELL_ENDPOINT`, `OPENSHELL_SSH_LISTEN_ADDR`, etc.). When callers do not provide custom `volumeClaimTemplates`, the server injects a default `workspace` PVC and mounts it at `/sandbox` so the default sandbox home/workdir survives pod rescheduling.
505505
- **Delete**: Calls the Kubernetes API to delete the CRD by name. Returns `false` if already gone (404).
506506
- **Pod IP resolution**: `agent_pod_ip()` fetches the agent pod and reads `status.podIP`.
507507

‎architecture/sandbox.md‎

Lines changed: 38 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ All paths are relative to `crates/openshell-sandbox/src/`.
2424
| `sandbox/mod.rs` | Platform abstraction -- dispatches to Linux or no-op |
2525
| `sandbox/linux/mod.rs` | Linux composition: Landlock then seccomp |
2626
| `sandbox/linux/landlock.rs` | Filesystem isolation via Landlock LSM (ABI V1) |
27-
| `sandbox/linux/seccomp.rs` | Syscall filtering via BPF on `SYS_socket` |
27+
| `sandbox/linux/seccomp.rs` | Syscall filtering via BPF: socket domain blocks, dangerous syscall blocks, conditional flag blocks |
2828
| `bypass_monitor.rs` | Background `/dev/kmsg` reader for iptables bypass detection events |
2929
| `sandbox/linux/netns.rs` | Network namespace creation, veth pair setup, bypass detection iptables rules, cleanup on drop |
3030
| `l7/mod.rs` | L7 types (`L7Protocol`, `TlsMode`, `EnforcementMode`, `L7EndpointConfig`), config parsing, validation, access preset expansion, deprecated `tls` value handling |
@@ -451,22 +451,52 @@ Kernel-level error behavior (e.g., Landlock ABI unavailable) depends on `Landloc
451451

452452
**File:** `crates/openshell-sandbox/src/sandbox/linux/seccomp.rs`
453453

454-
Seccomp blocks socket creation for specific address families. The filter targets a single syscall (`SYS_socket`) and inspects argument 0 (the domain).
455-
456-
**Always blocked** (regardless of network mode):
457-
- `AF_NETLINK`, `AF_PACKET`, `AF_BLUETOOTH`, `AF_VSOCK`
458-
459-
**Additionally blocked in `Block` mode** (no proxy):
460-
- `AF_INET`, `AF_INET6`
454+
Seccomp provides three layers of syscall restriction: socket domain blocks, unconditional syscall blocks, and conditional syscall blocks. The filter uses a default-allow policy (`SeccompAction::Allow`) with targeted rules that return `Errno(EPERM)`.
461455

462456
**Skipped entirely** in `Allow` mode.
463457

464458
Setup:
465459
1. `prctl(PR_SET_NO_NEW_PRIVS, 1)` -- required before seccomp
466460
2. `seccompiler::apply_filter()` with default action `Allow` and per-rule action `Errno(EPERM)`
467461

462+
#### Socket domain blocks
463+
464+
| Domain | Always blocked | Additionally blocked in Block mode |
465+
|--------|:-:|:-:|
466+
| `AF_PACKET` | Yes | |
467+
| `AF_BLUETOOTH` | Yes | |
468+
| `AF_VSOCK` | Yes | |
469+
| `AF_INET` | | Yes |
470+
| `AF_INET6` | | Yes |
471+
| `AF_NETLINK` | | Yes |
472+
468473
In `Proxy` mode, `AF_INET`/`AF_INET6` are allowed because the sandboxed process needs to connect to the proxy over the veth pair. The network namespace ensures it can only reach the proxy's IP (`10.200.0.1`).
469474

475+
#### Unconditional syscall blocks
476+
477+
These syscalls are blocked entirely (EPERM for any invocation):
478+
479+
| Syscall | Reason |
480+
|---------|--------|
481+
| `memfd_create` | Fileless binary execution bypasses Landlock filesystem restrictions |
482+
| `ptrace` | Cross-process memory inspection and code injection |
483+
| `bpf` | Kernel BPF program loading |
484+
| `process_vm_readv` | Cross-process memory read |
485+
| `io_uring_setup` | Async I/O subsystem with extensive CVE history |
486+
| `mount` | Filesystem mount could subvert Landlock or overlay writable paths |
487+
488+
#### Conditional syscall blocks
489+
490+
These syscalls are only blocked when specific flag patterns are present:
491+
492+
| Syscall | Condition | Reason |
493+
|---------|-----------|--------|
494+
| `execveat` | `AT_EMPTY_PATH` flag set (arg4) | Fileless execution from an anonymous fd |
495+
| `unshare` | `CLONE_NEWUSER` flag set (arg0) | User namespace creation enables privilege escalation |
496+
| `seccomp` | operation == `SECCOMP_SET_MODE_FILTER` (arg0) | Prevents sandboxed code from replacing the active filter |
497+
498+
Conditional blocks use `MaskedEq` for flag checks (bit-test) and `Eq` for exact-value matches. This allows normal use of these syscalls while blocking the dangerous flag combinations.
499+
470500
### Network namespace isolation
471501

472502
**File:** `crates/openshell-sandbox/src/sandbox/linux/netns.rs`

‎architecture/security-policy.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -850,6 +850,10 @@ The response includes an `X-OpenShell-Policy` header and `Connection: close`. Se
850850

851851
## Seccomp Filter Details
852852

853+
The seccomp filter uses a default-allow policy (`SeccompAction::Allow`) with targeted rules that return `EPERM`. It provides three layers of protection: socket domain blocks, unconditional syscall blocks, and conditional syscall blocks. See `crates/openshell-sandbox/src/sandbox/linux/seccomp.rs`.
854+
855+
### Blocked socket domains
856+
853857
Regardless of network mode, certain socket domains are always blocked:
854858

855859
| Domain | Constant | Reason |
@@ -861,7 +865,30 @@ Regardless of network mode, certain socket domains are always blocked:
861865

862866
In proxy mode (which is always active), `AF_INET` (2) and `AF_INET6` (10) are allowed so the sandbox process can reach the proxy.
863867

864-
The seccomp filter uses a default-allow policy (`SeccompAction::Allow`) with specific `socket()` syscall rules that return `EPERM` when the first argument (domain) matches a blocked value. See `crates/openshell-sandbox/src/sandbox/linux/seccomp.rs`.
868+
### Blocked syscalls
869+
870+
These syscalls are blocked unconditionally (EPERM for any invocation):
871+
872+
| Syscall | NR (x86-64) | Reason |
873+
|---------|-------------|--------|
874+
| `memfd_create` | 319 | Fileless binary execution bypasses Landlock filesystem restrictions |
875+
| `ptrace` | 101 | Cross-process memory inspection and code injection |
876+
| `bpf` | 321 | Kernel BPF program loading |
877+
| `process_vm_readv` | 310 | Cross-process memory read |
878+
| `io_uring_setup` | 425 | Async I/O subsystem with extensive CVE history |
879+
| `mount` | 165 | Filesystem mount could subvert Landlock or overlay writable paths |
880+
881+
### Conditionally blocked syscalls
882+
883+
These syscalls are blocked only when specific flag patterns are present in their arguments:
884+
885+
| Syscall | NR (x86-64) | Condition | Reason |
886+
|---------|-------------|-----------|--------|
887+
| `execveat` | 322 | `AT_EMPTY_PATH` (0x1000) set in flags (arg4) | Fileless execution from an anonymous fd |
888+
| `unshare` | 272 | `CLONE_NEWUSER` (0x10000000) set in flags (arg0) | User namespace creation enables privilege escalation |
889+
| `seccomp` | 317 | operation == `SECCOMP_SET_MODE_FILTER` (1) in arg0 | Prevents sandboxed code from replacing the active filter |
890+
891+
Flag checks use `MaskedEq` (`(arg & mask) == mask`) to detect the flag bit regardless of other bits. The `seccomp` syscall check uses `Eq` for exact value comparison on the operation argument.
865892

866893
---
867894

‎crates/openshell-bootstrap/src/constants.rs‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,15 +13,100 @@ pub const SERVER_CLIENT_CA_SECRET_NAME: &str = "openshell-server-client-ca";
1313
pub const CLIENT_TLS_SECRET_NAME: &str = "openshell-client-tls";
1414
/// K8s secret holding the SSH handshake HMAC secret (shared by gateway and sandbox pods).
1515
pub const SSH_HANDSHAKE_SECRET_NAME: &str = "openshell-ssh-handshake";
16+
const NODE_NAME_PREFIX: &str = "openshell-";
17+
const NODE_NAME_FALLBACK_SUFFIX: &str = "gateway";
18+
const KUBERNETES_MAX_NAME_LEN: usize = 253;
1619

1720
pub fn container_name(name: &str) -> String {
1821
format!("openshell-cluster-{name}")
1922
}
2023

24+
/// Deterministic k3s node name derived from the gateway name.
25+
///
26+
/// k3s defaults to using the container hostname (= Docker container ID) as
27+
/// the node name. When the container is recreated (e.g. after an image
28+
/// upgrade), the container ID changes, creating a new k3s node. The
29+
/// `clean_stale_nodes` function then deletes PVCs whose backing PVs have
30+
/// node affinity for the old node — wiping the server database and any
31+
/// sandbox persistent volumes.
32+
///
33+
/// By passing a deterministic `--node-name` to k3s, the node identity
34+
/// survives container recreation, and PVCs are never orphaned.
35+
///
36+
/// Gateway names allow Docker-friendly separators and uppercase characters,
37+
/// but Kubernetes node names must be DNS-safe. Normalize the gateway name into
38+
/// a single lowercase RFC 1123 label so previously accepted names such as
39+
/// `prod_us` or `Prod.US` still deploy successfully.
40+
pub fn node_name(name: &str) -> String {
41+
format!("{NODE_NAME_PREFIX}{}", normalize_node_name_suffix(name))
42+
}
43+
44+
fn normalize_node_name_suffix(name: &str) -> String {
45+
let mut normalized = String::with_capacity(name.len());
46+
let mut last_was_separator = false;
47+
48+
for ch in name.chars() {
49+
if ch.is_ascii_alphanumeric() {
50+
normalized.push(ch.to_ascii_lowercase());
51+
last_was_separator = false;
52+
} else if !last_was_separator {
53+
normalized.push('-');
54+
last_was_separator = true;
55+
}
56+
}
57+
58+
let mut normalized = normalized.trim_matches('-').to_string();
59+
if normalized.is_empty() {
60+
normalized.push_str(NODE_NAME_FALLBACK_SUFFIX);
61+
}
62+
63+
let max_suffix_len = KUBERNETES_MAX_NAME_LEN.saturating_sub(NODE_NAME_PREFIX.len());
64+
if normalized.len() > max_suffix_len {
65+
normalized.truncate(max_suffix_len);
66+
normalized.truncate(normalized.trim_end_matches('-').len());
67+
}
68+
69+
if normalized.is_empty() {
70+
normalized.push_str(NODE_NAME_FALLBACK_SUFFIX);
71+
}
72+
73+
normalized
74+
}
75+
2176
pub fn volume_name(name: &str) -> String {
2277
format!("openshell-cluster-{name}")
2378
}
2479

2580
pub fn network_name(name: &str) -> String {
2681
format!("openshell-cluster-{name}")
2782
}
83+
84+
#[cfg(test)]
85+
mod tests {
86+
use super::*;
87+
88+
#[test]
89+
fn node_name_normalizes_uppercase_and_underscores() {
90+
assert_eq!(node_name("Prod_US"), "openshell-prod-us");
91+
}
92+
93+
#[test]
94+
fn node_name_collapses_and_trims_separator_runs() {
95+
assert_eq!(node_name("._Prod..__-Gateway-."), "openshell-prod-gateway");
96+
}
97+
98+
#[test]
99+
fn node_name_falls_back_when_gateway_name_has_no_alphanumerics() {
100+
assert_eq!(node_name("...___---"), "openshell-gateway");
101+
}
102+
103+
#[test]
104+
fn node_name_truncates_to_kubernetes_name_limit() {
105+
let gateway_name = "A".repeat(400);
106+
let node_name = node_name(&gateway_name);
107+
108+
assert!(node_name.len() <= KUBERNETES_MAX_NAME_LEN);
109+
assert!(node_name.starts_with(NODE_NAME_PREFIX));
110+
assert!(node_name.ends_with('a'));
111+
}
112+
}

‎crates/openshell-bootstrap/src/container_runtime.rs‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,16 @@ fn podman_rootless_socket_path() -> Option<String> {
203203
Some(format!("{runtime_dir}/podman/podman.sock"))
204204
}
205205

206+
/// Check whether the current process is running as a non-root user.
207+
///
208+
/// Returns `true` when the effective UID is non-zero (rootless mode).
209+
/// Used to decide container configuration — for example, rootless Podman
210+
/// needs a private cgroup namespace while rootful Podman (and Docker) can
211+
/// use the host cgroup namespace.
212+
pub(crate) fn is_rootless() -> bool {
213+
current_uid().map_or(false, |uid| uid != 0)
214+
}
215+
206216
/// Get the current user's UID by reading `/proc/self/status`.
207217
///
208218
/// Returns `None` on non-Linux systems or if the file cannot be parsed.

0 commit comments

Comments
 (0)