Skip to content

Commit cd0477f

Browse files
committed
fix(kubernetes): preserve process identity in sidecar topology
Render sidecar pods with a shared process namespace, keep binary-aware network policy enabled, and move Kubernetes sidecar settings under the nested sidecar config table. Also apply unprivileged Landlock/seccomp setup in NetworkOnly supervisor mode so sidecar topology keeps sandbox child hardening without privileged process setup. Signed-off-by: Taylor Mutch <[email protected]>
1 parent 5681187 commit cd0477f

19 files changed

Lines changed: 338 additions & 151 deletions

File tree

‎architecture/compute-runtimes.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -107,8 +107,12 @@ log expressions can fail without rolling back the required table, chain, and
107107
reject rules.
108108
The agent container runs as the resolved sandbox UID/GID with no added Linux
109109
capabilities. Sidecar mode preserves gateway session and SSH behavior, but
110-
treats the process leaf as network-only: Landlock filesystem policy, process
111-
privilege dropping, and process/binary identity checks are not applied there.
110+
treats the process leaf as network-only: Landlock filesystem policy and child
111+
seccomp still apply where supported, while process privilege dropping and
112+
supervisor identity mount isolation do not run because the agent container is
113+
already unprivileged. Sidecar pods use a shared process namespace so the
114+
network sidecar can resolve workload process and binary identity through
115+
`/proc/<entrypoint-pid>`.
112116

113117
## Images
114118

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -62,15 +62,17 @@ container and runs the long-lived network sidecar as a non-root UID with no
6262
added Linux capabilities. The agent container also runs as the resolved sandbox
6363
UID/GID with `allowPrivilegeEscalation: false` and `capabilities.drop: ["ALL"]`.
6464
In this mode OpenShell preserves gateway session and SSH behavior, but the
65-
process supervisor runs in network-only mode and does not apply Landlock
66-
filesystem policy, process privilege dropping, or process/binary identity
67-
checks. Network endpoint and L7 policy remain enforced by the network sidecar.
68-
69-
Sidecar mode uses the pod `fsGroup` to make the projected service-account token
70-
and sandbox client TLS secret group-readable so the non-root process supervisor
71-
can authenticate to the gateway. Treat the agent container as trusted with
72-
respect to those in-pod gateway credentials until a narrower credential handoff
73-
exists.
65+
process supervisor does not perform root-to-sandbox privilege dropping or
66+
supervisor identity mount isolation. It still applies Landlock filesystem policy
67+
and child seccomp filters where the kernel/runtime supports them. Network
68+
endpoint and L7 policy remain enforced by the network sidecar, and
69+
sidecar pods use a shared process namespace so the network sidecar can resolve
70+
process/binary identity through `/proc/<entrypoint-pid>`.
71+
72+
Sidecar mode keeps gateway credentials in the network sidecar. The agent
73+
container does not mount the projected service-account token used for sandbox
74+
token bootstrap, does not mount the sandbox client TLS secret, and does not get
75+
gateway callback environment variables.
7476

7577
The driver can request a Kubernetes AppArmor profile through
7678
`app_armor_profile`.

‎crates/openshell-driver-kubernetes/src/config.rs‎

Lines changed: 84 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,35 @@ impl FromStr for SupervisorTopology {
8888
}
8989
}
9090

91+
#[derive(Debug, Clone, Serialize, Deserialize)]
92+
#[serde(default, deny_unknown_fields)]
93+
pub struct KubernetesSidecarConfig {
94+
/// UID used by the long-running network sidecar in `sidecar` topology.
95+
/// The network init container installs nftables rules that exempt this
96+
/// UID, so it must not match the sandbox workload UID.
97+
pub proxy_uid: u32,
98+
}
99+
100+
impl Default for KubernetesSidecarConfig {
101+
fn default() -> Self {
102+
Self {
103+
proxy_uid: DEFAULT_PROXY_UID,
104+
}
105+
}
106+
}
107+
108+
impl KubernetesSidecarConfig {
109+
pub fn validate_proxy_uid(&self) -> Result<(), String> {
110+
if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID {
111+
return Err(format!(
112+
"sidecar.proxy_uid must be at least {}",
113+
openshell_policy::MIN_SANDBOX_UID
114+
));
115+
}
116+
Ok(())
117+
}
118+
}
119+
91120
/// Kubernetes `AppArmor` profile requested for the sandbox agent container.
92121
#[derive(Debug, Clone, PartialEq, Eq)]
93122
pub enum AppArmorProfile {
@@ -213,11 +242,9 @@ pub struct KubernetesComputeConfig {
213242
/// How the supervisor binary is delivered into sandbox pods.
214243
pub supervisor_sideload_method: SupervisorSideloadMethod,
215244
/// How the supervisor is arranged for Kubernetes sandbox pods.
216-
pub supervisor_topology: SupervisorTopology,
217-
/// UID used by the long-running network sidecar in `sidecar` topology.
218-
/// The network init container installs nftables rules that exempt this
219-
/// UID, so it must not match the sandbox workload UID.
220-
pub proxy_uid: u32,
245+
pub topology: SupervisorTopology,
246+
/// Sidecar-only settings used when `topology = "sidecar"`.
247+
pub sidecar: KubernetesSidecarConfig,
221248
pub grpc_endpoint: String,
222249
pub ssh_socket_path: String,
223250
pub client_tls_secret_name: String,
@@ -303,8 +330,8 @@ impl Default for KubernetesComputeConfig {
303330
supervisor_image: DEFAULT_SUPERVISOR_IMAGE.to_string(),
304331
supervisor_image_pull_policy: String::new(),
305332
supervisor_sideload_method: SupervisorSideloadMethod::default(),
306-
supervisor_topology: SupervisorTopology::default(),
307-
proxy_uid: DEFAULT_PROXY_UID,
333+
topology: SupervisorTopology::default(),
334+
sidecar: KubernetesSidecarConfig::default(),
308335
grpc_endpoint: String::new(),
309336
ssh_socket_path: "/run/openshell/ssh.sock".to_string(),
310337
client_tls_secret_name: String::new(),
@@ -350,13 +377,7 @@ impl KubernetesComputeConfig {
350377
}
351378

352379
pub fn validate_proxy_uid(&self) -> Result<(), String> {
353-
if self.proxy_uid < openshell_policy::MIN_SANDBOX_UID {
354-
return Err(format!(
355-
"proxy_uid must be at least {}",
356-
openshell_policy::MIN_SANDBOX_UID
357-
));
358-
}
359-
Ok(())
380+
self.sidecar.validate_proxy_uid()
360381
}
361382

362383
/// Resolve the sandbox UID/GID pair.
@@ -486,65 +507,98 @@ mod tests {
486507
}
487508

488509
#[test]
489-
fn default_supervisor_topology_is_combined() {
510+
fn default_topology_is_combined() {
490511
let cfg = KubernetesComputeConfig::default();
491-
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
492-
assert_eq!(cfg.supervisor_topology.to_string(), "combined");
512+
assert_eq!(cfg.topology, SupervisorTopology::Combined);
513+
assert_eq!(cfg.topology.to_string(), "combined");
493514
}
494515

495516
#[test]
496517
fn default_proxy_uid_is_dedicated_non_root_uid() {
497518
let cfg = KubernetesComputeConfig::default();
498-
assert_eq!(cfg.proxy_uid, DEFAULT_PROXY_UID);
519+
assert_eq!(cfg.sidecar.proxy_uid, DEFAULT_PROXY_UID);
499520
}
500521

501522
#[test]
502-
fn serde_override_supervisor_topology_sidecar() {
523+
fn serde_override_topology_sidecar() {
503524
let json = serde_json::json!({
504-
"supervisor_topology": "sidecar"
525+
"topology": "sidecar"
505526
});
506527
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
507-
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Sidecar);
528+
assert_eq!(cfg.topology, SupervisorTopology::Sidecar);
508529
}
509530

510531
#[test]
511-
fn serde_override_supervisor_topology_combined() {
532+
fn serde_override_topology_combined() {
512533
let json = serde_json::json!({
513-
"supervisor_topology": "combined"
534+
"topology": "combined"
514535
});
515536
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
516-
assert_eq!(cfg.supervisor_topology, SupervisorTopology::Combined);
537+
assert_eq!(cfg.topology, SupervisorTopology::Combined);
538+
}
539+
540+
#[test]
541+
fn serde_rejects_sidecar_binary_identity_field() {
542+
let json = serde_json::json!({
543+
"sidecar": {
544+
"binary_identity": "shared-pid"
545+
}
546+
});
547+
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
548+
assert!(err.to_string().contains("unknown field"));
517549
}
518550

519551
#[test]
520-
fn serde_override_proxy_uid() {
552+
fn serde_override_sidecar_proxy_uid_nested() {
521553
let json = serde_json::json!({
522-
"proxy_uid": 2000
554+
"sidecar": {
555+
"proxy_uid": 2000
556+
}
523557
});
524558
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
525-
assert_eq!(cfg.proxy_uid, 2000);
559+
assert_eq!(cfg.sidecar.proxy_uid, 2000);
526560
cfg.validate_proxy_uid().unwrap();
527561
}
528562

529563
#[test]
530564
fn validate_proxy_uid_rejects_privileged_uid() {
531565
let cfg = KubernetesComputeConfig {
532-
proxy_uid: 999,
566+
sidecar: KubernetesSidecarConfig { proxy_uid: 999 },
533567
..KubernetesComputeConfig::default()
534568
};
535569
let err = cfg.validate_proxy_uid().unwrap_err();
536570
assert!(err.contains("proxy_uid"));
537571
}
538572

539573
#[test]
540-
fn serde_rejects_invalid_supervisor_topology() {
574+
fn serde_rejects_invalid_topology() {
541575
let json = serde_json::json!({
542-
"supervisor_topology": "unsupported"
576+
"topology": "unsupported"
543577
});
544578
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
545579
assert!(err.to_string().contains("unknown variant"));
546580
}
547581

582+
#[test]
583+
fn serde_rejects_removed_supervisor_topology_field() {
584+
let json = serde_json::json!({
585+
"supervisor_topology": "sidecar"
586+
});
587+
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
588+
assert!(err.to_string().contains("unknown field"));
589+
}
590+
591+
#[test]
592+
fn serde_rejects_removed_flat_sidecar_fields() {
593+
for json in [
594+
serde_json::json!({ "sidecar_binary_identity": "shared-pid" }),
595+
serde_json::json!({ "proxy_uid": 2000 }),
596+
] {
597+
let err = serde_json::from_value::<KubernetesComputeConfig>(json).unwrap_err();
598+
assert!(err.to_string().contains("unknown field"));
599+
}
600+
}
601+
548602
#[test]
549603
fn serde_rejects_removed_process_enforcement_field() {
550604
let json = serde_json::json!({

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 6 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -563,8 +563,8 @@ impl KubernetesComputeDriver {
563563
supervisor_image: &self.config.supervisor_image,
564564
supervisor_image_pull_policy: &self.config.supervisor_image_pull_policy,
565565
supervisor_sideload_method: self.config.supervisor_sideload_method,
566-
supervisor_topology: self.config.supervisor_topology,
567-
proxy_uid: self.config.proxy_uid,
566+
supervisor_topology: self.config.topology,
567+
proxy_uid: self.config.sidecar.proxy_uid,
568568
service_account_name: &self.config.service_account_name,
569569
sandbox_id: &sandbox.id,
570570
sandbox_name: &sandbox.name,
@@ -1334,11 +1334,6 @@ fn supervisor_sidecar_env(
13341334
openshell_core::sandbox_env::NETWORK_ENFORCEMENT_MODE,
13351335
"sidecar-nftables",
13361336
);
1337-
upsert_env(
1338-
&mut env,
1339-
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY,
1340-
"relaxed",
1341-
);
13421337
upsert_env(
13431338
&mut env,
13441339
openshell_core::sandbox_env::SUPERVISOR_READY_FILE,
@@ -1488,6 +1483,8 @@ fn apply_supervisor_sidecar_topology(
14881483
sc.insert("fsGroup".to_string(), serde_json::json!(params.sandbox_gid));
14891484
}
14901485

1486+
spec.insert("shareProcessNamespace".to_string(), serde_json::json!(true));
1487+
14911488
apply_supervisor_binary_source(
14921489
spec,
14931490
params.supervisor_image,
@@ -3124,10 +3121,7 @@ mod tests {
31243121
&params,
31253122
);
31263123

3127-
assert!(
3128-
pod_template["spec"]["shareProcessNamespace"].is_null(),
3129-
"sidecar mode no longer needs a shared process namespace when binary identity is relaxed"
3130-
);
3124+
assert_eq!(pod_template["spec"]["shareProcessNamespace"], true);
31313125
assert_eq!(pod_template["spec"]["securityContext"]["fsGroup"], 1500);
31323126
let containers = pod_template["spec"]["containers"].as_array().unwrap();
31333127
assert_eq!(containers.len(), 2);
@@ -3250,7 +3244,7 @@ mod tests {
32503244
sidecar,
32513245
openshell_core::sandbox_env::NETWORK_BINARY_IDENTITY
32523246
),
3253-
Some("relaxed")
3247+
None
32543248
);
32553249
assert_eq!(
32563250
rendered_env(sidecar, openshell_core::sandbox_env::ENTRYPOINT_PID_FILE),

‎crates/openshell-driver-kubernetes/src/lib.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,8 +7,8 @@ pub mod grpc;
77

88
pub use config::{
99
AppArmorProfile, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
10-
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, SupervisorSideloadMethod,
11-
SupervisorTopology,
10+
DEFAULT_WORKSPACE_STORAGE_SIZE, KubernetesComputeConfig, KubernetesSidecarConfig,
11+
SupervisorSideloadMethod, SupervisorTopology,
1212
};
1313
pub use driver::{KubernetesComputeDriver, KubernetesDriverError};
1414
pub use grpc::ComputeDriverService;

‎crates/openshell-driver-kubernetes/src/main.rs‎

Lines changed: 16 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,8 @@ use openshell_core::VERSION;
1111
use openshell_core::proto::compute::v1::compute_driver_server::ComputeDriverServer;
1212
use openshell_driver_kubernetes::{
1313
AppArmorProfile, ComputeDriverService, DEFAULT_PROXY_UID, DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME,
14-
KubernetesComputeConfig, KubernetesComputeDriver, SupervisorSideloadMethod, SupervisorTopology,
14+
KubernetesComputeConfig, KubernetesComputeDriver, KubernetesSidecarConfig,
15+
SupervisorSideloadMethod, SupervisorTopology,
1516
};
1617

1718
#[derive(Parser, Debug)]
@@ -82,13 +83,19 @@ struct Args {
8283

8384
#[arg(
8485
long,
85-
env = "OPENSHELL_SUPERVISOR_TOPOLOGY",
86+
alias = "supervisor-topology",
87+
env = "OPENSHELL_K8S_TOPOLOGY",
8688
default_value = "combined"
8789
)]
88-
supervisor_topology: SupervisorTopology,
90+
topology: SupervisorTopology,
8991

90-
#[arg(long, env = "OPENSHELL_PROXY_UID", default_value_t = DEFAULT_PROXY_UID)]
91-
proxy_uid: u32,
92+
#[arg(
93+
long = "sidecar-proxy-uid",
94+
alias = "proxy-uid",
95+
env = "OPENSHELL_K8S_SIDECAR_PROXY_UID",
96+
default_value_t = DEFAULT_PROXY_UID
97+
)]
98+
sidecar_proxy_uid: u32,
9299

93100
#[arg(long, env = "OPENSHELL_ENABLE_USER_NAMESPACES")]
94101
enable_user_namespaces: bool,
@@ -133,8 +140,10 @@ async fn main() -> Result<()> {
133140
.unwrap_or_else(|| openshell_core::config::DEFAULT_SUPERVISOR_IMAGE.to_string()),
134141
supervisor_image_pull_policy: args.supervisor_image_pull_policy.unwrap_or_default(),
135142
supervisor_sideload_method: args.supervisor_sideload_method,
136-
supervisor_topology: args.supervisor_topology,
137-
proxy_uid: args.proxy_uid,
143+
topology: args.topology,
144+
sidecar: KubernetesSidecarConfig {
145+
proxy_uid: args.sidecar_proxy_uid,
146+
},
138147
grpc_endpoint: args.grpc_endpoint.unwrap_or_default(),
139148
ssh_socket_path: args.sandbox_ssh_socket_path,
140149
client_tls_secret_name: args.client_tls_secret_name.unwrap_or_default(),

0 commit comments

Comments
 (0)