Skip to content

Commit 49f2d31

Browse files
committed
feat(kubernetes): support corporate proxy CA bundle
The Kubernetes driver had no way to supply a CA bundle for the corporate egress proxy, so an `https://` proxy with a private CA, or a TLS-intercepting proxy, could not be used. Podman and VM already expose `proxy_ca_bundle`. Add `proxy_ca_bundle` to `[openshell.drivers.kubernetes]` as a path the gateway Pod reads. The gateway stages the PEM into the existing per-generation supervisor bootstrap Secret and passes `--upstream-proxy-ca-bundle` on the supervisor argv. That Secret is already immutable, owner-referenced and garbage-collected, and its volume mounts every key at /.openshell/supervisor with no items filter, so this needs no new object kind, volume, mount, or RBAC verb, and works in shared, managed and operator workspace modes. The bundle is deliberately read from the gateway's filesystem rather than referenced as an object in the sandbox namespace. It becomes a trust anchor for every upstream the sandbox reaches, so it must stay in the gateway's trust domain; the immutable staging Secret also keeps the anchor from changing underneath a running sandbox. Bound the staged bundle at 256 KiB. The shared reader's limit is exactly the apiserver's own Secret limit and the bootstrap Secret carries four other keys, so a bundle between the two would pass gateway startup and then fail every sandbox create with an opaque `data: Too long`. Delegate the URL, no_proxy, connect_by_hostname and ca_bundle rules to the shared validate_upstream_proxy_settings, keeping the Secret-specific credential block local: this driver accepts an explicit `proxy_auth_allow_insecure = false` without credentials, which the shared rules reject. This also fixes the acknowledgement being demanded for an `https://` proxy, where the credential travels inside the verified TLS session. Add auth_setting_label so the inline-credential diagnostic names the Secret keys instead of proxy_auth_file, which this driver rejects as an unknown key. Document that the bundle should carry only the CA that signs the proxy's certificate, or that an intercepting proxy re-signs upstream certificates with. Public roots already reach the sandbox through the supervisor image and its TLS stack, and the bundle is concatenated with that system store into a single boundary control frame, so a full merged trust bundle spends the frame budget on duplicated roots. The frame, not the apiserver Secret limit, is the tighter of the two ceilings in practice; raising the staging bound requires checking it. Closes NVIDIA#3443 Signed-off-by: Philippe Martin <[email protected]>
1 parent c8a4ff5 commit 49f2d31

18 files changed

Lines changed: 871 additions & 70 deletions

File tree

‎architecture/sandbox.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -461,6 +461,15 @@ every intercepted upstream connection failing. The bundle is valid with either
461461
an `http://` or `https://` proxy (an intercepting proxy can be reached over
462462
plain HTTP) and is fail-closed: an unreadable or certificate-free file is fatal.
463463

464+
How the bundle reaches the supervisor is driver-specific. Drivers that run the
465+
supervisor locally bind-mount the operator's file. The Kubernetes driver cannot:
466+
the supervisor Pod is scheduled remotely, and a trust anchor for every upstream
467+
the sandbox reaches must not be sourced from the workload namespace, where it
468+
would widen to anyone holding write access there. The gateway instead reads the
469+
PEM from its own filesystem and stages it into the per-generation supervisor
470+
bootstrap Secret, which is immutable, so the anchor cannot change underneath a
471+
running sandbox.
472+
464473
Proxy credentials are never embedded in the URL: an inline `user:pass@` is
465474
rejected because it would be stored in `gateway.toml` and exposed in container
466475
metadata. Operators supply credentials via `proxy_auth_file`; the driver

‎crates/openshell-core/src/driver_utils.rs‎

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -561,7 +561,9 @@ fn read_regular_file_bounded(path: &str, max_bytes: u64) -> Result<String, Bound
561561
/// between them live in one place instead of being restated per driver.
562562
/// Field names map 1:1 onto the documented TOML keys `https_proxy`,
563563
/// `no_proxy`, `proxy_auth_file`, `proxy_auth_allow_insecure`,
564-
/// `proxy_connect_by_hostname`, and `proxy_ca_bundle`.
564+
/// `proxy_connect_by_hostname`, and `proxy_ca_bundle`. The trailing
565+
/// `auth_setting_label` is not a TOML key; it only names the credential
566+
/// setting in diagnostics.
565567
#[derive(Debug, Clone, Copy, Default)]
566568
pub struct UpstreamProxySettings<'a> {
567569
/// `https_proxy`: the corporate forward proxy URL.
@@ -578,6 +580,14 @@ pub struct UpstreamProxySettings<'a> {
578580
pub connect_by_hostname: Option<bool>,
579581
/// `proxy_ca_bundle`: host path to a PEM CA bundle trusted for the proxy.
580582
pub ca_bundle: Option<&'a str>,
583+
/// Operator-facing name of this driver's credential setting, used only in
584+
/// diagnostics. `None` means `proxy_auth_file`, the local-driver default.
585+
///
586+
/// Drivers that deliver the credential by another mechanism set this so an
587+
/// error never points the operator at a key their driver rejects: the
588+
/// Kubernetes driver takes credentials from a Secret and denies unknown
589+
/// config keys, so `proxy_auth_file` would be actively misleading there.
590+
pub auth_setting_label: Option<&'a str>,
581591
}
582592

583593
/// Validate operator-supplied corporate upstream-proxy settings, fail-closed.
@@ -603,9 +613,11 @@ pub fn validate_upstream_proxy_settings(
603613
let addr = parse_upstream_proxy_url(url).map_err(|err| match err {
604614
UpstreamProxyUrlError::Empty => "https_proxy must not be empty when set".to_string(),
605615
UpstreamProxyUrlError::InlineCredentials => {
606-
"https_proxy must not embed credentials in the URL; supply them via \
607-
proxy_auth_file so they are not stored in config or sandbox metadata"
608-
.to_string()
616+
let auth_setting = settings.auth_setting_label.unwrap_or("proxy_auth_file");
617+
format!(
618+
"https_proxy must not embed credentials in the URL; supply them via \
619+
{auth_setting} so they are not stored in config or sandbox metadata"
620+
)
609621
}
610622
err => format!("https_proxy {err}"),
611623
})?;
@@ -1333,6 +1345,24 @@ mod tests {
13331345
assert!(err.contains("proxy_auth_file"), "{err}");
13341346
}
13351347

1348+
#[test]
1349+
fn upstream_proxy_settings_name_the_driver_credential_setting_in_diagnostics() {
1350+
// A driver that takes credentials elsewhere must not send the operator
1351+
// looking for `proxy_auth_file`; the Kubernetes driver denies unknown
1352+
// config keys, so naming it there would be an actively wrong hint.
1353+
let err = validate_upstream_proxy_settings(&UpstreamProxySettings {
1354+
url: Some("http://u:p@proxy:3128"),
1355+
auth_setting_label: Some("proxy_auth_secret_name and proxy_auth_secret_key"),
1356+
..UpstreamProxySettings::default()
1357+
})
1358+
.expect_err("inline credentials would be stored in gateway config");
1359+
assert!(
1360+
err.contains("proxy_auth_secret_name and proxy_auth_secret_key"),
1361+
"{err}"
1362+
);
1363+
assert!(!err.contains("proxy_auth_file"), "{err}");
1364+
}
1365+
13361366
#[test]
13371367
fn upstream_proxy_settings_reject_an_empty_proxy_url() {
13381368
let err = validate_upstream_proxy_settings(&proxy_settings(Some(" ")))

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -80,7 +80,10 @@ CNI enforces ingress and egress `NetworkPolicy` for sandbox namespaces.
8080
Each sandbox generation uses two immutable bootstrap Secrets. A trusted init
8181
container stages the sandbox bootstrap into memory, and the sandbox removes it
8282
before starting untrusted code. The other Secret is mounted only by the
83-
supervisor. The TLS channel binds the namespace, Sandbox CR, workload Pod,
83+
supervisor; when `proxy_ca_bundle` is configured it also carries the operator's
84+
corporate proxy CA bundle, which the gateway reads from its own filesystem so
85+
the anchor stays in the gateway's trust domain rather than the sandbox
86+
namespace. The TLS channel binds the namespace, Sandbox CR, workload Pod,
8487
supervisor Pod, and shared network-policy identities. Stop deletes the workload
8588
and supervisor Pods. Start rotates both Secrets and creates a new supervisor
8689
Pod before releasing a new workload Pod. The shared network fence remains for

‎crates/openshell-driver-kubernetes/src/config.rs‎

Lines changed: 162 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,19 @@ pub const DEFAULT_GATEWAY_ID: &str = "openshell";
5555
/// Default Kubernetes namespace for sandbox resources.
5656
pub const DEFAULT_K8S_NAMESPACE: &str = "openshell";
5757

58+
/// Hard upper bound on a `proxy_ca_bundle` staged into a bootstrap Secret.
59+
///
60+
/// The shared reader already bounds the file at
61+
/// [`MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES`](openshell_core::driver_utils::MAX_UPSTREAM_PROXY_CA_BUNDLE_BYTES),
62+
/// but that bound is exactly the apiserver's own ~1 MiB limit on a Secret,
63+
/// and the bootstrap Secret carries four other keys besides this one. A
64+
/// bundle between the two limits would be accepted on the gateway host and
65+
/// then fail every sandbox create with an opaque apiserver `data: Too long`
66+
/// error, so this driver bounds it well clear of that ceiling and names the
67+
/// operator setting in the error. A bundle holding every corporate trust
68+
/// anchor is a few tens of kilobytes.
69+
pub const MAX_STAGED_PROXY_CA_BUNDLE_BYTES: usize = 256 * 1024;
70+
5871
/// Default Kubernetes `ServiceAccount` assigned to sandbox pods.
5972
pub const DEFAULT_SANDBOX_SERVICE_ACCOUNT_NAME: &str = "default";
6073

@@ -217,6 +230,25 @@ pub struct KubernetesComputeConfig {
217230
/// Send hostnames rather than validated IPs in CONNECT requests. This is a
218231
/// last-resort compatibility mode for hostname-filtering proxy ACLs.
219232
pub proxy_connect_by_hostname: Option<bool>,
233+
/// Path, on the gateway Pod's filesystem, to a PEM CA bundle trusted for
234+
/// the corporate proxy.
235+
///
236+
/// A CA certificate is not secret, so the gateway reads this file and
237+
/// stages it into the per-generation supervisor bootstrap Secret, which
238+
/// kubelet mounts read-only in the supervisor Pod. The supervisor trusts
239+
/// it for the TLS handshake with an `https://` proxy and, because a
240+
/// TLS-intercepting proxy re-signs tunneled server certificates with the
241+
/// same CA, folds it into the sandbox trust bundle and upstream
242+
/// verification.
243+
///
244+
/// The bundle is deliberately read from the gateway's own filesystem
245+
/// rather than referenced as an object in the sandbox namespace: it
246+
/// becomes a trust anchor for every upstream the sandbox reaches, so it
247+
/// must stay in the gateway's trust domain. The staging Secret is
248+
/// immutable, so the anchor cannot change underneath a running sandbox.
249+
/// Only meaningful with `https_proxy` set; the bundle must exist and
250+
/// contain at least one usable trust anchor.
251+
pub proxy_ca_bundle: Option<String>,
220252
pub grpc_endpoint: String,
221253
pub ssh_socket_path: String,
222254
pub client_tls_secret_name: String,
@@ -318,6 +350,7 @@ impl Default for KubernetesComputeConfig {
318350
proxy_auth_secret_key: None,
319351
proxy_auth_allow_insecure: None,
320352
proxy_connect_by_hostname: None,
353+
proxy_ca_bundle: None,
321354
grpc_endpoint: String::new(),
322355
ssh_socket_path: openshell_core::container_paths::SSH_SOCKET_PATH.to_string(),
323356
client_tls_secret_name: String::new(),
@@ -376,25 +409,46 @@ impl KubernetesComputeConfig {
376409
}
377410

378411
/// Validate the operator-owned corporate upstream proxy configuration.
412+
///
413+
/// The URL grammar and the pairing rules for `no_proxy`,
414+
/// `proxy_connect_by_hostname`, and `proxy_ca_bundle` are delegated to
415+
/// [`validate_upstream_proxy_settings`](openshell_core::driver_utils::validate_upstream_proxy_settings)
416+
/// so this driver can never accept a value the supervisor rejects, or
417+
/// drift from the Podman and VM drivers.
418+
///
419+
/// Credential validation stays local, because this driver takes
420+
/// credentials from a Kubernetes Secret reference rather than the
421+
/// `proxy_auth_file` the shared rules describe.
422+
///
423+
/// # Errors
424+
///
425+
/// Returns a message naming the offending key.
379426
pub fn validate_upstream_proxy_config(&self) -> Result<(), String> {
380-
use openshell_core::driver_utils::{UpstreamProxyUrlError, parse_upstream_proxy_url};
381-
382-
if let Some(url) = &self.https_proxy {
383-
parse_upstream_proxy_url(url).map_err(|err| match err {
384-
UpstreamProxyUrlError::Empty => "https_proxy must not be empty when set".to_string(),
385-
UpstreamProxyUrlError::InlineCredentials => "https_proxy must not embed credentials in the URL; supply them through proxy_auth_secret_name and proxy_auth_secret_key".to_string(),
386-
err => format!("https_proxy {err}"),
387-
})?;
388-
}
427+
use openshell_core::driver_utils::{
428+
UpstreamProxySettings, parse_upstream_proxy_url, validate_upstream_proxy_settings,
429+
};
389430

390-
if let Some(list) = self.no_proxy.as_deref() {
391-
if list.trim().is_empty() {
392-
return Err("no_proxy must not be empty when set; omit it instead".to_string());
393-
}
394-
if self.https_proxy.is_none() {
395-
return Err("no_proxy is set but no https_proxy is configured".to_string());
396-
}
397-
}
431+
validate_upstream_proxy_settings(&UpstreamProxySettings {
432+
url: self.https_proxy.as_deref(),
433+
no_proxy: self.no_proxy.as_deref(),
434+
connect_by_hostname: self.proxy_connect_by_hostname,
435+
ca_bundle: self.proxy_ca_bundle.as_deref(),
436+
// Credentials are validated below, so the shared auth fields stay
437+
// unset; this label only redirects the inline-credential
438+
// diagnostic away from `proxy_auth_file`, which this driver
439+
// rejects as an unknown key.
440+
auth_setting_label: Some("proxy_auth_secret_name and proxy_auth_secret_key"),
441+
..UpstreamProxySettings::default()
442+
})?;
443+
444+
// A credential sent to an `https://` proxy travels inside the verified
445+
// TLS session to that proxy, so the cleartext acknowledgement below is
446+
// only meaningful for an `http://` one. The URL already parsed above.
447+
let proxy_secure = self
448+
.https_proxy
449+
.as_deref()
450+
.and_then(|url| parse_upstream_proxy_url(url).ok())
451+
.is_some_and(|addr| addr.secure);
398452

399453
let secret_name = self.proxy_auth_secret_name.as_deref();
400454
let secret_key = self.proxy_auth_secret_key.as_deref();
@@ -444,8 +498,8 @@ impl KubernetesComputeConfig {
444498
.to_string(),
445499
);
446500
}
447-
if self.proxy_auth_allow_insecure != Some(true) {
448-
return Err("proxy credentials use cleartext Basic auth over the connection to the http:// proxy; set proxy_auth_allow_insecure = true to accept that exposure, or remove the credential Secret".to_string());
501+
if !proxy_secure && self.proxy_auth_allow_insecure != Some(true) {
502+
return Err("proxy credentials use cleartext Basic auth over the connection to the http:// proxy; set proxy_auth_allow_insecure = true to accept that exposure, use an https:// proxy, or remove the credential Secret".to_string());
449503
}
450504
}
451505
_ => {
@@ -456,11 +510,6 @@ impl KubernetesComputeConfig {
456510
}
457511
}
458512

459-
if self.proxy_connect_by_hostname.is_some() && self.https_proxy.is_none() {
460-
return Err(
461-
"proxy_connect_by_hostname is set but no https_proxy is configured".to_string(),
462-
);
463-
}
464513
Ok(())
465514
}
466515

@@ -1166,10 +1215,15 @@ mod tests {
11661215
proxy_auth_secret_key = "credentials"
11671216
proxy_auth_allow_insecure = true
11681217
proxy_connect_by_hostname = true
1218+
proxy_ca_bundle = "/etc/openshell-tls/proxy-ca/ca.crt"
11691219
"#,
11701220
)
11711221
.unwrap();
11721222
assert!(cfg.validate_upstream_proxy_config().is_ok());
1223+
assert_eq!(
1224+
cfg.proxy_ca_bundle.as_deref(),
1225+
Some("/etc/openshell-tls/proxy-ca/ca.crt")
1226+
);
11731227
assert_eq!(
11741228
cfg.https_proxy.as_deref(),
11751229
Some("http://proxy.corp.example:8080")
@@ -1317,6 +1371,90 @@ mod tests {
13171371
assert!(cfg.validate_upstream_proxy_config().is_ok());
13181372
}
13191373

1374+
#[test]
1375+
fn upstream_proxy_config_accepts_a_ca_bundle_with_an_https_proxy() {
1376+
let cfg = KubernetesComputeConfig {
1377+
https_proxy: Some("https://proxy.corp.example:3130".to_string()),
1378+
proxy_ca_bundle: Some("/etc/openshell-tls/proxy-ca/ca.crt".to_string()),
1379+
..KubernetesComputeConfig::default()
1380+
};
1381+
assert!(cfg.validate_upstream_proxy_config().is_ok());
1382+
}
1383+
1384+
#[test]
1385+
fn upstream_proxy_config_accepts_a_ca_bundle_for_an_intercepting_http_proxy() {
1386+
// A proxy reached over plain HTTP still re-signs tunneled upstream
1387+
// certificates when it intercepts TLS, so the bundle is meaningful
1388+
// without an `https://` proxy URL.
1389+
let cfg = KubernetesComputeConfig {
1390+
https_proxy: Some("http://proxy.corp.example:8080".to_string()),
1391+
proxy_ca_bundle: Some("/etc/openshell-tls/proxy-ca/ca.crt".to_string()),
1392+
..KubernetesComputeConfig::default()
1393+
};
1394+
assert!(cfg.validate_upstream_proxy_config().is_ok());
1395+
}
1396+
1397+
#[test]
1398+
fn upstream_proxy_config_rejects_a_ca_bundle_without_a_proxy() {
1399+
let cfg = KubernetesComputeConfig {
1400+
proxy_ca_bundle: Some("/etc/openshell-tls/proxy-ca/ca.crt".to_string()),
1401+
..KubernetesComputeConfig::default()
1402+
};
1403+
let err = cfg.validate_upstream_proxy_config().unwrap_err();
1404+
assert!(err.contains("proxy_ca_bundle"), "{err}");
1405+
}
1406+
1407+
#[test]
1408+
fn upstream_proxy_config_rejects_an_empty_ca_bundle() {
1409+
// Present-but-empty is a misconfiguration, not "unset": the supervisor
1410+
// treats an empty driver-supplied argument as a fatal error.
1411+
let cfg = KubernetesComputeConfig {
1412+
https_proxy: Some("https://proxy.corp.example:3130".to_string()),
1413+
proxy_ca_bundle: Some(" ".to_string()),
1414+
..KubernetesComputeConfig::default()
1415+
};
1416+
let err = cfg.validate_upstream_proxy_config().unwrap_err();
1417+
assert!(err.contains("proxy_ca_bundle"), "{err}");
1418+
}
1419+
1420+
#[test]
1421+
fn upstream_proxy_config_accepts_credentials_without_acknowledgement_for_an_https_proxy() {
1422+
// The credential travels inside the verified TLS session to the proxy,
1423+
// so the cleartext acknowledgement is not meaningful here.
1424+
let cfg = KubernetesComputeConfig {
1425+
https_proxy: Some("https://proxy.corp.example:3130".to_string()),
1426+
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
1427+
proxy_auth_secret_key: Some("credentials".to_string()),
1428+
..KubernetesComputeConfig::default()
1429+
};
1430+
assert!(cfg.validate_upstream_proxy_config().is_ok());
1431+
}
1432+
1433+
#[test]
1434+
fn upstream_proxy_config_still_requires_acknowledgement_for_an_http_proxy() {
1435+
let cfg = KubernetesComputeConfig {
1436+
https_proxy: Some("http://proxy.corp.example:8080".to_string()),
1437+
proxy_auth_secret_name: Some("corporate-proxy-auth".to_string()),
1438+
proxy_auth_secret_key: Some("credentials".to_string()),
1439+
..KubernetesComputeConfig::default()
1440+
};
1441+
let err = cfg.validate_upstream_proxy_config().unwrap_err();
1442+
assert!(err.contains("proxy_auth_allow_insecure"), "{err}");
1443+
}
1444+
1445+
#[test]
1446+
fn upstream_proxy_config_names_the_secret_keys_for_inline_credentials() {
1447+
// `proxy_auth_file` is a Podman/VM key that this driver rejects under
1448+
// `deny_unknown_fields`, so it must never appear in the diagnostic.
1449+
let cfg = KubernetesComputeConfig {
1450+
https_proxy: Some("http://user:[email protected]:8080".to_string()),
1451+
..KubernetesComputeConfig::default()
1452+
};
1453+
let err = cfg.validate_upstream_proxy_config().unwrap_err();
1454+
assert!(err.contains("proxy_auth_secret_name"), "{err}");
1455+
assert!(!err.contains("proxy_auth_file"), "{err}");
1456+
}
1457+
13201458
#[test]
13211459
fn namespace_for_workspace_shared() {
13221460
let cfg = KubernetesComputeConfig {

0 commit comments

Comments
 (0)