Skip to content

Commit 9a705b3

Browse files
feat(k8s): support configuring workspace PVC storageClassName
The Kubernetes driver's default workspace PVC never set storageClassName, so on clusters with no default StorageClass the PVC stayed Pending and sandbox creation failed. Add a workspace_storage_class option to KubernetesComputeConfig, wired through SandboxPodParams into the generated volumeClaimTemplates. When non-empty it sets storageClassName; empty preserves the current behavior of relying on the cluster default StorageClass. Expose it via the OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS env var on both the standalone driver and the embedded gateway runtime defaults, and via the server.workspaceStorageClass Helm value. Closes #2442 Signed-off-by: lr90 <[email protected]>
1 parent 21da343 commit 9a705b3

11 files changed

Lines changed: 120 additions & 12 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -393,6 +393,7 @@ openshell logs <sandbox-name>
393393
| Docker or Podman sandbox never registers | Wrong callback endpoint or supervisor startup failure | Gateway logs and sandbox container logs |
394394
| Docker GPU e2e fails before GPU sandbox comparison | NVIDIA CDI specs are missing or Docker has not discovered them | `docker info --format '{{json .DiscoveredDevices}}'`, `/etc/cdi`, `/var/run/cdi`, `nvidia-cdi-refresh.service` |
395395
| Kubernetes gateway pod pending | PVC unbound, taint, selector, or insufficient resources | `kubectl -n openshell describe pod <pod>` |
396+
| Kubernetes sandbox pod stuck pending, workspace PVC unbound | Cluster has no default `StorageClass` and OpenShell does not set `storageClassName` on the workspace PVC (clusters with a default `StorageClass` bind fine without it) | `kubectl -n openshell describe pvc`; set `server.workspaceStorageClass` (gateway config `workspace_storage_class`) to a valid `StorageClass` |
396397
| Kubernetes gateway pod crash loops | Missing secret, bad DB URL, bad TLS config | `kubectl -n openshell logs deployment/openshell -c openshell-gateway` or `kubectl -n openshell logs statefulset/openshell -c openshell-gateway` |
397398
| CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
398399
| Edge or OIDC gateway returns `Unauthenticated` | Stored login expired, audience/scopes mismatch, or gateway auth configuration changed | `openshell gateway info`, `openshell gateway login <name>`, gateway auth logs |

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,16 @@ This is a stopgap persistence model. It preserves user files across pod
3636
rescheduling but duplicates the base workspace and does not automatically apply
3737
image updates to existing PVCs. Future snapshotting should replace it.
3838

39+
The workspace PVC size defaults to `workspace_default_storage_size`. Set
40+
`workspace_storage_class` to pin the PVC to a specific `StorageClass`; an empty
41+
value omits `storageClassName` so the cluster's default `StorageClass` applies.
42+
Clusters with no default `StorageClass` must set this, otherwise the PVC stays
43+
`Pending` and the sandbox never starts. Both fields can also be supplied at
44+
runtime via `OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE` and
45+
`OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS`. Both apply only to the workspace PVC
46+
that OpenShell provisions automatically; they have no effect when a `driver_config`
47+
mount attaches an existing PVC under `/sandbox`, which skips the default PVC.
48+
3949
## Credentials, TLS, and Relay
4050

4151
The driver injects gateway callback configuration, sandbox identity, TLS client

‎crates/openshell-driver-kubernetes/src/config.rs‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,6 +267,12 @@ pub struct KubernetesComputeConfig {
267267
)]
268268
pub app_armor_profile: Option<AppArmorProfile>,
269269
pub workspace_default_storage_size: String,
270+
/// Kubernetes `StorageClass` name for the default workspace PVC.
271+
/// Empty string (default) = omit `storageClassName`, using the cluster's
272+
/// default `StorageClass`. Set this on clusters with no default
273+
/// `StorageClass`, otherwise the workspace PVC stays `Pending` and the
274+
/// sandbox never starts.
275+
pub workspace_storage_class: String,
270276
/// Default Kubernetes `runtimeClassName` for sandbox pods.
271277
/// Applied when a `CreateSandbox` request does not specify one.
272278
/// Empty string (default) = omit the field, using the cluster default.
@@ -347,6 +353,7 @@ impl Default for KubernetesComputeConfig {
347353
enable_user_namespaces: false,
348354
app_armor_profile: None,
349355
workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE.to_string(),
356+
workspace_storage_class: String::new(),
350357
default_runtime_class_name: String::new(),
351358
sa_token_ttl_secs: 3600,
352359
provider_spiffe_workload_api_socket_path: String::new(),
@@ -514,6 +521,12 @@ mod tests {
514521
);
515522
}
516523

524+
#[test]
525+
fn default_workspace_storage_class_is_empty() {
526+
let cfg = KubernetesComputeConfig::default();
527+
assert!(cfg.workspace_storage_class.is_empty());
528+
}
529+
517530
#[test]
518531
fn default_topology_is_combined() {
519532
let cfg = KubernetesComputeConfig::default();
@@ -658,6 +671,15 @@ mod tests {
658671
assert_eq!(cfg.workspace_default_storage_size, "10Gi");
659672
}
660673

674+
#[test]
675+
fn serde_override_workspace_storage_class() {
676+
let json = serde_json::json!({
677+
"workspace_storage_class": "fast-ssd"
678+
});
679+
let cfg: KubernetesComputeConfig = serde_json::from_value(json).unwrap();
680+
assert_eq!(cfg.workspace_storage_class, "fast-ssd");
681+
}
682+
661683
#[test]
662684
fn serde_override_service_account_name() {
663685
let json = serde_json::json!({

‎crates/openshell-driver-kubernetes/src/driver.rs‎

Lines changed: 68 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -848,6 +848,7 @@ impl KubernetesComputeDriver {
848848
enable_user_namespaces: self.config.enable_user_namespaces,
849849
app_armor_profile: self.config.app_armor_profile.as_ref(),
850850
workspace_default_storage_size: &self.config.workspace_default_storage_size,
851+
workspace_storage_class: &self.config.workspace_storage_class,
851852
default_runtime_class_name: &self.config.default_runtime_class_name,
852853
sa_token_ttl_secs: self.config.effective_sa_token_ttl_secs(),
853854
provider_spiffe_enabled: self.config.provider_spiffe_enabled(),
@@ -2155,24 +2156,36 @@ fn apply_workspace_persistence(
21552156
///
21562157
/// Provides a single PVC named "workspace" that backs the `/sandbox`
21572158
/// directory. The init container seeds it from the image on first use.
2158-
fn default_workspace_volume_claim_templates(storage_size: &str) -> serde_json::Value {
2159+
///
2160+
/// When `storage_class` is non-empty, it is written to the PVC's
2161+
/// `storageClassName`. An empty value omits the field so the cluster's
2162+
/// default `StorageClass` applies. Clusters with no default `StorageClass`
2163+
/// must set this to prevent the PVC from staying `Pending`.
2164+
fn default_workspace_volume_claim_templates(
2165+
storage_size: &str,
2166+
storage_class: &str,
2167+
) -> serde_json::Value {
21592168
let size = if storage_size.is_empty() {
21602169
DEFAULT_WORKSPACE_STORAGE_SIZE
21612170
} else {
21622171
storage_size
21632172
};
2173+
let mut spec = serde_json::json!({
2174+
"accessModes": ["ReadWriteOnce"],
2175+
"resources": {
2176+
"requests": {
2177+
"storage": size
2178+
}
2179+
}
2180+
});
2181+
if !storage_class.is_empty() {
2182+
spec["storageClassName"] = serde_json::json!(storage_class);
2183+
}
21642184
serde_json::json!([{
21652185
"metadata": {
21662186
"name": WORKSPACE_VOLUME_NAME
21672187
},
2168-
"spec": {
2169-
"accessModes": ["ReadWriteOnce"],
2170-
"resources": {
2171-
"requests": {
2172-
"storage": size
2173-
}
2174-
}
2175-
}
2188+
"spec": spec
21762189
}])
21772190
}
21782191

@@ -2197,6 +2210,7 @@ struct SandboxPodParams<'a> {
21972210
enable_user_namespaces: bool,
21982211
app_armor_profile: Option<&'a AppArmorProfile>,
21992212
workspace_default_storage_size: &'a str,
2213+
workspace_storage_class: &'a str,
22002214
default_runtime_class_name: &'a str,
22012215
/// Lifetime (seconds) of the projected `ServiceAccount` token used
22022216
/// for the bootstrap `IssueSandboxToken` exchange.
@@ -2231,6 +2245,7 @@ impl Default for SandboxPodParams<'_> {
22312245
enable_user_namespaces: false,
22322246
app_armor_profile: None,
22332247
workspace_default_storage_size: DEFAULT_WORKSPACE_STORAGE_SIZE,
2248+
workspace_storage_class: "",
22342249
default_runtime_class_name: "",
22352250
sa_token_ttl_secs: 3600,
22362251
provider_spiffe_enabled: false,
@@ -2328,7 +2343,10 @@ fn sandbox_to_k8s_spec(
23282343
if inject_workspace {
23292344
root.insert(
23302345
"volumeClaimTemplates".to_string(),
2331-
default_workspace_volume_claim_templates(params.workspace_default_storage_size),
2346+
default_workspace_volume_claim_templates(
2347+
params.workspace_default_storage_size,
2348+
params.workspace_storage_class,
2349+
),
23322350
);
23332351
}
23342352

@@ -5714,14 +5732,14 @@ mod tests {
57145732

57155733
#[test]
57165734
fn default_workspace_vct_uses_provided_storage_size() {
5717-
let vct = default_workspace_volume_claim_templates("5Gi");
5735+
let vct = default_workspace_volume_claim_templates("5Gi", "");
57185736
let storage = &vct[0]["spec"]["resources"]["requests"]["storage"];
57195737
assert_eq!(storage, "5Gi");
57205738
}
57215739

57225740
#[test]
57235741
fn default_workspace_vct_falls_back_to_const_when_empty() {
5724-
let vct = default_workspace_volume_claim_templates("");
5742+
let vct = default_workspace_volume_claim_templates("", "");
57255743
let storage = &vct[0]["spec"]["resources"]["requests"]["storage"];
57265744
assert_eq!(storage, DEFAULT_WORKSPACE_STORAGE_SIZE);
57275745
}
@@ -5921,4 +5939,42 @@ mod tests {
59215939
};
59225940
assert!(sandbox_id_from_object(&obj).is_err());
59235941
}
5942+
5943+
#[test]
5944+
fn default_workspace_vct_sets_storage_class_when_provided() {
5945+
let vct = default_workspace_volume_claim_templates("5Gi", "fast-ssd");
5946+
assert_eq!(vct[0]["spec"]["storageClassName"], "fast-ssd");
5947+
}
5948+
5949+
#[test]
5950+
fn default_workspace_vct_omits_storage_class_when_empty() {
5951+
let vct = default_workspace_volume_claim_templates("5Gi", "");
5952+
assert!(vct[0]["spec"].get("storageClassName").is_none());
5953+
}
5954+
5955+
#[test]
5956+
fn workspace_storage_class_propagates_to_generated_cr_spec() {
5957+
let params = SandboxPodParams {
5958+
workspace_storage_class: "fast-ssd",
5959+
..SandboxPodParams::default()
5960+
};
5961+
let cr = sandbox_to_k8s_spec_for_test(Some(&SandboxSpec::default()), &params);
5962+
assert_eq!(
5963+
cr["spec"]["volumeClaimTemplates"][0]["spec"]["storageClassName"],
5964+
"fast-ssd"
5965+
);
5966+
}
5967+
5968+
#[test]
5969+
fn workspace_storage_class_omitted_from_cr_spec_when_empty() {
5970+
let cr = sandbox_to_k8s_spec_for_test(
5971+
Some(&SandboxSpec::default()),
5972+
&SandboxPodParams::default(),
5973+
);
5974+
assert!(
5975+
cr["spec"]["volumeClaimTemplates"][0]["spec"]
5976+
.get("storageClassName")
5977+
.is_none()
5978+
);
5979+
}
59245980
}

‎crates/openshell-driver-kubernetes/src/main.rs‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,8 @@ async fn main() -> Result<()> {
160160
.unwrap_or_else(|_| {
161161
openshell_driver_kubernetes::DEFAULT_WORKSPACE_STORAGE_SIZE.to_string()
162162
}),
163+
workspace_storage_class: std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS")
164+
.unwrap_or_default(),
163165
default_runtime_class_name: std::env::var("OPENSHELL_K8S_DEFAULT_RUNTIME_CLASS_NAME")
164166
.unwrap_or_default(),
165167
sa_token_ttl_secs: args.sa_token_ttl_secs,

‎crates/openshell-server/src/compute/driver_config.rs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -145,6 +145,9 @@ fn apply_kubernetes_runtime_defaults(k8s: &mut KubernetesComputeConfig) {
145145
if let Ok(size) = std::env::var("OPENSHELL_K8S_WORKSPACE_DEFAULT_STORAGE_SIZE") {
146146
k8s.workspace_default_storage_size = size;
147147
}
148+
if let Ok(storage_class) = std::env::var("OPENSHELL_K8S_WORKSPACE_STORAGE_CLASS") {
149+
k8s.workspace_storage_class = storage_class;
150+
}
148151
}
149152

150153
fn apply_podman_runtime_defaults(

‎deploy/helm/openshell/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,7 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
229229
| server.tls.clientCaSecretName | string | `"openshell-server-client-ca"` | K8s secret with ca.crt for client certificate verification (mTLS). Set to "" to disable mTLS and run HTTPS-only (use OIDC for auth instead). |
230230
| server.tls.clientTlsSecretName | string | `"openshell-client-tls"` | K8s secret mounted into sandbox pods for mTLS to the server. |
231231
| server.workspaceDefaultStorageSize | string | `""` | Default storage size for the workspace PVC in sandbox pods. Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi"). Empty = built-in default (2Gi). |
232+
| server.workspaceStorageClass | string | `""` | Kubernetes StorageClass for the workspace PVC in sandbox pods. Empty (default) = omit storageClassName, using the cluster's default StorageClass. Set this on clusters with no default StorageClass, otherwise the workspace PVC stays Pending and the sandbox never starts. |
232233
| service.healthPort | int | `8081` | Gateway health service port. |
233234
| service.metricsPort | int | `9090` | Gateway metrics service port. |
234235
| service.port | int | `8080` | Gateway gRPC/HTTP service port. |

‎deploy/helm/openshell/templates/gateway-config.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,9 @@ data:
135135
{{- if .Values.server.workspaceDefaultStorageSize }}
136136
workspace_default_storage_size = {{ .Values.server.workspaceDefaultStorageSize | quote }}
137137
{{- end }}
138+
{{- if .Values.server.workspaceStorageClass }}
139+
workspace_storage_class = {{ .Values.server.workspaceStorageClass | quote }}
140+
{{- end }}
138141
{{- if .Values.server.defaultRuntimeClassName }}
139142
default_runtime_class_name = {{ .Values.server.defaultRuntimeClassName | quote }}
140143
{{- end }}

‎deploy/helm/openshell/values.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -187,6 +187,11 @@ server:
187187
# Uses Kubernetes quantity syntax (e.g. "2Gi", "10Gi", "500Mi").
188188
# Empty = built-in default (2Gi).
189189
workspaceDefaultStorageSize: ""
190+
# -- Kubernetes StorageClass for the workspace PVC in sandbox pods.
191+
# Empty (default) = omit storageClassName, using the cluster's default
192+
# StorageClass. Set this on clusters with no default StorageClass, otherwise
193+
# the workspace PVC stays Pending and the sandbox never starts.
194+
workspaceStorageClass: ""
190195
# -- Default Kubernetes runtimeClassName for sandbox pods.
191196
# Applied when a CreateSandbox request does not specify one.
192197
# Empty (default) = omit the field, using the cluster's default RuntimeClass.

‎docs/reference/gateway-config.mdx‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -277,6 +277,10 @@ host_gateway_ip = "10.0.0.1"
277277
enable_user_namespaces = false
278278
app_armor_profile = "Unconfined"
279279
workspace_default_storage_size = "10Gi"
280+
# Kubernetes StorageClass for the workspace PVC. Empty (default) omits the
281+
# field, using the cluster's default StorageClass. Set this on clusters with no
282+
# default StorageClass, otherwise the workspace PVC stays Pending.
283+
# workspace_storage_class = "fast-ssd"
280284
# Kubernetes RuntimeClass applied to sandbox pods when the API request does
281285
# not specify one. Empty (default) = omit the field, using the cluster default.
282286
# default_runtime_class_name = "kata-containers"

0 commit comments

Comments
 (0)