Skip to content

Commit 8f592ba

Browse files
authored
Merge pull request NVIDIA#42 from Ygnas/test/rhaieng-6877-selinux-qualification
test(odh): add SELinux enforcing qualification
2 parents 32f0315 + d4da8a6 commit 8f592ba

10 files changed

Lines changed: 650 additions & 73 deletions

File tree

‎e2e/rust/tests/odh/AGENTS.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ a rule and the README disagree, fix the drift rather than guessing.
3333
## Commit messages
3434

3535
- Prefix the commit title of any downstream-only (fork carry) commit with
36-
`CARRY: ` so it is identifiable when rebasing against `NVIDIA/OpenShell`.
36+
`CARRY:` so it is identifiable when rebasing against `NVIDIA/OpenShell`.
3737
- Keep the repository-root Conventional Commits format after the prefix, e.g.
3838
`CARRY: test(odh): add SELinux enforcing coverage`.
3939
- Sign off every commit for DCO (`git commit --signoff`) and never reference AI

‎e2e/rust/tests/odh/README.md‎

Lines changed: 40 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,8 @@ e2e/rust/tests/odh/
2525
│ ├── image_provenance.rs # sandbox/gateway/supervisor image registry + pull-policy check
2626
│ └── sandbox.rs # sandbox create/exec/delete + supervisor presence check
2727
├── tier1/ # Tier 1: high-priority tests, excluding Smoke
28-
│ └── mod.rs # empty — no scenarios yet
28+
│ ├── mod.rs
29+
│ └── selinux.rs # combined/sidecar process-supervisor SELinux label
2930
├── tier2/ # Tier 2: medium/low priority positive tests
3031
│ └── mod.rs # empty — no scenarios yet
3132
├── tier3/ # Tier 3: negative and destructive tests
@@ -56,9 +57,12 @@ plain module is enough — no new crate and no workspace change.
5657
when it is set, so every ODH test targets the same cluster the upstream
5758
harness does. `oc_json()` runs a query and parses `-o json` output, panicking
5859
with a descriptive message on any failure.
60+
- `odh_harness::selinux::SelinuxAudit` — an opt-in scenario guard that detects
61+
OpenShift, requires every Ready worker to report `Enforcing`, records
62+
node-local audit cutoffs, and rejects OpenShell AVCs on completion.
5963

60-
Put ODH-specific shared helpers here (the `oc` builder, and future node-level
61-
checks such as `getenforce` and the AVC-audit guard), and reuse them rather than
64+
Put ODH-specific shared helpers here (the `oc` builder and node-level SELinux
65+
checks), and reuse them rather than
6266
duplicating logic across tier files. Two rules keep this rebase-safe:
6367

6468
- **Do not** add ODH helpers to the upstream harness library
@@ -90,14 +94,11 @@ The upstream test assignments in `tiers.toml` are currently placeholders —
9094
they should be revisited based on measured execution time and actual test
9195
criticality, not just copied as-is.
9296

93-
**Current implementation status:** only the Smoke tier has real test
94-
functions (`smoke::gateway::test_reachable`, `smoke::sandbox::test_create_delete`,
95-
`smoke::image_provenance::test_sandbox_gateway_supervisor_images`).
96-
Tier 1–3 are empty modules with no scenarios yet — running those tiers today
97-
executes 0 ODH tests (a legitimate `ok` result, not a failure) plus whatever
98-
upstream tests are mapped to them, plus the image provenance test (see
99-
below). Add scenarios by creating a `.rs` file under the tier's directory and
100-
declaring it with a `mod` line in that tier's `mod.rs`.
97+
Smoke covers gateway reachability, sandbox lifecycle, and image provenance.
98+
Tier 1 checks the process-supervisor SELinux label. Tier 2 and Tier 3 have no
99+
ODH scenarios yet; they run their mapped upstream tests and the image
100+
provenance check. Add scenarios by creating a `.rs` file under the tier's
101+
directory and declaring it with a `mod` line in that tier's `mod.rs`.
101102

102103
## Prerequisites
103104

@@ -165,6 +166,25 @@ defaults (`openshell`/`openshell`). The Helm chart's
165166
defaults to `""`, i.e. Kubernetes' own default of `Always` for the
166167
`:latest`-tagged sandbox image) — see below.
167168

169+
### SELinux-enforcing OCP validation
170+
171+
Tier 1 includes the SELinux checks when it runs against an OpenShift cluster.
172+
The tests expect an already-deployed, working gateway. The named preflight
173+
checks `getenforce` on every Ready worker. Each audited SELinux scenario uses
174+
`SelinuxAudit`, which self-skips outside OpenShift, records node-local cutoffs,
175+
and queries `ausearch -m AVC -x` through `oc debug node … chroot /host` after
176+
the scenario.
177+
The audit covers the gateway and both supervisor executable paths, and the
178+
supervisor test checks the stable `container_t` type without hard-coding pod
179+
MCS categories.
180+
181+
The runner requires permission to create debug pods and access the host through
182+
`chroot /host`. Deployment-specific Helm values and proxy fixtures are owned by
183+
the environment that deploys the gateway; tier1 does not create them. The
184+
custom egress control is intentionally outside SELinux scope; egress behavior
185+
remains covered by the existing proxy tests, and this tier does not create a
186+
SELinux-specific proxy fixture.
187+
168188
### Why `e2e:odh` / `e2e:odh:full` run more than you might expect
169189

170190
`e2e-odh` is defined as `e2e-odh = ["e2e-kubernetes"]` in `Cargo.toml`, so it
@@ -194,7 +214,7 @@ already covers it.
194214
`smoke::image_provenance::test_sandbox_gateway_supervisor_images` creates a
195215
sandbox, then verifies that its image, the gateway's image, and the
196216
supervisor image (read from the rendered gateway config, since the
197-
supervisor never runs as its own pod) all came from an authorized downstream
217+
supervisor never runs as its own pod) all came from an authorized
198218
registry, and that no container — including ephemeral containers — has
199219
regressed away from `imagePullPolicy: IfNotPresent`.
200220

@@ -208,33 +228,32 @@ ALLOWED_IMAGE_REGISTRY_PREFIXES="quay.io/opendatahub/,ghcr.io/nvidia/openshell-c
208228
must end in `/`) — no default, since an empty list would silently approve
209229
any image. This should include every registry prefix your deployment
210230
legitimately pulls from:
211-
- `quay.io/opendatahub/` — confirmed in practice for this project's
212-
gateway/supervisor/CLI builds.
231+
- `ghcr.io/nvidia/openshell/` — the standard upstream gateway,
232+
supervisor, and CLI image prefix.
213233
- `ghcr.io/nvidia/openshell-community/sandboxes/` — the sandbox default
214234
image (`server.sandboxImage` in the Helm chart). There is no
215235
downstream-built sandbox base image yet (only `gateway`, `supervisor`,
216236
and `cli` have Tekton pipelines under `.tekton/`), so this upstream
217237
prefix has to stay allowed until one exists. This is a different path
218-
than the upstream gateway/CLI images
219-
(`ghcr.io/nvidia/openshell/*`), which this check is meant to reject.
238+
alongside the upstream gateway and CLI images.
220239

221240
A prefix without a trailing `/` is rejected outright, since it could
222241
otherwise match a lookalike host (e.g. `registry.redhat.io` would also
223242
match `registry.redhat.io.attacker.example/image`).
224243
- `NAMESPACE`/`RELEASE` env vars default to `openshell`/`openshell`.
225244
- The check is a registry-prefix allowlist, not an exact image/digest match.
226-
It works because the downstream pipeline only ever publishes to one
227-
registry — matching that prefix is sufficient proof an image (including
228-
the supervisor) is the downstream build and not an upstream
229-
`ghcr.io/nvidia/openshell/*` reference.
245+
The allowlist is explicit about both the upstream gateway images and the
246+
upstream sandbox image.
230247
- The `imagePullPolicy: IfNotPresent` check applies to every container,
231248
including the sandbox pod's. The Helm chart's `server.sandboxImagePullPolicy`
232249
defaults to `""` (Kubernetes' own default, which is `Always` for a
233250
`:latest`-tagged image like the default sandbox image) — deployments must
234251
set it explicitly, e.g. `--set server.sandboxImagePullPolicy=IfNotPresent`,
235252
or this check fails on a freshly installed chart.
236253
- Requires the `oc` CLI in PATH with a kubeconfig targeting the cluster (same
237-
as the rest of this suite).
254+
as the rest of this suite). When `OPENSHELL_E2E_KUBE_CONTEXT_ACTIVE` is set,
255+
all provenance queries use that context; otherwise they use the kubeconfig's
256+
current context.
238257
- Skip it locally with `SKIP_IMAGE_PROVENANCE=1 mise run e2e:odh:tier1` (e.g.
239258
if `oc` isn't configured for the target cluster in your current shell) —
240259
this only affects the tiered tasks' extra step; `e2e:odh`/`e2e:odh:full`/

‎e2e/rust/tests/odh/odh_harness/mod.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@
1010
//! collide with the upstream `harness` module.
1111
1212
pub mod oc;
13+
pub mod selinux;

‎e2e/rust/tests/odh/odh_harness/oc.rs‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,3 +45,229 @@ pub async fn oc_json(args: &[&str]) -> Value {
4545
serde_json::from_slice(&output.stdout)
4646
.unwrap_or_else(|e| panic!("oc {args:?} did not return valid JSON: {e}"))
4747
}
48+
49+
/// Resolve the supervisor Pod paired with a named Sandbox resource.
50+
pub async fn paired_supervisor_pod(namespace: &str, sandbox_name: &str) -> Result<String, String> {
51+
let sandbox_selector = format!("openshell.ai/sandbox-name={sandbox_name}");
52+
let sandbox = oc_get_json(&[
53+
"get",
54+
"sandboxes.agents.x-k8s.io",
55+
"-n",
56+
namespace,
57+
"-l",
58+
&sandbox_selector,
59+
"-o",
60+
"json",
61+
])
62+
.await?;
63+
let sandbox_id = sandbox_id_from_json(&sandbox)
64+
.ok_or_else(|| format!("Sandbox {sandbox_name:?} has no openshell.ai/sandbox-id"))?;
65+
66+
let supervisor_selector =
67+
format!("openshell.ai/sandbox-id={sandbox_id},openshell.ai/boundary-role=supervisor");
68+
let pods = oc_get_json(&[
69+
"get",
70+
"pods",
71+
"-n",
72+
namespace,
73+
"-l",
74+
&supervisor_selector,
75+
"-o",
76+
"json",
77+
])
78+
.await?;
79+
supervisor_pod_from_json(&pods)
80+
.map(str::to_owned)
81+
.ok_or_else(|| {
82+
format!(
83+
"expected exactly one supervisor Pod for Sandbox {sandbox_name:?} ({sandbox_id})"
84+
)
85+
})
86+
}
87+
88+
/// Read the SELinux label of the supervisor process from its host node.
89+
pub async fn supervisor_selinux_label(namespace: &str, pod: &str) -> Result<String, String> {
90+
let pod_json = oc_get_json(&["get", "pod", pod, "-n", namespace, "-o", "json"]).await?;
91+
let (node, pod_uid) = pod_node_and_uid(&pod_json)
92+
.ok_or_else(|| format!("supervisor Pod {pod:?} has no node name or UID"))?;
93+
let cgroup_pod_uid = pod_uid_cgroup_form(pod_uid);
94+
let script = r#"
95+
pod_uid="$1"
96+
cgroup_pod_uid="$2"
97+
found=0
98+
for process in /proc/[0-9]*; do
99+
executable=$(readlink "$process/exe" 2>/dev/null) || continue
100+
[ "$executable" = "/openshell-supervisor" ] || continue
101+
grep -q "pod${pod_uid}" "$process/cgroup" 2>/dev/null ||
102+
grep -q "pod${cgroup_pod_uid}" "$process/cgroup" 2>/dev/null || continue
103+
label=
104+
IFS= read -r label < "$process/attr/current" || true
105+
printf '%s %s\n' "$executable" "$label"
106+
found=$((found + 1))
107+
done
108+
[ "$found" -eq 1 ]
109+
"#;
110+
oc_debug_node(
111+
node,
112+
&[
113+
"sh",
114+
"-ec",
115+
script,
116+
"supervisor-label",
117+
pod_uid,
118+
&cgroup_pod_uid,
119+
],
120+
)
121+
.await
122+
}
123+
124+
pub(crate) fn pod_node_and_uid(value: &Value) -> Option<(&str, &str)> {
125+
Some((
126+
value["spec"]["nodeName"].as_str()?,
127+
value["metadata"]["uid"].as_str()?,
128+
))
129+
}
130+
131+
pub(crate) fn pod_uid_cgroup_form(pod_uid: &str) -> String {
132+
pod_uid.replace('-', "_")
133+
}
134+
135+
/// Run a command in the host namespace of an OpenShift node debug pod.
136+
pub(crate) async fn oc_debug_node(node: &str, args: &[&str]) -> Result<String, String> {
137+
let output = oc_command()
138+
.args([
139+
"debug",
140+
&format!("node/{node}"),
141+
"--quiet",
142+
"--no-stdin",
143+
"--no-tty",
144+
"--",
145+
"chroot",
146+
"/host",
147+
])
148+
.args(args)
149+
.output()
150+
.await
151+
.map_err(|error| format!("failed to run oc debug node/{node}: {error}"))?;
152+
if !output.status.success() {
153+
return Err(format!(
154+
"oc debug node/{node} failed: {}",
155+
command_output(&output)
156+
));
157+
}
158+
Ok(String::from_utf8_lossy(&output.stdout).trim().to_string())
159+
}
160+
161+
async fn oc_get_json(args: &[&str]) -> Result<Value, String> {
162+
let output = oc_command()
163+
.args(args)
164+
.output()
165+
.await
166+
.map_err(|error| format!("failed to run oc {args:?}: {error}"))?;
167+
if !output.status.success() {
168+
return Err(format!("oc {args:?} failed: {}", command_output(&output)));
169+
}
170+
serde_json::from_slice(&output.stdout)
171+
.map_err(|error| format!("oc {args:?} returned invalid JSON: {error}"))
172+
}
173+
174+
fn sandbox_id_from_json(value: &Value) -> Option<&str> {
175+
value
176+
.get("items")
177+
.and_then(Value::as_array)
178+
.and_then(|items| (items.len() == 1).then(|| items.first()).flatten())
179+
.and_then(|sandbox| sandbox["metadata"]["labels"]["openshell.ai/sandbox-id"].as_str())
180+
}
181+
182+
fn supervisor_pod_from_json(value: &Value) -> Option<&str> {
183+
value
184+
.get("items")
185+
.and_then(Value::as_array)
186+
.and_then(|items| (items.len() == 1).then(|| items.first()).flatten())
187+
.and_then(|pod| pod["metadata"]["name"].as_str())
188+
}
189+
190+
pub(crate) fn command_output(output: &std::process::Output) -> String {
191+
let stdout = String::from_utf8_lossy(&output.stdout).trim().to_string();
192+
let stderr = String::from_utf8_lossy(&output.stderr).trim().to_string();
193+
match (stdout.is_empty(), stderr.is_empty()) {
194+
(true, true) => "no output".to_string(),
195+
(false, true) => format!("stdout: {stdout}"),
196+
(true, false) => format!("stderr: {stderr}"),
197+
(false, false) => format!("stdout: {stdout}; stderr: {stderr}"),
198+
}
199+
}
200+
201+
#[cfg(test)]
202+
mod tests {
203+
use serde_json::json;
204+
205+
use super::{
206+
pod_node_and_uid, pod_uid_cgroup_form, sandbox_id_from_json, supervisor_pod_from_json,
207+
};
208+
209+
#[test]
210+
fn resolves_sandbox_id_from_named_sandbox() {
211+
let sandbox = json!({
212+
"items": [{
213+
"metadata": {
214+
"labels": {
215+
"openshell.ai/sandbox-id": "sandbox-123"
216+
}
217+
}
218+
}]
219+
});
220+
221+
assert_eq!(sandbox_id_from_json(&sandbox), Some("sandbox-123"));
222+
}
223+
224+
#[test]
225+
fn resolves_the_paired_supervisor_pod() {
226+
let pods = json!({
227+
"items": [{
228+
"metadata": {"name": "os-supervisor-sandbox-123"},
229+
"status": {"phase": "Running"}
230+
}]
231+
});
232+
233+
assert_eq!(
234+
supervisor_pod_from_json(&pods),
235+
Some("os-supervisor-sandbox-123")
236+
);
237+
}
238+
239+
#[test]
240+
fn rejects_missing_or_ambiguous_supervisor_pods() {
241+
assert_eq!(supervisor_pod_from_json(&json!({"items": []})), None);
242+
assert_eq!(
243+
supervisor_pod_from_json(&json!({
244+
"items": [
245+
{"metadata": {"name": "one"}},
246+
{"metadata": {"name": "two"}}
247+
]
248+
})),
249+
None
250+
);
251+
}
252+
253+
#[test]
254+
fn extracts_node_and_uid_from_pod() {
255+
let pod = json!({
256+
"metadata": {"uid": "123e4567-e89b-12d3-a456-426614174000"},
257+
"spec": {"nodeName": "worker-a"}
258+
});
259+
260+
assert_eq!(
261+
pod_node_and_uid(&pod),
262+
Some(("worker-a", "123e4567-e89b-12d3-a456-426614174000"))
263+
);
264+
}
265+
266+
#[test]
267+
fn formats_pod_uid_for_systemd_cgroup() {
268+
assert_eq!(
269+
pod_uid_cgroup_form("123e4567-e89b-12d3-a456-426614174000"),
270+
"123e4567_e89b_12d3_a456_426614174000"
271+
);
272+
}
273+
}

0 commit comments

Comments
 (0)