Skip to content

Commit 3a31220

Browse files
test(conformance): tighten credential gating checks from review
- The WebSocket recorder reads frames until the connection ends, and the test waits for that before checking that no binary frame arrived. - The HTTP recorder keeps the request head and body apart. The REST tests compare the body exactly and check the secret appears nowhere, and failure messages redact it. - testkit errors keep the gateway's gRPC status code, and the admission test checks for FailedPrecondition. - The policy create request destructures the SDK spec, so a new field fails to compile instead of being dropped. - host_gateway no longer retries its connection. - Docs state that conformance requires sandboxes to reach the test machine as host.openshell.internal. Signed-off-by: Matthew Grossman <[email protected]>
1 parent 61504a3 commit 3a31220

19 files changed

Lines changed: 232 additions & 123 deletions

File tree

‎TESTING.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -657,8 +657,8 @@ explains the isolation backend separation behind the capability-free runtime.
657657
On a **remote** cluster, drop the `e2e-host-gateway` feature. Those tests rely
658658
on the sandbox-side `host.openshell.internal` alias reaching the machine running
659659
the tests, which is unreachable from pods on a remote cluster, so they fail.
660-
The conformance suite's `credential_gating` tests rely on the alias too and
661-
fail on a remote cluster; the feature selection doesn't skip them.
660+
The conformance suite requires the alias to reach that machine too, for its
661+
`credential_gating` tests; the feature selection doesn't skip them.
662662
Left enabled, the `host_gateway_alias` suite fails because
663663
`host.openshell.internal` does not resolve inside the pod, so the gateway
664664
SSRF-denies the request (`DNS resolution failed` / `ssrf_denied`) — a networking

‎e2e/helm-plugins/openshell-external-compute-driver/post-renderer.sh‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,10 @@ cp "${plugin_dir}/kustomization.yaml" "${work_dir}/kustomization.yaml"
1616
cp "${plugin_dir}/workload-patch.yaml" "${work_dir}/workload-patch.yaml"
1717
tee "${work_dir}/rendered.yaml" >/dev/null
1818

19-
# server.hostGatewayIP renders the gateway pod's hostAliases entry.
20-
if grep -q 'host\.openshell\.internal' "${work_dir}/rendered.yaml" \
21-
&& grep -q '^ *hostAliases:' "${work_dir}/rendered.yaml"; then
19+
# server.hostGatewayIP renders the gateway pod's only hostAliases entry
20+
# (deploy/helm/openshell/templates/_gateway-workload.tpl), which the
21+
# host-gateway component copies from hostAliases.0.ip.
22+
if grep -q '^ *hostAliases:' "${work_dir}/rendered.yaml"; then
2223
cp -R "${plugin_dir}/host-gateway" "${work_dir}/host-gateway"
2324
printf 'components:\n - host-gateway\n' >>"${work_dir}/kustomization.yaml"
2425
fi

‎tests/suites/conformance/Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/conformance/README.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -23,10 +23,9 @@ tmachine environment on CI. The other suites are ports of the old
2323
tests, so a test environment needs it installed
2424
(`tests/tmachine/ansible/playbooks/test-tools.yaml` does that for the tmachine
2525
environments). `credential-gating`, ported from NVIDIA/OpenShell#5264, starts
26-
servers (`tests/support/host-server`) on the machine that runs the tests and reaches them from sandboxes
27-
created from `testkit::python_image()` through `host.openshell.internal`, so
28-
it fails against a gateway whose sandboxes can't reach that machine, such as
29-
a remote Kubernetes cluster.
26+
servers (`tests/support/host-server`) on the machine that runs the tests and
27+
reaches them from sandboxes created from `testkit::python_image()` through
28+
`host.openshell.internal`.
3029

3130
## Running it
3231

@@ -47,7 +46,9 @@ tests/suites/conformance/target/debug/openshell-conformance run --junit conforma
4746
```
4847

4948
The program runs only this suite: passing it means a gateway meets the
50-
conformance contract below. Feature and driver suites are not part of it. The
49+
conformance contract below. The machine that runs it must be reachable from
50+
sandboxes as `host.openshell.internal`, which rules out a gateway on a remote
51+
Kubernetes cluster. Feature and driver suites are not part of it. The
5152
[testkit README](../../testkit/README.md#test-programs) describes the commands
5253
and exit codes. `tasks/scripts/check-conformance-program.sh` checks that it
5354
lists the same tests as nextest.

‎tests/suites/conformance/src/credential_gating.rs‎

Lines changed: 54 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,17 @@
55
//! what the supervisor forwards upstream in REST bodies and WebSocket frames.
66
//!
77
//! Ported from the e2e tests added in NVIDIA/OpenShell#5264. The upstream
8-
//! servers run on the test host ([`host_server_support`]), so a gateway on a
9-
//! remote cluster can't run this suite.
8+
//! servers run on the test host ([`host_server_support`]), so the suite
9+
//! requires sandboxes to reach the test host as `host.openshell.internal`.
1010
1111
use std::time::Duration;
1212

13-
use host_server_support::{HttpRecorder, WebSocketRecorder};
13+
use host_server_support::{HttpRecorder, Request, WebSocketRecorder};
1414
use testkit::policy::{self, Access, Endpoint as _};
1515
use testkit::sdk::raw::proto;
16-
use testkit::{Context, Error, PYTHON_BINARIES, Poll, Result, ResultExt as _, Sandbox, YamlFile};
16+
use testkit::{
17+
Context, Error, GrpcCode, PYTHON_BINARIES, Poll, Result, ResultExt as _, Sandbox, YamlFile,
18+
};
1719

1820
const TEST_HOST: &str = "host.openshell.internal";
1921
const TOKEN_ENV: &str = "E2E_GATING_TOKEN";
@@ -71,15 +73,15 @@ fi"#;
7173
async fn profile_rest_endpoint_keeps_body_placeholder(cx: &mut Context) -> testkit::Result {
7274
let server = HttpRecorder::start("ok").await?;
7375
let websocket = WebSocketRecorder::start().await?;
74-
let provider = install_provider(cx, Some((server.port, websocket.port))).await?;
76+
let provider = install_gating_provider(cx, Some((server.port, websocket.port))).await?;
7577

7678
let mut policy = policy::base();
7779
allow(&mut policy, server.port)
7880
.rest(Access::Full)
7981
.request_body_credential_rewrite(false);
8082
let sandbox = create_sandbox(cx, "rest", policy, &provider).await?;
8183
run_python(&sandbox, BODY_CLIENT, server.port).await?;
82-
expect_one_body(&server, PLACEHOLDER_PREFIX, TEST_SECRET).await
84+
expect_placeholder_body(&server).await
8385
}
8486

8587
/// A provider's WebSocket endpoint completes the handshake but doesn't
@@ -89,7 +91,7 @@ async fn profile_rest_endpoint_keeps_body_placeholder(cx: &mut Context) -> testk
8991
async fn profile_websocket_endpoint_denies_binary_frames(cx: &mut Context) -> testkit::Result {
9092
let server = HttpRecorder::start("ok").await?;
9193
let websocket = WebSocketRecorder::start().await?;
92-
let provider = install_provider(cx, Some((server.port, websocket.port))).await?;
94+
let provider = install_gating_provider(cx, Some((server.port, websocket.port))).await?;
9395

9496
let mut policy = policy::base();
9597
allow(&mut policy, websocket.port).websocket(Access::ReadWrite);
@@ -99,8 +101,9 @@ async fn profile_websocket_endpoint_denies_binary_frames(cx: &mut Context) -> te
99101
.wait_for_handshake()
100102
.await
101103
.context("upstream should receive the WebSocket handshake")?;
104+
let binary_seen = websocket.wait_until_finished().await?;
102105
testkit::ensure!(
103-
!websocket.binary_seen(),
106+
!binary_seen,
104107
"credentialed WebSocket binary frame reached upstream"
105108
);
106109
Ok(())
@@ -112,10 +115,11 @@ async fn profile_websocket_endpoint_denies_binary_frames(cx: &mut Context) -> te
112115
#[testkit::test]
113116
async fn policy_binding_requires_inspection_or_opt_in(cx: &mut Context) -> testkit::Result {
114117
let server = HttpRecorder::start("ok").await?;
115-
let provider = install_provider(cx, None).await?;
118+
let provider = install_gating_provider(cx, None).await?;
116119

117120
let mut l4 = policy::base();
118121
allow(&mut l4, server.port).credential_binding(&provider);
122+
// The wording checks follow the #5264 tests.
119123
let error = expect_rejected(cx, "l4", l4, &provider).await?;
120124
testkit::ensure!(error.contains("credentialed endpoint"), "{error}");
121125
testkit::ensure!(error.contains("L4-only"), "{error}");
@@ -145,7 +149,7 @@ async fn policy_binding_rest_body_keeps_placeholder_without_rewrite(
145149
cx: &mut Context,
146150
) -> testkit::Result {
147151
let server = HttpRecorder::start("ok").await?;
148-
let provider = install_provider(cx, None).await?;
152+
let provider = install_gating_provider(cx, None).await?;
149153

150154
let mut policy = policy::base();
151155
allow(&mut policy, server.port)
@@ -154,7 +158,7 @@ async fn policy_binding_rest_body_keeps_placeholder_without_rewrite(
154158
.credential_binding(&provider);
155159
let sandbox = create_sandbox(cx, "body", policy, &provider).await?;
156160
run_python(&sandbox, BODY_CLIENT, server.port).await?;
157-
expect_one_body(&server, PLACEHOLDER_PREFIX, TEST_SECRET).await
161+
expect_placeholder_body(&server).await
158162
}
159163

160164
/// A policy binding with `request_body_credential_rewrite: true` replaces
@@ -164,7 +168,7 @@ async fn policy_binding_rest_body_rewrite_sends_secret_upstream(
164168
cx: &mut Context,
165169
) -> testkit::Result {
166170
let server = HttpRecorder::start("ok").await?;
167-
let provider = install_provider(cx, None).await?;
171+
let provider = install_gating_provider(cx, None).await?;
168172

169173
let mut policy = policy::base();
170174
allow(&mut policy, server.port)
@@ -173,15 +177,21 @@ async fn policy_binding_rest_body_rewrite_sends_secret_upstream(
173177
.credential_binding(&provider);
174178
let sandbox = create_sandbox(cx, "rw", policy, &provider).await?;
175179
run_python(&sandbox, BODY_CLIENT, server.port).await?;
176-
expect_one_body(&server, TEST_SECRET, PLACEHOLDER_PREFIX).await
180+
let request = one_request(&server).await?;
181+
testkit::ensure!(
182+
request.body == format!("prefix-{TEST_SECRET}-suffix"),
183+
"expected the secret in the body upstream; got {}",
184+
redact(&request)
185+
);
186+
Ok(())
177187
}
178188

179189
/// Binding and unbinding a provider with `policy set --wait` adds and
180190
/// removes its environment variable in a running sandbox.
181191
#[testkit::test]
182192
async fn policy_binding_updates_provider_environment_live(cx: &mut Context) -> testkit::Result {
183193
let server = HttpRecorder::start("ok").await?;
184-
let provider = install_provider(cx, None).await?;
194+
let provider = install_gating_provider(cx, None).await?;
185195

186196
let policy_binding = |bind: bool| {
187197
let mut policy = policy::base();
@@ -223,7 +233,7 @@ fn allow(policy: &mut proto::SandboxPolicy, port: u16) -> &mut proto::NetworkEnd
223233
/// Install a run-unique provider holding `TEST_SECRET` in `TOKEN_ENV`. With
224234
/// `endpoints`, its profile declares the REST and WebSocket ports as its
225235
/// endpoints; without, a policy must bind the provider explicitly.
226-
async fn install_provider(cx: &Context, endpoints: Option<(u16, u16)>) -> Result<String> {
236+
async fn install_gating_provider(cx: &Context, endpoints: Option<(u16, u16)>) -> Result<String> {
227237
let name = cx.name("cg");
228238
let endpoints = endpoints.map_or_else(String::new, |(rest, websocket)| {
229239
format!(
@@ -288,7 +298,8 @@ async fn create_sandbox(
288298
Ok(sandbox)
289299
}
290300

291-
/// Fail unless the gateway rejects the sandbox, and return the rejection.
301+
/// Fail unless the gateway rejects the sandbox as a failed precondition, and
302+
/// return the rejection.
292303
async fn expect_rejected(
293304
cx: &Context,
294305
suffix: &str,
@@ -300,7 +311,10 @@ async fn expect_rejected(
300311
"the gateway accepted sandbox {}",
301312
sandbox.name()
302313
))),
303-
Err(error) => Ok(error.to_string()),
314+
Err(error) if error.grpc_code() == Some(GrpcCode::FailedPrecondition) => {
315+
Ok(error.to_string())
316+
}
317+
Err(error) => Err(error.context("expected a failed precondition")),
304318
}
305319
}
306320

@@ -315,23 +329,39 @@ async fn run_python(sandbox: &Sandbox, script: &str, port: u16) -> Result {
315329
Ok(())
316330
}
317331

318-
/// Fail unless `server` saw exactly one request, containing `present` and
319-
/// not `absent`.
320-
async fn expect_one_body(server: &HttpRecorder, present: &str, absent: &str) -> Result {
321-
let requests = server.wait_for_requests(1).await?;
332+
/// The one request `server` saw.
333+
async fn one_request(server: &HttpRecorder) -> Result<Request> {
334+
let mut requests = server.wait_for_requests(1).await?;
322335
testkit::ensure!(
323336
requests.len() == 1,
324337
"expected one request upstream; got {}",
325338
requests.len()
326339
);
327-
let request = &requests[0];
340+
Ok(requests.remove(0))
341+
}
342+
343+
/// Fail unless `server` saw one request whose body has the credential
344+
/// placeholder in place of the token, and the secret appears nowhere in it.
345+
async fn expect_placeholder_body(server: &HttpRecorder) -> Result {
346+
let request = one_request(server).await?;
328347
testkit::ensure!(
329-
request.contains(present) && !request.contains(absent),
330-
"expected {present:?} and not {absent:?} upstream"
348+
request
349+
.body
350+
.starts_with(&format!("prefix-{PLACEHOLDER_PREFIX}"))
351+
&& request.body.ends_with("-suffix")
352+
&& !request.head.contains(TEST_SECRET)
353+
&& !request.body.contains(TEST_SECRET),
354+
"expected the placeholder in the body upstream and no secret; got {}",
355+
redact(&request)
331356
);
332357
Ok(())
333358
}
334359

360+
/// `request` for a failure message, with the secret replaced.
361+
fn redact(request: &Request) -> String {
362+
format!("{}\n\n{}", request.head, request.body).replace(TEST_SECRET, "<secret>")
363+
}
364+
335365
/// Wait until new exec sessions report `expected` from [`TOKEN_PRESENCE`].
336366
async fn wait_for_token(sandbox: &Sandbox, expected: &str) -> Result {
337367
testkit::poll_until(

‎tests/suites/drivers/podman/Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/features/Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎tests/suites/features/host-gateway/tests/host_gateway.rs‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -20,23 +20,13 @@ use testkit::{Context, PYTHON_BINARIES, Result, ResultExt as _, YamlFile};
2020
const PYTHON: &str = "python3";
2121

2222
/// Requests `/` from `host.openshell.internal` on the port in `argv[1]` and
23-
/// prints the response. A failed connection is retried for 20 seconds, so a
24-
/// slow start doesn't fail the test, and reported on stderr.
23+
/// prints the response. The server listens before the sandbox is created,
24+
/// and a Ready sandbox's network is up, so the client doesn't retry.
2525
const CLIENT: &str = r#"
26-
import socket, sys, time
26+
import socket, sys
2727
2828
host, port = "host.openshell.internal", int(sys.argv[1])
29-
deadline = time.monotonic() + 20
30-
while True:
31-
try:
32-
conn = socket.create_connection((host, port), timeout=10)
33-
break
34-
except OSError as error:
35-
print(f"connect {host}:{port}: {error}", file=sys.stderr, flush=True)
36-
if time.monotonic() > deadline:
37-
sys.exit(1)
38-
time.sleep(1)
39-
with conn:
29+
with socket.create_connection((host, port), timeout=10) as conn:
4030
conn.sendall(f"GET / HTTP/1.1\r\nHost: {host}:{port}\r\nConnection: close\r\n\r\n".encode())
4131
response = b""
4232
while chunk := conn.recv(4096):

‎tests/support/Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)