Skip to content

Commit 363f081

Browse files
fix(runner): wait for the Agent Sandbox controller after a k3s guest boots
The k3s installers check that the Agent Sandbox controller reconciles a gated probe Sandbox, but the runner then boots a fresh VM from the installed disk, which restarts k3s and the controller. The runner only waited for the gateway, so the first sandbox create on the k3s and k3s-ha rows could race the controller's leader election again. The old conformance playbook ran the probe after boot, before the tests. After the gateway is ready on a guest with /usr/local/bin/k3s, run the same probe (wait-k3s-controller.sh): create a scheduling-gated Sandbox, wait for the controller-owned gated Pod, and delete it. A failure is an environment error, not a test failure. Signed-off-by: Matthew Grossman <[email protected]>
1 parent ce953dd commit 363f081

3 files changed

Lines changed: 94 additions & 1 deletion

File tree

‎tests/runner/src/execute.rs‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,25 @@ async fn run_tests(session: &Session, run: &Run<'_>) -> Result<Status> {
257257
.await?,
258258
)?;
259259

260+
// A booted guest also restarts k3s and the Agent Sandbox controller, and
261+
// a gateway that is healthy doesn't show the controller is reconciling.
262+
if session
263+
.exec(&["test", "-x", K3S_BINARY])
264+
.await?
265+
.status
266+
.success()
267+
{
268+
checked(
269+
"wait for the Agent Sandbox controller",
270+
session
271+
.exec_timeout(
272+
&["sudo", "-n", "bash", "-c", WAIT_K3S_CONTROLLER],
273+
WAIT_K3S_CONTROLLER_TIMEOUT,
274+
)
275+
.await?,
276+
)?;
277+
}
278+
260279
let archive = format!("{guest}/tests.tar.zst");
261280
let extract = format!("{guest}/extract");
262281
let config = format!("{guest}/nextest.toml");
@@ -351,6 +370,12 @@ fn print_log_tail(log: &Path) {
351370
const COLLECT_K3S: &str = include_str!("collect-k3s.sh");
352371
/// Where the k3s installers put the k3s binary; its presence marks a k3s guest.
353372
const K3S_BINARY: &str = "/usr/local/bin/k3s";
373+
/// The check that the Agent Sandbox controller reconciles Sandboxes, run on a
374+
/// k3s guest after it boots.
375+
const WAIT_K3S_CONTROLLER: &str = include_str!("wait-k3s-controller.sh");
376+
/// The script retries for up to ten minutes; this catches one that stalls.
377+
const WAIT_K3S_CONTROLLER_TIMEOUT: Duration = Duration::from_mins(12);
378+
354379
/// Where the guest keeps the script's output until it is copied out.
355380
const COLLECT_K3S_OUTPUT: &str = "/var/tmp/openshell-k3s-diagnostics.txt";
356381
/// How long the guest lets the script run before stopping it. The script
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
#!/usr/bin/env bash
2+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
# SPDX-License-Identifier: Apache-2.0
4+
5+
# Waits until the Agent Sandbox controller reconciles Sandboxes on a k3s guest.
6+
# Deployment readiness does not establish that the controller is the leader or
7+
# reconciling, and a booted guest restarts it, so the first test would
8+
# otherwise race it. A Sandbox with a scheduling-gated Pod proves the
9+
# controller reconciles without running the Pod or pulling its image.
10+
# The same check as tests/tmachine/ansible/playbooks/k3s-controller-ready.yaml.
11+
set -u
12+
k3s=/usr/local/bin/k3s
13+
gate=openshell.ai/conformance-startup
14+
name="conformance-startup-$RANDOM$RANDOM"
15+
retries=60
16+
delay=5
17+
18+
cleanup() {
19+
timeout 30s "$k3s" kubectl --request-timeout=5s --namespace=openshell \
20+
delete sandboxes.agents.x-k8s.io "$name" --ignore-not-found=true \
21+
--cascade=foreground --wait=true --timeout=20s >/dev/null 2>&1
22+
}
23+
trap cleanup EXIT
24+
25+
uid=
26+
for _ in $(seq "$retries"); do
27+
uid=$(timeout 15s "$k3s" kubectl --request-timeout=10s apply --filename=- \
28+
--output=jsonpath='{.metadata.uid}' <<EOF
29+
apiVersion: agents.x-k8s.io/v1beta1
30+
kind: Sandbox
31+
metadata:
32+
name: "$name"
33+
namespace: openshell
34+
spec:
35+
podTemplate:
36+
spec:
37+
automountServiceAccountToken: false
38+
schedulingGates:
39+
- name: $gate
40+
containers:
41+
- name: probe
42+
image: registry.k8s.io/pause:3.9
43+
EOF
44+
) && [ -n "$uid" ] && break
45+
uid=
46+
sleep "$delay"
47+
done
48+
if [ -z "$uid" ]; then
49+
echo "could not create the controller probe Sandbox $name" >&2
50+
exit 1
51+
fi
52+
53+
# The Sandbox (by uid) controls the Pod the controller creates, and the gate
54+
# keeps it unscheduled.
55+
want="$uid|$gate|"
56+
got=
57+
for _ in $(seq "$retries"); do
58+
got=$(timeout 10s "$k3s" kubectl --request-timeout=5s --namespace=openshell \
59+
get pod "$name" --output=jsonpath='{.metadata.ownerReferences[?(@.controller==true)].uid}|{.spec.schedulingGates[*].name}|{.spec.nodeName}' 2>/dev/null)
60+
if [ "$got" = "$want" ]; then
61+
exit 0
62+
fi
63+
sleep "$delay"
64+
done
65+
echo "the Agent Sandbox controller did not reconcile the probe Sandbox $name; last observation: ${got:-none}" >&2
66+
exit 1

‎tests/tmachine/ansible/playbooks/openshell-k3s.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -168,7 +168,9 @@
168168
fail_msg: "Unexpected gateway Service address: {{ k3s_gateway_address.stdout }}"
169169

170170
# Deployment readiness doesn't establish that the controller reconciles
171-
# Sandboxes, so the first test would otherwise race it. Tests don't wait.
171+
# Sandboxes. This checks the installed cluster; a booted guest restarts the
172+
# controller, so the test runner checks again after it boots
173+
# (tests/runner/src/wait-k3s-controller.sh). Tests don't wait.
172174
- name: Verify Agent Sandbox reconciliation
173175
ansible.builtin.include_tasks: k3s-controller-ready.yaml
174176

0 commit comments

Comments
 (0)