Repository navigation
Commit 7362e56
authored
test(e2e): stop the kind HCL case's cleanup from waiting out namespace deletion (#14177)
## Failure
On `linux_amd64`, the `kind-openshift-agents` HCL case
(`hcl_live::case`, which replaced the three-agent Rust test) took
110.7-132.6 s in the last nine `v1` Live / Kind runs, against a per-test
limit raised to 180 s while #14087 is investigated.
A local `cargo ci live-kind` on `linux_arm64` (20 CPUs) split one run of
the case like this:
| Run | Seconds | Where the time goes |
|---|---|---|
| cluster | 6.6 | Issuer key material and the fixture release, which
waits for the stub model and issuer |
| apply | 36.2 | Gateway chart about 14 s, sandbox pods about 9 s,
Fabric starting the agents about 10 s |
| reply | 3.4 | |
| unchanged_plan | 3.3 | |
| destroy | 39.0 | Stopping the agents about 6.5 s, then 30 s for the
sandbox pods' grace period (#13959, NVIDIA/OpenShell#5025) |
| (cleanup) | 12.3 | The fixture release's uninstall waiting for
Kubernetes to finalize its namespaces |
The cleanup is the only large phase that this repository's test code
controls.
The kind module's single Helm release held the three namespaces with
`wait = true`, and the Helm provider applies `wait` to uninstall too, so
`tofu test` waited for the namespaces to finish deleting from a cluster
that the runner deletes next.
## Decision
- Release the chart twice: the namespaces, the pull Secret, and the
issuer's CA without waiting, since they are ready once created; the stub
model and the issuer with `wait = true`, as before, so the first apply
still starts after both are ready.
- On cleanup, the waiting release now waits only for its own objects,
and the namespaces release deletes the three namespaces without waiting
for them to finish deleting.
- The namespaces release now holds the stub model's namespace as well.
Its record is in `default`, which outlives the namespaces it deletes.
Before, Helm created that namespace outside the release, and nothing
deleted it until the cluster was deleted.
- The case checks the same things as before.
- The rest of the time belongs to other components: the destroy's 30 s
grace period to OpenShell (#13959, NVIDIA/OpenShell#5025), agent start
and stop to Fabric and OpenClaw, and the gateway install to the
OpenShell chart and kind's volume provisioner.
The per-test limit stays at 180 s for now; it should go back to 150 s
once Live / Kind runs on `linux_amd64` show the margin.
Part of #14087. Still open: the destroy grace period (#13959, waiting on
NVIDIA/OpenShell#5025), and returning the Live / Kind limits to 150 s
and 270 s.
## Validation
Same machine (`linux_arm64`, 20 CPUs, no CPU limit), `cargo ci
live-kind`, `hcl_live::case`:
| | Run 1 | Run 2 | Cleanup |
|---|---|---|---|
| Before (`v1` at 259d08b) | 100.9 s | 98.4 s | 12.3 s, 12.3 s |
| After | 84.6 s | 83.1 s | 0.4 s, 0.4 s |
- `cargo ci live-kind`: passed twice with the change. A pod and
namespace monitor on the kind node showed all three namespaces
`Terminating` after cleanup.
- `cargo ci` (fmt, clippy, build, test, schema, bundle, lifecycle):
passed.
- `cargo run --locked -p nemoclaw-build -- fern check`: passed, 0 errors
(Node 25.9.0 rather than the pinned 24.22.1).
- `tofu fmt -check` on `kind/main.tf`: passed.
- CI Live / Kind on this PR, one run: `hcl_live::case` took 101.9 s on
`linux_amd64` and 101.1 s on `linux_arm64`, against 110.7-132.6 s and
113.3-119.7 s in the last nine `v1` runs. One run doesn't justify
lowering the limit to 150 s, so that stays open.1 parent 7126ef2 commit 7362e56
3 files changed
Lines changed: 49 additions & 14 deletions
File tree
- crates/nemoclaw-e2e/tests/e2e/hcl/kind
- model/templates
- docs/contributing
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
7 | 7 | | |
8 | 8 | | |
9 | 9 | | |
10 | | - | |
| 10 | + | |
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| |||
120 | 120 | | |
121 | 121 | | |
122 | 122 | | |
123 | | - | |
124 | | - | |
125 | | - | |
126 | | - | |
127 | | - | |
128 | | - | |
129 | | - | |
130 | | - | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
131 | 126 | | |
| 127 | + | |
132 | 128 | | |
133 | 129 | | |
134 | 130 | | |
| |||
144 | 140 | | |
145 | 141 | | |
146 | 142 | | |
147 | | - | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
148 | 170 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | 3 | | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
4 | 7 | | |
5 | 8 | | |
6 | 9 | | |
| |||
29 | 32 | | |
30 | 33 | | |
31 | 34 | | |
32 | | - | |
33 | | - | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
38 | 48 | | |
39 | 49 | | |
40 | 50 | | |
| |||
138 | 148 | | |
139 | 149 | | |
140 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
55 | | - | |
| 55 | + | |
56 | 56 | | |
57 | 57 | | |
58 | 58 | | |
| |||
0 commit comments