Skip to content

Commit 7362e56

Browse files
authored
test(e2e): stop the kind HCL case's cleanup from waiting out namespace deletion (#14177)
## Failure On `linux_amd64`, the `kind-openshift-agents` HCL case (`hcl_live::case`, which replaced the three-agent Rust test) took 110.7-132.6 s in the last nine `v1` Live / Kind runs, against a per-test limit raised to 180 s while #14087 is investigated. A local `cargo ci live-kind` on `linux_arm64` (20 CPUs) split one run of the case like this: | Run | Seconds | Where the time goes | |---|---|---| | cluster | 6.6 | Issuer key material and the fixture release, which waits for the stub model and issuer | | apply | 36.2 | Gateway chart about 14 s, sandbox pods about 9 s, Fabric starting the agents about 10 s | | reply | 3.4 | | | unchanged_plan | 3.3 | | | destroy | 39.0 | Stopping the agents about 6.5 s, then 30 s for the sandbox pods' grace period (#13959, NVIDIA/OpenShell#5025) | | (cleanup) | 12.3 | The fixture release's uninstall waiting for Kubernetes to finalize its namespaces | The cleanup is the only large phase that this repository's test code controls. The kind module's single Helm release held the three namespaces with `wait = true`, and the Helm provider applies `wait` to uninstall too, so `tofu test` waited for the namespaces to finish deleting from a cluster that the runner deletes next. ## Decision - Release the chart twice: the namespaces, the pull Secret, and the issuer's CA without waiting, since they are ready once created; the stub model and the issuer with `wait = true`, as before, so the first apply still starts after both are ready. - On cleanup, the waiting release now waits only for its own objects, and the namespaces release deletes the three namespaces without waiting for them to finish deleting. - The namespaces release now holds the stub model's namespace as well. Its record is in `default`, which outlives the namespaces it deletes. Before, Helm created that namespace outside the release, and nothing deleted it until the cluster was deleted. - The case checks the same things as before. - The rest of the time belongs to other components: the destroy's 30 s grace period to OpenShell (#13959, NVIDIA/OpenShell#5025), agent start and stop to Fabric and OpenClaw, and the gateway install to the OpenShell chart and kind's volume provisioner. The per-test limit stays at 180 s for now; it should go back to 150 s once Live / Kind runs on `linux_amd64` show the margin. Part of #14087. Still open: the destroy grace period (#13959, waiting on NVIDIA/OpenShell#5025), and returning the Live / Kind limits to 150 s and 270 s. ## Validation Same machine (`linux_arm64`, 20 CPUs, no CPU limit), `cargo ci live-kind`, `hcl_live::case`: | | Run 1 | Run 2 | Cleanup | |---|---|---|---| | Before (`v1` at 259d08b) | 100.9 s | 98.4 s | 12.3 s, 12.3 s | | After | 84.6 s | 83.1 s | 0.4 s, 0.4 s | - `cargo ci live-kind`: passed twice with the change. A pod and namespace monitor on the kind node showed all three namespaces `Terminating` after cleanup. - `cargo ci` (fmt, clippy, build, test, schema, bundle, lifecycle): passed. - `cargo run --locked -p nemoclaw-build -- fern check`: passed, 0 errors (Node 25.9.0 rather than the pinned 24.22.1). - `tofu fmt -check` on `kind/main.tf`: passed. - CI Live / Kind on this PR, one run: `hcl_live::case` took 101.9 s on `linux_amd64` and 101.1 s on `linux_arm64`, against 110.7-132.6 s and 113.3-119.7 s in the last nine `v1` runs. One run doesn't justify lowering the limit to 150 s, so that stays open.
1 parent 7126ef2 commit 7362e56

3 files changed

Lines changed: 49 additions & 14 deletions

File tree

‎crates/nemoclaw-e2e/tests/e2e/hcl/kind/main.tf‎

Lines changed: 32 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@
77
# its UID range, holding the registry pull Secret and the issuer's CA; a stub
88
# model in its own namespace that answers every chat completion with
99
# `reply`; and an existing OIDC issuer in its own namespace. The test runner
10-
# removes the release, and with it the three namespaces, after the case.
10+
# removes both releases, and with them the three namespaces, after the case.
1111

1212
terraform {
1313
required_version = "= 1.12.9"
@@ -120,15 +120,11 @@ provider "helm" {
120120
}
121121
}
122122

123-
resource "helm_release" "model" {
124-
name = "model"
125-
chart = "${path.module}/model"
126-
namespace = local.model
127-
create_namespace = true
128-
wait = true
129-
timeout = 180
130-
values = [yamlencode({
123+
locals {
124+
# Both releases render the one chart; `part` selects their objects.
125+
values = {
131126
deploymentNamespace = local.kubernetes.namespace
127+
modelNamespace = local.model
132128
# kind does not enforce OpenShift's constraints; the SDK and OpenShell
133129
# read this range as they do on OpenShift.
134130
uidRange = "1000680000/10000"
@@ -144,5 +140,31 @@ resource "helm_release" "model" {
144140
caConfigMap = local.kubernetes.authentication.caConfigMap
145141
ca = local.issuer_ca
146142
}
147-
})]
143+
}
144+
}
145+
146+
# The three namespaces, the pull Secret, and the issuer's CA, which are ready
147+
# once created. Removing the release deletes the namespaces without waiting
148+
# for Kubernetes to finalize them, about 10 s of the case, because nothing
149+
# after the case needs them gone and the runner deletes the cluster. Its
150+
# record is in `default`, which outlives the namespaces it deletes.
151+
resource "helm_release" "namespaces" {
152+
name = "${local.kubernetes.namespace}-namespaces"
153+
chart = "${path.module}/model"
154+
namespace = "default"
155+
wait = false
156+
timeout = 180
157+
values = [yamlencode(merge(local.values, { part = "namespaces" }))]
158+
}
159+
160+
# The stub model and the issuer. The release waits until both are ready,
161+
# before the case's first apply.
162+
resource "helm_release" "model" {
163+
depends_on = [helm_release.namespaces]
164+
name = "model"
165+
chart = "${path.module}/model"
166+
namespace = local.model
167+
wait = true
168+
timeout = 180
169+
values = [yamlencode(merge(local.values, { part = "workloads" }))]
148170
}

‎crates/nemoclaw-e2e/tests/e2e/hcl/kind/model/templates/objects.yaml‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,9 @@
11
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
22
# SPDX-License-Identifier: Apache-2.0
33

4+
# Two releases render this chart. The `namespaces` part holds what is ready
5+
# once created; the `workloads` part, what its release waits for.
6+
{{- if eq .Values.part "namespaces" }}
47
# The deployment's namespace, annotated as OpenShift annotates a project.
58
apiVersion: v1
69
kind: Namespace
@@ -29,12 +32,19 @@ metadata:
2932
data:
3033
ca.crt: {{ .Values.issuer.ca | quote }}
3134
---
32-
# The existing issuer: the SDK's development issuer objects, in their own
33-
# namespace as an identity provider's would be.
35+
# The existing issuer's namespace, as an identity provider's would be.
3436
apiVersion: v1
3537
kind: Namespace
3638
metadata:
3739
name: {{ .Values.issuer.namespace }}
40+
---
41+
# The stub model's namespace.
42+
apiVersion: v1
43+
kind: Namespace
44+
metadata:
45+
name: {{ .Values.modelNamespace }}
46+
{{- else if eq .Values.part "workloads" }}
47+
# The existing issuer: the SDK's development issuer objects.
3848
{{- range .Values.issuer.objects }}
3949
---
4050
{{ toYaml . }}
@@ -138,3 +148,6 @@ spec:
138148
- name: scratch
139149
emptyDir:
140150
sizeLimit: 16Mi
151+
{{- else }}
152+
{{- fail "part must be namespaces or workloads" }}
153+
{{- end }}

‎docs/contributing/live-tests.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,7 +42,7 @@ The `kind-openshift-agents` [HCL E2E scenario](#hcl-e2e-scenarios) deploys two a
4242
Before the scenario, `cargo ci live-kind` starts a pinned registry that requires a login on the run's kind network and pushes the agent image there under a new image-index digest.
4343
The node may pull from that registry over HTTP, and its configuration for it holds no credentials, so a pod can pull the image only with a pull Secret.
4444
The new digest makes the node fetch the index from the registry, although the node already has the image's layers.
45-
Before its first apply, the `tests/e2e/hcl/kind` module releases a chart with the bundled Helm provider, as a platform and an operator would prepare the cluster:
45+
Before its first apply, the `tests/e2e/hcl/kind` module releases a chart twice with the bundled Helm provider, as a platform and an operator would prepare the cluster: once for the namespaces, the pull Secret, and the CA, and once for the stub model and the issuer, whose release waits until both are ready:
4646

4747
- The deployment's namespace with OpenShift UID-range annotations, as OpenShift prepares a project. It holds the registry's image pull Secret and the CA of the existing issuer.
4848
- A stub model server in its own namespace: the pinned `nginx-unprivileged` image that the development issuer runs, behind a Service named `model` on port 8000. It answers every `POST /v1/chat/completions` with the same streamed chat completion. The inference profile names that Service by its cluster DNS name, without a key, as [Use a Model Server in the Cluster](../kubernetes.md#use-a-model-server-in-the-cluster) describes.
@@ -52,7 +52,7 @@ Before its first apply, the `tests/e2e/hcl/kind` module releases a chart with th
5252
The token stays in a private credentials file in the run's `work` directory, which the `cli` and `verifier` modules export as the case's `token.env` variable.
5353
The gateway uses the `oidc` authentication profile, so the apply and the replies show that it accepts the issuer's tokens and that each sandbox pulled its image with the pull Secret.
5454
Each agent must answer from its recorded runtime with the stub's reply, a plan must report no changes, and destroy must keep only the gateway's storage.
55-
The test runner then uninstalls the release, which deletes the three namespaces.
55+
The test runner then uninstalls both releases, which deletes the three namespaces; it does not wait for Kubernetes to finish deleting them.
5656
`cargo ci live-kind` binds `cases/kind-openshift-agents.yaml` to the cluster's context, a fresh UID, gateway port, and namespace, and the image in the registry.
5757
It passes the kubeconfig as `TF_VAR_kubeconfig` and the registry's fixture login as `TF_VAR_pull_credentials`.
5858
The scenario does not check pod UIDs; kind does not enforce OpenShift security context constraints, and OpenShell applies the range.

0 commit comments

Comments
 (0)