Repository navigation
Commit 2d647a5
authored
ci: raise the Live / Kind per-test budget to 150 s (#13960)
## Failure
Live / Kind on linux_amd64 fails its 120 s per-test budget
intermittently. The three-agent OpenShift test took 112.3 s before
#13947, then 123.2 s and 121.7 s in two of four runs after it added a
runtime check to each agent refresh. About 30 s of every Kubernetes
test's destroy waits out the sandbox pods' termination grace period:
their processes catch SIGTERM but don't exit (#13959,
NVIDIA/OpenShell#5025).
## Decision
Raise the Live / Kind per-test budget and the `live-kind` profile's
slow-timeout together to 150 s, as `.config/test-budgets.yaml` requires,
and record the measurements and when to lower it. The step's wall budget
stays at 270 s; the slowest failing run's nextest step took 217 s.
The test can't get faster in NemoClaw: the destroy wait is OpenShell's
(NVIDIA/OpenShell#5025, the Kubernetes form of the not-planned
NVIDIA/OpenShell#4635), and #13959 tracks lowering the budget once a
fixed OpenShell is pinned.
## Validation
- `cargo nextest run -p nemoclaw-build`: 204 passed, including the check
that each step's slow-timeout matches its budget.
- Live / Kind on linux_arm64 (DGX Spark), measured while investigating:
the three-agent test took 81 s, of which the first apply took 30–36 s
and destroy 34 s; every sandbox's agent and supervisor pods stayed
running until the 30 s grace deadline.
- Deferred to CI: Live / Kind on linux_amd64, where the budget was
exceeded.1 parent 32b92f1 commit 2d647a5
2 files changed
Lines changed: 6 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
88 | 88 | | |
89 | 89 | | |
90 | 90 | | |
91 | | - | |
| 91 | + | |
92 | 92 | | |
93 | 93 | | |
94 | 94 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
36 | 36 | | |
37 | 37 | | |
38 | 38 | | |
39 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
40 | 43 | | |
41 | 44 | | |
42 | | - | |
| 45 | + | |
43 | 46 | | |
44 | 47 | | |
45 | 48 | | |
| |||
0 commit comments