Skip to content

Commit 2d647a5

Browse files
authored
ci: raise the Live / Kind per-test budget to 150 s (#13960)
## Failure Live / Kind on linux_amd64 fails its 120 s per-test budget intermittently. The three-agent OpenShift test took 112.3 s before #13947, then 123.2 s and 121.7 s in two of four runs after it added a runtime check to each agent refresh. About 30 s of every Kubernetes test's destroy waits out the sandbox pods' termination grace period: their processes catch SIGTERM but don't exit (#13959, NVIDIA/OpenShell#5025). ## Decision Raise the Live / Kind per-test budget and the `live-kind` profile's slow-timeout together to 150 s, as `.config/test-budgets.yaml` requires, and record the measurements and when to lower it. The step's wall budget stays at 270 s; the slowest failing run's nextest step took 217 s. The test can't get faster in NemoClaw: the destroy wait is OpenShell's (NVIDIA/OpenShell#5025, the Kubernetes form of the not-planned NVIDIA/OpenShell#4635), and #13959 tracks lowering the budget once a fixed OpenShell is pinned. ## Validation - `cargo nextest run -p nemoclaw-build`: 204 passed, including the check that each step's slow-timeout matches its budget. - Live / Kind on linux_arm64 (DGX Spark), measured while investigating: the three-agent test took 81 s, of which the first apply took 30–36 s and destroy 34 s; every sandbox's agent and supervisor pods stayed running until the 30 s grace deadline. - Deferred to CI: Live / Kind on linux_amd64, where the budget was exceeded.
1 parent 32b92f1 commit 2d647a5

2 files changed

Lines changed: 6 additions & 3 deletions

File tree

‎.config/nextest.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -88,7 +88,7 @@ default-filter = '''
8888
[profile.live-kind]
8989
inherits = "ci"
9090
test-threads = 4
91-
slow-timeout = { period = "120s", terminate-after = 2 }
91+
slow-timeout = { period = "150s", terminate-after = 2 }
9292
default-filter = '''
9393
package(nemoclaw-e2e) & binary(=integration) & (test(/^kubernetes_live::/) | test(/^kubernetes_chart::/))
9494
'''

‎.config/test-budgets.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -36,10 +36,13 @@ live-docker:
3636
test_seconds: 180
3737

3838
# Live / Kind: at most 199 s on linux_amd64 and 171 s on linux_arm64 in 11 v1
39-
# runs; slowest test 83 s.
39+
# runs. The slowest test, three agents on OpenShift, took 112 s on linux_amd64
40+
# and 123 s after refresh began checking agent runtimes (#13947); about 30 s of
41+
# each test's destroy waits out the sandbox pods' termination grace period
42+
# (#13959, NVIDIA/OpenShell#5025). Lower test_seconds when that is fixed.
4043
live-kind:
4144
wall_seconds: 270
42-
test_seconds: 120
45+
test_seconds: 150
4346

4447
# Each provider's contract tests, measured on linux_amd64 and linux_arm64 in
4548
# two runs after providers and the fake OpenShell gateway set TCP_NODELAY.

0 commit comments

Comments
 (0)