Skip to content

ci: raise the Live / Kind per-test budget to 150 s - #13960

Merged
cv merged 1 commit into
v1from
test/live-kind-budget
Oct 29, 2026
Merged

cv merged 1 commit into
v1from
test/live-kind-budget

Conversation

@cv

@cv cv commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator

Failure

Live / Kind on linux_amd64 fails its 120 s per-test budget intermittently. The three-agent OpenShift test took 112.3 s before #13947, then 123.2 s and 121.7 s in two of four runs after it added a runtime check to each agent refresh. About 30 s of every Kubernetes test's destroy waits out the sandbox pods' termination grace period: their processes catch SIGTERM but don't exit (#13959, NVIDIA/OpenShell#5025).

Decision

Raise the Live / Kind per-test budget and the live-kind profile's slow-timeout together to 150 s, as .config/test-budgets.yaml requires, and record the measurements and when to lower it. The step's wall budget stays at 270 s; the slowest failing run's nextest step took 217 s.

The test can't get faster in NemoClaw: the destroy wait is OpenShell's (NVIDIA/OpenShell#5025, the Kubernetes form of the not-planned NVIDIA/OpenShell#4635), and #13959 tracks lowering the budget once a fixed OpenShell is pinned.

Validation

  • cargo nextest run -p nemoclaw-build: 204 passed, including the check that each step's slow-timeout matches its budget.
  • Live / Kind on linux_arm64 (DGX Spark), measured while investigating: the three-agent test took 81 s, of which the first apply took 30–36 s and destroy 34 s; every sandbox's agent and supervisor pods stayed running until the 30 s grace deadline.
  • Deferred to CI: Live / Kind on linux_amd64, where the budget was exceeded.

@cv

cv commented Oct 29, 2026

Copy link
Copy Markdown
Collaborator Author
Authoring session — opening
Field Status Value Evidence or limit
Harness recorded Pi 1.4.0 Session metadata and harness version output
Skills recorded nemoclaw-contributor-create-pr (complete read); test-driven-development (complete read); add-fabric-harness (complete read) Skill reads in the active branch to cutoff
Instruction files recorded AGENTS.md (injected); WRITING.md (complete read); .github/PULL_REQUEST_TEMPLATE.md (complete read); .github/SESSION_REPORT_TEMPLATE.md (complete read) Startup injection and reads to cutoff
Tool calls partial typescript: 418 whole session; 19 in this task Wrapper calls in the active branch, including failures; the wrapper's nested host calls are not recorded
Session window recorded 2026-10-28T17:24Z to 2026-10-29T05:51Z; task from 2026-10-29T05:40Z First event to cutoff, including idle time; the session also wrote #13946, #13947, and #13952
Compactions recorded 0 Complete record to cutoff; context elisions are not compactions
Models recorded azure/anthropic/claude-opus-5-5 Model change events; no changes
Reasoning levels recorded high Thinking-level events; no changes

@github-actions

Copy link
Copy Markdown
Contributor

@cv
cv enabled auto-merge (squash) October 29, 2026 06:00
@cv
cv merged commit 2d647a5 into v1 Oct 29, 2026
92 of 94 checks passed
@cv
cv deleted the test/live-kind-budget branch October 29, 2026 06:14
@wscurran wscurran added area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance platform: k8s Affects Kubernetes deployments v1 NemoClaw v1 branch labels Oct 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure chore Build, CI, dependency, or tooling maintenance platform: k8s Affects Kubernetes deployments v1 NemoClaw v1 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants