Skip to content

DC6 — Reliability + gap-fills #1279

Description

@njhensley

Parent: #1264 · Labels: enhancement, area/ci (auto), area/tests
(auto), area/docs (auto) · Size: S–M (three mechanical fixes) · Depends on:
DC1 (no-hard-fail and superseded-run surfacing fold into the broker); the rest are
independent

Goal. Close the long-tail reliability and orphaned-asset gaps so the lane is
operable and nothing dead lingers. DC6 is scoped to the three mechanical fixes;
the two items that carry independent design debate are peeled out (Azure → DC7;
merge-gate smoke → Deferred proposal).

Scope.

  • No hard-fail on a busy reservation — confirm DC1/DC2's queue-aware acquisition
    fully retires the AWS ::error::Insufficient capacity; exit 1 race
    (uat-aws.yaml:144-147); add a regression guard. (GCP has no such race today —
    this item is AWS-specific; DC2 covers the GCP capacity posture.) Couple this with
    DC1's superseded-run surfacing so a contended-and-dropped run is visible, not silent.
  • Orphaned Kind snapshot testtests/chainsaw/snapshot/deploy-agent/ has no
    chainsaw/Make/CI invocation (only deploy-agent-talos is wired, Makefile:741);
    the equivalent behavior is exercised by test_snapshot() in tests/e2e/run.sh,
    which does not run the chainsaw assets. Either wire the chainsaw test into the
    Makefile/CI alongside the Talos sibling or remove it; either way, correct the
    stale tests/chainsaw/README.md cross-reference
    at :48/:75/:96 (which
    falsely claims test_snapshot() exercises this dir). Document the decision.
  • Opt-in per-PR GPU run/ok-to-test exists (.github/workflows/ok-to-test.yaml,
    fanning out to qualification.yaml at :85-99) and the nvkind GPU lane is
    separately reachable on PRs via the pull-request/[0-9]+ push mirror that
    maintainer approval creates (gpu-h100-training-test.yaml:17-23, comment at
    :101-103). Extend one of these mechanisms so a maintainer can opt a single PR
    into a reservation-lane UAT run (through the DC1 broker, so it queues) — without
    making it a required gate; do not assume the GPU lane is already on /ok-to-test.

Out of scope. Cost governance (deferred). Promoting any GPU check to required.
Azure revive-or-retire (DC7). The merge-gate cluster-smoke proposal (Deferred).

Key files. .github/workflows/uat-aws.yaml (capacity :122-147),
tests/chainsaw/snapshot/deploy-agent/, tests/e2e/run.sh (test_snapshot()),
Makefile (:741), .github/workflows/ok-to-test.yaml (:85-99),
.github/workflows/gpu-h100-training-test.yaml (:17-23, :101-103),
docs/contributor/uat.md, tests/chainsaw/README.md (:48/:75/:96).

Tests. A contention scenario no longer errors and a superseded run is reported
(DC1 regression guard); the Kind test is wired-and-green or removed-and-referenced-nowhere
with the stale README pointer corrected; /ok-to-test-or-push-mirror opt-in reaches
the reservation lane in a manual trial.

Acceptance. No AWS hard-fail on contention and dropped runs are surfaced;
orphaned Kind test resolved and its stale README cross-reference corrected; opt-in
per-PR GPU available through the broker.


Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC6).

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions