Skip to content

DC4 — CLI-boundary coverage #1277

Description

@njhensley

Parent: #1264 · Labels: enhancement, area/tests (auto), area/ci
(auto) · Size: M · Depends on: DC2 (for the cluster a real snapshot
needs); DC3 not required. For end-to-end TestGrid visibility: the TestGrid epic TG2/TG5 (the
publisher + ingestion job) — the CTRF/JUnit artifact is verifiable from this child
alone, but "appears as a TestGrid column" depends on the TestGrid epic shipping.

Goal. Bring the CLI surfaces no CUJ exercises under real-or-simulated coverage
that emits a consumable result: query, diff, evidence verify/publish,
real-cluster snapshot, and at least one additional bundle deployer.

Scope.

  • query — add a coverage step (KWOK or --no-cluster, hardware-independent)
    asserting aicr query --selector <path> returns the expected hydrated value for a
    launch recipe.
  • diff — add a step exercising aicr diff between two recipe/snapshot inputs.
  • evidence verify / publishverify is already exercised in
    phase_verify (tests/uat/aws/run:258-287); add explicit coverage of the
    decoupled aicr evidence publish <dir> --push path (ADR-007 surface item 1's
    post-V1 decoupled-publish addition, 007-recipe-evidence.md:198-207) so the
    split validate-then-publish leg is tested, not only the one-shot --push.
  • Real-cluster snapshotphase_prep runs snapshot --config against the
    live cluster (tests/uat/aws/run:86, inside phase_prep :79-130); add an
    assertion step that the captured snapshot.yaml carries the expected fingerprint
    dimensions (service/accelerator/os) so snapshot fidelity is a checked result, not
    an implicit precondition.
  • Additional deployer — the UAT runner uses deployer: helmfile
    (h100-training-config.yaml:67); add at least one run (KWOK-level is fine) that
    bundles with argocd or flux and asserts the bundle renders, broadening deployer
    coverage beyond helmfile.
  • Result shape — one build or many. These boundary checks emit CTRF/JUnit-shaped
    results, but the TestGrid epic TG2 emits exactly one artifacts/junit.xml per build, and
    everything under one build dir merges into one column with row-name collisions
    feeding flaky detection (the wire reference's per-build merge semantics). Pick
    one and state it in DC4 and TG2 together
    : either (a) these checks are emitted as
    separate builds/columns (their own coordinate), or (b) they fold into the UAT
    build as namespaced deployment/<check> rows that TG2's converter merges
    deterministically without name collision (pulling TG2's deferred "merge multiple
    CTRF sources" need forward). The default for hardware-independent KWOK-level checks
    is separate columns; only the real-cluster snapshot fidelity check, which runs
    inside an existing UAT phase, folds into that run's build as a namespaced row.

Out of scope. mirror, verify (bundle), trust, skill — these are not
CUJ-adjacent; leave them to unit coverage unless a pull-trigger fires (see
Deferred). New CLI features (implement exactly the existing surface).

Key files. pkg/cli/query.go, pkg/cli/diff.go, pkg/cli/evidence_publish.go,
pkg/cli/snapshot.go, tests/uat/{aws,gcp}/run (phase_prep :79-130,
phase_verify :258-287), KWOK harness (kwok-recipes.yaml, Makefile:681-694),
tests/uat/aws/tests/h100-training-config.yaml (deployer :67).

Tests. Each boundary has a passing assertion in KWOK or a UAT phase; the new
evidence publish path produces a bundle that evidence verify accepts; the
additional-deployer bundle renders; multi-source CTRF results either land as distinct
columns or as namespaced non-colliding rows per the chosen TG2 contract.

Acceptance. query, diff, evidence publish, real-cluster snapshot
fidelity, and a non-helmfile deployer each have an emitting test (CTRF/JUnit
artifact verifiable independent of the TestGrid epic); the build-vs-column shape for multi-source
results is agreed with TG2; the deliberately-uncovered surfaces are documented as such.


Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC4).

Metadata

Metadata

Assignees

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions