Parent: #1264 · Labels: enhancement, area/tests (auto), area/ci
(auto) · Size: M · Depends on: DC2 (for the cluster a real snapshot
needs); DC3 not required. For end-to-end TestGrid visibility: the TestGrid epic TG2/TG5 (the
publisher + ingestion job) — the CTRF/JUnit artifact is verifiable from this child
alone, but "appears as a TestGrid column" depends on the TestGrid epic shipping.
Goal. Bring the CLI surfaces no CUJ exercises under real-or-simulated coverage
that emits a consumable result: query, diff, evidence verify/publish,
real-cluster snapshot, and at least one additional bundle deployer.
Scope.
query — add a coverage step (KWOK or --no-cluster, hardware-independent)
asserting aicr query --selector <path> returns the expected hydrated value for a
launch recipe.
diff — add a step exercising aicr diff between two recipe/snapshot inputs.
evidence verify / publish — verify is already exercised in
phase_verify (tests/uat/aws/run:258-287); add explicit coverage of the
decoupled aicr evidence publish <dir> --push path (ADR-007 surface item 1's
post-V1 decoupled-publish addition, 007-recipe-evidence.md:198-207) so the
split validate-then-publish leg is tested, not only the one-shot --push.
- Real-cluster
snapshot — phase_prep runs snapshot --config against the
live cluster (tests/uat/aws/run:86, inside phase_prep :79-130); add an
assertion step that the captured snapshot.yaml carries the expected fingerprint
dimensions (service/accelerator/os) so snapshot fidelity is a checked result, not
an implicit precondition.
- Additional deployer — the UAT runner uses
deployer: helmfile
(h100-training-config.yaml:67); add at least one run (KWOK-level is fine) that
bundles with argocd or flux and asserts the bundle renders, broadening deployer
coverage beyond helmfile.
- Result shape — one build or many. These boundary checks emit CTRF/JUnit-shaped
results, but the TestGrid epic TG2 emits exactly one artifacts/junit.xml per build, and
everything under one build dir merges into one column with row-name collisions
feeding flaky detection (the wire reference's per-build merge semantics). Pick
one and state it in DC4 and TG2 together: either (a) these checks are emitted as
separate builds/columns (their own coordinate), or (b) they fold into the UAT
build as namespaced deployment/<check> rows that TG2's converter merges
deterministically without name collision (pulling TG2's deferred "merge multiple
CTRF sources" need forward). The default for hardware-independent KWOK-level checks
is separate columns; only the real-cluster snapshot fidelity check, which runs
inside an existing UAT phase, folds into that run's build as a namespaced row.
Out of scope. mirror, verify (bundle), trust, skill — these are not
CUJ-adjacent; leave them to unit coverage unless a pull-trigger fires (see
Deferred). New CLI features (implement exactly the existing surface).
Key files. pkg/cli/query.go, pkg/cli/diff.go, pkg/cli/evidence_publish.go,
pkg/cli/snapshot.go, tests/uat/{aws,gcp}/run (phase_prep :79-130,
phase_verify :258-287), KWOK harness (kwok-recipes.yaml, Makefile:681-694),
tests/uat/aws/tests/h100-training-config.yaml (deployer :67).
Tests. Each boundary has a passing assertion in KWOK or a UAT phase; the new
evidence publish path produces a bundle that evidence verify accepts; the
additional-deployer bundle renders; multi-source CTRF results either land as distinct
columns or as namespaced non-colliding rows per the chosen TG2 contract.
Acceptance. query, diff, evidence publish, real-cluster snapshot
fidelity, and a non-helmfile deployer each have an emitting test (CTRF/JUnit
artifact verifiable independent of the TestGrid epic); the build-vs-column shape for multi-source
results is agreed with TG2; the deliberately-uncovered surfaces are documented as such.
Child of #1264 · staged from docs/design/011-uat-dynamic-clusters.md (DC4).
Parent: #1264 · Labels:
enhancement,area/tests(auto),area/ci(auto) · Size: M · Depends on: DC2 (for the cluster a real
snapshotneeds); DC3 not required. For end-to-end TestGrid visibility: the TestGrid epic TG2/TG5 (the
publisher + ingestion job) — the CTRF/JUnit artifact is verifiable from this child
alone, but "appears as a TestGrid column" depends on the TestGrid epic shipping.
Goal. Bring the CLI surfaces no CUJ exercises under real-or-simulated coverage
that emits a consumable result:
query,diff,evidence verify/publish,real-cluster
snapshot, and at least one additional bundle deployer.Scope.
query— add a coverage step (KWOK or--no-cluster, hardware-independent)asserting
aicr query --selector <path>returns the expected hydrated value for alaunch recipe.
diff— add a step exercisingaicr diffbetween two recipe/snapshot inputs.evidence verify/publish—verifyis already exercised inphase_verify(tests/uat/aws/run:258-287); add explicit coverage of thedecoupled
aicr evidence publish <dir> --pushpath (ADR-007 surface item 1'spost-V1 decoupled-publish addition,
007-recipe-evidence.md:198-207) so thesplit validate-then-publish leg is tested, not only the one-shot
--push.snapshot—phase_preprunssnapshot --configagainst thelive cluster (
tests/uat/aws/run:86, insidephase_prep:79-130); add anassertion step that the captured
snapshot.yamlcarries the expected fingerprintdimensions (service/accelerator/os) so snapshot fidelity is a checked result, not
an implicit precondition.
deployer: helmfile(
h100-training-config.yaml:67); add at least one run (KWOK-level is fine) thatbundles with
argocdorfluxand asserts the bundle renders, broadening deployercoverage beyond helmfile.
results, but the TestGrid epic TG2 emits exactly one
artifacts/junit.xmlper build, andeverything under one build dir merges into one column with row-name collisions
feeding flaky detection (the wire reference's per-build merge semantics). Pick
one and state it in DC4 and TG2 together: either (a) these checks are emitted as
separate builds/columns (their own coordinate), or (b) they fold into the UAT
build as namespaced
deployment/<check>rows that TG2's converter mergesdeterministically without name collision (pulling TG2's deferred "merge multiple
CTRF sources" need forward). The default for hardware-independent KWOK-level checks
is separate columns; only the real-cluster
snapshotfidelity check, which runsinside an existing UAT phase, folds into that run's build as a namespaced row.
Out of scope.
mirror,verify(bundle),trust,skill— these are notCUJ-adjacent; leave them to unit coverage unless a pull-trigger fires (see
Deferred). New CLI features (implement exactly the existing surface).
Key files.
pkg/cli/query.go,pkg/cli/diff.go,pkg/cli/evidence_publish.go,pkg/cli/snapshot.go,tests/uat/{aws,gcp}/run(phase_prep:79-130,phase_verify:258-287), KWOK harness (kwok-recipes.yaml,Makefile:681-694),tests/uat/aws/tests/h100-training-config.yaml(deployer:67).Tests. Each boundary has a passing assertion in KWOK or a UAT phase; the new
evidence publishpath produces a bundle thatevidence verifyaccepts; theadditional-deployer bundle renders; multi-source CTRF results either land as distinct
columns or as namespaced non-colliding rows per the chosen TG2 contract.
Acceptance.
query,diff,evidence publish, real-clustersnapshotfidelity, and a non-helmfile deployer each have an emitting test (CTRF/JUnit
artifact verifiable independent of the TestGrid epic); the build-vs-column shape for multi-source
results is agreed with TG2; the deliberately-uncovered surfaces are documented as such.
Child of #1264 · staged from
docs/design/011-uat-dynamic-clusters.md(DC4).