Skip to content

feat(ci): add UAT reservation registry and broker helper (#1274)#1559

Merged
njhensley merged 2 commits into
NVIDIA:mainfrom
njhensley:ci/uat-reservation-broker
Jun 30, 2026
Merged

feat(ci): add UAT reservation registry and broker helper (#1274)#1559
njhensley merged 2 commits into
NVIDIA:mainfrom
njhensley:ci/uat-reservation-broker

Conversation

@njhensley

Copy link
Copy Markdown
Member

Summary

Add the UAT reservation registry (infra/uat/reservations.yaml) and a uat-broker helper that resolves reservation rows and expands the nightly version-matrix schedule. Data + tooling only — no live workflow is changed.

Motivation / Context

First of four PRs implementing DC1 — day/night scheduler + reservation broker (#1274), part of the comprehensive UAT epic (#1264). It establishes the data-driven foundation the rest of the broker builds on: the reservation registry is the lease's source of truth (each row's name becomes the GitHub Actions concurrency-group key uat-<name>), and the helper provides the two operations the later PRs consume — resolving a row for the dispatch surface (PR-2) and expanding the time-boxed version matrix for the nightly controller (PR-3). Onboarding new reserved capacity is a one-row data edit, no code change.

Fixes: N/A
Related: #1274, #1264

Type of Change

  • New feature (non-breaking change that adds functionality)
  • Build/CI/tooling

Component(s) Affected

  • Other: infra/uat/ reservation registry; new pkg/uatbroker + tools/uat-broker; .github/labeler.yml + CODEOWNERS

Implementation Notes

  • Go helper, mirroring the pkg/corroborate + tools/corroborate precedent: all logic lives in pkg/uatbroker (ParseRegistry/LoadRegistryFile/Validate/Lookup/Names, ExpandSchedule), with a thin tools/uat-broker CLI exposing reservations and schedule subcommands.
  • Schedule ordering is the testable core: main-first, then the previous N stable releases in descending semver order (Masterminds/semver/v3, not shell sort), with pre-release and non-semver tags dropped and the oldest dropped first when truncating — so the nightly controller sheds the oldest releases when its time-box closes.
  • reservations emits GITHUB_OUTPUT-style key=value lines so PR-2's resolve job is simply uat-broker reservations --name … >> "$GITHUB_OUTPUT"; schedule emits JSON from a stdin tag list.
  • Coded exits (0 ok / 2 invalid-request / 3 not-found) so workflows can branch on failure mode.
  • reservations.yaml is intentionally not CODEOWNERS-gated (kept under the default owner) so "new infra extends by data edit" stays low-friction; only the Go helper is maintainer-owned. gpu-count is forward-looking metadata (consumed by DC2) and is validated positive.

Testing

unset GITLAB_TOKEN && make qualify
  • make qualify green end-to-end: tests pass, aggregate coverage 77.5% ≥ 75%, lint clean (golangci-lint v2.12.2), e2e 23/0, grype reports no vulnerabilities.
  • New-package coverage: pkg/uatbroker 95.9%, tools/uat-broker 87.3% — table-driven tests cover registry validation, lookup, schedule ordering/filtering/drop-oldest, the CLI subcommands and their exit codes, and a guard that the committed registry stays valid.

Risk Assessment

  • Low — Isolated change, well-tested, easy to revert

Rollout notes: No runtime/behavior change — the helper is not wired into any workflow until PR-2 (the broker cutover). Safe to merge independently.

Checklist

  • Tests pass locally (make test with -race)
  • Linter passes (make lint)
  • I did not skip/disable tests to make CI green
  • I added/updated tests for new functionality
  • I updated docs if user-facing behavior changed (N/A — no user-facing behavior; the broker docs page lands with PR-2)
  • Changes follow existing patterns in the codebase
  • Commits are cryptographically signed (git commit -S)

@njhensley njhensley requested review from a team as code owners June 30, 2026 19:08
@njhensley njhensley added the theme/ci-dx CI pipelines, developer experience, and build tooling label Jun 30, 2026
@coderabbitai

coderabbitai Bot commented Jun 30, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds a new uatbroker Go package and a uat-broker CLI tool for UAT reservation registry parsing and nightly schedule expansion. It adds committed reservation registry data, package documentation, model types, registry validation and lookup, semver-based schedule expansion, CLI subcommands, README documentation, CODEOWNERS and labeler updates, and tests for the package and CLI.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 4
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the main change: adding the UAT reservation registry and broker helper.
Description check ✅ Passed The description is directly related and accurately describes the registry, helper, tooling, and scope of the PR.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@njhensley

Copy link
Copy Markdown
Member Author

@coderabbitai full review

@mchmarny mchmarny left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid DC1 foundation — clean, well-tested, and it mirrors the pkg/corroborate + tools/corroborate precedent nicely (coded exits, size-bounded reads, registry self-validation, ~96%/87% coverage). CI is fully green.

One medium note on the CLI side: signal.NotifyContext is set up but the context is never consumed, which actually makes the tool less interruptible on a blocking stdin read than no signal handling at all. Plus two questions on the committed infra data (personal-repo dependency and public exposure of prod GCP/reservation IDs) — worth confirming those are intentional. None of it blocks merge.

Comment thread tools/uat-broker/main.go
func realMain() int {
// Cancel on Ctrl-C / SIGTERM (CI cancellation) rather than imposing an
// arbitrary deadline: the tool only does local file + stdin I/O.
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

signal.NotifyContext is installed but ctx is never consumed — runReservations doesn't take it and runSchedule discards it (_ context.Context). NotifyContext suppresses the default SIGINT/SIGTERM termination, but since nothing checks ctx.Done(), a signal arriving while readTags blocks on an open stdin (interactive invocation, or CI cancelling mid-read) is swallowed and the process hangs until SIGKILL — strictly worse than no signal handling.

Either wire ctx into the stdin read (read in a goroutine, select on ctx.Done()), or drop the signal handling entirely since the tool only does fast local file + stdin I/O and the default termination behavior is fine.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed in the latest push. runSchedule now threads ctx, and readTags runs the stdin read in a goroutine and selects on ctx.Done(), so a SIGINT/SIGTERM (interactive run, or CI cancelling mid-read) interrupts it instead of hanging until SIGKILL. Added TestReadTagsCanceledContext. runReservations stays ctx-free on purpose — it only does fast, size-bounded file I/O with no blocking call to interrupt.

Comment thread infra/uat/reservations.yaml
Comment thread infra/uat/reservations.yaml
Comment thread pkg/uatbroker/schedule.go
previousN = 0
}
stable := sortedStableDescending(rawTags)
if previousN < len(stable) {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: duplicate names in the reservations slice silently dedupe via the output map, so --reservations aws-h100,aws-h100 produces one key with no error. Harmless today; optional to reject duplicates for symmetry with the registry's uniqueness check.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done — resolveReservations now rejects duplicate --reservations names with ErrCodeInvalidRequest instead of silently collapsing via the output map, mirroring the registry's own name-uniqueness check. Added a test.

@njhensley njhensley force-pushed the ci/uat-reservation-broker branch from cd0f98f to 4820195 Compare June 30, 2026 20:18
@njhensley

Copy link
Copy Markdown
Member Author

Thanks for the review @mchmarny 🙏 — pushed fixes addressing all four (replied inline on each thread):

  • [medium] signal/ctxreadTags now does the stdin read in a goroutine and selects on ctx.Done(), so a signal interrupts a blocking read instead of hanging to SIGKILL (+ TestReadTagsCanceledContext).
  • dup --reservations — now rejected with ErrCodeInvalidRequest instead of silently deduping (+ test).
  • public exposure of GCP/reservation IDs — added a NOTE to reservations.yaml clarifying these are identifiers, not credentials (access is IAM/ACL-gated), so they're safe to commit and don't need re-litigating.
  • personal-repo dependency — pre-existing (cluster-config + actuator already use it); out of scope for DC1, no migration planned now.

CI was green on the prior push; tools/uat-broker coverage ticked up to 89.6%.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/uat-broker/main.go`:
- Around line 128-130: After FlagSet.Parse in both subcommand handlers, check
fs.Args() and return ExitInvalidInput when any unexpected positional arguments
remain. Update the reservations and schedule command handlers in main.go so they
reject extra trailing args instead of silently succeeding, using the existing
flagParseErr/invalid-input path and the handler functions that call
fs.Parse(args).
- Around line 137-140: The stdout writes in the `reservations` flow are ignoring
`Fprintln`/`Fprintf` errors, so failures like broken pipes or an unwritable
`$GITHUB_OUTPUT` can still return success. Update the `reservations` handling in
`main` to check and propagate every write error from `stdout` instead of
discarding it, and return a non-zero failure when any output write fails. Keep
the fix localized around the `list` branch and the later output-emission path so
the tool fails fast with the underlying write error.
- Around line 193-200: The warning in uat-broker’s schedule expansion path is
too broad: when ExpandSchedule produces empty cell lists because
--include-main=false and there are no stable tags, it should not be reported as
“main-only.” Update the warning logic around scheduleHasNoReleases in main to
distinguish an empty schedule from a degraded main-only schedule, and only emit
the main-only message when the schedule actually contains main rows; otherwise
use a precise empty/none-available warning.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: a58e6f1a-8c2a-4c1a-b179-5d112d7210fc

📥 Commits

Reviewing files that changed from the base of the PR and between 09cb46e and 4820195.

📒 Files selected for processing (12)
  • .github/CODEOWNERS
  • .github/labeler.yml
  • infra/uat/reservations.yaml
  • pkg/uatbroker/doc.go
  • pkg/uatbroker/model.go
  • pkg/uatbroker/registry.go
  • pkg/uatbroker/registry_test.go
  • pkg/uatbroker/schedule.go
  • pkg/uatbroker/schedule_test.go
  • tools/uat-broker/README.md
  • tools/uat-broker/main.go
  • tools/uat-broker/main_test.go

Comment thread tools/uat-broker/main.go
Comment thread tools/uat-broker/main.go Outdated
Comment thread tools/uat-broker/main.go Outdated
mchmarny
mchmarny previously approved these changes Jun 30, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/uat-broker/main.go`:
- Around line 282-284: The stdin size-limit guard in readTags is untested, so
add a unit test in main_test.go that exercises the oversized-input path by
calling readTags with a reader larger than maxTagsBytes and asserting it returns
an error; use the readTags function and the maxTagsBytes limit so the OOM
safeguard branch is covered.
- Around line 146-165: The reservations command in main/runReservations
currently lets `--list` override `--name`, so the conflicting flags are silently
accepted. Update the flag handling in the `switch` around `list`, `name == ""`,
and `reg.Lookup` to explicitly reject the `--list` + `--name` combination with
an invalid request error before listing or looking up reservations. Make sure
the validation sits alongside the existing argument checks in the same function
so workflow templating mistakes are surfaced instead of masked.

In `@tools/uat-broker/README.md`:
- Around line 62-65: The Exit codes section in the uat-broker README is missing
codes returned by the implementation. Update the documentation for the
runSchedule and runReservations behavior to include ErrCodeInternal for stdout
write/JSON encoding failures and ErrCodeTimeout for ctx-canceled stdin reads,
alongside the existing 0, 2, and 3 entries. Keep the table/wording aligned with
the actual pkg/errors coded contract so callers can branch correctly.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: f7634ebf-e0c8-40d9-bd04-9b5b08b46805

📥 Commits

Reviewing files that changed from the base of the PR and between 4820195 and d1d9962.

📒 Files selected for processing (12)
  • .github/CODEOWNERS
  • .github/labeler.yml
  • infra/uat/reservations.yaml
  • pkg/uatbroker/doc.go
  • pkg/uatbroker/model.go
  • pkg/uatbroker/registry.go
  • pkg/uatbroker/registry_test.go
  • pkg/uatbroker/schedule.go
  • pkg/uatbroker/schedule_test.go
  • tools/uat-broker/README.md
  • tools/uat-broker/main.go
  • tools/uat-broker/main_test.go

Comment thread tools/uat-broker/main.go
Comment thread tools/uat-broker/main.go
Comment thread tools/uat-broker/README.md Outdated
@njhensley njhensley force-pushed the ci/uat-reservation-broker branch from d1d9962 to b531b55 Compare June 30, 2026 20:44

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.github/labeler.yml (1)

15-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Consider whether pkg/uatbroker/** and infra/uat/** should also be labeled.

Only tools/uat-broker/** was added to area/ci. Changes to pkg/uatbroker/** (business logic) or infra/uat/reservations.yaml (registry data) won't get this label unless another existing rule already covers them.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/labeler.yml around lines 15 - 23, The area/ci label rule is too
narrow and misses other UAT Broker-related changes. Update the labeler
configuration in the area/ci rule so it also matches pkg/uatbroker/** and
infra/uat/** alongside tools/uat-broker/**, ensuring UAT broker business logic
and registry data changes get labeled consistently. Use the existing area/ci
entry in the labeler config as the place to expand the affected globs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/CODEOWNERS:
- Around line 44-47: The CODEOWNERS entry only covers tools/uat-broker/**, so
the new UAT directories are still inheriting `@nvidia/aicr-write`. Add explicit
owner rules for pkg/uatbroker/** and infra/uat/** in CODEOWNERS, or replace with
broader pkg/** and infra/** rules if those areas should require
`@nvidia/aicr-maintainer` approval; keep the existing tools/uat-broker/** rule
unchanged.

In `@pkg/uatbroker/registry.go`:
- Around line 105-112: Lookup currently returns a pointer to an element inside
Registry.Reservations, which lets callers mutate the registry’s internal state.
Update Registry.Lookup to return a copy of the matching Reservation instead of
&r.Reservations[i], while keeping the not-found error behavior unchanged. Use
the Lookup method and Reservation type as the main touchpoints when making the
change.

In `@pkg/uatbroker/schedule.go`:
- Around line 63-86: The sortedStableDescending helper in schedule.go should
guard against normalized-semver duplicates, since semver.NewVersion can coerce
different raw tags into the same semantic version and let both consume previousN
slots. Update the stable tag collection logic in sortedStableDescending to
de-dup by the normalized version identity (for example, the parsed semver
value’s canonical string) before sorting and returning the list, and keep the
existing filtering for blanks, invalid tags, and prereleases.

---

Outside diff comments:
In @.github/labeler.yml:
- Around line 15-23: The area/ci label rule is too narrow and misses other UAT
Broker-related changes. Update the labeler configuration in the area/ci rule so
it also matches pkg/uatbroker/** and infra/uat/** alongside tools/uat-broker/**,
ensuring UAT broker business logic and registry data changes get labeled
consistently. Use the existing area/ci entry in the labeler config as the place
to expand the affected globs.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: d7833e58-c476-43a6-b587-33986671b2b2

📥 Commits

Reviewing files that changed from the base of the PR and between d1d9962 and b531b55.

📒 Files selected for processing (12)
  • .github/CODEOWNERS
  • .github/labeler.yml
  • infra/uat/reservations.yaml
  • pkg/uatbroker/doc.go
  • pkg/uatbroker/model.go
  • pkg/uatbroker/registry.go
  • pkg/uatbroker/registry_test.go
  • pkg/uatbroker/schedule.go
  • pkg/uatbroker/schedule_test.go
  • tools/uat-broker/README.md
  • tools/uat-broker/main.go
  • tools/uat-broker/main_test.go

Comment thread .github/CODEOWNERS
Comment thread pkg/uatbroker/registry.go
Comment thread pkg/uatbroker/schedule.go
DC1 of the day/night UAT broker epic (NVIDIA#1264): the data and tooling layer
the broker will consume, with no live-workflow change.

- infra/uat/reservations.yaml: the reservation registry — one row per
  reserved GPU pool ({name, cloud, reservation-id, accelerator,
  gpu-count, cluster-config-path, test-config-dir}), seeded with the AWS
  H100 and GCP H100 rows. The name is the lease key; new infrastructure
  is onboarded by adding a row, no code change.
- pkg/uatbroker: registry parse/validate/lookup plus the nightly
  version-matrix schedule expansion (main-first, then the previous N
  stable releases in descending semver order; pre-releases and
  non-semver tags dropped; oldest dropped first), with table-driven
  tests including a guard that the committed registry stays valid.
- tools/uat-broker: thin CLI over the package — `reservations` (resolve
  a row to GITHUB_OUTPUT key=value lines, or --list) and `schedule`
  (emit the ordered matrix as JSON from a stdin tag list). Coded exits:
  0 ok, 2 invalid-request, 3 not-found.
- labeler.yml + CODEOWNERS: route tools/uat-broker/**.

Signed-off-by: Nathan Hensley <[email protected]>
@njhensley njhensley force-pushed the ci/uat-reservation-broker branch from b531b55 to cff6ab4 Compare June 30, 2026 21:42

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/uat-broker/main.go`:
- Around line 117-167: In runReservations, validate the flag combination before
calling uatbroker.LoadRegistryFile so usage errors are returned first; move the
--list/--name mutual-exclusion check and the required-name check ahead of the
registry load, then only load the registry in the branch that actually needs it
for lookup or listing. Keep the existing error handling paths in runReservations
and flag parsing unchanged, but ensure the filesystem is not touched for invalid
flag combinations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 265506af-a6f0-4fe0-accb-cf1823f6c778

📥 Commits

Reviewing files that changed from the base of the PR and between b531b55 and cff6ab4.

📒 Files selected for processing (12)
  • .github/CODEOWNERS
  • .github/labeler.yml
  • infra/uat/reservations.yaml
  • pkg/uatbroker/doc.go
  • pkg/uatbroker/model.go
  • pkg/uatbroker/registry.go
  • pkg/uatbroker/registry_test.go
  • pkg/uatbroker/schedule.go
  • pkg/uatbroker/schedule_test.go
  • tools/uat-broker/README.md
  • tools/uat-broker/main.go
  • tools/uat-broker/main_test.go

Comment thread tools/uat-broker/main.go
Comment on lines +117 to +167
func runReservations(args []string, stdout, stderr io.Writer) error {
fs := flag.NewFlagSet("reservations", flag.ContinueOnError)
fs.SetOutput(stderr)
var (
file string
name string
list bool
)
fs.StringVar(&file, "file", defaultRegistryPath, "path to the reservation registry")
fs.StringVar(&name, "name", "", "reservation name to resolve")
fs.BoolVar(&list, "list", false, "list all reservation names, one per line")
if err := fs.Parse(args); err != nil {
return flagParseErr(err, "reservations")
}
if fs.NArg() != 0 {
return errors.New(errors.ErrCodeInvalidRequest,
"reservations: unexpected positional arguments: "+strings.Join(fs.Args(), " "))
}

reg, err := uatbroker.LoadRegistryFile(file)
if err != nil {
return err
}

// Build the output first, then write it in one checked call so a broken
// pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
// silent exit-0 that leaves downstream jobs without their inputs. Writes to
// the strings.Builder cannot fail.
var b strings.Builder
switch {
case list && name != "":
return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
case list:
for _, n := range reg.Names() {
fmt.Fprintln(&b, n)
}
case name == "":
return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
default:
res, lookupErr := reg.Lookup(name)
if lookupErr != nil {
return lookupErr
}
// GITHUB_OUTPUT-style key=value lines; every value is single-line.
fmt.Fprintf(&b, "cloud=%s\n", res.Cloud)
fmt.Fprintf(&b, "reservation-id=%s\n", res.ReservationID)
fmt.Fprintf(&b, "accelerator=%s\n", res.Accelerator)
fmt.Fprintf(&b, "gpu-count=%d\n", res.GPUCount)
fmt.Fprintf(&b, "cluster-config-path=%s\n", res.ClusterConfigPath)
fmt.Fprintf(&b, "test-config-dir=%s\n", res.TestConfigDir)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Registry is loaded before flag-combination validation, masking usage errors with file errors.

LoadRegistryFile (Line 136) runs before the --list/--name mutual-exclusion and "name required" checks (Lines 147, 153-154). A call like reservations --list --name x --file /bad/path reports a registry-load failure instead of the actual flag-usage error, which is misleading when debugging a CI workflow misconfiguration. Two of the four switch arms (list && name != "", name == "") don't need reg at all.

♻️ Suggested fix: validate flags before touching the filesystem
-	reg, err := uatbroker.LoadRegistryFile(file)
-	if err != nil {
-		return err
-	}
-
-	// Build the output first, then write it in one checked call so a broken
-	// pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
-	// silent exit-0 that leaves downstream jobs without their inputs. Writes to
-	// the strings.Builder cannot fail.
-	var b strings.Builder
-	switch {
-	case list && name != "":
-		return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
-	case list:
+	switch {
+	case list && name != "":
+		return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
+	case name == "" && !list:
+		return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
+	}
+
+	reg, err := uatbroker.LoadRegistryFile(file)
+	if err != nil {
+		return err
+	}
+
+	// Build the output first, then write it in one checked call so a broken
+	// pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
+	// silent exit-0 that leaves downstream jobs without their inputs. Writes to
+	// the strings.Builder cannot fail.
+	var b strings.Builder
+	switch {
+	case list:
 		for _, n := range reg.Names() {
 			fmt.Fprintln(&b, n)
 		}
-	case name == "":
-		return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
 	default:
 		res, lookupErr := reg.Lookup(name)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
func runReservations(args []string, stdout, stderr io.Writer) error {
fs := flag.NewFlagSet("reservations", flag.ContinueOnError)
fs.SetOutput(stderr)
var (
file string
name string
list bool
)
fs.StringVar(&file, "file", defaultRegistryPath, "path to the reservation registry")
fs.StringVar(&name, "name", "", "reservation name to resolve")
fs.BoolVar(&list, "list", false, "list all reservation names, one per line")
if err := fs.Parse(args); err != nil {
return flagParseErr(err, "reservations")
}
if fs.NArg() != 0 {
return errors.New(errors.ErrCodeInvalidRequest,
"reservations: unexpected positional arguments: "+strings.Join(fs.Args(), " "))
}
reg, err := uatbroker.LoadRegistryFile(file)
if err != nil {
return err
}
// Build the output first, then write it in one checked call so a broken
// pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
// silent exit-0 that leaves downstream jobs without their inputs. Writes to
// the strings.Builder cannot fail.
var b strings.Builder
switch {
case list && name != "":
return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
case list:
for _, n := range reg.Names() {
fmt.Fprintln(&b, n)
}
case name == "":
return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
default:
res, lookupErr := reg.Lookup(name)
if lookupErr != nil {
return lookupErr
}
// GITHUB_OUTPUT-style key=value lines; every value is single-line.
fmt.Fprintf(&b, "cloud=%s\n", res.Cloud)
fmt.Fprintf(&b, "reservation-id=%s\n", res.ReservationID)
fmt.Fprintf(&b, "accelerator=%s\n", res.Accelerator)
fmt.Fprintf(&b, "gpu-count=%d\n", res.GPUCount)
fmt.Fprintf(&b, "cluster-config-path=%s\n", res.ClusterConfigPath)
fmt.Fprintf(&b, "test-config-dir=%s\n", res.TestConfigDir)
}
func runReservations(args []string, stdout, stderr io.Writer) error {
fs := flag.NewFlagSet("reservations", flag.ContinueOnError)
fs.SetOutput(stderr)
var (
file string
name string
list bool
)
fs.StringVar(&file, "file", defaultRegistryPath, "path to the reservation registry")
fs.StringVar(&name, "name", "", "reservation name to resolve")
fs.BoolVar(&list, "list", false, "list all reservation names, one per line")
if err := fs.Parse(args); err != nil {
return flagParseErr(err, "reservations")
}
if fs.NArg() != 0 {
return errors.New(errors.ErrCodeInvalidRequest,
"reservations: unexpected positional arguments: "+strings.Join(fs.Args(), " "))
}
switch {
case list && name != "":
return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
case name == "" && !list:
return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
}
reg, err := uatbroker.LoadRegistryFile(file)
if err != nil {
return err
}
// Build the output first, then write it in one checked call so a broken
// pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
// silent exit-0 that leaves downstream jobs without their inputs. Writes to
// the strings.Builder cannot fail.
var b strings.Builder
switch {
case list:
for _, n := range reg.Names() {
fmt.Fprintln(&b, n)
}
default:
res, lookupErr := reg.Lookup(name)
if lookupErr != nil {
return lookupErr
}
// GITHUB_OUTPUT-style key=value lines; every value is single-line.
fmt.Fprintf(&b, "cloud=%s\n", res.Cloud)
fmt.Fprintf(&b, "reservation-id=%s\n", res.ReservationID)
fmt.Fprintf(&b, "accelerator=%s\n", res.Accelerator)
fmt.Fprintf(&b, "gpu-count=%d\n", res.GPUCount)
fmt.Fprintf(&b, "cluster-config-path=%s\n", res.ClusterConfigPath)
fmt.Fprintf(&b, "test-config-dir=%s\n", res.TestConfigDir)
}
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/uat-broker/main.go` around lines 117 - 167, In runReservations,
validate the flag combination before calling uatbroker.LoadRegistryFile so usage
errors are returned first; move the --list/--name mutual-exclusion check and the
required-name check ahead of the registry load, then only load the registry in
the branch that actually needs it for lookup or listing. Keep the existing error
handling paths in runReservations and flag parsing unchanged, but ensure the
filesystem is not touched for invalid flag combinations.

@njhensley njhensley merged commit a363ce4 into NVIDIA:main Jun 30, 2026
30 checks passed
@njhensley njhensley deleted the ci/uat-reservation-broker branch June 30, 2026 22:01
njhensley added a commit to njhensley/aicr that referenced this pull request Jul 1, 2026
DC1 PR-2 (the broker cutover), building on the reservation registry +
uat-broker helper from NVIDIA#1559. Every UAT run now goes through a shared
dispatch surface that holds a per-reservation concurrency lease, so
contending runs QUEUE instead of racing the reserved GPU hardware.

- uat-run.yaml (new): the lease owner + dispatch surface. Holds the
  concurrency group uat-<reservation>, resolves the reservation row from
  infra/uat/reservations.yaml via the uat-broker helper, and invokes the
  cloud-appropriate reusable pipeline with the permission superset it needs.
  Dispatchable by a human (workflow_dispatch) or the nightly batch
  (workflow_call).
- uat-aws.yaml / uat-gcp.yaml: converted in place to workflow_call-only
  (dropped the per-workflow cron + self-concurrency; CLUSTER_CONFIG /
  TEST_CONFIG now come from the registry via inputs). Filenames and the
  keyless-cosign identity pins are unchanged; the pipeline bodies are
  otherwise untouched, and evidence-ingest stays nested.
- uat-nightly-batch.yaml (new): the night side of the cycle — a cron that
  enumerates reservations from the registry and runs one main pass per
  reservation (data-driven matrix -> uat-run.yaml). Different reservations
  run in parallel; same-reservation runs serialize on the lease.
- docs/contributor/uat.md (new, registered in docs/index.yml): the day/night
  model, how to request a run, how queuing + the one-pending limit work, and
  how to add a reservation. Also fixes the stale gh-workflow-run command in
  the CUJ1 demo and clarifies the two infra-account READMEs.

Scope: exercises the main x training cell on both reservations. The version
matrix (PR-3/DC5), the inference intent (DC2/DC3), and superseded-run
surfacing (DC6) light up the remaining axes; the mechanism already scales to
them.

Signed-off-by: Nathan Hensley <[email protected]>
njhensley added a commit to njhensley/aicr that referenced this pull request Jul 1, 2026
DC1 PR-2 (the broker cutover), building on the reservation registry +
uat-broker helper from NVIDIA#1559. Every UAT run now goes through a shared
dispatch surface that holds a per-reservation concurrency lease, so
contending runs QUEUE instead of racing the reserved GPU hardware.

- uat-run.yaml (new): the lease owner + dispatch surface. Holds the
  concurrency group uat-<reservation>, resolves the reservation row from
  infra/uat/reservations.yaml via the uat-broker helper, and invokes the
  cloud-appropriate reusable pipeline with the permission superset it needs.
  Dispatchable by a human (workflow_dispatch) or the nightly batch
  (workflow_call).
- uat-aws.yaml / uat-gcp.yaml: converted in place to workflow_call-only
  (dropped the per-workflow cron + self-concurrency; CLUSTER_CONFIG /
  TEST_CONFIG now come from the registry via inputs). Filenames and the
  keyless-cosign identity pins are unchanged; the pipeline bodies are
  otherwise untouched, and evidence-ingest stays nested.
- uat-nightly-batch.yaml (new): the night side of the cycle — a cron that
  enumerates reservations from the registry and runs one main pass per
  reservation (data-driven matrix -> uat-run.yaml). Different reservations
  run in parallel; same-reservation runs serialize on the lease.
- docs/contributor/uat.md (new, registered in docs/index.yml): the day/night
  model, how to request a run, how queuing + the one-pending limit work, and
  how to add a reservation. Also fixes the stale gh-workflow-run command in
  the CUJ1 demo and clarifies the two infra-account READMEs.

Scope: exercises the main x training cell on both reservations. The version
matrix (PR-3/DC5), the inference intent (DC2/DC3), and superseded-run
surfacing (DC6) light up the remaining axes; the mechanism already scales to
them.

Signed-off-by: Nathan Hensley <[email protected]>
njhensley added a commit to njhensley/aicr that referenced this pull request Jul 1, 2026
DC1 PR-2 (the broker cutover), building on the reservation registry +
uat-broker helper from NVIDIA#1559. Every UAT run now goes through a shared
dispatch surface that holds a per-reservation concurrency lease, so
contending runs QUEUE instead of racing the reserved GPU hardware.

- uat-run.yaml (new): the lease owner + dispatch surface. Holds the
  concurrency group uat-<reservation>, resolves the reservation row from
  infra/uat/reservations.yaml via the uat-broker helper, and invokes the
  cloud-appropriate reusable pipeline with the permission superset it needs.
  Dispatchable by a human (workflow_dispatch) or the nightly batch
  (workflow_call).
- uat-aws.yaml / uat-gcp.yaml: converted in place to workflow_call-only
  (dropped the per-workflow cron + self-concurrency; CLUSTER_CONFIG /
  TEST_CONFIG now come from the registry via inputs). Filenames and the
  keyless-cosign identity pins are unchanged; the pipeline bodies are
  otherwise untouched, and evidence-ingest stays nested.
- uat-nightly-batch.yaml (new): the night side of the cycle — a cron that
  enumerates reservations from the registry and runs one main pass per
  reservation (data-driven matrix -> uat-run.yaml). Different reservations
  run in parallel; same-reservation runs serialize on the lease.
- docs/contributor/uat.md (new, registered in docs/index.yml): the day/night
  model, how to request a run, how queuing + the one-pending limit work, and
  how to add a reservation. Also fixes the stale gh-workflow-run command in
  the CUJ1 demo and clarifies the two infra-account READMEs.

Scope: exercises the main x training cell on both reservations. The version
matrix (PR-3/DC5), the inference intent (DC2/DC3), and superseded-run
surfacing (DC6) light up the remaining axes; the mechanism already scales to
them.

Signed-off-by: Nathan Hensley <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci area/infra size/XL theme/ci-dx CI pipelines, developer experience, and build tooling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants