feat(ci): add UAT reservation registry and broker helper (#1274)#1559
Conversation
📝 WalkthroughWalkthroughThis PR adds a new Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
@coderabbitai full review |
mchmarny
left a comment
There was a problem hiding this comment.
Solid DC1 foundation — clean, well-tested, and it mirrors the pkg/corroborate + tools/corroborate precedent nicely (coded exits, size-bounded reads, registry self-validation, ~96%/87% coverage). CI is fully green.
One medium note on the CLI side: signal.NotifyContext is set up but the context is never consumed, which actually makes the tool less interruptible on a blocking stdin read than no signal handling at all. Plus two questions on the committed infra data (personal-repo dependency and public exposure of prod GCP/reservation IDs) — worth confirming those are intentional. None of it blocks merge.
| func realMain() int { | ||
| // Cancel on Ctrl-C / SIGTERM (CI cancellation) rather than imposing an | ||
| // arbitrary deadline: the tool only does local file + stdin I/O. | ||
| ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) |
There was a problem hiding this comment.
signal.NotifyContext is installed but ctx is never consumed — runReservations doesn't take it and runSchedule discards it (_ context.Context). NotifyContext suppresses the default SIGINT/SIGTERM termination, but since nothing checks ctx.Done(), a signal arriving while readTags blocks on an open stdin (interactive invocation, or CI cancelling mid-read) is swallowed and the process hangs until SIGKILL — strictly worse than no signal handling.
Either wire ctx into the stdin read (read in a goroutine, select on ctx.Done()), or drop the signal handling entirely since the tool only does fast local file + stdin I/O and the default termination behavior is fine.
There was a problem hiding this comment.
Good catch — fixed in the latest push. runSchedule now threads ctx, and readTags runs the stdin read in a goroutine and selects on ctx.Done(), so a SIGINT/SIGTERM (interactive run, or CI cancelling mid-read) interrupts it instead of hanging until SIGKILL. Added TestReadTagsCanceledContext. runReservations stays ctx-free on purpose — it only does fast, size-bounded file I/O with no blocking call to interrupt.
| previousN = 0 | ||
| } | ||
| stable := sortedStableDescending(rawTags) | ||
| if previousN < len(stable) { |
There was a problem hiding this comment.
nit: duplicate names in the reservations slice silently dedupe via the output map, so --reservations aws-h100,aws-h100 produces one key with no error. Harmless today; optional to reject duplicates for symmetry with the registry's uniqueness check.
There was a problem hiding this comment.
Done — resolveReservations now rejects duplicate --reservations names with ErrCodeInvalidRequest instead of silently collapsing via the output map, mirroring the registry's own name-uniqueness check. Added a test.
cd0f98f to
4820195
Compare
|
Thanks for the review @mchmarny 🙏 — pushed fixes addressing all four (replied inline on each thread):
CI was green on the prior push; |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/uat-broker/main.go`:
- Around line 128-130: After FlagSet.Parse in both subcommand handlers, check
fs.Args() and return ExitInvalidInput when any unexpected positional arguments
remain. Update the reservations and schedule command handlers in main.go so they
reject extra trailing args instead of silently succeeding, using the existing
flagParseErr/invalid-input path and the handler functions that call
fs.Parse(args).
- Around line 137-140: The stdout writes in the `reservations` flow are ignoring
`Fprintln`/`Fprintf` errors, so failures like broken pipes or an unwritable
`$GITHUB_OUTPUT` can still return success. Update the `reservations` handling in
`main` to check and propagate every write error from `stdout` instead of
discarding it, and return a non-zero failure when any output write fails. Keep
the fix localized around the `list` branch and the later output-emission path so
the tool fails fast with the underlying write error.
- Around line 193-200: The warning in uat-broker’s schedule expansion path is
too broad: when ExpandSchedule produces empty cell lists because
--include-main=false and there are no stable tags, it should not be reported as
“main-only.” Update the warning logic around scheduleHasNoReleases in main to
distinguish an empty schedule from a degraded main-only schedule, and only emit
the main-only message when the schedule actually contains main rows; otherwise
use a precise empty/none-available warning.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: a58e6f1a-8c2a-4c1a-b179-5d112d7210fc
📒 Files selected for processing (12)
.github/CODEOWNERS.github/labeler.ymlinfra/uat/reservations.yamlpkg/uatbroker/doc.gopkg/uatbroker/model.gopkg/uatbroker/registry.gopkg/uatbroker/registry_test.gopkg/uatbroker/schedule.gopkg/uatbroker/schedule_test.gotools/uat-broker/README.mdtools/uat-broker/main.gotools/uat-broker/main_test.go
4820195 to
d1d9962
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/uat-broker/main.go`:
- Around line 282-284: The stdin size-limit guard in readTags is untested, so
add a unit test in main_test.go that exercises the oversized-input path by
calling readTags with a reader larger than maxTagsBytes and asserting it returns
an error; use the readTags function and the maxTagsBytes limit so the OOM
safeguard branch is covered.
- Around line 146-165: The reservations command in main/runReservations
currently lets `--list` override `--name`, so the conflicting flags are silently
accepted. Update the flag handling in the `switch` around `list`, `name == ""`,
and `reg.Lookup` to explicitly reject the `--list` + `--name` combination with
an invalid request error before listing or looking up reservations. Make sure
the validation sits alongside the existing argument checks in the same function
so workflow templating mistakes are surfaced instead of masked.
In `@tools/uat-broker/README.md`:
- Around line 62-65: The Exit codes section in the uat-broker README is missing
codes returned by the implementation. Update the documentation for the
runSchedule and runReservations behavior to include ErrCodeInternal for stdout
write/JSON encoding failures and ErrCodeTimeout for ctx-canceled stdin reads,
alongside the existing 0, 2, and 3 entries. Keep the table/wording aligned with
the actual pkg/errors coded contract so callers can branch correctly.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: f7634ebf-e0c8-40d9-bd04-9b5b08b46805
📒 Files selected for processing (12)
.github/CODEOWNERS.github/labeler.ymlinfra/uat/reservations.yamlpkg/uatbroker/doc.gopkg/uatbroker/model.gopkg/uatbroker/registry.gopkg/uatbroker/registry_test.gopkg/uatbroker/schedule.gopkg/uatbroker/schedule_test.gotools/uat-broker/README.mdtools/uat-broker/main.gotools/uat-broker/main_test.go
d1d9962 to
b531b55
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.github/labeler.yml (1)
15-23: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider whether
pkg/uatbroker/**andinfra/uat/**should also be labeled.Only
tools/uat-broker/**was added toarea/ci. Changes topkg/uatbroker/**(business logic) orinfra/uat/reservations.yaml(registry data) won't get this label unless another existing rule already covers them.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.github/labeler.yml around lines 15 - 23, The area/ci label rule is too narrow and misses other UAT Broker-related changes. Update the labeler configuration in the area/ci rule so it also matches pkg/uatbroker/** and infra/uat/** alongside tools/uat-broker/**, ensuring UAT broker business logic and registry data changes get labeled consistently. Use the existing area/ci entry in the labeler config as the place to expand the affected globs.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/CODEOWNERS:
- Around line 44-47: The CODEOWNERS entry only covers tools/uat-broker/**, so
the new UAT directories are still inheriting `@nvidia/aicr-write`. Add explicit
owner rules for pkg/uatbroker/** and infra/uat/** in CODEOWNERS, or replace with
broader pkg/** and infra/** rules if those areas should require
`@nvidia/aicr-maintainer` approval; keep the existing tools/uat-broker/** rule
unchanged.
In `@pkg/uatbroker/registry.go`:
- Around line 105-112: Lookup currently returns a pointer to an element inside
Registry.Reservations, which lets callers mutate the registry’s internal state.
Update Registry.Lookup to return a copy of the matching Reservation instead of
&r.Reservations[i], while keeping the not-found error behavior unchanged. Use
the Lookup method and Reservation type as the main touchpoints when making the
change.
In `@pkg/uatbroker/schedule.go`:
- Around line 63-86: The sortedStableDescending helper in schedule.go should
guard against normalized-semver duplicates, since semver.NewVersion can coerce
different raw tags into the same semantic version and let both consume previousN
slots. Update the stable tag collection logic in sortedStableDescending to
de-dup by the normalized version identity (for example, the parsed semver
value’s canonical string) before sorting and returning the list, and keep the
existing filtering for blanks, invalid tags, and prereleases.
---
Outside diff comments:
In @.github/labeler.yml:
- Around line 15-23: The area/ci label rule is too narrow and misses other UAT
Broker-related changes. Update the labeler configuration in the area/ci rule so
it also matches pkg/uatbroker/** and infra/uat/** alongside tools/uat-broker/**,
ensuring UAT broker business logic and registry data changes get labeled
consistently. Use the existing area/ci entry in the labeler config as the place
to expand the affected globs.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: d7833e58-c476-43a6-b587-33986671b2b2
📒 Files selected for processing (12)
.github/CODEOWNERS.github/labeler.ymlinfra/uat/reservations.yamlpkg/uatbroker/doc.gopkg/uatbroker/model.gopkg/uatbroker/registry.gopkg/uatbroker/registry_test.gopkg/uatbroker/schedule.gopkg/uatbroker/schedule_test.gotools/uat-broker/README.mdtools/uat-broker/main.gotools/uat-broker/main_test.go
DC1 of the day/night UAT broker epic (NVIDIA#1264): the data and tooling layer the broker will consume, with no live-workflow change. - infra/uat/reservations.yaml: the reservation registry — one row per reserved GPU pool ({name, cloud, reservation-id, accelerator, gpu-count, cluster-config-path, test-config-dir}), seeded with the AWS H100 and GCP H100 rows. The name is the lease key; new infrastructure is onboarded by adding a row, no code change. - pkg/uatbroker: registry parse/validate/lookup plus the nightly version-matrix schedule expansion (main-first, then the previous N stable releases in descending semver order; pre-releases and non-semver tags dropped; oldest dropped first), with table-driven tests including a guard that the committed registry stays valid. - tools/uat-broker: thin CLI over the package — `reservations` (resolve a row to GITHUB_OUTPUT key=value lines, or --list) and `schedule` (emit the ordered matrix as JSON from a stdin tag list). Coded exits: 0 ok, 2 invalid-request, 3 not-found. - labeler.yml + CODEOWNERS: route tools/uat-broker/**. Signed-off-by: Nathan Hensley <[email protected]>
b531b55 to
cff6ab4
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/uat-broker/main.go`:
- Around line 117-167: In runReservations, validate the flag combination before
calling uatbroker.LoadRegistryFile so usage errors are returned first; move the
--list/--name mutual-exclusion check and the required-name check ahead of the
registry load, then only load the registry in the branch that actually needs it
for lookup or listing. Keep the existing error handling paths in runReservations
and flag parsing unchanged, but ensure the filesystem is not touched for invalid
flag combinations.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 265506af-a6f0-4fe0-accb-cf1823f6c778
📒 Files selected for processing (12)
.github/CODEOWNERS.github/labeler.ymlinfra/uat/reservations.yamlpkg/uatbroker/doc.gopkg/uatbroker/model.gopkg/uatbroker/registry.gopkg/uatbroker/registry_test.gopkg/uatbroker/schedule.gopkg/uatbroker/schedule_test.gotools/uat-broker/README.mdtools/uat-broker/main.gotools/uat-broker/main_test.go
| func runReservations(args []string, stdout, stderr io.Writer) error { | ||
| fs := flag.NewFlagSet("reservations", flag.ContinueOnError) | ||
| fs.SetOutput(stderr) | ||
| var ( | ||
| file string | ||
| name string | ||
| list bool | ||
| ) | ||
| fs.StringVar(&file, "file", defaultRegistryPath, "path to the reservation registry") | ||
| fs.StringVar(&name, "name", "", "reservation name to resolve") | ||
| fs.BoolVar(&list, "list", false, "list all reservation names, one per line") | ||
| if err := fs.Parse(args); err != nil { | ||
| return flagParseErr(err, "reservations") | ||
| } | ||
| if fs.NArg() != 0 { | ||
| return errors.New(errors.ErrCodeInvalidRequest, | ||
| "reservations: unexpected positional arguments: "+strings.Join(fs.Args(), " ")) | ||
| } | ||
|
|
||
| reg, err := uatbroker.LoadRegistryFile(file) | ||
| if err != nil { | ||
| return err | ||
| } | ||
|
|
||
| // Build the output first, then write it in one checked call so a broken | ||
| // pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a | ||
| // silent exit-0 that leaves downstream jobs without their inputs. Writes to | ||
| // the strings.Builder cannot fail. | ||
| var b strings.Builder | ||
| switch { | ||
| case list && name != "": | ||
| return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive") | ||
| case list: | ||
| for _, n := range reg.Names() { | ||
| fmt.Fprintln(&b, n) | ||
| } | ||
| case name == "": | ||
| return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)") | ||
| default: | ||
| res, lookupErr := reg.Lookup(name) | ||
| if lookupErr != nil { | ||
| return lookupErr | ||
| } | ||
| // GITHUB_OUTPUT-style key=value lines; every value is single-line. | ||
| fmt.Fprintf(&b, "cloud=%s\n", res.Cloud) | ||
| fmt.Fprintf(&b, "reservation-id=%s\n", res.ReservationID) | ||
| fmt.Fprintf(&b, "accelerator=%s\n", res.Accelerator) | ||
| fmt.Fprintf(&b, "gpu-count=%d\n", res.GPUCount) | ||
| fmt.Fprintf(&b, "cluster-config-path=%s\n", res.ClusterConfigPath) | ||
| fmt.Fprintf(&b, "test-config-dir=%s\n", res.TestConfigDir) | ||
| } |
There was a problem hiding this comment.
🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win
Registry is loaded before flag-combination validation, masking usage errors with file errors.
LoadRegistryFile (Line 136) runs before the --list/--name mutual-exclusion and "name required" checks (Lines 147, 153-154). A call like reservations --list --name x --file /bad/path reports a registry-load failure instead of the actual flag-usage error, which is misleading when debugging a CI workflow misconfiguration. Two of the four switch arms (list && name != "", name == "") don't need reg at all.
♻️ Suggested fix: validate flags before touching the filesystem
- reg, err := uatbroker.LoadRegistryFile(file)
- if err != nil {
- return err
- }
-
- // Build the output first, then write it in one checked call so a broken
- // pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
- // silent exit-0 that leaves downstream jobs without their inputs. Writes to
- // the strings.Builder cannot fail.
- var b strings.Builder
- switch {
- case list && name != "":
- return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
- case list:
+ switch {
+ case list && name != "":
+ return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive")
+ case name == "" && !list:
+ return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
+ }
+
+ reg, err := uatbroker.LoadRegistryFile(file)
+ if err != nil {
+ return err
+ }
+
+ // Build the output first, then write it in one checked call so a broken
+ // pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a
+ // silent exit-0 that leaves downstream jobs without their inputs. Writes to
+ // the strings.Builder cannot fail.
+ var b strings.Builder
+ switch {
+ case list:
for _, n := range reg.Names() {
fmt.Fprintln(&b, n)
}
- case name == "":
- return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)")
default:
res, lookupErr := reg.Lookup(name)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| func runReservations(args []string, stdout, stderr io.Writer) error { | |
| fs := flag.NewFlagSet("reservations", flag.ContinueOnError) | |
| fs.SetOutput(stderr) | |
| var ( | |
| file string | |
| name string | |
| list bool | |
| ) | |
| fs.StringVar(&file, "file", defaultRegistryPath, "path to the reservation registry") | |
| fs.StringVar(&name, "name", "", "reservation name to resolve") | |
| fs.BoolVar(&list, "list", false, "list all reservation names, one per line") | |
| if err := fs.Parse(args); err != nil { | |
| return flagParseErr(err, "reservations") | |
| } | |
| if fs.NArg() != 0 { | |
| return errors.New(errors.ErrCodeInvalidRequest, | |
| "reservations: unexpected positional arguments: "+strings.Join(fs.Args(), " ")) | |
| } | |
| reg, err := uatbroker.LoadRegistryFile(file) | |
| if err != nil { | |
| return err | |
| } | |
| // Build the output first, then write it in one checked call so a broken | |
| // pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a | |
| // silent exit-0 that leaves downstream jobs without their inputs. Writes to | |
| // the strings.Builder cannot fail. | |
| var b strings.Builder | |
| switch { | |
| case list && name != "": | |
| return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive") | |
| case list: | |
| for _, n := range reg.Names() { | |
| fmt.Fprintln(&b, n) | |
| } | |
| case name == "": | |
| return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)") | |
| default: | |
| res, lookupErr := reg.Lookup(name) | |
| if lookupErr != nil { | |
| return lookupErr | |
| } | |
| // GITHUB_OUTPUT-style key=value lines; every value is single-line. | |
| fmt.Fprintf(&b, "cloud=%s\n", res.Cloud) | |
| fmt.Fprintf(&b, "reservation-id=%s\n", res.ReservationID) | |
| fmt.Fprintf(&b, "accelerator=%s\n", res.Accelerator) | |
| fmt.Fprintf(&b, "gpu-count=%d\n", res.GPUCount) | |
| fmt.Fprintf(&b, "cluster-config-path=%s\n", res.ClusterConfigPath) | |
| fmt.Fprintf(&b, "test-config-dir=%s\n", res.TestConfigDir) | |
| } | |
| func runReservations(args []string, stdout, stderr io.Writer) error { | |
| fs := flag.NewFlagSet("reservations", flag.ContinueOnError) | |
| fs.SetOutput(stderr) | |
| var ( | |
| file string | |
| name string | |
| list bool | |
| ) | |
| fs.StringVar(&file, "file", defaultRegistryPath, "path to the reservation registry") | |
| fs.StringVar(&name, "name", "", "reservation name to resolve") | |
| fs.BoolVar(&list, "list", false, "list all reservation names, one per line") | |
| if err := fs.Parse(args); err != nil { | |
| return flagParseErr(err, "reservations") | |
| } | |
| if fs.NArg() != 0 { | |
| return errors.New(errors.ErrCodeInvalidRequest, | |
| "reservations: unexpected positional arguments: "+strings.Join(fs.Args(), " ")) | |
| } | |
| switch { | |
| case list && name != "": | |
| return errors.New(errors.ErrCodeInvalidRequest, "reservations: --list and --name are mutually exclusive") | |
| case name == "" && !list: | |
| return errors.New(errors.ErrCodeInvalidRequest, "reservations: --name is required (or pass --list)") | |
| } | |
| reg, err := uatbroker.LoadRegistryFile(file) | |
| if err != nil { | |
| return err | |
| } | |
| // Build the output first, then write it in one checked call so a broken | |
| // pipe or an unwritable $GITHUB_OUTPUT surfaces as a failure instead of a | |
| // silent exit-0 that leaves downstream jobs without their inputs. Writes to | |
| // the strings.Builder cannot fail. | |
| var b strings.Builder | |
| switch { | |
| case list: | |
| for _, n := range reg.Names() { | |
| fmt.Fprintln(&b, n) | |
| } | |
| default: | |
| res, lookupErr := reg.Lookup(name) | |
| if lookupErr != nil { | |
| return lookupErr | |
| } | |
| // GITHUB_OUTPUT-style key=value lines; every value is single-line. | |
| fmt.Fprintf(&b, "cloud=%s\n", res.Cloud) | |
| fmt.Fprintf(&b, "reservation-id=%s\n", res.ReservationID) | |
| fmt.Fprintf(&b, "accelerator=%s\n", res.Accelerator) | |
| fmt.Fprintf(&b, "gpu-count=%d\n", res.GPUCount) | |
| fmt.Fprintf(&b, "cluster-config-path=%s\n", res.ClusterConfigPath) | |
| fmt.Fprintf(&b, "test-config-dir=%s\n", res.TestConfigDir) | |
| } |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tools/uat-broker/main.go` around lines 117 - 167, In runReservations,
validate the flag combination before calling uatbroker.LoadRegistryFile so usage
errors are returned first; move the --list/--name mutual-exclusion check and the
required-name check ahead of the registry load, then only load the registry in
the branch that actually needs it for lookup or listing. Keep the existing error
handling paths in runReservations and flag parsing unchanged, but ensure the
filesystem is not touched for invalid flag combinations.
DC1 PR-2 (the broker cutover), building on the reservation registry + uat-broker helper from NVIDIA#1559. Every UAT run now goes through a shared dispatch surface that holds a per-reservation concurrency lease, so contending runs QUEUE instead of racing the reserved GPU hardware. - uat-run.yaml (new): the lease owner + dispatch surface. Holds the concurrency group uat-<reservation>, resolves the reservation row from infra/uat/reservations.yaml via the uat-broker helper, and invokes the cloud-appropriate reusable pipeline with the permission superset it needs. Dispatchable by a human (workflow_dispatch) or the nightly batch (workflow_call). - uat-aws.yaml / uat-gcp.yaml: converted in place to workflow_call-only (dropped the per-workflow cron + self-concurrency; CLUSTER_CONFIG / TEST_CONFIG now come from the registry via inputs). Filenames and the keyless-cosign identity pins are unchanged; the pipeline bodies are otherwise untouched, and evidence-ingest stays nested. - uat-nightly-batch.yaml (new): the night side of the cycle — a cron that enumerates reservations from the registry and runs one main pass per reservation (data-driven matrix -> uat-run.yaml). Different reservations run in parallel; same-reservation runs serialize on the lease. - docs/contributor/uat.md (new, registered in docs/index.yml): the day/night model, how to request a run, how queuing + the one-pending limit work, and how to add a reservation. Also fixes the stale gh-workflow-run command in the CUJ1 demo and clarifies the two infra-account READMEs. Scope: exercises the main x training cell on both reservations. The version matrix (PR-3/DC5), the inference intent (DC2/DC3), and superseded-run surfacing (DC6) light up the remaining axes; the mechanism already scales to them. Signed-off-by: Nathan Hensley <[email protected]>
DC1 PR-2 (the broker cutover), building on the reservation registry + uat-broker helper from NVIDIA#1559. Every UAT run now goes through a shared dispatch surface that holds a per-reservation concurrency lease, so contending runs QUEUE instead of racing the reserved GPU hardware. - uat-run.yaml (new): the lease owner + dispatch surface. Holds the concurrency group uat-<reservation>, resolves the reservation row from infra/uat/reservations.yaml via the uat-broker helper, and invokes the cloud-appropriate reusable pipeline with the permission superset it needs. Dispatchable by a human (workflow_dispatch) or the nightly batch (workflow_call). - uat-aws.yaml / uat-gcp.yaml: converted in place to workflow_call-only (dropped the per-workflow cron + self-concurrency; CLUSTER_CONFIG / TEST_CONFIG now come from the registry via inputs). Filenames and the keyless-cosign identity pins are unchanged; the pipeline bodies are otherwise untouched, and evidence-ingest stays nested. - uat-nightly-batch.yaml (new): the night side of the cycle — a cron that enumerates reservations from the registry and runs one main pass per reservation (data-driven matrix -> uat-run.yaml). Different reservations run in parallel; same-reservation runs serialize on the lease. - docs/contributor/uat.md (new, registered in docs/index.yml): the day/night model, how to request a run, how queuing + the one-pending limit work, and how to add a reservation. Also fixes the stale gh-workflow-run command in the CUJ1 demo and clarifies the two infra-account READMEs. Scope: exercises the main x training cell on both reservations. The version matrix (PR-3/DC5), the inference intent (DC2/DC3), and superseded-run surfacing (DC6) light up the remaining axes; the mechanism already scales to them. Signed-off-by: Nathan Hensley <[email protected]>
DC1 PR-2 (the broker cutover), building on the reservation registry + uat-broker helper from NVIDIA#1559. Every UAT run now goes through a shared dispatch surface that holds a per-reservation concurrency lease, so contending runs QUEUE instead of racing the reserved GPU hardware. - uat-run.yaml (new): the lease owner + dispatch surface. Holds the concurrency group uat-<reservation>, resolves the reservation row from infra/uat/reservations.yaml via the uat-broker helper, and invokes the cloud-appropriate reusable pipeline with the permission superset it needs. Dispatchable by a human (workflow_dispatch) or the nightly batch (workflow_call). - uat-aws.yaml / uat-gcp.yaml: converted in place to workflow_call-only (dropped the per-workflow cron + self-concurrency; CLUSTER_CONFIG / TEST_CONFIG now come from the registry via inputs). Filenames and the keyless-cosign identity pins are unchanged; the pipeline bodies are otherwise untouched, and evidence-ingest stays nested. - uat-nightly-batch.yaml (new): the night side of the cycle — a cron that enumerates reservations from the registry and runs one main pass per reservation (data-driven matrix -> uat-run.yaml). Different reservations run in parallel; same-reservation runs serialize on the lease. - docs/contributor/uat.md (new, registered in docs/index.yml): the day/night model, how to request a run, how queuing + the one-pending limit work, and how to add a reservation. Also fixes the stale gh-workflow-run command in the CUJ1 demo and clarifies the two infra-account READMEs. Scope: exercises the main x training cell on both reservations. The version matrix (PR-3/DC5), the inference intent (DC2/DC3), and superseded-run surfacing (DC6) light up the remaining axes; the mechanism already scales to them. Signed-off-by: Nathan Hensley <[email protected]>
Summary
Add the UAT reservation registry (
infra/uat/reservations.yaml) and auat-brokerhelper that resolves reservation rows and expands the nightly version-matrix schedule. Data + tooling only — no live workflow is changed.Motivation / Context
First of four PRs implementing DC1 — day/night scheduler + reservation broker (#1274), part of the comprehensive UAT epic (#1264). It establishes the data-driven foundation the rest of the broker builds on: the reservation registry is the lease's source of truth (each row's
namebecomes the GitHub Actions concurrency-group keyuat-<name>), and the helper provides the two operations the later PRs consume — resolving a row for the dispatch surface (PR-2) and expanding the time-boxed version matrix for the nightly controller (PR-3). Onboarding new reserved capacity is a one-row data edit, no code change.Fixes: N/A
Related: #1274, #1264
Type of Change
Component(s) Affected
infra/uat/reservation registry; newpkg/uatbroker+tools/uat-broker;.github/labeler.yml+CODEOWNERSImplementation Notes
pkg/corroborate+tools/corroborateprecedent: all logic lives inpkg/uatbroker(ParseRegistry/LoadRegistryFile/Validate/Lookup/Names,ExpandSchedule), with a thintools/uat-brokerCLI exposingreservationsandschedulesubcommands.Masterminds/semver/v3, not shellsort), with pre-release and non-semver tags dropped and the oldest dropped first when truncating — so the nightly controller sheds the oldest releases when its time-box closes.reservationsemitsGITHUB_OUTPUT-stylekey=valuelines so PR-2's resolve job is simplyuat-broker reservations --name … >> "$GITHUB_OUTPUT";scheduleemits JSON from a stdin tag list.0ok /2invalid-request /3not-found) so workflows can branch on failure mode.reservations.yamlis intentionally not CODEOWNERS-gated (kept under the default owner) so "new infra extends by data edit" stays low-friction; only the Go helper is maintainer-owned.gpu-countis forward-looking metadata (consumed by DC2) and is validated positive.Testing
make qualifygreen end-to-end: tests pass, aggregate coverage 77.5% ≥ 75%, lint clean (golangci-lint v2.12.2), e2e 23/0, grype reports no vulnerabilities.pkg/uatbroker95.9%,tools/uat-broker87.3% — table-driven tests cover registry validation, lookup, schedule ordering/filtering/drop-oldest, the CLI subcommands and their exit codes, and a guard that the committed registry stays valid.Risk Assessment
Rollout notes: No runtime/behavior change — the helper is not wired into any workflow until PR-2 (the broker cutover). Safe to merge independently.
Checklist
make testwith-race)make lint)git commit -S)