Summary

Centralize how the dynamo runtime image (dynamo-frontend / vllm-runtime) is referenced across the inference-perf path, closing three related gaps with one refactor:

1. Single source of truth for the image (repo + tag) — today it's scattered string literals in multiple files.
2. Registry-override parity — AICR_VALIDATOR_IMAGE_REGISTRY rewrites the AIPerf image but not the dynamo runtime images, so a registry override silently leaves them pointing at nvcr.io. (absorbs inference-perf: route cacheWorkerImage through ResolveImage for registry-override parity #1159*)*
3. Drift guard — the runtime image tag must not skew from the dynamo-platform operator chart version (the 0.9.0-vs-1.0.2 skew that caused inference-perf: validator times out on healthy cluster when dynamo frontend discovery bootstrap races (false negative) #1192).

This issue supersedes #1159 — the natural fix is a single refactor of how the image is referenced, so doing them separately would mean two PRs fighting over the same code.

Background

• Version skew (the inference-perf: validator times out on healthy cluster when dynamo frontend discovery bootstrap races (false negative) #1192 root cause): AICR pins the dynamo version in two independent places — the dynamo-platform operator chart (recipes/registry.yaml, bumped to 1.0.2) and the runtime image tags (hardcoded literals, left at 0.9.0). Nothing tied them together, so a 1.0.2 operator scheduled 0.9.0 pods carrying a known panic.
• **Registry parity gap (**inference-perf: route cacheWorkerImage through ResolveImage for registry-override parity #1159): cacheWorkerImage and the DGD worker/frontend images are pinned to nvcr.io and not routed through catalog.ResolveImage, while the AIPerf image is. So AICR_VALIDATOR_IMAGE_REGISTRY redirects AIPerf but not the runtime images — observed repeatedly in testing (validator image came from a mirror/ECR while the dynamo runtime still pulled from nvcr.io).

Two surfaces — different mechanisms (important)

The dynamo runtime image is referenced in two places that need different override mechanisms:

Proposed change

1. Single source of truth for the dynamo runtime image — repo + tag (e.g. pkg/defaults.DynamoRuntimeImageRepo / DynamoRuntimeImageTag).
2. Use it for cacheWorkerImage (via ResolveImage) and substitute it into the DGD template (e.g. ${DYNAMO_RUNTIME_IMAGE} / ${DYNAMO_RUNTIME_TAG}) honoring the registry override — so one AICR_VALIDATOR_IMAGE_REGISTRY redirects all images the inference-perf path pulls, matching AIPerf.
3. Drift guard (make check wired into make lint/qualify) asserting the runtime image tag == dynamo-platform chart defaultVersion in recipes/registry.yaml. Keep TestCacheWorkerImageMatchesTemplate.

Scope note (from #1159) — not air-gap enablement

Registry-override parity is necessary but not sufficient for air-gap: the populate Job still huggingface_hub.snapshot_download()s weights from huggingface.co. Offline weights (pre-staged / internal registry / Dynamo ModelExpress) are out of scope here.

Acceptance criteria

• Single source of truth for the dynamo runtime image repo+tag (no scattered literals).
• cacheWorkerImage and the DGD-template images both honor AICR_VALIDATOR_IMAGE_REGISTRY (a registry override redirects every inference-perf image, not just AIPerf).
• make check fails if the runtime image tag ≠ dynamo-platform chart version (drift guard); deliberate mismatch is caught locally.
• Existing TestCacheWorkerImageMatchesTemplate retained.
• Existing dynamo behavior unchanged; make qualify green.

Supersedes: #1159
Related: #1192 (version skew this prevents), #1133 (introduced cacheWorkerImage)