Summary

cacheWorkerImage (nvcr.io/nvidia/ai-dynamo/vllm-runtime:<tag>) — used by the
inference-perf model-cache populate Job and the Dynamo inference workers — is
pinned to nvcr.io and is not routed through catalog.ResolveImage. The
AIPerf benchmark image is, so AICR_VALIDATOR_IMAGE_REGISTRY rewrites the
AIPerf image but silently leaves the vLLM-runtime image pointing at nvcr.io.

This is a registry-override consistency gap, not air-gap enablement (see the
scope note below). It is a pre-existing pattern — the Dynamo worker image was
already pinned in testdata/inference/dynamo-deployment.yaml; PR #1133 only
surfaced it by adding cacheWorkerImage. Drift between the two is guarded by
TestCacheWorkerImageMatchesTemplate. Documented inline in
validators/performance/model_cache.go.

Proposed change (scoped)

Route cacheWorkerImage (and, ideally, the Dynamo worker image used in the
template) through catalog.ResolveImage — or expose a dedicated override — so a
single AICR_VALIDATOR_IMAGE_REGISTRY consistently redirects all images the
inference-perf path pulls, matching how the AIPerf image already behaves. Keep
the TestCacheWorkerImageMatchesTemplate drift guard.

Scope note — this is necessary but NOT sufficient for air-gap

Registry-override parity alone does not make inference-perf air-gap-capable.
The populate Job runs huggingface_hub.snapshot_download(), which reaches
huggingface.co for the model weights — an inherent internet dependency that
dominates the air-gap story. Full air-gapped inference-perf would additionally
need a model-weights source that works offline (pre-staged weights / an internal
model registry / Dynamo ModelExpress). That is out of scope here and, given that
the validator framework already assumes registry connectivity (it pulls
validator images from ghcr.io), may be a separate decision about whether
air-gapped validation is a goal at all.

Raised in review of #1133 (Nathan Hensley, Mark Chmarny). Current state is fine
to ship.

Refs #1133