Two Phase-2 follow-ups from NVIDIA#698, batched together because both are
small chart-pin changes coupled to a single non-pin tweak each.

Components bumped:

  kai-scheduler           v0.13.0 -> v0.14.1
  kubeflow-trainer        2.1.0   -> 2.2.0

kai-scheduler — chart bump and OCI registry namespace migration
(NVIDIA#698 follow-up NVIDIA#3):

KAI-Scheduler was transferred from the NVIDIA org to its own
`kai-scheduler` org and chart publishing moved with it. The old
namespace `oci://ghcr.io/nvidia/kai-scheduler` is frozen at v0.13.0;
the new namespace `oci://ghcr.io/kai-scheduler/kai-scheduler` carries
the full release stream. v0.14.1 verified clean: 41/41 templates and
identical kinds/counts vs v0.13.0; only values.yaml addition is an
opt-in `vpa:` block (`enabled: false` default). Our customizations
(`global.tolerations`, `admission.gpuPodRuntimeClassName`,
`postCleanup.enabled`) all still apply unchanged.

kubeflow-trainer — chart bump, validator fallback URL update, and
demo migration to the new RuntimePatches API
(NVIDIA#698 follow-up NVIDIA#5):

The chart pin in `recipes/registry.yaml` and the hardcoded fallback
archive URL in `validators/performance/trainer_lifecycle.go` are
coupled: the validator's no-CRD install path downloads
`https://github.com/kubeflow/trainer/archive/refs/tags/<version>.tar.gz`
and applies the `manifests/overlays/manager` kustomize. If the chart
pin moves but the validator URL doesn't, the fallback installs the
old release while the chart deploys the new one. v2.2.0 archive
layout is unchanged from v2.1.0 (same `manifests/overlays/manager`
kustomize, same `trainjobs.trainer.kubeflow.org/v1alpha1` CRD); the
only difference is the controller-manager image tag.

v2.2.0 ships a breaking API change to TrainJob: `podTemplateOverrides`
is replaced by `runtimePatches` (kubeflow/trainer#3309). The CRD still
admits the old field name for compat, but the controller no longer
applies it — pods come out with no override fields, and on AICR's
tainted GPU nodes the `tolerations: [{operator: Exists}]` shorthand
the demo previously used silently no-ops, leaving pods Pending.

The `pytorch-mnist` demo TrainJob in `demos/cuj1-eks.md` and
`demos/cuj1-gke.md` is migrated to the new shape:

  spec:
    runtimePatches:
      - manager: aicr.nvidia.com/demo
        trainingRuntimeSpec:
          template:
            spec:
              replicatedJobs:
                - name: node
                  template:
                    spec:
                      template:
                        spec:
                          nodeSelector: {nodeGroup: gpu-worker}
                          tolerations:
                            - {key: dedicated, operator: Equal,
                               value: worker-workload, effect: NoSchedule}
                            - {key: dedicated, operator: Equal,
                               value: worker-workload, effect: NoExecute}

Validated end-to-end on a real EKS H100 cluster (aicr1) post-upgrade:
TrainJob admitted, pod scheduled to the GPU node with the expected
tolerations + nodeSelector, training completed in 2m39s with
accuracy=0.7413 (matches pre-upgrade baseline).

Verified locally:

  $ helm pull oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler --version v0.14.1
  $ helm pull oci://ghcr.io/kubeflow/charts/kubeflow-trainer --version 2.2.0
  $ make tidy && make lint && go test -count=1 ./pkg/recipe/... ./validators/performance/...

Two Phase-2 follow-ups from NVIDIA#698, batched together because both are
small chart-pin changes coupled to a single non-pin tweak each.

Components bumped:

  kai-scheduler           v0.13.0 -> v0.14.1
  kubeflow-trainer        2.1.0   -> 2.2.0

kai-scheduler — chart bump and OCI registry namespace migration
(NVIDIA#698 follow-up NVIDIA#3):

KAI-Scheduler was transferred from the NVIDIA org to its own
`kai-scheduler` org and chart publishing moved with it. The old
namespace `oci://ghcr.io/nvidia/kai-scheduler` is frozen at v0.13.0;
the new namespace `oci://ghcr.io/kai-scheduler/kai-scheduler` carries
the full release stream. v0.14.1 verified clean: 41/41 templates and
identical kinds/counts vs v0.13.0; only values.yaml addition is an
opt-in `vpa:` block (`enabled: false` default). Our customizations
(`global.tolerations`, `admission.gpuPodRuntimeClassName`,
`postCleanup.enabled`) all still apply unchanged.

kubeflow-trainer — chart bump, validator fallback URL update, and
demo migration to the new RuntimePatches API
(NVIDIA#698 follow-up NVIDIA#5):

The chart pin in `recipes/registry.yaml` and the hardcoded fallback
archive URL in `validators/performance/trainer_lifecycle.go` are
coupled: the validator's no-CRD install path downloads
`https://github.com/kubeflow/trainer/archive/refs/tags/<version>.tar.gz`
and applies the `manifests/overlays/manager` kustomize. If the chart
pin moves but the validator URL doesn't, the fallback installs the
old release while the chart deploys the new one. v2.2.0 archive
layout is unchanged from v2.1.0 (same `manifests/overlays/manager`
kustomize, same `trainjobs.trainer.kubeflow.org/v1alpha1` CRD); the
only difference is the controller-manager image tag.

v2.2.0 ships a breaking API change to TrainJob: `podTemplateOverrides`
is replaced by `runtimePatches` (kubeflow/trainer#3309). The CRD still
admits the old field name for compat, but the controller no longer
applies it — pods come out with no override fields, and on AICR's
tainted GPU nodes the `tolerations: [{operator: Exists}]` shorthand
the demo previously used silently no-ops, leaving pods Pending.

The `pytorch-mnist` demo TrainJob in `demos/cuj1-eks.md` and
`demos/cuj1-gke.md` is migrated to the new shape:

  spec:
    runtimePatches:
      - manager: aicr.nvidia.com/demo
        trainingRuntimeSpec:
          template:
            spec:
              replicatedJobs:
                - name: node
                  template:
                    spec:
                      template:
                        spec:
                          nodeSelector: {nodeGroup: gpu-worker}
                          tolerations:
                            - {key: dedicated, operator: Equal,
                               value: worker-workload, effect: NoSchedule}
                            - {key: dedicated, operator: Equal,
                               value: worker-workload, effect: NoExecute}

Validated end-to-end on a real EKS H100 cluster (aicr1) post-upgrade:
TrainJob admitted, pod scheduled to the GPU node with the expected
tolerations + nodeSelector, training completed in 2m39s with
accuracy=0.7413 (matches pre-upgrade baseline).

Verified locally:

  $ helm pull oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler --version v0.14.1
  $ helm pull oci://ghcr.io/kubeflow/charts/kubeflow-trainer --version 2.2.0
  $ make tidy && make lint && go test -count=1 ./pkg/recipe/... ./validators/performance/...

Two Phase-2 follow-ups from NVIDIA#698, batched together because both are
small chart-pin changes coupled to a single non-pin tweak each.

Components bumped:

  kai-scheduler           v0.13.0 -> v0.14.1
  kubeflow-trainer        2.1.0   -> 2.2.0

kai-scheduler — chart bump and OCI registry namespace migration
(NVIDIA#698 follow-up NVIDIA#3):

KAI-Scheduler was transferred from the NVIDIA org to its own
`kai-scheduler` org and chart publishing moved with it. The old
namespace `oci://ghcr.io/nvidia/kai-scheduler` is frozen at v0.13.0;
the new namespace `oci://ghcr.io/kai-scheduler/kai-scheduler` carries
the full release stream. v0.14.1 verified clean: 41/41 templates and
identical kinds/counts vs v0.13.0; only values.yaml addition is an
opt-in `vpa:` block (`enabled: false` default). Our customizations
(`global.tolerations`, `admission.gpuPodRuntimeClassName`,
`postCleanup.enabled`) all still apply unchanged.

kubeflow-trainer — chart bump, validator fallback URL update, demo
migration to RuntimePatches, and ClusterTrainingRuntime alignment
(NVIDIA#698 follow-up NVIDIA#5):

The chart pin in `recipes/registry.yaml` and the hardcoded fallback
archive URL in `validators/performance/trainer_lifecycle.go` are
coupled: the validator's no-CRD install path downloads
`https://github.com/kubeflow/trainer/archive/refs/tags/<version>.tar.gz`
and applies the `manifests/overlays/manager` kustomize. If the chart
pin moves but the validator URL doesn't, the fallback installs the
old release while the chart deploys the new one. v2.2.0 archive
layout is unchanged from v2.1.0 (same `manifests/overlays/manager`
kustomize, same `trainjobs.trainer.kubeflow.org/v1alpha1` CRD); the
only difference is the controller-manager image tag.

v2.2.0 ships two breaking API changes that touch AICR:

  1. PodTemplateOverrides → RuntimePatches (kubeflow/trainer#3309).
     The CRD still admits the old field for compat but the v2.2
     controller no longer applies it. The pytorch-mnist demo TrainJob
     in `demos/cuj1-eks.md` and `demos/cuj1-gke.md` is migrated to
     the `runtimePatches` shape with `manager: aicr.nvidia.com/demo`
     and explicit per-cluster scheduling (the EKS demo carries the
     AICR-standard `dedicated=worker-workload` tolerations + NoExecute
     effect; the GKE demo carries `dedicated=gpu-workload:NoSchedule`
     and `nvidia.com/gpu=present:NoSchedule` to match the rest of the
     GKE flow).

  2. mlPolicy.torch.numProcPerNode removal (kubeflow/trainer#3239).
     Upstream removed the field from the Torch policy because it now
     infers parallelism from the container's `nvidia.com/gpu` limit.
     `mlPolicy.mpi.numProcPerNode` is unaffected, so the existing MPI
     test fixtures stay as-is. AICR's `torch-distributed`
     ClusterTrainingRuntime is updated from
     `mlPolicy.torch: { numProcPerNode: auto }` to
     `mlPolicy.torch: {}`, matching the v2.2.0 reference runtime.

Validated end-to-end on a real EKS H100 cluster (aicr1) post-upgrade:
demo TrainJob admitted, pod scheduled with the migrated runtimePatches,
training completed in 2m39s with accuracy=0.7413 (matches pre-upgrade
baseline). 2-replica Deployment with `schedulerName: kai-scheduler` +
DRA `ResourceClaimTemplate` referencing `gpu.nvidia.com` also
scheduled cleanly with `priorityClassName: train` (each replica got
its own H100 via DRA).

Verified locally:

  $ helm pull oci://ghcr.io/kai-scheduler/kai-scheduler/kai-scheduler --version v0.14.1
  $ helm pull oci://ghcr.io/kubeflow/charts/kubeflow-trainer --version 2.2.0
  $ make tidy && make lint && go test -count=1 ./pkg/recipe/... ./validators/performance/... ./pkg/bundler/deployer/helm/...

The pytorch demo TrainJobs in demos/cuj1-{eks,gke}.md carry per-cluster
scheduling boilerplate (`podTemplateOverrides` with cluster-specific
tolerations) so the resulting pods land on AICR's tainted GPU nodes.
Each TrainJob author has to repeat this; each demo has to be edited
per-cluster vocabulary; and the override mechanism keeps changing
upstream (PodTemplateOverrides was deprecated in v2.1, replaced by
RuntimePatches in v2.2 — kubeflow/trainer#3309).

Move the per-cluster scheduling into the runtime instead. AICR's
existing `nodeScheduling.accelerated` bundler injection (already used
by gpu-operator, nfd, nodewright-customizations, kgateway) writes the
CLI flag values into the chart's values.yaml at the listed paths.
kubeflow-trainer was the only manifestFiles-using component without an
`accelerated:` block. This commit adds it and templates the
torch-distributed ClusterTrainingRuntime to consume the injected
values, mirroring nodewright-customizations/manifests/tuning.yaml.

Three coordinated changes:

1. recipes/registry.yaml — add `nodeScheduling.accelerated` block to
   the kubeflow-trainer entry. Targets top-level keys
   `acceleratedNodeSelector` and `acceleratedTolerations`.

2. recipes/components/kubeflow-trainer/manifests/
   torch-distributed-cluster-training-runtime.yaml — replace the
   static pod-spec scheduling region with Helm template directives:

       {{- $kft := index .Values "kubeflow-trainer" }}
       {{- with $kft.acceleratedNodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 20 }}
       {{- end }}
       {{- with $kft.acceleratedTolerations }}
       tolerations:
         {{- toYaml . | nindent 20 }}
       {{- end }}

   `index .Values "kubeflow-trainer"` matches the bundler's
   `manifest.RenderInput.Values` shape (values nested under
   ComponentName). The bundler renders this template at bundle time —
   the artifact in `bundle/<NNN>-kubeflow-trainer-post/templates/`
   is plain YAML with concrete values substituted.

3. demos/cuj1-eks.md and demos/cuj1-gke.md — drop the entire
   `podTemplateOverrides` block. Demo TrainJob is just `trainer:` +
   `runtimeRef:`.

API-version-agnostic: works on kubeflow-trainer v2.1 (PodTemplateOverrides
era) and v2.2+ (RuntimePatches era) identically, because the TrainJob
no longer overrides anything — the runtime carries the scheduling.

Validated end-to-end on a real EKS H100 cluster:
helm-upgrade kubeflow-trainer-post → CTR live with baked tolerations
+ nodeSelector → bare pytorch-mnist TrainJob admits, schedules with
the correct tolerations + nodeSelector inherited from the runtime,
trains to completion (accuracy=0.7424 in 21s).

`pkg/recipe.TestManifestHelmHooksRequired` still passes — the
`helm.sh/hook` annotations are preserved.

The pytorch demo TrainJobs in demos/cuj1-{eks,gke}.md carry per-cluster
scheduling boilerplate (`podTemplateOverrides` with cluster-specific
tolerations) so the resulting pods land on AICR's tainted GPU nodes.
Each TrainJob author has to repeat this; each demo has to be edited
per-cluster vocabulary; and the override mechanism keeps changing
upstream (PodTemplateOverrides was deprecated in v2.1, replaced by
RuntimePatches in v2.2 — kubeflow/trainer#3309).

Move the per-cluster scheduling into the runtime instead. AICR's
existing `nodeScheduling.accelerated` bundler injection (already used
by gpu-operator, nfd, nodewright-customizations, kgateway) writes the
CLI flag values into the chart's values.yaml at the listed paths.
kubeflow-trainer was the only manifestFiles-using component without an
`accelerated:` block. This commit adds it and templates the
torch-distributed ClusterTrainingRuntime to consume the injected
values, mirroring nodewright-customizations/manifests/tuning.yaml.

Three coordinated changes:

1. recipes/registry.yaml — add `nodeScheduling.accelerated` block to
   the kubeflow-trainer entry. Targets top-level keys
   `acceleratedNodeSelector` and `acceleratedTolerations`.

2. recipes/components/kubeflow-trainer/manifests/
   torch-distributed-cluster-training-runtime.yaml — replace the
   static pod-spec scheduling region with Helm template directives:

       {{- $kft := index .Values "kubeflow-trainer" }}
       {{- with $kft.acceleratedNodeSelector }}
       nodeSelector:
         {{- toYaml . | nindent 20 }}
       {{- end }}
       {{- with $kft.acceleratedTolerations }}
       tolerations:
         {{- toYaml . | nindent 20 }}
       {{- end }}

   `index .Values "kubeflow-trainer"` matches the bundler's
   `manifest.RenderInput.Values` shape (values nested under
   ComponentName). The bundler renders this template at bundle time —
   the artifact in `bundle/<NNN>-kubeflow-trainer-post/templates/`
   is plain YAML with concrete values substituted.

3. demos/cuj1-eks.md and demos/cuj1-gke.md — drop the entire
   `podTemplateOverrides` block. Demo TrainJob is just `trainer:` +
   `runtimeRef:`.

API-version-agnostic: works on kubeflow-trainer v2.1 (PodTemplateOverrides
era) and v2.2+ (RuntimePatches era) identically, because the TrainJob
no longer overrides anything — the runtime carries the scheduling.

Validated end-to-end on a real EKS H100 cluster:
helm-upgrade kubeflow-trainer-post → CTR live with baked tolerations
+ nodeSelector → bare pytorch-mnist TrainJob admits, schedules with
the correct tolerations + nodeSelector inherited from the runtime,
trains to completion (accuracy=0.7424 in 21s).

`pkg/recipe.TestManifestHelmHooksRequired` still passes — the
`helm.sh/hook` annotations are preserved.

…flow#3309)

* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Update goal with RuntimePatches API

Signed-off-by: Andrey Velichkevich <[email protected]>

* Add diagram for RuntimePatches

Signed-off-by: Andrey Velichkevich <[email protected]>

* feat(api): Replace PodTemplateOverrides with RuntimePatches API

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix kube lint errors

Signed-off-by: Antonin Stefanutti <[email protected]>

* Update Python models

Signed-off-by: Antonin Stefanutti <[email protected]>

* Enable maxlength Kube-API-Linter rule

Signed-off-by: Antonin Stefanutti <[email protected]>

* Restore manger field IsDomainPrefixedPath validation

Signed-off-by: Antonin Stefanutti <[email protected]>

* Fix integration tests

Signed-off-by: Antonin Stefanutti <[email protected]>

* Apply StrategicMergePatch to batchv1.JobTemplateSpec

Signed-off-by: Antonin Stefanutti <[email protected]>

---------

Signed-off-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Antonin Stefanutti <[email protected]>
Co-authored-by: Andrey Velichkevich <[email protected]>
Signed-off-by: Pulkit Agrawal <[email protected]>

Exercises wider TrainJob API surface in the upgrade test so spec
integrity checks can detect API migrations like kubeflow/trainer#3309
(PodTemplateOverrides → RuntimePatches).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Exercises wider TrainJob API surface in the upgrade test so spec
integrity checks can detect API migrations like kubeflow/trainer#3309
(PodTemplateOverrides → RuntimePatches).

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>