Add GB200 overlays for OKE by OguzPastirmaci · Pull Request #497 · NVIDIA/aicr

OguzPastirmaci · 2026-04-06T22:20:34Z

Summary

Add GB200 recipes for OKE using BM.GPU.GB200-v3.4 shape.

Motivation / Context

Fixes:
Related:

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Refactoring (no functional changes)
Build/CI/tooling

Component(s) Affected

CLI (cmd/aicr, pkg/cli)
API server (cmd/aicrd, pkg/api, pkg/server)
Recipe engine / data (pkg/recipe)
Bundlers (pkg/bundler, pkg/component/*)
Collectors / snapshotter (pkg/collector, pkg/snapshotter)
Validator (pkg/validator)
Core libraries (pkg/errors, pkg/k8s)
Docs/examples (docs/, examples/)
Other: ____________

Implementation Notes

Testing

# Commands run (prefer `make qualify` for non-trivial changes)
make qualify

Risk Assessment

Low — Isolated change, well-tested, easy to revert
Medium — Touches multiple components or has broader impact
High — Breaking change, affects critical paths, or complex rollout

Rollout notes:

Checklist

Tests pass locally (make test with -race)
Linter passes (make lint)
I did not skip/disable tests to make CI green
I added/updated tests for new functionality
I updated docs if user-facing behavior changed
Changes follow existing patterns in the codebase
Commits are cryptographically signed (git commit -S) — GPG signing info

copy-pr-bot · 2026-04-06T22:20:38Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-04-06T22:20:43Z

Welcome to AICR, @OguzPastirmaci! Thanks for your first pull request.

Before review, please ensure:

All commits are signed off per the DCO
CI checks pass (tests, lint, security scan)
The PR description explains the why behind your changes

A maintainer will review this soon.

yuanchen8911 · 2026-04-06T22:53:23Z

Review

Findings

Medium: gb200-oke training validation is placed in the Ubuntu leaf instead of the intent parent, which is inconsistent with the validation lift-up that just landed in refactor(recipes): lift validation blocks from ubuntu leaves to intent overlays #493. The conformance block in gb200-oke-ubuntu-training.yaml should move to gb200-oke-training.yaml so validation remains OS-independent and follows the current overlay pattern.

Open questions / notes

Confirm that OKE should constrain Ubuntu to 22.04 rather than 24.04; this would be the only Ubuntu service family using 22.04.
cdi.enabled: true appears in both values-oke.yaml and values-oke-training.yaml. That is redundant for training overlays, though harmless.
No Skyhook customization is consistent with the AKS-style pattern.
nvidiaDriverRoot: / and the worker-node affinity look coherent for host-installed OKE drivers.

yuanchen8911

thanks for the PR. left some comments.

ayuskauskas · 2026-04-06T23:37:55Z

@OguzPastirmaci would you be interested in working with me to add Nodewright optimizations here like I have done with AWS? There are two levels we could do:

Being able to bring a vanilla ubuntu OKE worker up to spec like with AWS we set the kernel, install EFA and setup a raid for container storage. nvidia-setup
Add optimizations, we could do basic nvidia tunings or an OKE specific one. nvidia-tuned

I would just need access to a node for an hour or two to make sure the package work correctly or I could work with you via a call or asynchronously. Looking forward to enabling Nodewright for OKE!

OguzPastirmaci · 2026-04-07T06:17:30Z

thanks for the PR. left some comments.

Thanks @yuanchen8911, updated based on your feedback.

OguzPastirmaci · 2026-04-07T06:20:27Z

@OguzPastirmaci would you be interested in working with me to add Nodewright optimizations here like I have done with AWS? There are two levels we could do:

Being able to bring a vanilla ubuntu OKE worker up to spec like with AWS we set the kernel, install EFA and setup a raid for container storage. nvidia-setup

Add optimizations, we could do basic nvidia tunings or an OKE specific one. nvidia-tuned

I would just need access to a node for an hour or two to make sure the package work correctly or I could work with you via a call or asynchronously. Looking forward to enabling Nodewright for OKE!

@ayuskauskas happy to help. Could you elaborate what you mean by "bringing a vanilla OKE worker up to spec"?

Here's the repo with the Packer templates that we use to build our HPC/GPU images: https://github.com/oracle-quickstart/oci-hpc-images

ayuskauskas · 2026-04-07T16:13:27Z

@OguzPastirmaci would you be interested in working with me to add Nodewright optimizations here like I have done with AWS? There are two levels we could do:
...

@ayuskauskas happy to help. Could you elaborate what you mean by "bringing a vanilla OKE worker up to spec"?

Here's the repo with the Packer templates that we use to build our HPC/GPU images: https://github.com/oracle-quickstart/oci-hpc-images

bringing a vanilla OKE worker up to spec

I mean if I start an OKE cluster and select an off the shelf worker image and install/configure all the things to make it support AI workloads. That repository you linked seems like exactly that so would make a great starting point. There are a few options to take:

Transcribe the ansible playbooks into shellscripts to use in a package like nvidia-setup
Create a new package (oke-setup for example) in nodewright-packages that clones your repo and runs the ansible playbooks directly whenever the package runs.
Your repo could publish a nodewright-package that does the same as the above.

I have a few questions:

I see you setup lustre in that playbook we chose not to do that in our initial recipes as it is dependent on how the infrastructure is setup. In OKE is lustre the only way and so making that choice for users isn't restrictive?
Can we turn parts of the ansible playbook off? Things like mellanox, gpu driver, dcgm etc are better managed by Network and GPU Operator in an aicr style cluster.

yuanchen8911 · 2026-04-07T16:27:23Z

Thanks for the updates. The structure follows existing patterns, validation placement is now consistent with #493, and the Ubuntu version aligns with other services.Please rebase.

yuanchen8911

KWOK Tests Failing: DRA Driver Scheduling Regression

All OKE KWOK tests (both Tier 1 and Tier 2) are failing because nvidia-dra-driver-gpu-controller can't be scheduled on any KWOK node.

Root cause: This PR adds a new OKE-specific values file (recipes/components/nvidia-dra-driver-gpu/values-oke.yaml) that hardcodes a hard node affinity requiring node-role.kubernetes.io/node:

controller:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: "node-role.kubernetes.io/node"
            operator: "Exists"

The nvidia-dra-driver-gpu component already exists in the codebase and works fine — this new OKE-specific values file introduces the regression. KWOK simulated GPU worker nodes don't have the node-role.kubernetes.io/node label, resulting in 0/7 nodes are available: 7 node(s) didn't match Pod's node affinity/selector for every OKE recipe.

Suggested fixes (pick one):

Add the node-role.kubernetes.io/node label to the KWOK simulated node templates (in kwok/ config files)
Use the system nodeScheduling mechanism from the registry instead of hardcoding affinity in values, so the KWOK test harness can set it appropriately
Use preferredDuringScheduling (soft) affinity instead of requiredDuringScheduling (hard)

yuanchen8911 · 2026-04-07T18:51:16Z

/lgtm
@OguzPastirmaci , please fix the affinity issue.

OguzPastirmaci · 2026-04-07T19:23:55Z

/lgtm
@OguzPastirmaci , please fix the affinity issue.

Updated and confirmed with a local Kwok run.

OguzPastirmaci · 2026-04-07T19:29:16Z

@OguzPastirmaci would you be interested in working with me to add Nodewright optimizations here like I have done with AWS? There are two levels we could do:
...

@ayuskauskas happy to help. Could you elaborate what you mean by "bringing a vanilla OKE worker up to spec"?
Here's the repo with the Packer templates that we use to build our HPC/GPU images: https://github.com/oracle-quickstart/oci-hpc-images

bringing a vanilla OKE worker up to spec

I mean if I start an OKE cluster and select an off the shelf worker image and install/configure all the things to make it support AI workloads. That repository you linked seems like exactly that so would make a great starting point. There are a few options to take:

Transcribe the ansible playbooks into shellscripts to use in a package like nvidia-setup

Create a new package (oke-setup for example) in nodewright-packages that clones your repo and runs the ansible playbooks directly whenever the package runs.

Your repo could publish a nodewright-package that does the same as the above.

I have a few questions:

I see you setup lustre in that playbook we chose not to do that in our initial recipes as it is dependent on how the infrastructure is setup. In OKE is lustre the only way and so making that choice for users isn't restrictive?

Can we turn parts of the ansible playbook off? Things like mellanox, gpu driver, dcgm etc are better managed by Network and GPU Operator in an aicr style cluster.

I think what we have wouldn't work easily at this stage, as our builds take more than an hour. We pre-install the GPU & OFED drivers and other things like the Lustre client in our images.

We install the Lustre client so users won't need to if they prefer to use our managed Lustre service.
We support GPU Operator and Network Operator, but we don't support running the GPU and network drivers as containers today.

yuanchen8911 · 2026-04-07T20:29:22Z

Please rebase and squash.

Add OKE service family support with GB200 accelerator overlays for training and inference workloads. Includes GPU Operator and DRA driver configurations for OKE's host-installed driver model.

yuanchen8911

/lgtm

OguzPastirmaci requested a review from a team as a code owner April 6, 2026 22:20

github-actions Bot added area/recipes size/XL labels Apr 6, 2026

yuanchen8911 requested review from yuanchen8911 April 6, 2026 22:53

yuanchen8911 requested changes Apr 6, 2026

View reviewed changes

mchmarny assigned OguzPastirmaci Apr 7, 2026

yuanchen8911 previously approved these changes Apr 7, 2026

View reviewed changes

OguzPastirmaci dismissed yuanchen8911’s stale review via 5f34df5 April 7, 2026 16:32

OguzPastirmaci force-pushed the oke-gb200 branch from 7785913 to 5f34df5 Compare April 7, 2026 16:32

yuanchen8911 approved these changes Apr 7, 2026

View reviewed changes

yuanchen8911 requested changes Apr 7, 2026

View reviewed changes

OguzPastirmaci requested a review from yuanchen8911 April 7, 2026 19:25

OguzPastirmaci force-pushed the oke-gb200 branch from 7ab2e70 to c5756a9 Compare April 7, 2026 20:35

ayuskauskas reviewed Apr 7, 2026

View reviewed changes

Comment thread recipes/components/gpu-operator/values-oke.yaml

Add GB200 overlays for OKE

aadd07c

Add OKE service family support with GB200 accelerator overlays for training and inference workloads. Includes GPU Operator and DRA driver configurations for OKE's host-installed driver model.

OguzPastirmaci force-pushed the oke-gb200 branch from c5756a9 to aadd07c Compare April 7, 2026 21:13

yuanchen8911 requested a review from ayuskauskas April 7, 2026 21:37

yuanchen8911 approved these changes Apr 7, 2026

View reviewed changes

yuanchen8911 merged commit fa73b5b into NVIDIA:main Apr 7, 2026
38 checks passed

OguzPastirmaci deleted the oke-gb200 branch April 7, 2026 21:52

yuanchen8911 mentioned this pull request Apr 7, 2026

feat(recipe): add mixin composition for OS and platform fragments #501

Merged

23 tasks

Uh oh!

Conversation

OguzPastirmaci commented Apr 6, 2026

Summary

Motivation / Context

Type of Change

Component(s) Affected

Implementation Notes

Testing

Risk Assessment

Checklist

Uh oh!

copy-pr-bot Bot commented Apr 6, 2026

Uh oh!

github-actions Bot commented Apr 6, 2026

Uh oh!

yuanchen8911 commented Apr 6, 2026

Review

Findings

Open questions / notes

Uh oh!

yuanchen8911 left a comment

Choose a reason for hiding this comment

Uh oh!

ayuskauskas commented Apr 6, 2026

Uh oh!

OguzPastirmaci commented Apr 7, 2026

Uh oh!

OguzPastirmaci commented Apr 7, 2026

Uh oh!

ayuskauskas commented Apr 7, 2026

Uh oh!

yuanchen8911 commented Apr 7, 2026

Uh oh!

yuanchen8911 left a comment

Choose a reason for hiding this comment

KWOK Tests Failing: DRA Driver Scheduling Regression

Uh oh!

yuanchen8911 commented Apr 7, 2026

Uh oh!

OguzPastirmaci commented Apr 7, 2026

Uh oh!

OguzPastirmaci commented Apr 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yuanchen8911 commented Apr 7, 2026

Uh oh!

Uh oh!

yuanchen8911 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

OguzPastirmaci commented Apr 7, 2026 •

edited

Loading