KubeFlow patches#94
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a Kubeflow Trainer patch to include an external manifest so Eidos can install a torch-distributed ClusterTrainingRuntime alongside the kubeflow-trainer Helm release, and updates the CUJ1 demo to reference that runtime.
Changes:
- Adds
manifestFilesentries forkubeflow-trainerin the Kubeflow training and inference overlays. - Introduces a new
ClusterTrainingRuntimemanifest fortorch-distributed. - Updates the CUJ1 demo TrainJob example to use the
torch-distributedruntime and a different example image/path.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| pkg/recipe/data/overlays/h100-eks-ubuntu-training-kubeflow.yaml | Adds the kubeflow-trainer extra manifest file for Kubeflow training overlay. |
| pkg/recipe/data/overlays/h100-eks-ubuntu-inference.yaml | Adds the same kubeflow-trainer extra manifest file for inference overlay. |
| pkg/recipe/data/components/kubeflow-trainer/manifests/torch-distributed-cluster-training-runtime.yaml | New cluster-scoped runtime manifest intended to be installed with kubeflow-trainer. |
| examples/demos/cuj1.md | Updates demo TrainJob snippet to reference torch-distributed runtime and adjust image/paths/resources. |
coffeepac
force-pushed
the
kubeflow-works
branch
from
62d9ced to
1e9eccd
Compare
coffeepac
force-pushed
the
kubeflow-works
branch
from
dd5616e to
510ce78
Compare
coffeepac
force-pushed
the
kubeflow-works
branch
from
68e167c to
e48d0ab
Compare
mchmarny
approved these changes
Feb 11, 2026
10 tasks
yuanchen8911
added a commit
to yuanchen8911/aicr
that referenced
this pull request
Apr 30, 2026
The torch-distributed ClusterTrainingRuntime declared its delete policy as hook-delete-policy: before-hook-creation,hook-succeeded. Helm interprets hook-succeeded literally — after the post-install hook runs successfully, the resource is deleted. So every install would create the CTR and immediately delete it, leaving the cluster with no torch-distributed runtime. Symptom: any TrainJob referencing runtimeRef.name=torch-distributed (e.g. the pytorch-mnist demo in demos/cuj1-eks.md) is rejected by the trainer admission webhook with "ClusterTrainingRuntime torch-distributed not found". Fix: drop only ,hook-succeeded. Keep the helm.sh/hook annotation (project convention enforced by pkg/recipe.TestManifestHelmHooksRequired) and before-hook-creation (idempotent re-install). Without hook-succeeded, the CTR persists between installs. Verified end-to-end on a real EKS H100 cluster: with the fix applied the CTR is present after install, the demo TrainJob is admitted, and a 1-epoch pytorch-mnist run completes (accuracy 0.74). The bug has existed since the manifest was first introduced in NVIDIA#94 (Feb 2026); confirmed by git log -p on both the original embedded path and the current recipes/ path.
Contributor
|
This pull request has been automatically locked since it has been closed for 90 days with no further activity. Please open a new pull request for related changes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
adds external manifest to kubeflow helm chart to install torch-distributed
ClusterTrainingRuntimeas part of eidosMotivation / Context
easily support CUJ1
Fixes:
Related:
Type of Change
Component(s) Affected
cmd/eidos,pkg/cli)cmd/eidosd,pkg/api,pkg/server)pkg/recipe)pkg/bundler,pkg/component/*)pkg/collector,pkg/snapshotter)pkg/validator)pkg/errors,pkg/k8s)docs/,examples/)Implementation Notes
Testing
# Commands run (prefer `make qualify` for non-trivial changes) make qualifyRisk Assessment
Rollout notes:
Checklist
make testwith-race)make lint)git commit -s) — DCO info