Skip to content

Commit f45facc

Browse files
lexfreilexfrei
authored
feat(hami): add HAMi GPU virtualization system package (#2484)
## What this PR does Documentation: cozystack/website#517 Integrates [HAMi](https://github.com/Project-HAMi/HAMi) v2.8.1 (CNCF Sandbox) into Cozystack as a system-level package for fractional GPU sharing in tenant Kubernetes clusters. The integration covers three layers: - **System chart** (`packages/system/hami/`): Vendors upstream HAMi Helm chart with device plugin, scheduler extender, mutating webhook, and RuntimeClass configuration. The broken DRA subchart was removed — it renders resources even when disabled and references unpublished images. - **Kubernetes app addon** (`packages/apps/kubernetes/`): HAMi exposed as an optional toggle (`hami.enabled`). Automatically disables GPU Operator's native device plugin when active to avoid conflicts. Enforces hard dependency on GPU Operator. - **Platform registration** (`packages/core/platform/`): HAMi declared as PackageSource with gpu-operator dependency, included in the iaas bundle. **Known limitation**: HAMi-core relies on a private glibc symbol (`_dl_sym`) removed in glibc 2.34, which breaks compute isolation on modern container images (Ubuntu 22.04+, PyTorch/TensorFlow official images). Alpine/musl is entirely incompatible. See the package README for details and upstream issue references. ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(hami): add HAMi GPU virtualization as an optional system package for fractional GPU sharing. Enables memory and compute isolation for NVIDIA GPUs across tenant workloads. Integrates with GPU Operator and can be enabled per-cluster via the hami.enabled toggle. Note: compute isolation requires glibc < 2.34 in workload containers. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added HAMi GPU virtualization middleware as an optional cluster addon to enable fractional GPU sharing. * HAMi addon includes an enable toggle (default: disabled) and a customizable Helm values override; when enabled it enforces GPU Operator presence and adjusts GPU Operator values accordingly. * New HelmRelease integration for deploying HAMi with dependency ordering and conditional rendering. * **Documentation** * Added HAMi setup guide, compatibility notes, and updated cluster addon configuration reference. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 69ad74e + 5d58d0f commit f45facc

50 files changed

Lines changed: 3281 additions & 4 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 12 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 33 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,9 @@ See the reference for components utilized in this service:
128128
| `addons.gpuOperator` | NVIDIA GPU Operator. | `object` | `{}` |
129129
| `addons.gpuOperator.enabled` | Enable GPU Operator. | `bool` | `false` |
130130
| `addons.gpuOperator.valuesOverride` | Custom Helm values overrides. | `object` | `{}` |
131+
| `addons.hami` | HAMi GPU virtualization middleware. | `object` | `{}` |
132+
| `addons.hami.enabled` | Enable HAMi (requires GPU Operator). | `bool` | `false` |
133+
| `addons.hami.valuesOverride` | Custom Helm values overrides. | `object` | `{}` |
131134
| `addons.fluxcd` | FluxCD GitOps operator. | `object` | `{}` |
132135
| `addons.fluxcd.enabled` | Enable FluxCD. | `bool` | `false` |
133136
| `addons.fluxcd.valuesOverride` | Custom Helm values overrides. | `object` | `{}` |

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 13 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,11 @@
1+
{{- define "cozystack.defaultGpuOperatorValues" -}}
2+
{{- if .Values.addons.hami.enabled }}
3+
gpu-operator:
4+
devicePlugin:
5+
enabled: false
6+
{{- end }}
7+
{{- end }}
8+
19
{{- if and .Values.addons.gpuOperator.enabled .Values._namespace.etcd }}
210
apiVersion: helm.toolkit.fluxcd.io/v2
311
kind: HelmRelease
@@ -29,9 +37,12 @@ spec:
2937
force: true
3038
remediation:
3139
retries: -1
32-
{{- with .Values.addons.gpuOperator.valuesOverride }}
40+
{{- $defaults := fromYaml (include "cozystack.defaultGpuOperatorValues" .) }}
41+
{{- $overrides := deepCopy (default (dict) .Values.addons.gpuOperator.valuesOverride) }}
42+
{{- $merged := mergeOverwrite (default (dict) $defaults) $overrides }}
43+
{{- if $merged }}
3344
values:
34-
{{- toYaml . | nindent 4 }}
45+
{{- toYaml $merged | nindent 4 }}
3546
{{- end }}
3647

3748
dependsOn:

‎packages/apps/kubernetes/templates/helmreleases/hami.yaml‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{{- if and .Values.addons.hami.enabled .Values._namespace.etcd }}
2+
{{- if not .Values.addons.gpuOperator.enabled }}
3+
{{- fail "addons.hami requires addons.gpuOperator to be enabled" }}
4+
{{- end }}
5+
apiVersion: helm.toolkit.fluxcd.io/v2
6+
kind: HelmRelease
7+
metadata:
8+
name: {{ .Release.Name }}-hami
9+
labels:
10+
cozystack.io/repository: system
11+
cozystack.io/target-cluster-name: {{ .Release.Name }}
12+
sharding.fluxcd.io/key: tenants
13+
spec:
14+
releaseName: hami
15+
chartRef:
16+
kind: ExternalArtifact
17+
name: cozystack-kubernetes-application-kubevirt-kubernetes-hami
18+
namespace: cozy-system
19+
kubeConfig:
20+
secretRef:
21+
name: {{ .Release.Name }}-admin-kubeconfig
22+
key: super-admin.svc
23+
targetNamespace: cozy-hami
24+
storageNamespace: cozy-hami
25+
interval: 5m
26+
timeout: 10m
27+
install:
28+
createNamespace: true
29+
remediation:
30+
retries: -1
31+
upgrade:
32+
force: true
33+
remediation:
34+
retries: -1
35+
{{- with .Values.addons.hami.valuesOverride }}
36+
values:
37+
{{- toYaml . | nindent 4 }}
38+
{{- end }}
39+
40+
dependsOn:
41+
{{- if lookup "helm.toolkit.fluxcd.io/v2" "HelmRelease" .Release.Namespace .Release.Name }}
42+
- name: {{ .Release.Name }}
43+
namespace: {{ .Release.Namespace }}
44+
{{- end }}
45+
- name: {{ .Release.Name }}-cilium
46+
namespace: {{ .Release.Namespace }}
47+
- name: {{ .Release.Name }}-gpu-operator
48+
namespace: {{ .Release.Namespace }}
49+
{{- end }}

‎packages/apps/kubernetes/tests/gpu_operator_hami_test.yaml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
suite: GPU Operator HelmRelease HAMi integration tests
2+
templates:
3+
- templates/helmreleases/gpu-operator.yaml
4+
values:
5+
- values-ci.yaml
6+
tests:
7+
- it: should disable devicePlugin when hami is enabled
8+
set:
9+
addons:
10+
gpuOperator:
11+
enabled: true
12+
valuesOverride: {}
13+
hami:
14+
enabled: true
15+
valuesOverride: {}
16+
asserts:
17+
- equal:
18+
path: spec.values.gpu-operator.devicePlugin.enabled
19+
value: false
20+
21+
- it: should not have values when hami is disabled and no overrides
22+
set:
23+
addons:
24+
gpuOperator:
25+
enabled: true
26+
valuesOverride: {}
27+
hami:
28+
enabled: false
29+
valuesOverride: {}
30+
asserts:
31+
- notExists:
32+
path: spec.values
33+
34+
- it: should apply hami defaults when valuesOverride key is omitted
35+
set:
36+
addons:
37+
gpuOperator:
38+
enabled: true
39+
hami:
40+
enabled: true
41+
asserts:
42+
- equal:
43+
path: spec.values.gpu-operator.devicePlugin.enabled
44+
value: false
45+
46+
- it: should allow user overrides to merge with hami defaults
47+
set:
48+
addons:
49+
gpuOperator:
50+
enabled: true
51+
valuesOverride:
52+
gpu-operator:
53+
driver:
54+
enabled: false
55+
hami:
56+
enabled: true
57+
valuesOverride: {}
58+
asserts:
59+
- equal:
60+
path: spec.values.gpu-operator.devicePlugin.enabled
61+
value: false
62+
- equal:
63+
path: spec.values.gpu-operator.driver.enabled
64+
value: false
65+
66+
- it: should let user explicitly override devicePlugin.enabled to true with hami enabled
67+
set:
68+
addons:
69+
gpuOperator:
70+
enabled: true
71+
valuesOverride:
72+
gpu-operator:
73+
devicePlugin:
74+
enabled: true
75+
driver:
76+
enabled: false
77+
hami:
78+
enabled: true
79+
valuesOverride: {}
80+
asserts:
81+
- equal:
82+
path: spec.values.gpu-operator.devicePlugin.enabled
83+
value: true
84+
- equal:
85+
path: spec.values.gpu-operator.driver.enabled
86+
value: false
87+
88+
- it: should not render when gpuOperator is disabled
89+
set:
90+
addons:
91+
gpuOperator:
92+
enabled: false
93+
valuesOverride: {}
94+
hami:
95+
enabled: false
96+
valuesOverride: {}
97+
asserts:
98+
- hasDocuments:
99+
count: 0

0 commit comments

Comments
 (0)