Skip to content

Commit 8db92d5

Browse files
kvapskvaps
authored
[kubernetes] Add gpu-operator and introduce GPU support for tenant Kubernetes clusters (#834)
Signed-off-by: Andrei Kvapil <[email protected]> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for GPU resources in Kubernetes clusters, including the ability to specify GPUs per node group and deploy the NVIDIA GPU Operator as an optional addon. - Introduced new configuration options for customizing Kamaji control plane resources and presets. - Added support for vertical pod autoscaler customization via override values. - **Bug Fixes** - Corrected typographical errors in label keys across multiple HelmRelease manifests to ensure consistent labeling. - **Documentation** - Updated documentation to describe new GPU and control plane configuration options, removed the instance type feature matrix, and added detailed parameter explanations. - **Chores** - Incremented Kubernetes app chart version to 0.19.0 and updated version mappings. - Fixed typos in parameter descriptions and comments. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents e7ded52 + 7537235 commit 8db92d5

18 files changed

Lines changed: 239 additions & 31 deletions

‎packages/apps/kubernetes/Chart.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@ type: application
1616
# This is the chart version. This version number should be incremented each time you make changes
1717
# to the chart and its templates, including the app version.
1818
# Versions are expected to follow Semantic Versioning (https://semver.org/)
19-
version: 0.18.1
19+
version: 0.19.0
2020

2121
# This is the version number of the application being deployed. This version number should be
2222
# incremented each time you make changes to the application. Versions are not expected to

‎packages/apps/kubernetes/README.md‎

Lines changed: 40 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -27,20 +27,46 @@ How to access to deployed cluster:
2727
kubectl get secret -n <namespace> kubernetes-<clusterName>-admin-kubeconfig -o go-template='{{ printf "%s\n" (index .data "super-admin.conf" | base64decode) }}' > test
2828
```
2929

30-
# Series
31-
32-
<!-- source: https://github.com/kubevirt/common-instancetypes/blob/main/README.md -->
33-
34-
. | U | O | CX | M | RT
35-
----------------------------|-----|-----|------|-----|------
36-
*Has GPUs* | | | | |
37-
*Hugepages* | | | ✓ | ✓ | ✓
38-
*Overcommitted Memory* | | ✓ | | |
39-
*Dedicated CPU* | | | ✓ | | ✓
40-
*Burstable CPU performance* | ✓ | ✓ | | ✓ |
41-
*Isolated emulator threads* | | | ✓ | | ✓
42-
*vNUMA* | | | ✓ | | ✓
43-
*vCPU-To-Memory Ratio* | 1:4 | 1:4 | 1:2 | 1:8 | 1:4
30+
## Parameters
31+
32+
### Common parameters
33+
34+
| Name | Description | Value |
35+
| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ------------ |
36+
| `host` | The hostname used to access the Kubernetes cluster externally (defaults to using the cluster name as a subdomain for the tenant host). | `""` |
37+
| `controlPlane.replicas` | Number of replicas for Kubernetes control-plane components | `2` |
38+
| `storageClass` | StorageClass used to store user data | `replicated` |
39+
| `nodeGroups` | nodeGroups configuration | `{}` |
40+
41+
### Cluster Addons
42+
43+
| Name | Description | Value |
44+
| --------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
45+
| `addons.certManager.enabled` | Enables the cert-manager | `false` |
46+
| `addons.certManager.valuesOverride` | Custom values to override | `{}` |
47+
| `addons.ingressNginx.enabled` | Enable Ingress-NGINX controller (expect nodes with 'ingress-nginx' role) | `false` |
48+
| `addons.ingressNginx.valuesOverride` | Custom values to override | `{}` |
49+
| `addons.ingressNginx.hosts` | List of domain names that should be passed through to the cluster by upper cluster | `[]` |
50+
| `addons.gpuOperator.enabled` | Enables the gpu-operator | `false` |
51+
| `addons.gpuOperator.valuesOverride` | Custom values to override | `{}` |
52+
| `addons.fluxcd.enabled` | Enables Flux CD | `false` |
53+
| `addons.fluxcd.valuesOverride` | Custom values to override | `{}` |
54+
| `addons.monitoringAgents.enabled` | Enables MonitoringAgents (fluentbit, vmagents for sending logs and metrics to storage) if tenant monitoring enabled, send to tenant storage, else to root storage | `false` |
55+
| `addons.monitoringAgents.valuesOverride` | Custom values to override | `{}` |
56+
| `addons.verticalPodAutoscaler.valuesOverride` | Custom values to override | `{}` |
57+
58+
### Kamaji control plane
59+
60+
| Name | Description | Value |
61+
| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
62+
| `kamajiControlPlane.apiServer.resources` | Resources | `{}` |
63+
| `kamajiControlPlane.apiServer.resourcesPreset` | Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if resources is set (resources is recommended for production). | `small` |
64+
| `kamajiControlPlane.controllerManager.resources` | Resources | `{}` |
65+
| `kamajiControlPlane.controllerManager.resourcesPreset` | Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if resources is set (resources is recommended for production). | `micro` |
66+
| `kamajiControlPlane.scheduler.resources` | Resources | `{}` |
67+
| `kamajiControlPlane.scheduler.resourcesPreset` | Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if resources is set (resources is recommended for production). | `micro` |
68+
| `kamajiControlPlane.addons.konnectivity.server.resources` | Resources | `{}` |
69+
| `kamajiControlPlane.addons.konnectivity.server.resourcesPreset` | Set container resources according to one common preset (allowed values: none, nano, micro, small, medium, large, xlarge, 2xlarge). This is ignored if resources is set (resources is recommended for production). | `micro` |
4470

4571

4672
## U Series

‎packages/apps/kubernetes/templates/cluster.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,13 @@ spec:
3939
sockets: 1
4040
{{- end }}
4141
devices:
42+
{{- if .group.gpus }}
43+
gpus:
44+
{{- range $i, $gpu := .group.gpus }}
45+
- name: gpu{{ add $i 1 }}
46+
deviceName: {{ $gpu.name }}
47+
{{- end }}
48+
{{- end }}
4249
disks:
4350
- name: system
4451
disk:

‎packages/apps/kubernetes/templates/helmreleases/cert-manager-crds.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ metadata:
44
name: {{ .Release.Name }}-cert-manager-crds
55
labels:
66
cozystack.io/repository: system
7-
coztstack.io/target-cluster-name: {{ .Release.Name }}
7+
cozystack.io/target-cluster-name: {{ .Release.Name }}
88
spec:
99
interval: 5m
1010
releaseName: cert-manager-crds

‎packages/apps/kubernetes/templates/helmreleases/cert-manager.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ metadata:
55
name: {{ .Release.Name }}-cert-manager
66
labels:
77
cozystack.io/repository: system
8-
coztstack.io/target-cluster-name: {{ .Release.Name }}
8+
cozystack.io/target-cluster-name: {{ .Release.Name }}
99
spec:
1010
interval: 5m
1111
releaseName: cert-manager

‎packages/apps/kubernetes/templates/helmreleases/cilium.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ metadata:
44
name: {{ .Release.Name }}-cilium
55
labels:
66
cozystack.io/repository: system
7-
coztstack.io/target-cluster-name: {{ .Release.Name }}
7+
cozystack.io/target-cluster-name: {{ .Release.Name }}
88
spec:
99
interval: 5m
1010
releaseName: cilium

‎packages/apps/kubernetes/templates/helmreleases/csi.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ metadata:
44
name: {{ .Release.Name }}-csi
55
labels:
66
cozystack.io/repository: system
7-
coztstack.io/target-cluster-name: {{ .Release.Name }}
7+
cozystack.io/target-cluster-name: {{ .Release.Name }}
88
spec:
99
interval: 5m
1010
releaseName: csi

‎packages/apps/kubernetes/templates/helmreleases/delete.yaml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@ spec:
3838
{{ .Release.Name }}-ingress-nginx
3939
{{ .Release.Name }}-fluxcd-operator
4040
{{ .Release.Name }}-fluxcd
41+
{{ .Release.Name }}-gpu-operator
4142
-p '{"spec": {"suspend": true}}'
4243
--type=merge --field-manager=flux-client-side-apply || true
4344
---
@@ -76,6 +77,7 @@ rules:
7677
- {{ .Release.Name }}-ingress-nginx
7778
- {{ .Release.Name }}-fluxcd-operator
7879
- {{ .Release.Name }}-fluxcd
80+
- {{ .Release.Name }}-gpu-operator
7981
---
8082
apiVersion: rbac.authorization.k8s.io/v1
8183
kind: RoleBinding

‎packages/apps/kubernetes/templates/helmreleases/fluxcd.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ metadata:
55
name: {{ .Release.Name }}-fluxcd-operator
66
labels:
77
cozystack.io/repository: system
8-
coztstack.io/target-cluster-name: {{ .Release.Name }}
8+
cozystack.io/target-cluster-name: {{ .Release.Name }}
99
spec:
1010
interval: 5m
1111
releaseName: fluxcd-operator
@@ -49,7 +49,7 @@ metadata:
4949
name: {{ .Release.Name }}-fluxcd
5050
labels:
5151
cozystack.io/repository: system
52-
coztstack.io/target-cluster-name: {{ .Release.Name }}
52+
cozystack.io/target-cluster-name: {{ .Release.Name }}
5353
spec:
5454
interval: 5m
5555
releaseName: fluxcd

‎packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
{{- if .Values.addons.gpuOperator.enabled }}
2+
apiVersion: helm.toolkit.fluxcd.io/v2
3+
kind: HelmRelease
4+
metadata:
5+
name: {{ .Release.Name }}-gpu-operator
6+
labels:
7+
cozystack.io/repository: system
8+
cozystack.io/target-cluster-name: {{ .Release.Name }}
9+
spec:
10+
interval: 5m
11+
releaseName: gpu-operator
12+
chart:
13+
spec:
14+
chart: cozy-gpu-operator
15+
reconcileStrategy: Revision
16+
sourceRef:
17+
kind: HelmRepository
18+
name: cozystack-system
19+
namespace: cozy-system
20+
kubeConfig:
21+
secretRef:
22+
name: {{ .Release.Name }}-admin-kubeconfig
23+
key: super-admin.svc
24+
targetNamespace: cozy-gpu-operator
25+
storageNamespace: cozy-gpu-operator
26+
install:
27+
createNamespace: true
28+
remediation:
29+
retries: -1
30+
upgrade:
31+
remediation:
32+
retries: -1
33+
{{- if .Values.addons.gpuOperator.valuesOverride }}
34+
valuesFrom:
35+
- kind: Secret
36+
name: {{ .Release.Name }}-gpu-operator-values-override
37+
valuesKey: values
38+
{{- end }}
39+
40+
dependsOn:
41+
{{- if lookup "helm.toolkit.fluxcd.io/v2" "HelmRelease" .Release.Namespace .Release.Name }}
42+
- name: {{ .Release.Name }}
43+
namespace: {{ .Release.Namespace }}
44+
{{- end }}
45+
- name: {{ .Release.Name }}-cilium
46+
namespace: {{ .Release.Namespace }}
47+
{{- end }}
48+
{{- if .Values.addons.gpuOperator.valuesOverride }}
49+
---
50+
apiVersion: v1
51+
kind: Secret
52+
metadata:
53+
name: {{ .Release.Name }}-gpu-operator-values-override
54+
stringData:
55+
values: |
56+
{{- toYaml .Values.addons.gpuOperator.valuesOverride | nindent 4 }}
57+
{{- end }}

0 commit comments

Comments
 (0)