Skip to content

Commit 7ab6283

Browse files
authored
feat(scheduler): integrate SchedulingClass support for tenant workloads (#2223)
## What this PR does Adds infrastructure support for restricting tenant workloads to specific data centers (or node groups) via SchedulingClass CRs. - Adds `schedulingClass` parameter to the Tenant model, inherited by child tenants (children cannot override parent's scheduling constraints) - Adds `scheduling.cozystack.io/class` label to tenant namespaces - Adds SchedulingClass dropdown to the Tenant creation form in the dashboard - Extends lineage-controller-webhook to inject `schedulerName: cozystack-scheduler` and `scheduler.cozystack.io/scheduling-class` annotation into all Pods in namespaces with the scheduling label - Webhook verifies the referenced SchedulingClass CR exists before injection; if missing (scheduler not installed), pods are left untouched ### Operators without native schedulerName support The following operators do not support `schedulerName` in their CRDs, but are still covered by the webhook-based approach since it mutates pods directly: - etcd-operator - redis-operator (spotahome) - mariadb-operator - clickhouse-operator (altinity) ### Release note ```release-note [scheduler] Add SchedulingClass support for tenant workloads. Administrators can assign a SchedulingClass to a tenant, restricting where its workloads are scheduled. The lineage-controller-webhook injects the custom scheduler into all pods in the tenant namespace. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a schedulingClass parameter for tenant configuration; it can be set in values, appears in schema/docs, inherits to namespaces, and is included in exported tenant manifests. * When present, the schedulingClass is propagated to namespace metadata and applied to workloads so scheduler selection and scheduling annotations reflect the setting. * **Bug Fixes** * Improved owner-resolution and labeling so resources and scheduling are associated more accurately. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 199ccfc + bdeb3a7 commit 7ab6283

10 files changed

Lines changed: 170 additions & 44 deletions

File tree

‎go.mod‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,7 @@ require (
4242
github.com/cespare/xxhash/v2 v2.3.0 // indirect
4343
github.com/coreos/go-semver v0.3.1 // indirect
4444
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
45+
github.com/cozystack/cozystack-scheduler/pkg/apis v0.1.1 // indirect
4546
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
4647
github.com/emicklei/go-restful/v3 v3.12.2 // indirect
4748
github.com/evanphx/json-patch v4.12.0+incompatible // indirect

‎go.sum‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,8 @@ github.com/coreos/go-systemd/v22 v22.5.0 h1:RrqgGjYQKalulkV8NGVIfkXQf6YYmOyiJKk8
2020
github.com/coreos/go-systemd/v22 v22.5.0/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
2121
github.com/cozystack/apimachinery v0.0.0-20251219010959-1f91eabae46c h1:C2wIfH/OzhU9XOK/e6Ik9cg7nZ1z6fN4lf6a3yFdik8=
2222
github.com/cozystack/apimachinery v0.0.0-20251219010959-1f91eabae46c/go.mod h1:/GwIlEcWuTX9zKIg2mbw0LRFIsXwrfoVxn+ef0X13lw=
23+
github.com/cozystack/cozystack-scheduler/pkg/apis v0.1.1 h1:9uLa/8J4lRx3sNuaVH8UZqgZvnskHATPo5GxEKh0iSM=
24+
github.com/cozystack/cozystack-scheduler/pkg/apis v0.1.1/go.mod h1:kPeS9YPB4ENbvNINEkkp0SX8FB+gvwoOHGOHMT3Tg9Y=
2325
github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=
2426
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
2527
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=

‎internal/controller/dashboard/customformsoverride.go‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -230,6 +230,10 @@ func applyListInputOverrides(schema map[string]any, kind string, openAPIProps ma
230230
kafkaProps["storageClass"] = storageClassListInput()
231231
zkProps := ensureSchemaPath(schema, "spec", "zookeeper")
232232
zkProps["storageClass"] = storageClassListInput()
233+
234+
case "Tenant":
235+
specProps := ensureSchemaPath(schema, "spec")
236+
specProps["schedulingClass"] = schedulingClassListInput()
233237
}
234238
}
235239

@@ -246,6 +250,20 @@ func storageClassListInput() map[string]any {
246250
}
247251
}
248252

253+
// schedulingClassListInput returns a listInput field config for a schedulingClass dropdown
254+
// backed by the cluster's available SchedulingClass CRs.
255+
func schedulingClassListInput() map[string]any {
256+
return map[string]any{
257+
"type": "listInput",
258+
"customProps": map[string]any{
259+
"valueUri": "/api/clusters/{cluster}/k8s/apis/cozystack.io/v1alpha1/schedulingclasses",
260+
"keysToValue": []any{"metadata", "name"},
261+
"keysToLabel": []any{"metadata", "name"},
262+
"allowEmpty": true,
263+
},
264+
}
265+
}
266+
249267
// ensureArrayItemProps ensures that parentProps[fieldName].items.properties exists
250268
// and returns the items properties map. Used for overriding fields inside array items.
251269
func ensureArrayItemProps(parentProps map[string]any, fieldName string) map[string]any {

‎internal/lineagecontrollerwebhook/webhook.go‎

Lines changed: 112 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -8,22 +8,28 @@ import (
88
"strings"
99

1010
"github.com/cozystack/cozystack/pkg/lineage"
11+
corev1 "k8s.io/api/core/v1"
12+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
1113
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
14+
"k8s.io/apimachinery/pkg/runtime"
1215
"k8s.io/apimachinery/pkg/runtime/schema"
1316
"k8s.io/client-go/dynamic"
1417
"k8s.io/client-go/rest"
1518
ctrl "sigs.k8s.io/controller-runtime"
19+
"sigs.k8s.io/controller-runtime/pkg/client"
1620
"sigs.k8s.io/controller-runtime/pkg/client/apiutil"
1721
"sigs.k8s.io/controller-runtime/pkg/log"
1822
"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
1923

2024
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
25+
appsv1alpha1 "github.com/cozystack/cozystack/pkg/apis/apps/v1alpha1"
2126
corev1alpha1 "github.com/cozystack/cozystack/pkg/apis/core/v1alpha1"
27+
schedulerapi "github.com/cozystack/cozystack-scheduler/pkg/apis/v1alpha1"
2228
)
2329

2430
var (
25-
NoAncestors = fmt.Errorf("no managed apps found in lineage")
26-
AncestryAmbiguous = fmt.Errorf("object ancestry is ambiguous")
31+
NoAncestors = errors.New("no managed apps found in lineage")
32+
AncestryAmbiguous = errors.New("object ancestry is ambiguous")
2733
)
2834

2935
const (
@@ -95,26 +101,29 @@ func (h *LineageControllerWebhook) Handle(ctx context.Context, req admission.Req
95101
return admission.Errored(400, fmt.Errorf("decode object: %w", err))
96102
}
97103

98-
labels, err := h.computeLabels(ctx, obj)
99-
for {
100-
if err != nil && errors.Is(err, NoAncestors) {
101-
break // not a problem, mark object as unmanaged
102-
}
103-
if err != nil && errors.Is(err, AncestryAmbiguous) {
104-
warn = append(warn, "object ancestry ambiguous, using first ancestor found")
105-
break
106-
}
107-
if err != nil {
108-
logger.Error(err, "error computing lineage labels")
109-
return admission.Errored(500, fmt.Errorf("error computing lineage labels: %w", err))
110-
}
111-
if err == nil {
112-
break
113-
}
104+
owner, err := h.getOwner(ctx, obj)
105+
switch {
106+
case err != nil && errors.Is(err, AncestryAmbiguous):
107+
warn = append(warn, "object ancestry ambiguous, using first ancestor found")
108+
case err != nil && errors.Is(err, NoAncestors):
109+
// not a problem, mark object as unmanaged
110+
case err != nil:
111+
logger.Error(err, "error computing lineage labels")
112+
return admission.Errored(500, fmt.Errorf("error computing lineage labels: %w", err))
113+
}
114+
labels, err := h.computeLabels(ctx, obj, owner)
115+
if err != nil {
116+
logger.Error(err, "error computing lineage labels")
117+
return admission.Errored(500, fmt.Errorf("error computing lineage labels: %w", err))
114118
}
115119

116120
h.applyLabels(obj, labels)
117121

122+
if err := h.applySchedulingClass(ctx, obj, owner, req.Namespace); err != nil {
123+
logger.Error(err, "error applying scheduling class")
124+
return admission.Errored(500, fmt.Errorf("error applying scheduling class: %w", err))
125+
}
126+
118127
mutated, err := json.Marshal(obj)
119128
if err != nil {
120129
return admission.Errored(500, fmt.Errorf("marshal mutated pod: %w", err))
@@ -123,23 +132,30 @@ func (h *LineageControllerWebhook) Handle(ctx context.Context, req admission.Req
123132
return admission.PatchResponseFromRaw(req.Object.Raw, mutated).WithWarnings(warn...)
124133
}
125134

126-
func (h *LineageControllerWebhook) computeLabels(ctx context.Context, o *unstructured.Unstructured) (map[string]string, error) {
135+
func (h *LineageControllerWebhook) getOwner(ctx context.Context, o *unstructured.Unstructured) (*unstructured.Unstructured, error) {
127136
owners := lineage.WalkOwnershipGraph(ctx, h.dynClient, h.mapper, h, o)
128137
if len(owners) == 0 {
129-
return map[string]string{ManagedObjectKey: "false"}, NoAncestors
138+
return nil, NoAncestors
130139
}
131140
obj, err := owners[0].GetUnstructured(ctx, h.dynClient, h.mapper)
132141
if err != nil {
133142
return nil, err
134143
}
135-
gv, err := schema.ParseGroupVersion(obj.GetAPIVersion())
136-
if err != nil {
137-
// should never happen, we got an APIVersion right from the API
138-
return nil, fmt.Errorf("could not parse APIVersion %s to a group and version: %w", obj.GetAPIVersion(), err)
139-
}
140144
if len(owners) > 1 {
141145
err = AncestryAmbiguous
142146
}
147+
return obj, err
148+
}
149+
150+
func (h *LineageControllerWebhook) computeLabels(ctx context.Context, obj *unstructured.Unstructured, owner *unstructured.Unstructured) (map[string]string, error) {
151+
if owner == nil {
152+
return nil, nil
153+
}
154+
gv, err := schema.ParseGroupVersion(owner.GetAPIVersion())
155+
if err != nil {
156+
// should never happen, we got an APIVersion right from the API
157+
return nil, fmt.Errorf("could not parse APIVersion %s to a group and version: %w", owner.GetAPIVersion(), err)
158+
}
143159
labels := map[string]string{
144160
// truncate apigroup to first 63 chars
145161
ManagedObjectKey: "true",
@@ -153,25 +169,25 @@ func (h *LineageControllerWebhook) computeLabels(ctx context.Context, o *unstruc
153169
}
154170
return s
155171
}(gv.Group),
156-
ManagerKindKey: obj.GetKind(),
157-
ManagerNameKey: obj.GetName(),
172+
ManagerKindKey: owner.GetKind(),
173+
ManagerNameKey: owner.GetName(),
158174
}
159175
templateLabels := map[string]string{
160-
"kind": strings.ToLower(obj.GetKind()),
161-
"name": obj.GetName(),
162-
"namespace": o.GetNamespace(),
176+
"kind": strings.ToLower(owner.GetKind()),
177+
"name": owner.GetName(),
178+
"namespace": obj.GetNamespace(),
163179
}
164180
cfg := h.config.Load().(*runtimeConfig)
165-
crd := cfg.appCRDMap[appRef{gv.Group, obj.GetKind()}]
166-
resourceSelectors := h.getResourceSelectors(o.GroupVersionKind().GroupKind(), crd)
181+
crd := cfg.appCRDMap[appRef{gv.Group, owner.GetKind()}]
182+
resourceSelectors := h.getResourceSelectors(obj.GroupVersionKind().GroupKind(), crd)
167183

168184
labels[corev1alpha1.TenantResourceLabelKey] = func(b bool) string {
169185
if b {
170186
return corev1alpha1.TenantResourceLabelValue
171187
}
172188
return "false"
173-
}(matchResourceToExcludeInclude(ctx, o.GetName(), templateLabels, o.GetLabels(), resourceSelectors))
174-
return labels, err
189+
}(matchResourceToExcludeInclude(ctx, obj.GetName(), templateLabels, obj.GetLabels(), resourceSelectors))
190+
return labels, nil
175191
}
176192

177193
func (h *LineageControllerWebhook) applyLabels(o *unstructured.Unstructured, labels map[string]string) {
@@ -185,6 +201,68 @@ func (h *LineageControllerWebhook) applyLabels(o *unstructured.Unstructured, lab
185201
o.SetLabels(existing)
186202
}
187203

204+
// applySchedulingClass injects schedulerName and scheduling-class annotation
205+
// into Pods whose namespace carries the scheduler.cozystack.io/scheduling-class label.
206+
// If the referenced SchedulingClass CR does not exist (e.g. the scheduler
207+
// package is not installed), the injection is silently skipped so that pods
208+
// are not left Pending.
209+
func (h *LineageControllerWebhook) applySchedulingClass(ctx context.Context, obj, owner *unstructured.Unstructured, namespace string) error {
210+
if obj.GetKind() != "Pod" {
211+
return nil
212+
}
213+
214+
// Determine scheduling class: owner Application field takes priority,
215+
// then fall back to namespace label.
216+
var schedulingClass string
217+
if owner != nil {
218+
var app appsv1alpha1.Application
219+
if err := runtime.DefaultUnstructuredConverter.FromUnstructured(owner.Object, &app); err == nil {
220+
schedulingClass = app.SchedulingClass()
221+
}
222+
}
223+
if schedulingClass == "" {
224+
ns := &corev1.Namespace{}
225+
if err := h.Get(ctx, client.ObjectKey{Name: namespace}, ns); err != nil {
226+
return fmt.Errorf("getting namespace %s: %w", namespace, err)
227+
}
228+
schedulingClass = ns.Labels[schedulerapi.SchedulingClassLabel]
229+
}
230+
231+
if schedulingClass == "" {
232+
return nil
233+
}
234+
235+
// Verify that the referenced SchedulingClass CR exists.
236+
// If the CRD is not installed or the CR is missing, skip injection
237+
// so that pods are not stuck Pending on a non-existent scheduler.
238+
_, err := h.dynClient.Resource(schedulingClassGVR).Get(ctx, schedulingClass, metav1.GetOptions{})
239+
if err != nil {
240+
logger := log.FromContext(ctx)
241+
logger.Info("SchedulingClass not found, skipping scheduler injection",
242+
"schedulingClass", schedulingClass, "namespace", namespace)
243+
return nil
244+
}
245+
246+
if err := unstructured.SetNestedField(obj.Object, schedulerapi.SchedulerName, "spec", "schedulerName"); err != nil {
247+
return fmt.Errorf("setting schedulerName: %w", err)
248+
}
249+
250+
annotations := obj.GetAnnotations()
251+
if annotations == nil {
252+
annotations = make(map[string]string)
253+
}
254+
annotations[schedulerapi.SchedulingClassAnnotation] = schedulingClass
255+
obj.SetAnnotations(annotations)
256+
257+
return nil
258+
}
259+
260+
var schedulingClassGVR = schema.GroupVersionResource{
261+
Group: schedulerapi.Group,
262+
Version: schedulerapi.Version,
263+
Resource: schedulerapi.Resource,
264+
}
265+
188266
func (h *LineageControllerWebhook) decodeUnstructured(req admission.Request, out *unstructured.Unstructured) error {
189267
if h.decoder != nil {
190268
if err := h.decoder.Decode(req, out); err == nil {

‎packages/apps/tenant/README.md‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -74,14 +74,15 @@ tenant-u1
7474

7575
### Common parameters
7676

77-
| Name | Description | Type | Value |
78-
| ---------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------- |
79-
| `host` | The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). | `string` | `""` |
80-
| `etcd` | Deploy own Etcd cluster. | `bool` | `false` |
81-
| `monitoring` | Deploy own Monitoring Stack. | `bool` | `false` |
82-
| `ingress` | Deploy own Ingress Controller. | `bool` | `false` |
83-
| `seaweedfs` | Deploy own SeaweedFS. | `bool` | `false` |
84-
| `resourceQuotas` | Define resource quotas for the tenant. | `map[string]quantity` | `{}` |
77+
| Name | Description | Type | Value |
78+
| ----------------- | -------------------------------------------------------------------------------------------------------------------------- | --------------------- | ------- |
79+
| `host` | The hostname used to access tenant services (defaults to using the tenant name as a subdomain for its parent tenant host). | `string` | `""` |
80+
| `etcd` | Deploy own Etcd cluster. | `bool` | `false` |
81+
| `monitoring` | Deploy own Monitoring Stack. | `bool` | `false` |
82+
| `ingress` | Deploy own Ingress Controller. | `bool` | `false` |
83+
| `seaweedfs` | Deploy own SeaweedFS. | `bool` | `false` |
84+
| `schedulingClass` | The name of a SchedulingClass CR to apply scheduling constraints for this tenant's workloads. | `string` | `""` |
85+
| `resourceQuotas` | Define resource quotas for the tenant. | `map[string]quantity` | `{}` |
8586

8687

8788
## Configuration

‎packages/apps/tenant/templates/namespace.yaml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,12 @@
3838
{{- if .Values.seaweedfs }}
3939
{{- $seaweedfs = $tenantName }}
4040
{{- end }}
41+
42+
{{/* SchedulingClass: inherited from parent, can be set if parent has none */}}
43+
{{- $schedulingClass := $parentNamespace.schedulingClass | default "" }}
44+
{{- if and (not $schedulingClass) .Values.schedulingClass }}
45+
{{- $schedulingClass = .Values.schedulingClass }}
46+
{{- end }}
4147
---
4248
apiVersion: v1
4349
kind: Namespace
@@ -58,6 +64,9 @@ metadata:
5864
namespace.cozystack.io/monitoring: {{ $monitoring | quote }}
5965
namespace.cozystack.io/seaweedfs: {{ $seaweedfs | quote }}
6066
namespace.cozystack.io/host: {{ $computedHost | quote }}
67+
{{- with $schedulingClass }}
68+
scheduler.cozystack.io/scheduling-class: {{ . | quote }}
69+
{{- end }}
6170
alpha.kubevirt.io/auto-memory-limits-ratio: "1.0"
6271
ownerReferences:
6372
- apiVersion: v1
@@ -86,4 +95,7 @@ stringData:
8695
monitoring: {{ $monitoring | quote }}
8796
seaweedfs: {{ $seaweedfs | quote }}
8897
host: {{ $computedHost | quote }}
98+
{{- with $schedulingClass }}
99+
schedulingClass: {{ . | quote }}
100+
{{- end }}
89101
{{- end }}

‎packages/apps/tenant/values.schema.json‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,11 @@
3939
"x-kubernetes-int-or-string": true
4040
}
4141
},
42+
"schedulingClass": {
43+
"description": "The name of a SchedulingClass CR to apply scheduling constraints for this tenant's workloads.",
44+
"type": "string",
45+
"default": ""
46+
},
4247
"seaweedfs": {
4348
"description": "Deploy own SeaweedFS.",
4449
"type": "boolean",

‎packages/apps/tenant/values.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,5 +17,8 @@ ingress: false
1717
## @param {bool} seaweedfs - Deploy own SeaweedFS.
1818
seaweedfs: false
1919

20+
## @param {string} [schedulingClass] - The name of a SchedulingClass CR to apply scheduling constraints for this tenant's workloads.
21+
schedulingClass: ""
22+
2023
## @param {map[string]quantity} resourceQuotas - Define resource quotas for the tenant.
2124
resourceQuotas: {}

0 commit comments

Comments
 (0)