Skip to content

Commit 5e9c392

Browse files
lexfreilexfrei
authored
feat(api): add the reservation evaluator for tenant quotas (#4983)
## What this PR does First phase of the accepted [tenant quotas as reservation limits](cozystack/community#48) design: a standalone, importable `pkg/reservation` package that computes what an application consumes from a tenant's budget from its declared values. Nothing calls it yet; the usage oracle, the generalized admission gate and the per-kind declarations come in the following phases. Today a tenant quota is declared in instance-type units but enforced against `ResourceQuota.status.used`, which counts virt-launcher pods with their additive virtualization overhead and drifts without recomputation. The design makes the declared reservation the accounting authority instead, and this package is the evaluator both the apiserver and the controller will read. - **Declarations are data, not Go.** A kind describes its consumption as CEL over its values (`exempt`, ordered `let` bindings, one expression per quota key), so the evaluator holds no per-kind knowledge and a kind from an out-of-tree catalog participates without a change here. The `Declaration` type is ready to be embedded in `ApplicationDefinitionSpec` in phase 2. Environment functions are `instanceType(name)`, `preset(name)` and `quantity(x)`, on top of the Kubernetes CEL quantity library; scaling is `quantity.mul(n)`, because the standard `_*_` cannot take a quantity overload. - **Instance types resolve to scalars keyed by class**, read from the type's spec and never from its name: `cpu`, `dedicated-cpu` (guest cores plus the isolated emulator thread, the extra parity core when the type itself carries the `EmulatorThreadCompleteToEvenParity` annotation, and a supplemental I/O pool), `memory`, `hugepages-<size>`, `overcommitted-memory`. The test pins the classification against every `VirtualMachineClusterInstancetype` shipped in `packages/system/kubevirt-instancetypes`. - **Fail closed.** An unknown instance type or preset, a value of the wrong type, an unguarded absent key, an unparseable quantity or a negative figure is an error, and values are checked against the kind's schema before defaulting, which would otherwise replace a mistyped value with its default, never a smaller reservation. A definition that declares neither its consumption nor an exemption yields `ErrNoDeclaration`. - **Stored values are defaulted first.** The aggregated apiserver persists a tenant's spec without its schema defaults, so an evaluator reading them raw would count zero for `kubernetes-nodes.maxReplicas` on every pool created with defaults. The structural-schema defaulting moves from `pkg/registry/apps/application` into `pkg/apis/apps/defaulting` so the registry and the evaluator apply the same defaults; the registry now only delegates to it, with no behaviour change. - **Aggregation** sums reservations per namespace and returns the applications that could not be evaluated separately, so a caller never mistakes an incomplete sum for a complete one. The evaluator tests run against the shipped OpenAPI schemas of `postgres`, `vm-instance` and `kubernetes-nodes`: the cozy-lib per-key preset fill, a `{cpu: 4, sockets: 2}` block reserving 8 vCPUs, the autoscaling ceiling, `external: true` reserving a load balancer, and a ComputePlane-shaped `KubernetesNodes` release carrying only `{roles, minReplicas}` reserving exactly what the same pool with its defaults written out reserves. Left for the next phases on purpose: where presets are resolved from (the design's direction is platform-shipped environment objects; here they come through the `Environment` interface), the `ApplicationDefinition` field itself, and any consumer. ### Screenshots Not a UI change. ### Downstream repositories The diff adds two Go packages and moves the defaulting helpers without changing behaviour. It touches no package under `packages/`, no `values.yaml`, no CRD and no `ApplicationDefinition` field, so none of the triggers in the map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(api): add the reservation evaluator that computes an application's tenant-quota consumption from its declared values. Internal only, with no consumer yet and no change in behaviour. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - Added reservation evaluation for application resources, including CPU, memory, hugepages, and overcommitted memory. - Added support for instance types, resource presets, schema-based defaults, and CEL expressions. - Added namespace-level reservation aggregation with clear reporting for applications that cannot be evaluated. - Added validation for invalid, negative, unsupported, or excessive resource values. - Added classification of KubeVirt instance types, including dedicated CPUs, emulator threads, supplemental I/O threads, and hugepages. - **Bug Fixes** - Schema defaults are applied consistently to nested application values. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b53ca1f + 0a310cc commit 5e9c392

12 files changed

Lines changed: 1918 additions & 156 deletions

File tree

‎go.mod‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ require (
1616
github.com/go-logr/logr v1.4.4
1717
github.com/go-logr/zapr v1.3.0
1818
github.com/go-task/slim-sprig/v3 v3.0.0
19+
github.com/google/cel-go v0.29.2
1920
github.com/minio/minio-go/v7 v7.0.98
2021
github.com/onsi/ginkgo/v2 v2.33.0
2122
github.com/onsi/gomega v1.43.1
@@ -83,7 +84,6 @@ require (
8384
github.com/go-openapi/testify/enable/yaml/v2 v2.8.0 // indirect
8485
github.com/go-openapi/testify/v2 v2.8.0 // indirect
8586
github.com/golang/protobuf v1.5.4 // indirect
86-
github.com/google/cel-go v0.29.2 // indirect
8787
github.com/google/gnostic-models v0.7.1 // indirect
8888
github.com/google/go-cmp v0.7.0 // indirect
8989
github.com/google/pprof v0.0.0-20260906184651-6331bc6350fe // indirect

‎pkg/apis/apps/defaulting/defaulting.go‎

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,201 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// Package defaulting applies an ApplicationDefinition's structural-schema
18+
// defaults to application values. It is shared by the aggregated apiserver,
19+
// which defaults on the read path, and by anything that has to reason about
20+
// stored values, which carry only the keys a tenant actually wrote.
21+
package defaulting
22+
23+
import (
24+
"encoding/json"
25+
"fmt"
26+
"strings"
27+
28+
internalapiext "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions"
29+
apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
30+
structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema"
31+
"k8s.io/apimachinery/pkg/runtime"
32+
)
33+
34+
// BuildSchema parses an OpenAPI-v3 JSON schema string and returns the
35+
// structural-schema form used for defaulting. Returns (nil, nil) when raw
36+
// is empty after trimming. Returns (nil, error) for malformed JSON,
37+
// v1→internal conversion failures, or structural-schema construction
38+
// errors — callers can decide whether to log-and-continue or fail hard.
39+
func BuildSchema(raw string) (*structuralschema.Structural, error) {
40+
raw = strings.TrimSpace(raw)
41+
if raw == "" {
42+
return nil, nil
43+
}
44+
var v1js apiextv1.JSONSchemaProps
45+
if err := json.Unmarshal([]byte(raw), &v1js); err != nil {
46+
return nil, fmt.Errorf("unmarshal v1 OpenAPI schema: %w", err)
47+
}
48+
scheme := runtime.NewScheme()
49+
_ = internalapiext.AddToScheme(scheme)
50+
_ = apiextv1.AddToScheme(scheme)
51+
var ijs internalapiext.JSONSchemaProps
52+
if err := scheme.Convert(&v1js, &ijs, nil); err != nil {
53+
return nil, fmt.Errorf("convert v1->internal JSONSchemaProps: %w", err)
54+
}
55+
s, err := structuralschema.NewStructural(&ijs)
56+
if err != nil {
57+
return nil, fmt.Errorf("build structural schema: %w", err)
58+
}
59+
return s, nil
60+
}
61+
62+
// Apply defaults root in place against s the way the Kubernetes apiserver
63+
// defaults a custom resource. A nil schema leaves root untouched.
64+
func Apply(root *map[string]any, s *structuralschema.Structural) error {
65+
if s == nil {
66+
return nil
67+
}
68+
v := any(*root)
69+
nv, err := applyDefaults(v, s, true)
70+
if err != nil {
71+
return err
72+
}
73+
obj, ok := nv.(map[string]any)
74+
if !ok && nv != nil {
75+
return fmt.Errorf("internal error: applyDefaults returned non-map type %T for object root", nv)
76+
}
77+
if obj == nil {
78+
obj = map[string]any{}
79+
}
80+
*root = obj
81+
return nil
82+
}
83+
84+
func applyDefaults(v any, s *structuralschema.Structural, top bool) (any, error) {
85+
if s == nil {
86+
return v, nil
87+
}
88+
89+
effType := s.Type
90+
if effType == "" {
91+
switch {
92+
case len(s.Properties) > 0 || (s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil):
93+
effType = "object"
94+
case s.Items != nil:
95+
effType = "array"
96+
default:
97+
// scalar
98+
}
99+
}
100+
101+
switch effType {
102+
case "object":
103+
mv, isMap := v.(map[string]any)
104+
if !isMap || mv == nil {
105+
if s.Default.Object != nil && !top {
106+
if dm, ok := s.Default.Object.(map[string]any); ok {
107+
mv = cloneMap(dm)
108+
}
109+
}
110+
if mv == nil {
111+
mv = map[string]any{}
112+
}
113+
}
114+
115+
for name, ps := range s.Properties {
116+
if _, ok := mv[name]; !ok {
117+
if ps.Default.Object != nil {
118+
mv[name] = clone(ps.Default.Object)
119+
}
120+
}
121+
if cur, ok := mv[name]; ok {
122+
cv, err := applyDefaults(cur, &ps, false)
123+
if err != nil {
124+
return nil, err
125+
}
126+
mv[name] = cv
127+
}
128+
}
129+
130+
if s.AdditionalProperties != nil && s.AdditionalProperties.Structural != nil {
131+
ap := s.AdditionalProperties.Structural
132+
for k, cur := range mv {
133+
if _, isKnown := s.Properties[k]; isKnown {
134+
continue
135+
}
136+
cv, err := applyDefaults(cur, ap, false)
137+
if err != nil {
138+
return nil, err
139+
}
140+
mv[k] = cv
141+
}
142+
}
143+
return mv, nil
144+
145+
case "array":
146+
sl, isSlice := v.([]any)
147+
if !isSlice || sl == nil {
148+
if s.Default.Object != nil {
149+
if ds, ok := s.Default.Object.([]any); ok {
150+
sl = cloneSlice(ds)
151+
}
152+
}
153+
if sl == nil {
154+
sl = []any{}
155+
}
156+
}
157+
if s.Items != nil {
158+
for i := range sl {
159+
cv, err := applyDefaults(sl[i], s.Items, false)
160+
if err != nil {
161+
return nil, err
162+
}
163+
sl[i] = cv
164+
}
165+
}
166+
return sl, nil
167+
168+
default:
169+
if v == nil && s.Default.Object != nil {
170+
return clone(s.Default.Object), nil
171+
}
172+
return v, nil
173+
}
174+
}
175+
176+
func clone(x any) any {
177+
switch t := x.(type) {
178+
case map[string]any:
179+
return cloneMap(t)
180+
case []any:
181+
return cloneSlice(t)
182+
default:
183+
return t
184+
}
185+
}
186+
187+
func cloneMap(m map[string]any) map[string]any {
188+
out := make(map[string]any, len(m))
189+
for k, v := range m {
190+
out[k] = clone(v)
191+
}
192+
return out
193+
}
194+
195+
func cloneSlice(s []any) []any {
196+
out := make([]any, len(s))
197+
for i := range s {
198+
out[i] = clone(s[i])
199+
}
200+
return out
201+
}

‎pkg/apis/apps/defaulting/defaulting_test.go‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package defaulting
18+
19+
import "testing"
20+
21+
// A nil map[string]any in an interface is not == nil, so it has to be
22+
// recognised as empty explicitly, at the root and below it, or writing the
23+
// first default into it panics.
24+
func TestApplyDefaultsTypedNilMaps(t *testing.T) {
25+
s, err := BuildSchema(`{"type":"object","properties":{
26+
"replicas":{"type":"integer","default":2},
27+
"roles":{"type":"array","default":["worker"],"items":{"type":"string"}},
28+
"resources":{"type":"object","default":{},"properties":{"preset":{"type":"string","default":"t1.micro"}}}}}`)
29+
if err != nil {
30+
t.Fatal(err)
31+
}
32+
33+
var root map[string]any
34+
if err := Apply(&root, s); err != nil {
35+
t.Fatal(err)
36+
}
37+
if root["replicas"] != int64(2) {
38+
t.Errorf("root: replicas = %#v, want 2", root["replicas"])
39+
}
40+
41+
var nested map[string]any
42+
var roles []any
43+
root = map[string]any{"resources": nested, "roles": roles}
44+
if err := Apply(&root, s); err != nil {
45+
t.Fatal(err)
46+
}
47+
resources, _ := root["resources"].(map[string]any)
48+
if resources["preset"] != "t1.micro" {
49+
t.Errorf("nested: resources = %#v, want the preset default", root["resources"])
50+
}
51+
if got, _ := root["roles"].([]any); len(got) != 1 || got[0] != "worker" {
52+
t.Errorf("nested: roles = %#v, want the array default", root["roles"])
53+
}
54+
}

‎pkg/registry/apps/application/rest.go‎

Lines changed: 2 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -50,14 +50,14 @@ import (
5050
"sigs.k8s.io/controller-runtime/pkg/client"
5151

5252
fluxshard "github.com/cozystack/cozystack/internal/fluxshardoperator"
53+
"github.com/cozystack/cozystack/pkg/apis/apps/defaulting"
5354
"github.com/cozystack/cozystack/pkg/apis/apps/presets"
5455
appsv1alpha1 "github.com/cozystack/cozystack/pkg/apis/apps/v1alpha1"
5556
"github.com/cozystack/cozystack/pkg/apis/apps/validation"
5657
"github.com/cozystack/cozystack/pkg/config"
5758
"github.com/cozystack/cozystack/pkg/registry"
5859
fieldfilter "github.com/cozystack/cozystack/pkg/registry/fields"
5960
"github.com/cozystack/cozystack/pkg/registry/sorting"
60-
internalapiext "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions"
6161
apiextv1 "k8s.io/apiextensions-apiserver/pkg/apis/apiextensions/v1"
6262
structuralschema "k8s.io/apiextensions-apiserver/pkg/apiserver/schema"
6363

@@ -110,26 +110,7 @@ type REST struct {
110110
// errors — callers can decide whether to log-and-continue (NewREST does)
111111
// or fail hard (tests do).
112112
func buildSpecSchema(raw string) (*structuralschema.Structural, error) {
113-
raw = strings.TrimSpace(raw)
114-
if raw == "" {
115-
return nil, nil
116-
}
117-
var v1js apiextv1.JSONSchemaProps
118-
if err := json.Unmarshal([]byte(raw), &v1js); err != nil {
119-
return nil, fmt.Errorf("unmarshal v1 OpenAPI schema: %w", err)
120-
}
121-
scheme := runtime.NewScheme()
122-
_ = internalapiext.AddToScheme(scheme)
123-
_ = apiextv1.AddToScheme(scheme)
124-
var ijs internalapiext.JSONSchemaProps
125-
if err := scheme.Convert(&v1js, &ijs, nil); err != nil {
126-
return nil, fmt.Errorf("convert v1->internal JSONSchemaProps: %w", err)
127-
}
128-
s, err := structuralschema.NewStructural(&ijs)
129-
if err != nil {
130-
return nil, fmt.Errorf("build structural schema: %w", err)
131-
}
132-
return s, nil
113+
return defaulting.BuildSchema(raw)
133114
}
134115

135116
// NewREST creates a new REST storage for Application with specific configuration

0 commit comments

Comments
 (0)