Skip to content

Commit f301cb5

Browse files
authored
feat(api): add configurable resources and replicas for cozystack-engine components (#2956)
## What this PR does Exposes standard Helm `resources` values for the three core workloads in the `cozystack-engine` package, and makes `cozystack-controller` replicas configurable (was hardcoded to `1`). **Changes by component:** | Component | `resources` | `replicas` | |---|---|---| | `cozystack-api` | ✅ NEW (`cozystackAPI.resources`) | already configurable | | `cozystack-controller` | ✅ NEW (`cozystackController.resources`) | ✅ NEW (`cozystackController.replicas`, default `1`) | | `lineage-controller-webhook` | ✅ NEW (`lineageControllerWebhook.resources`) | already configurable | Each template uses `{{- with .Values.xxx.resources }}` so the `resources:` block is only rendered when values are provided. Default `resources: {}` produces no output, so resource requests/limits stay opt-in. **Leader election:** `cozystack-controller` now always runs with leader election — the chart passes `--leader-elect` unconditionally and ships a namespaced `Role`/`RoleBinding` for the Lease in `cozy-system` (where controller-runtime creates it). Raising `cozystackController.replicas` therefore adds standbys instead of a second active writer over the reconcilers that patch HelmReleases, Gateways, Certificates, ResourceQuotas and Secrets; this holds whatever `replicas` is. A restart can pause reconciliation for up to the Lease duration (15s by default), because `LeaderElectionReleaseOnCancel` is off. With `replicas: 1` the change for an existing install is the flag plus the two RBAC objects. **Why:** Without these values, operators cannot set QoS guarantees and must resort to `kubectl patch`, Flux Kustomize SSA patches, or Kyverno mutations — all fragile under Helm upgrades. Closes #2955 ### Release note ```release-note feat(api): add configurable `resources` (requests/limits) for cozystack-api, cozystack-controller, and lineage-controller-webhook. `cozystack-controller` replicas is now configurable via `cozystackController.replicas` (default: 1). The `cozystack-controller` Deployment now always runs with leader election and creates a Lease in `cozy-system`, so raising `replicas` adds standbys rather than a second active controller; a restart can pause reconciliation for up to the Lease duration (15s by default). An empty `resources: {}` renders no resources block, so requests and limits remain opt-in. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Container resource requests and limits can now be configured for the API, controller, and webhook services; defaults remain unchanged. * The controller replica count is configurable, and leader election is enabled for multi-replica operation. * Controller leader-election access is limited to the `cozy-system` namespace. * **Documentation** * Documented the webhook resource configuration option. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 6d759cc + cb8728d commit f301cb5

13 files changed

Lines changed: 272 additions & 1 deletion

File tree

‎packages/system/cozystack-api/templates/deployment.yaml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,10 @@ spec:
3939
- name: cozystack-api-certs
4040
mountPath: /tmp/cozystack-api-certs
4141
readOnly: true
42+
{{- with .Values.cozystackAPI.resources }}
43+
resources:
44+
{{- toYaml . | nindent 10 }}
45+
{{- end }}
4246
volumes:
4347
- name: cozystack-api-certs
4448
secret:
Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
suite: cozystack-api resources knob
2+
templates:
3+
- templates/deployment.yaml
4+
5+
release:
6+
name: cozystack-api
7+
namespace: cozy-system
8+
9+
# The shipped default is `{}`, which renders no resources key at all, so no
10+
# existing pod moves on upgrade; only an admin who sets the value under the
11+
# Package CR gets requests or limits. The empty default is what makes the change
12+
# safe to land, hence a case on each half.
13+
14+
tests:
15+
- it: the shipped default renders no resources key
16+
asserts:
17+
- isKind:
18+
of: Deployment
19+
- notExists:
20+
path: spec.template.spec.containers[0].resources
21+
22+
- it: a set value renders on the container, not the pod
23+
set:
24+
cozystackAPI.resources:
25+
requests:
26+
cpu: 5m
27+
memory: 160Mi
28+
limits:
29+
cpu: 500m
30+
memory: 512Mi
31+
asserts:
32+
- equal:
33+
path: spec.template.spec.containers[0].resources
34+
value:
35+
requests:
36+
cpu: 5m
37+
memory: 160Mi
38+
limits:
39+
cpu: 500m
40+
memory: 512Mi

‎packages/system/cozystack-api/values.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,3 +5,6 @@ _cluster: {}
55
cozystackAPI:
66
image: ghcr.io/cozystack/cozystack/cozystack-api:v1.6.0@sha256:c4d374a9bc5e6f6e66d24b29396f1eae9d7f06fd268ff767d05c45ca35e84e2e
77
replicas: 2
8+
# Empty by default: no resources block is rendered, so requests and limits are
9+
# opt-in. Idle usage (measured) is ~130-143Mi — keep a limit above that.
10+
resources: {}

‎packages/system/cozystack-controller/templates/deployment.yaml‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ metadata:
55
labels:
66
app: cozystack-controller
77
spec:
8-
replicas: 1
8+
replicas: {{ .Values.cozystackController.replicas }}
99
selector:
1010
matchLabels:
1111
app: cozystack-controller
@@ -19,6 +19,7 @@ spec:
1919
- name: cozystack-controller
2020
image: "{{ .Values.cozystackController.image }}"
2121
args:
22+
- --leader-elect
2223
{{- if .Values.cozystackController.debug }}
2324
- --zap-log-level=debug
2425
{{- else }}
@@ -30,3 +31,7 @@ spec:
3031
{{- with .Values.cozystackController.seaweedfsMetricsEndpoint }}
3132
- --seaweedfs-metrics-endpoint={{ . }}
3233
{{- end }}
34+
{{- with .Values.cozystackController.resources }}
35+
resources:
36+
{{- toYaml . | nindent 10 }}
37+
{{- end }}
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
# Leader election (--leader-elect) writes a Lease in the manager's own namespace
2+
# — controller-runtime defaults LeaderElectionNamespace to the pod's namespace —
3+
# so the grant stays namespaced rather than giving the controller Lease writes in
4+
# every namespace, where a compromised controller could disturb another
5+
# component's election.
6+
apiVersion: rbac.authorization.k8s.io/v1
7+
kind: Role
8+
metadata:
9+
name: cozystack-controller-leader-election
10+
namespace: cozy-system
11+
rules:
12+
- apiGroups: ["coordination.k8s.io"]
13+
resources: ["leases"]
14+
verbs: ["get", "list", "watch", "create", "update", "patch"]
15+
---
16+
apiVersion: rbac.authorization.k8s.io/v1
17+
kind: RoleBinding
18+
metadata:
19+
name: cozystack-controller-leader-election
20+
namespace: cozy-system
21+
roleRef:
22+
apiGroup: rbac.authorization.k8s.io
23+
kind: Role
24+
name: cozystack-controller-leader-election
25+
subjects:
26+
- kind: ServiceAccount
27+
name: cozystack-controller
28+
namespace: cozy-system

‎packages/system/cozystack-controller/templates/rbac.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,9 @@ rules:
6868
- apiGroups: [""]
6969
resources: ["secrets"]
7070
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
71+
# Leader election (--leader-elect) is granted by a namespaced Role in cozy-system
72+
# (see leader-election-role.yaml): the Lease lives in the controller's own
73+
# namespace, so the write does not need to reach every namespace.
7174
- apiGroups: ['*']
7275
resources: ['*']
7376
verbs: ["get", "list", "watch"]
Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
suite: cozystack-controller runs one active manager
2+
templates:
3+
- templates/deployment.yaml
4+
- templates/rbac.yaml
5+
- templates/leader-election-role.yaml
6+
7+
release:
8+
name: cozystack-controller
9+
namespace: cozy-system
10+
11+
# Eight reconcilers in this manager write (HelmRelease patches, Gateway and
12+
# Certificate renders, ResourceQuota, Secret replication, CA projection), so a
13+
# second replica is only safe behind leader election. --leader-elect is passed
14+
# unconditionally and the Lease grant ships with it; raising replicas then adds a
15+
# standby rather than a second writer.
16+
#
17+
# The Lease grant is namespaced on purpose: the lock lives in the manager's own
18+
# namespace, so the ClusterRole does not hand out Lease writes in every
19+
# namespace.
20+
21+
tests:
22+
- it: the manager always asks for leader election
23+
template: templates/deployment.yaml
24+
asserts:
25+
- contains:
26+
path: spec.template.spec.containers[0].args
27+
content: --leader-elect
28+
29+
- it: replicas defaults to the previously hardcoded 1
30+
template: templates/deployment.yaml
31+
asserts:
32+
- equal:
33+
path: spec.replicas
34+
value: 1
35+
36+
- it: replicas value drives spec.replicas
37+
set:
38+
cozystackController.replicas: 2
39+
template: templates/deployment.yaml
40+
asserts:
41+
- equal:
42+
path: spec.replicas
43+
value: 2
44+
45+
- it: the Lease verbs come from a namespaced Role in cozy-system
46+
template: templates/leader-election-role.yaml
47+
documentSelector:
48+
path: kind
49+
value: Role
50+
asserts:
51+
- equal:
52+
path: metadata.name
53+
value: cozystack-controller-leader-election
54+
- equal:
55+
path: metadata.namespace
56+
value: cozy-system
57+
- contains:
58+
path: rules
59+
content:
60+
apiGroups: ["coordination.k8s.io"]
61+
resources: ["leases"]
62+
verbs: ["get", "list", "watch", "create", "update", "patch"]
63+
64+
- it: the Lease Role reaches the controller ServiceAccount in cozy-system
65+
template: templates/leader-election-role.yaml
66+
documentSelector:
67+
path: kind
68+
value: RoleBinding
69+
asserts:
70+
- equal:
71+
path: roleRef.name
72+
value: cozystack-controller-leader-election
73+
- contains:
74+
path: subjects
75+
content:
76+
kind: ServiceAccount
77+
name: cozystack-controller
78+
namespace: cozy-system
79+
80+
- it: the ClusterRole grants no Lease write, so no other namespace is reachable
81+
template: templates/rbac.yaml
82+
documentSelector:
83+
path: kind
84+
value: ClusterRole
85+
asserts:
86+
# Match on the resource alone (any: true), not on the exact rule spelling, so
87+
# the assertion fails for any ClusterRole rule that hands out leases — e.g.
88+
# one with verbs: ["*"] — instead of only for the rule shape used above.
89+
- notContains:
90+
path: rules
91+
content:
92+
resources: ["leases"]
93+
any: true
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
suite: cozystack-controller resources knob
2+
templates:
3+
- templates/deployment.yaml
4+
5+
release:
6+
name: cozystack-controller
7+
namespace: cozy-system
8+
9+
# Same contract as cozystack-api: the shipped `{}` renders no resources key, so
10+
# the default render stays byte-identical to the hardcoded chart and only an
11+
# admin who sets the value gets requests or limits.
12+
13+
tests:
14+
- it: the shipped default renders no resources key
15+
asserts:
16+
- isKind:
17+
of: Deployment
18+
- notExists:
19+
path: spec.template.spec.containers[0].resources
20+
21+
- it: a set value renders on the container, not the pod
22+
set:
23+
cozystackController.resources:
24+
requests:
25+
cpu: 1m
26+
memory: 96Mi
27+
limits:
28+
cpu: 500m
29+
memory: 512Mi
30+
asserts:
31+
- equal:
32+
path: spec.template.spec.containers[0].resources
33+
value:
34+
requests:
35+
cpu: 1m
36+
memory: 96Mi
37+
limits:
38+
cpu: 500m
39+
memory: 512Mi

‎packages/system/cozystack-controller/values.yaml‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,3 +7,11 @@ cozystackController:
77
# stack that scrapes it run in a separate cluster. Empty keeps discovery
88
# via the namespace.cozystack.io/monitoring label.
99
seaweedfsMetricsEndpoint: ""
10+
# Raising this gives the reconciliation loop a standby, not a second writer:
11+
# the chart always passes --leader-elect, so exactly one manager holds the
12+
# Lease and runs the reconcilers. With the default of 1 the extra Lease is
13+
# the only difference from the previous hardcoded value.
14+
replicas: 1
15+
# Empty by default: no resources block is rendered, so requests and limits are
16+
# opt-in. Idle usage (measured) is ~69Mi — keep a limit above that.
17+
resources: {}

‎packages/system/lineage-controller-webhook/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,7 @@ All keys live under the top-level `lineageControllerWebhook:` map.
5252
| `image` | Container image (digest-pinned) | `ghcr.io/cozystack/cozystack/lineage-controller-webhook:v1.3.0@sha256:e898…6fb0` |
5353
| `debug` | Enable `--zap-log-level=debug` instead of `info` | `false` |
5454
| `replicas` | Deployment replica count | `2` |
55+
| `resources` | Container requests/limits; `{}` renders no key | `{}` |
5556
| `localK8sAPIEndpoint.enabled` | **Deprecated.** See note below. | `false` |
5657

5758
### `localK8sAPIEndpoint.enabled` (deprecated)

0 commit comments

Comments
 (0)