Skip to content

Commit d2e3eb4

Browse files
authored
fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does Makes the backupcontroller test package compile again on main. #4587 added a flavor argument to `createCNPGBackupArtifact` and `marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added two test calls with the old argument lists. Both PRs were green on their own, and together they break `go test ./internal/backupcontroller/`. The two calls now pass the same default flavor as the other tests in the file. The second commit fixes an e2e suite that fails the required E2E check here. It is the `computeplane` suite from #4713, and its worker pool was one node with a single vCPU. After the kubelet reservations that node has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi, ingress-nginx, metrics-server, cert-manager) request exactly 900m between them. If cert-manager's startupapicheck hook pod is created last, it can't be scheduled and the install times out after 10m. The reinstall after remediation runs into the same full node. Whether that pod comes last depends on install order, so the suite passed on #4713 and failed here. The pool now has two workers with one vCPU each, the same shape the kubernetes suites use. One vCPU per worker stays on purpose, because of #3513. Two workers exposed a deadlock in the tenant cluster-autoscaler. The suite's pool is created at zero replicas and then gets a min size of 2, which CAPI does not apply to the live replica count on upgrade (#4786). The autoscaler's scale-up for a pending pod asks for one node, and the clusterapi provider refuses it because one is under the min size, although it is an increase. That refusal backs the group off, and the min-size enforcement skips groups in backoff, so the pool never leaves zero. Upstream had the same report in kubernetes/autoscaler#4942, closed as stale, and it is now tracked in kubernetes/autoscaler#10400. Our autoscaler image already patches the mirror case for max size, and the patch now refuses only a decrease below the min size. `go vet` and `go test ./internal/backupcontroller/...` pass, and so do the bats pins for the computeplane suite. The autoscaler patch adds a test for each direction, and the clusterapi package tests pass with it applied to 1.32.0. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cluster autoscaling for worker groups outside configured size limits: groups below their minimum can scale up toward it, and nodes can be removed when a group exceeds its maximum. Scale-down requests below the minimum remain restricted. * **Tests** * Updated backup and restore test calls with PostgreSQL and empty flavor values. * Updated ComputePlane checks for a two-worker pool and clarified cert-manager installation timing expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 9620be0 + 82ae9f6 commit d2e3eb4

3 files changed

Lines changed: 152 additions & 26 deletions

File tree

‎hack/e2e-chainsaw/computeplane/chainsaw-test.yaml‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,10 @@
1717
# the container lane does not have, so that path is left to the render tests.
1818
# The pool shape copies the kubernetes suites' worker sizing (see
1919
# run_kubernetes_test in _lib/run-kubernetes.sh for why each field is there),
20-
# at one worker rather than two.
20+
# two workers included. On one single-vCPU worker the tenant addons' CPU
21+
# requests take nearly all it can allocate, so cert-manager's startupapicheck
22+
# hook Job can find no room and stay Pending, which fails the cert-manager
23+
# install.
2124
#
2225
# Budget: the try ops' ceilings add up to 147m, this suite's own catch to 12m
2326
# (the events and describe collectors take timeouts.exec, 5m each, and the
@@ -115,8 +118,8 @@ spec:
115118
diskSize: 20Gi
116119
instanceType: u1.medium
117120
logSerialConsole: true
118-
minReplicas: 1
119-
maxReplicas: 1
121+
minReplicas: 2
122+
maxReplicas: 2
120123
podCpuLimit: 2
121124
podCpuRequest: 100m
122125
resources:
@@ -183,7 +186,7 @@ spec:
183186
sc=$(cozy_e2e_storage_class)
184187
values=$(timeout -k 5 30 kubectl -n tenant-cplane get helmrelease kubernetes-nodes-computeplane-cluster-md0 -o json | jq -c '.spec.values')
185188
echo "pool values: $values"
186-
echo "$values" | jq -e --arg sc "$sc" '.storageClass == $sc and .minReplicas == 1 and .cluster == "computeplane-cluster" and .clusterReleaseName == "computeplane-cluster"'
189+
echo "$values" | jq -e --arg sc "$sc" '.storageClass == $sc and .minReplicas == 2 and .cluster == "computeplane-cluster" and .clusterReleaseName == "computeplane-cluster"'
187190
extra=$(echo "$values" | jq -r 'keys - ["cluster","clusterReleaseName","minReplicas","maxReplicas","instanceType","diskSize","storageClass","roles","resources","gpus","kubelet","logSerialConsole","maxUnhealthy","nodeStartupTimeout","readyUnknownTimeout","readyFalseTimeout","podCpuLimit","podCpuRequest","maxNodeProvisionTime"] | .[]')
188191
if [ -n "$extra" ]; then
189192
echo "pool HelmRelease carries keys outside the module allowlist: $extra" >&2
@@ -245,8 +248,9 @@ spec:
245248

246249
# ingress-nginx needs a joined worker, so it is waited for first: 29m for
247250
# the worker, the kubernetes suites' node-join deadline, plus room for the
248-
# install that follows it. cert-manager installs alongside and has had that
249-
# time already; its own budget clears its 10m install timeout.
251+
# install that follows it. cert-manager also waits on the tenant cilium, so
252+
# the worker join is not time it has had; its own budget clears its 10m
253+
# install timeout.
250254
- assert:
251255
timeout: 35m
252256
resource:

‎internal/backupcontroller/cnpgstrategy_controller_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3673,7 +3673,7 @@ func TestCreateCNPGBackupArtifact_RecordsMajorVersion(t *testing.T) {
36733673
rendered := &strategyv1alpha1.CNPGTemplate{
36743674
BarmanObjectStore: strategyv1alpha1.BarmanObjectStoreTemplate{DestinationPath: "s3://b/"},
36753675
}
3676-
got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "postgres-pg", rendered, newPostgresApp("pg", "tenant"))
3676+
got, err := r.createCNPGBackupArtifact(context.Background(), j, resolved, cnpgBk, "postgres-pg", "postgres-pg", "postgresql", rendered, newPostgresApp("pg", "tenant"))
36773677
if err != nil {
36783678
t.Fatalf("createCNPGBackupArtifact: %v", err)
36793679
}
@@ -3736,7 +3736,7 @@ func TestReconcileCNPGRestore_RefusesCrossMajorRestore(t *testing.T) {
37363736
}
37373737
targetCluster := cnpgClusterNameForApp(targetApp)
37383738

3739-
snap, err := marshalCNPGBackupSnapshot(newPostgresApp(sourceApp, ns), nil)
3739+
snap, err := marshalCNPGBackupSnapshot(newPostgresApp(sourceApp, ns), nil, "")
37403740
if err != nil {
37413741
t.Fatalf("marshal snapshot: %v", err)
37423742
}

‎packages/apps/kubernetes/images/cluster-autoscaler/fix-downscale.diff‎

Lines changed: 140 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,8 @@
1-
diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go
2-
index 4eec0e4bf..f28fd9241 100644
3-
--- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go
4-
+++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go
5-
@@ -106,8 +106,6 @@ func (r unstructuredScalableResource) Replicas() (int, error) {
6-
7-
func (r unstructuredScalableResource) SetSize(nreplicas int) error {
8-
switch {
9-
- case nreplicas > r.maxSize:
10-
- return fmt.Errorf("size increase too large - desired:%d max:%d", nreplicas, r.maxSize)
11-
case nreplicas < r.minSize:
12-
return fmt.Errorf("size decrease too large - desired:%d min:%d", nreplicas, r.minSize)
13-
}
141
diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
2+
index 962ac11..5a10bde 100644
153
--- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
164
+++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_test.go
17-
@@ -241,11 +241,6 @@
5+
@@ -241,11 +241,6 @@ func TestNodeGroupIncreaseSizeErrors(t *testing.T) {
186
delta: -1,
197
initial: 3,
208
errorMsg: "size increase must be positive",
@@ -26,11 +14,10 @@ diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_te
2614
}}
2715

2816
test := func(t *testing.T, tc *testCase, testConfig *testConfig) {
29-
@@ -767,7 +762,59 @@
30-
)
17+
@@ -768,6 +763,58 @@ func TestNodeGroupDeleteNodes(t *testing.T) {
3118
})
3219
}
33-
+
20+
3421
+// A group can stand above its max size, for one when the annotation is
3522
+// lowered under it, and deleting a node from it must not be refused as an
3623
+// increase.
@@ -43,7 +30,7 @@ diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_te
4330
+ if err != nil {
4431
+ t.Fatalf("unexpected error: %v", err)
4532
+ }
46-
33+
+
4734
+ if l := len(nodegroups); l != 1 {
4835
+ t.Fatalf("expected 1 nodegroup, got %d", l)
4936
+ }
@@ -86,3 +73,138 @@ diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_nodegroup_te
8673
func TestNodeGroupMachineSetDeleteNodesWithMismatchedNodes(t *testing.T) {
8774
test := func(t *testing.T, expected int, testConfigs []*testConfig) {
8875
testConfig0, testConfig1 := testConfigs[0], testConfigs[1]
76+
@@ -1631,3 +1678,103 @@ func TestNodeGroupGetOptions(t *testing.T) {
77+
})
78+
}
79+
}
80+
+
81+
+// A group can stand below its min size, for one when the annotation is
82+
+// raised over it, and adding a node to it must not be refused as a
83+
+// decrease: the refusal backs the group off, which also stops the min-size
84+
+// enforcement from lifting it.
85+
+func TestNodeGroupIncreaseSizeBelowMinSize(t *testing.T) {
86+
+ test := func(t *testing.T, testConfig *testConfig) {
87+
+ controller, stop := mustCreateTestController(t, testConfig)
88+
+ defer stop()
89+
+
90+
+ nodegroups, err := controller.nodeGroups()
91+
+ if err != nil {
92+
+ t.Fatalf("unexpected error: %v", err)
93+
+ }
94+
+
95+
+ if l := len(nodegroups); l != 1 {
96+
+ t.Fatalf("expected 1 nodegroup, got %d", l)
97+
+ }
98+
+
99+
+ ng := nodegroups[0].(*nodegroup)
100+
+ if err := ng.IncreaseSize(1); err != nil {
101+
+ t.Fatalf("unexpected error: %v", err)
102+
+ }
103+
+
104+
+ gvr, err := ng.scalableResource.GroupVersionResource()
105+
+ if err != nil {
106+
+ t.Fatalf("unexpected error: %v", err)
107+
+ }
108+
+
109+
+ scalableResource, err := ng.machineController.managementScaleClient.Scales(testConfig.spec.namespace).
110+
+ Get(context.TODO(), gvr.GroupResource(), ng.scalableResource.Name(), metav1.GetOptions{})
111+
+ if err != nil {
112+
+ t.Fatalf("unexpected error: %v", err)
113+
+ }
114+
+
115+
+ if scalableResource.Spec.Replicas != 2 {
116+
+ t.Errorf("expected 2, got %v", scalableResource.Spec.Replicas)
117+
+ }
118+
+ }
119+
+
120+
+ annotations := map[string]string{
121+
+ nodeGroupMinSizeAnnotationKey: "3",
122+
+ nodeGroupMaxSizeAnnotationKey: "10",
123+
+ }
124+
+
125+
+ t.Run("MachineSet", func(t *testing.T) {
126+
+ test(t, createMachineSetTestConfig(RandomString(6), RandomString(6), RandomString(6), 1, annotations, nil))
127+
+ })
128+
+
129+
+ t.Run("MachineDeployment", func(t *testing.T) {
130+
+ test(t, createMachineDeploymentTestConfig(RandomString(6), RandomString(6), RandomString(6), 1, annotations, nil))
131+
+ })
132+
+}
133+
+
134+
+func TestNodeGroupSetSizeRefusesDecreaseBelowMinSize(t *testing.T) {
135+
+ test := func(t *testing.T, desired int, testConfig *testConfig) {
136+
+ controller, stop := mustCreateTestController(t, testConfig)
137+
+ defer stop()
138+
+
139+
+ nodegroups, err := controller.nodeGroups()
140+
+ if err != nil {
141+
+ t.Fatalf("unexpected error: %v", err)
142+
+ }
143+
+
144+
+ if l := len(nodegroups); l != 1 {
145+
+ t.Fatalf("expected 1 nodegroup, got %d", l)
146+
+ }
147+
+
148+
+ ng := nodegroups[0].(*nodegroup)
149+
+ err = ng.scalableResource.SetSize(desired)
150+
+ if err == nil {
151+
+ t.Fatal("expected an error")
152+
+ }
153+
+ if !strings.Contains(err.Error(), fmt.Sprintf("size decrease too large - desired:%d min:3", desired)) {
154+
+ t.Errorf("unexpected error: %v", err)
155+
+ }
156+
+ }
157+
+
158+
+ annotations := map[string]string{
159+
+ nodeGroupMinSizeAnnotationKey: "3",
160+
+ nodeGroupMaxSizeAnnotationKey: "10",
161+
+ }
162+
+
163+
+ for _, tc := range []struct {
164+
+ description string
165+
+ initial int
166+
+ desired int
167+
+ }{
168+
+ {"from above min size", 4, 2},
169+
+ {"from below min size", 2, 1},
170+
+ } {
171+
+ t.Run(tc.description+"/MachineSet", func(t *testing.T) {
172+
+ test(t, tc.desired, createMachineSetTestConfig(RandomString(6), RandomString(6), RandomString(6), tc.initial, annotations, nil))
173+
+ })
174+
+
175+
+ t.Run(tc.description+"/MachineDeployment", func(t *testing.T) {
176+
+ test(t, tc.desired, createMachineDeploymentTestConfig(RandomString(6), RandomString(6), RandomString(6), tc.initial, annotations, nil))
177+
+ })
178+
+ }
179+
+}
180+
diff --git a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go
181+
index f374dc7..0e4e469 100644
182+
--- a/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go
183+
+++ b/cluster-autoscaler/cloudprovider/clusterapi/clusterapi_unstructured.go
184+
@@ -106,13 +106,6 @@ func (r unstructuredScalableResource) Replicas() (int, error) {
185+
}
186+
187+
func (r unstructuredScalableResource) SetSize(nreplicas int) error {
188+
- switch {
189+
- case nreplicas > r.maxSize:
190+
- return fmt.Errorf("size increase too large - desired:%d max:%d", nreplicas, r.maxSize)
191+
- case nreplicas < r.minSize:
192+
- return fmt.Errorf("size decrease too large - desired:%d min:%d", nreplicas, r.minSize)
193+
- }
194+
-
195+
gvr, err := r.GroupVersionResource()
196+
if err != nil {
197+
return err
198+
@@ -127,6 +120,12 @@ func (r unstructuredScalableResource) SetSize(nreplicas int) error {
199+
return fmt.Errorf("unknown %s %s/%s", r.Kind(), r.Namespace(), r.Name())
200+
}
201+
202+
+ // Only a decrease is refused below min size: a group standing under it
203+
+ // has to be able to grow toward it one step at a time.
204+
+ if nreplicas < r.minSize && nreplicas < int(s.Spec.Replicas) {
205+
+ return fmt.Errorf("size decrease too large - desired:%d min:%d", nreplicas, r.minSize)
206+
+ }
207+
+
208+
s.Spec.Replicas = int32(nreplicas)
209+
_, updateErr := r.controller.managementScaleClient.Scales(r.Namespace()).Update(context.TODO(), gvr.GroupResource(), s, metav1.UpdateOptions{})
210+

0 commit comments

Comments
 (0)