Skip to content

Commit 938abdc

Browse files
authored
fix(platform)!: refuse a networking variant change on a live cluster (cozystack#4790)
## What this PR does Changing the platform variant on a running cluster can delete its CNI, and nothing stopped that before. A new platform variant (`spec.variant` of the `cozystack.cozystack-platform` Package) can give the `cozystack.networking` Package a different variant, and the operator then deletes every HelmRelease of that Package the new variant does not install. Moving from `isp-full` to `isp-hosted` deletes Cilium and Kube-OVN, so every pod loses networking. Moving from `isp-full` to `isp-slim` deletes Kube-OVN. With this PR the platform render looks up the live `cozystack.networking` Package. If its variant differs from the one about to be rendered, the render fails, and the error names the value that lets it through: `networking.acknowledgeVariantChange: "<new variant>"`. I made the acknowledgement a variant name instead of a boolean like `publishing.proxyProtocolAcknowledgeUnclean`. A leftover `true` would let every later switch through unchecked, while a variant name covers only the switch it names. The guard does nothing when there is no live Package to compare against: `helm template`, a client-side dry-run, or a first install. It is also skipped when `cozystack.networking` is listed in `bundles.disabledPackages`, because the platform does not render the Package then. It lives in the shared Package helper and not in `bundles/system.yaml`, so it covers both places that emit the networking Package, including the `noop` call in the `isp-hosted` branch. The other Packages whose variant follows the bundle, `cozystack-scheduler` and `tenant-application`, keep the same release names in every variant, so changing their variant deletes nothing and they need no guard. The new helm-unittest cases cover the refused switches, an unchanged variant, a matching and a mismatching acknowledgement, a bare boolean, `disabledPackages`, an offline render, and a live cluster with no networking Package yet. A cluster where someone set the Helm-owned networking Package to `cilium-kilo` by hand now fails its platform upgrade, and the error points it at `bundles.disabledPackages`. Acknowledging does not help there, because the render writes the bundle's variant over `cilium-kilo`. helm-controller v1.5.0 installs a release with server-side apply by default, which takes over every rendered field. On a release it installed, the upgrade replaces the variant and the operator deletes Kilo. A release first installed with Helm 3 keeps the apply method recorded in its history, so there the outcome depends on that history. Known gap, left for a follow-up: the guard fires on any variant change, including Talos to generic (`kubeovn-cilium` to `kubeovn-cilium-generic`). Those pairs share release names, so nothing is deleted and only values change. Refusing is safe, and the error does not claim a deletion for such a switch. Closes cozystack#4596 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories This PR adds a key to `packages/core/platform/values.yaml`, and the website keeps a hand-written table of those keys in `operations/configuration/platform-package.md`. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#746 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): the platform render now refuses to change the variant of a live cozystack.networking Package, because the operator would then delete the CNI HelmReleases the new variant does not install (isp-full to isp-hosted removes Cilium and Kube-OVN). To go ahead with such a change, set networking.acknowledgeVariantChange to the new networking variant named in the error. A cluster whose cozystack.networking variant was set by hand (for example cilium-kilo) needs cozystack.networking in bundles.disabledPackages before the upgrade, otherwise the platform render refuses. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Changing the networking variant on a running cluster now requires explicit confirmation naming the requested variant. A change without a matching confirmation is blocked. * The confirmation applies only to the specified switch and must be updated for a later variant change. If the running package has no recorded variant, it is treated as using the default variant. * New installations without an existing networking package are unaffected, and networking can still be disabled through its package setting. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 953b965 + 1245b32 commit 938abdc

3 files changed

Lines changed: 267 additions & 0 deletions

File tree

‎packages/core/platform/templates/_helpers.tpl‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,9 @@
88
{{- end -}}
99
{{- $disabled := default (list) $root.Values.bundles.disabledPackages -}}
1010
{{- if not (has $name $disabled) -}}
11+
{{- if eq $name "cozystack.networking" -}}
12+
{{- include "cozystack.platform.networking.variantGuard" (list $root $variant) -}}
13+
{{- end }}
1114
---
1215
apiVersion: cozystack.io/v1alpha1
1316
kind: Package
@@ -24,6 +27,29 @@ spec:
2427
{{- end }}
2528
{{ end }}
2629

30+
{{/*
31+
Refuses to change the variant of a live cozystack.networking Package. The
32+
operator deletes every HelmRelease of a Package that its new variant does not
33+
install, so isp-full -> isp-hosted removes cilium and kubeovn and leaves the
34+
cluster without a CNI. The acknowledgement names the target variant rather than
35+
being a boolean, so a value left set after one switch does not cover the next.
36+
Inert where lookup returns nothing: helm template, client-side dry-run, first
37+
install.
38+
Call as (list $ <variant about to be rendered>).
39+
*/}}
40+
{{- define "cozystack.platform.networking.variantGuard" -}}
41+
{{- $root := index . 0 -}}
42+
{{- $variant := index . 1 -}}
43+
{{- $live := lookup "cozystack.io/v1alpha1" "Package" "" "cozystack.networking" -}}
44+
{{- if $live -}}
45+
{{- $liveVariant := default "default" (dig "spec" "variant" "" $live) -}}
46+
{{- $ack := dig "acknowledgeVariantChange" "" ($root.Values.networking | default dict) | toString -}}
47+
{{- if and (ne $liveVariant $variant) (ne $ack $variant) -}}
48+
{{- fail (printf "the live cozystack.networking Package runs variant %q and this render would switch it to %q (bundles.system.variant: %s). The operator then deletes any HelmRelease of that Package the new variant does not install (none if it installs everything the current one does): isp-full to isp-hosted removes Cilium and Kube-OVN and leaves every pod without networking, isp-full to isp-slim removes Kube-OVN. Either restore the previous spec.variant of the cozystack.cozystack-platform Package, or, once the cluster has another CNI or no longer needs the removed one, set networking.acknowledgeVariantChange: %q in its spec.components.platform.values to confirm this exact switch. If the live variant was set by hand and has to stay, add cozystack.networking to bundles.disabledPackages: acknowledging lets this render replace it." $liveVariant $variant $root.Values.bundles.system.variant $variant) -}}
49+
{{- end -}}
50+
{{- end -}}
51+
{{- end }}
52+
2753
{{- define "cozystack.platform.package.default" -}}
2854
{{- $name := index . 0 -}}
2955
{{- $root := index . 1 -}}
Lines changed: 230 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,230 @@
1+
suite: networking variant change guard (lookup of the live cozystack.networking Package)
2+
3+
# A new networking variant is not a migration. The operator reconciles the
4+
# Package against the new variant and deletes every HelmRelease of the Package
5+
# that the new variant does not install: isp-full -> isp-hosted (noop) removes
6+
# cilium and kubeovn, isp-full -> isp-slim (cilium) removes kubeovn. The guard
7+
# reads the live Package and is inert where lookup returns nothing (helm
8+
# template, client-side dry-run, fresh install).
9+
templates:
10+
- templates/bundles/system.yaml
11+
release:
12+
name: cozystack
13+
namespace: cozy-system
14+
set:
15+
bundles:
16+
system:
17+
enabled: true
18+
variant: isp-full
19+
tests:
20+
- it: refuses isp-full -> isp-hosted on a live kubeovn-cilium cluster and names the acknowledge value
21+
set:
22+
bundles:
23+
system:
24+
variant: isp-hosted
25+
kubernetesProvider:
26+
scheme: &scheme
27+
"cozystack.io/v1alpha1/Package":
28+
gvr: {group: "cozystack.io", version: "v1alpha1", resource: "packages"}
29+
namespaced: false
30+
objects: &liveKubeovnCilium
31+
- kind: Package
32+
apiVersion: cozystack.io/v1alpha1
33+
metadata: {name: cozystack.networking}
34+
spec: {variant: kubeovn-cilium}
35+
asserts:
36+
- failedTemplate:
37+
errorPattern: 'cozystack.networking Package runs variant "kubeovn-cilium".*render would switch it to "noop".*restore the previous spec.variant of the cozystack.cozystack-platform Package.*networking.acknowledgeVariantChange: "noop" in its spec.components.platform.values'
38+
39+
- it: refuses isp-full -> isp-slim on a live kubeovn-cilium cluster
40+
set:
41+
bundles:
42+
system:
43+
variant: isp-slim
44+
kubernetesProvider:
45+
scheme: *scheme
46+
objects: *liveKubeovnCilium
47+
asserts:
48+
- failedTemplate:
49+
errorPattern: 'runs variant "kubeovn-cilium".*switch it to "cilium".*networking.acknowledgeVariantChange: "cilium"'
50+
51+
# Nothing is deleted on this switch, so the error must not say otherwise.
52+
- it: refuses isp-hosted -> isp-full on a live noop cluster
53+
kubernetesProvider:
54+
scheme: *scheme
55+
objects:
56+
- kind: Package
57+
apiVersion: cozystack.io/v1alpha1
58+
metadata: {name: cozystack.networking}
59+
spec: {variant: noop}
60+
asserts:
61+
- failedTemplate:
62+
errorPattern: 'runs variant "noop".*switch it to "kubeovn-cilium".*deletes any HelmRelease of that Package the new variant does not install \(none if it installs everything the current one does\)'
63+
64+
- it: treats a live Package without spec.variant as the default variant
65+
kubernetesProvider:
66+
scheme: *scheme
67+
objects:
68+
- kind: Package
69+
apiVersion: cozystack.io/v1alpha1
70+
metadata: {name: cozystack.networking}
71+
spec: {}
72+
asserts:
73+
- failedTemplate:
74+
errorPattern: 'runs variant "default".*switch it to "kubeovn-cilium"'
75+
76+
# No bundle renders cilium-kilo, so a live one was set by hand. Acknowledging
77+
# lets the render write the bundle's variant over it, so the error has to
78+
# point such a cluster at disabledPackages instead.
79+
- it: points a hand-set live variant at disabledPackages
80+
kubernetesProvider:
81+
scheme: *scheme
82+
objects: &liveCiliumKilo
83+
- kind: Package
84+
apiVersion: cozystack.io/v1alpha1
85+
metadata: {name: cozystack.networking}
86+
spec: {variant: cilium-kilo}
87+
asserts:
88+
- failedTemplate:
89+
errorPattern: 'runs variant "cilium-kilo".*restore the previous spec.variant of the cozystack.cozystack-platform Package.*networking.acknowledgeVariantChange: "kubeovn-cilium".*If the live variant was set by hand and has to stay, add cozystack.networking to bundles.disabledPackages: acknowledging lets this render replace it'
90+
91+
- it: lets an acknowledged switch replace a hand-set live variant
92+
set:
93+
networking:
94+
acknowledgeVariantChange: kubeovn-cilium
95+
kubernetesProvider:
96+
scheme: *scheme
97+
objects: *liveCiliumKilo
98+
documentSelector:
99+
path: metadata.name
100+
value: cozystack.networking
101+
asserts:
102+
- equal:
103+
path: spec.variant
104+
value: kubeovn-cilium
105+
106+
- it: renders when the live variant matches the rendered one
107+
kubernetesProvider:
108+
scheme: *scheme
109+
objects: *liveKubeovnCilium
110+
documentSelector:
111+
path: metadata.name
112+
value: cozystack.networking
113+
asserts:
114+
- equal:
115+
path: spec.variant
116+
value: kubeovn-cilium
117+
118+
- it: renders the change when acknowledgeVariantChange names the new variant
119+
set:
120+
bundles:
121+
system:
122+
variant: isp-hosted
123+
networking:
124+
acknowledgeVariantChange: noop
125+
kubernetesProvider:
126+
scheme: *scheme
127+
objects: *liveKubeovnCilium
128+
documentSelector:
129+
path: metadata.name
130+
value: cozystack.networking
131+
asserts:
132+
- equal:
133+
path: spec.variant
134+
value: noop
135+
136+
# The acknowledgement is bound to one target, so a value left behind after
137+
# one switch does not wave through the next one.
138+
- it: still refuses when acknowledgeVariantChange names a different variant
139+
set:
140+
bundles:
141+
system:
142+
variant: isp-hosted
143+
networking:
144+
acknowledgeVariantChange: cilium
145+
kubernetesProvider:
146+
scheme: *scheme
147+
objects: *liveKubeovnCilium
148+
asserts:
149+
- failedTemplate:
150+
errorPattern: 'networking.acknowledgeVariantChange: "noop"'
151+
152+
- it: still refuses when acknowledgeVariantChange is a bare boolean
153+
set:
154+
bundles:
155+
system:
156+
variant: isp-hosted
157+
networking:
158+
acknowledgeVariantChange: true
159+
kubernetesProvider:
160+
scheme: *scheme
161+
objects: *liveKubeovnCilium
162+
asserts:
163+
- failedTemplate:
164+
errorPattern: 'networking.acknowledgeVariantChange: "noop"'
165+
166+
# A Package left out of the render is not re-rendered, so Helm does not
167+
# touch its variant; helm.sh/resource-policy: keep leaves it as it is.
168+
- it: renders when cozystack.networking is in bundles.disabledPackages
169+
set:
170+
bundles:
171+
system:
172+
variant: isp-hosted
173+
disabledPackages:
174+
- cozystack.networking
175+
kubernetesProvider:
176+
scheme: *scheme
177+
objects: *liveKubeovnCilium
178+
asserts:
179+
- containsDocument:
180+
apiVersion: cozystack.io/v1alpha1
181+
kind: Package
182+
name: cozystack.networking
183+
any: true
184+
not: true
185+
186+
- it: renders offline (no live Package) for isp-hosted
187+
set:
188+
bundles:
189+
system:
190+
variant: isp-hosted
191+
documentSelector:
192+
path: metadata.name
193+
value: cozystack.networking
194+
asserts:
195+
- equal:
196+
path: spec.variant
197+
value: noop
198+
199+
- it: renders offline (no live Package) for isp-slim
200+
set:
201+
bundles:
202+
system:
203+
variant: isp-slim
204+
documentSelector:
205+
path: metadata.name
206+
value: cozystack.networking
207+
asserts:
208+
- equal:
209+
path: spec.variant
210+
value: cilium
211+
212+
- it: renders on a cluster that has other Packages but no cozystack.networking yet
213+
set:
214+
bundles:
215+
system:
216+
variant: isp-full-generic
217+
kubernetesProvider:
218+
scheme: *scheme
219+
objects:
220+
- kind: Package
221+
apiVersion: cozystack.io/v1alpha1
222+
metadata: {name: cozystack.cert-manager}
223+
spec: {variant: default}
224+
documentSelector:
225+
path: metadata.name
226+
value: cozystack.networking
227+
asserts:
228+
- equal:
229+
path: spec.variant
230+
value: kubeovn-cilium-generic

‎packages/core/platform/values.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -99,6 +99,17 @@ networking:
9999
# Set this to comma-separated list of master node IPs to override.
100100
kubeovn:
101101
MASTER_NODES: ""
102+
# Confirms a change of the cozystack.networking Package variant on a running
103+
# cluster. Changing the variant of the cozystack.cozystack-platform Package
104+
# can change that variant (isp-full runs kubeovn-cilium, isp-slim runs cilium,
105+
# isp-hosted runs noop), and the operator then deletes the networking
106+
# HelmReleases the new variant does not install: isp-full -> isp-hosted
107+
# removes Cilium and Kube-OVN. The platform render refuses such a change
108+
# while the live Package has another variant, unless this is set to the NEW
109+
# networking variant, as named in the render error. It confirms that one
110+
# switch only; a leftover value does not cover a later switch to a different
111+
# variant.
112+
acknowledgeVariantChange: ""
102113
# In-cluster (east-west) transport encryption. Opt-in, OFF by default.
103114
#
104115
# When enabled, pod-to-pod traffic that crosses nodes is encrypted at the

0 commit comments

Comments
 (0)