Skip to content

Commit 338a925

Browse files
committed
fix(installer): let the operator Deployment report Available honestly
At replicas: 1 with maxUnavailable: 1 the Deployment controller takes the minimum available replica count as 1 - 1 = 0, so the Available condition reports True with no running pod at all. That reading is what every consumer of the status sees: an alert on Available, a health dashboard, or a person reading kubectl get deploy all show green while the operator is dead. The replacement is not a tuned pair, it is the platform default written down. At replicas: 1 the default 25% / 25% resolves to maxSurge 1 and maxUnavailable 0, because the surge rounds up and the unavailability rounds down. So this change stops overriding the default rather than introducing a configuration of its own, and Kubernetes rejects maxUnavailable: 0 together with maxSurge: 0, so the surge could not have gone the other way regardless. The overridden pair was carried over from the earlier cozystack Deployment, where a darkhttpd sidecar published a host port and hostNetwork left a surge pod unschedulable on a single-node cluster. It arrived here in two steps: by the commit before this file was created the sidecar was already gone and no port was declared, yet the pair stayed, so it had outlived its own reason before it was ever copied. That is why looking for a justification in this file's history finds none. Checked on a single-node cluster that the surge pod is scheduled beside the outgoing one and the rollout completes, and that declaring a port brings the scheduling failure straight back. Stay on RollingUpdate rather than move to Recreate, which would fix the condition too, since the minimum available replica count is 0 for any strategy that is not RollingUpdate and needs no surge to get there. Every live release already carries an explicit rollingUpdate block, and server-side apply leaves that block in place when the type changes, so the apiserver refuses the upgrade with spec.strategy.rollingUpdate: Forbidden. This project has already shipped a fix for exactly that failure, in ouroboros 0.8.1. Editing the numbers inside the block is upgrade-safe from every state the field can already be in. The surge introduces new windows, and leader election covers less of them than it looks. The operator runs four privileged install steps before mgr.Start(), and the lease is only acquired inside mgr.Start(): the CRD install, the Flux install, the platform source resource and the platform PackageSource. All four therefore run in the incoming pod outside leader election entirely. What the lease does cover is the reconcile loops, the part that would otherwise conflict continuously instead of once at startup. That leaves three seams, and only the first two happen on a healthy upgrade. The routine one is the incoming pod's install steps against the outgoing pod's live reconcile loops, since the outgoing pod ran its own installs at its startup and has been inside mgr.Start() ever since. Those do not contend for the same fields: the installs write as field manager cozystack-operator, while the reconcilers write as cozystack-packagesource-controller and cozystack-package-controller and go mostly to status subresources. Different owners, different objects, so this seam is safe by separation rather than by any shared manager. The second is version skew rather than field contention, and maxSurge: 1 is what opens it. The incoming pod reports Ready the instant its container starts, because there is no readiness probe, so its CRD install runs while the outgoing binary's reconcile loops are still live and now writing against a schema the incoming pod has already moved. Old code against a new schema, which no field manager separates. It is bounded rather than benign: the outgoing pod exits on SIGTERM within seconds, a rejected write is transient and the next reconcile repairs it, and no object is left corrupt. Widening the skew, by adding a probe that delays Ready or by a migration the old code cannot round-trip, is what would make this seam matter. The third seam is install against install, and it needs the outgoing pod to be restarting, since a crashlooping pod re-runs its four steps on every restart, which is exactly the broken operator this gate exists to catch. Three of the four steps use server-side apply under the same field manager, so they never conflict and resolve last-writer-wins whenever the two pods carry different manifests. That ordering matters in one direction, and it is accepted here rather than guarded against: the CRD install and the Flux install both apply with Force: true, so an outgoing pod that writes last silently overwrites what the incoming pod has just applied, with no conflict and no error to exit on. The incoming pod is inside mgr.Start() by then and will not apply again until it restarts, so that revert of the upgrade's install phase persists rather than healing itself. Reaching it needs a restart landing in the seconds between the incoming pod's installs and the outgoing pod's SIGTERM. installPlatformSourceResource is the remaining exception, a read-modify-write of a Get followed by Create or by Update with the observed resourceVersion, so two pods running it at once can collide on AlreadyExists or on a conflict; that failure is loud and self-healing, since any error there exits the process and the pod restarts, and the object is not left corrupt. Pin the strategy and all four preconditions the surge makes load-bearing in a unit test: no container or initContainer declares a port, both listener arguments stay disabled, leader election stays on, and no probe uses httpGet. They fail differently and none is visible to the others, since a declared port stops the scheduler, a bound one only fails once the incoming process starts, and a second manager without a lease fails nothing at all until it writes. The httpGet precondition follows from the two disabled listeners leaving no port to target, and it is the worst of the four to get wrong: the other three wedge an upgrade, while a livenessProbe against a disabled listener crashloops the pod on every cluster, including a fresh install where no surge happens. Only the httpGet field is forbidden rather than the probes themselves, because an exec probe stays legitimate. All four are checked against all three variants, because the template branches on variant inside the container spec and a port added under one branch renders for that variant alone. The strategy type is pinned alongside its two values, since maxSurge and maxUnavailable say nothing on their own once the type is no longer RollingUpdate. The listener preconditions are pinned less tightly than the other two, and the test says so rather than implying coverage it does not have. A membership assert has no notion of position while Go's flag package keeps the last occurrence, so a duplicate flag anywhere in the list overrides the disabled one and the assert still passes; reordering the arguments therefore fixes nothing. Rejecting the two addresses this binary defaults to closes the shape a values-gated block to enable metrics would take, and nothing available here closes an arbitrary address, since helm-unittest cannot express that a flag appears at most once. Note in hack/helm-unit-tests.sh that the loop covers packages/core, which its comment omitted. The suite added here lives there, so a later edit reconciling the loop with the stale comment would have silently stopped running it. Split the rationale by audience rather than keeping it all in shipped comments. What an operator applying the rendered manifest has to respect stays in plain comments and travels into the install artifacts: that the pair is the default, why maxUnavailable must stay 0, that no port may be declared or bound, that no probe may use httpGet, and that leader election has to stay on. What stays in the template comment is what the chart's own suite pins, so it cannot drift out of step with the chart without a test going red: the four preconditions and which assert covers each, why the variant loop is needed, and why the webhook server configured on 9443 is not a port binder, controller-runtime adding it to the runnables only inside GetWebhookServer(), which this binary never calls. The seam analysis above deliberately stays here instead of in the template. It describes another file's internals, nothing pins it, and a commit message is fixed in time, so it cannot quietly fall out of date the way a comment can. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 517dd11 commit 338a925

3 files changed

Lines changed: 233 additions & 5 deletions

File tree

‎hack/helm-unit-tests.sh‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@
22
set -eu
33

44
# Script to run unit tests for all Helm charts.
5-
# It iterates through directories in packages/apps, packages/extra,
6-
# packages/system, and packages/library and runs the 'test' Makefile
7-
# target if it exists.
5+
# It iterates through directories in packages/apps, packages/core,
6+
# packages/extra, packages/system, and packages/library and runs the 'test'
7+
# Makefile target if it exists. Keep this list in step with the loop below:
8+
# packages/core carries suites of its own, so dropping it to match a stale
9+
# comment would silently stop running them.
810

911
FAILED_DIRS_FILE="$(mktemp)"
1012
trap 'rm -f "$FAILED_DIRS_FILE"' EXIT

‎packages/core/installer/templates/cozystack-operator.yaml‎

Lines changed: 85 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,62 @@
1+
{{/*
2+
Maintainer notes on the rollout strategy. The operator-facing constraints it
3+
implies are in plain # comments further down, because those ship to whoever
4+
applies the rendered _out/assets/cozystack-operator-*.yaml. Everything here
5+
is about this repository and stays inside the template.
6+
7+
tests/rollout_strategy_test.yaml pins the four preconditions below against
8+
every variant the chart ships, and the strategy values themselves on talos
9+
alone — which suffices only because the strategy sits outside every variant
10+
branch, so move it inside one and that coverage silently narrows. Three
11+
preconditions are pinned airtight; precondition 2 is not, and says where the
12+
gap is rather than leaving the reader to assume coverage that is not there:
13+
14+
1. No container or initContainer declares a port. Asserted with a
15+
containers[*] wildcard rather than containers[0], because the constraint
16+
binds every container in the pod and not only the first: under hostNetwork
17+
a declared containerPort defaults to the same hostPort, and the surge pod
18+
then has nowhere to go on a single-node cluster.
19+
2. Neither listener is bound at runtime. --metrics-bind-address=0 and
20+
--health-probe-bind-address=0 are asserted explicitly, because the
21+
operator's own defaults are :8080 and :8081. The arguments are the only
22+
thing holding this, and the scheduler cannot see it either way. This is
23+
the loosely pinned one, and the reason is worth stating precisely so
24+
nobody tries the repair that does not work. contains is a membership
25+
test with no notion of position, while Go's flag package keeps the last
26+
occurrence, so a duplicate of either flag anywhere in the list overrides
27+
the 0 and the assert still passes. Reordering args therefore fixes
28+
nothing. notContains on :8080 and :8081 closes that for those two
29+
addresses, again regardless of position, and they are the ones a
30+
values-gated block added to let users enable metrics would reach for,
31+
since they are this binary's own flag defaults. Any other address slips
32+
through: helm-unittest cannot express "this flag appears no more than
33+
once". So changing a value in place is caught, and duplicating one is
34+
caught only for those two addresses.
35+
3. --leader-elect stays on. maxSurge: 1 means a second manager process
36+
exists by construction on every upgrade, so the lease is what keeps the
37+
incoming pod's reconcile loops off the cluster until the outgoing one is
38+
gone. Turning it off lets two managers reconcile at once.
39+
4. No probe uses httpGet. This follows from 2: with both listeners disabled
40+
an httpGet probe has no port left to target. Only the httpGet field is
41+
asserted absent, not the probes themselves, so an exec probe stays
42+
legitimate.
43+
44+
The variant loop is not ceremony: the template branches on variant inside the
45+
container spec, so a port added under one branch renders for that variant
46+
alone and a single-variant suite would not see it. The container-name assert
47+
pins containers[0] for the three argument asserts, which would otherwise start
48+
describing a different container if one were inserted ahead of the operator.
49+
50+
The manager is also configured with a webhook server on 9443, which reads
51+
like a third binder and is not one: controller-runtime adds that server to
52+
its runnables only inside GetWebhookServer(), and this binary registers no
53+
webhook and never calls it, so nothing listens. Registering one would bind
54+
9443 in the host namespace and wedge the rollout the same way, and no
55+
chart-level test can reach that — it is a Go-side change. PprofBindAddress is
56+
a fourth binder of the same kind: unset today, with no flag exposing it and
57+
no path to it from this chart, so it is inert for the same reason and would
58+
wedge the rollout for the same reason if it were ever set.
59+
*/}}
160
{{- $validVariants := list "talos" "generic" "hosted" -}}
261
{{- if not (has .Values.cozystackOperator.variant $validVariants) -}}
362
{{- fail (printf "Invalid cozystackOperator.variant %q: must be one of talos, generic, hosted" .Values.cozystackOperator.variant) -}}
@@ -35,8 +94,28 @@ spec:
3594
strategy:
3695
type: RollingUpdate
3796
rollingUpdate:
38-
maxSurge: 0
39-
maxUnavailable: 1
97+
# This pair is what Kubernetes itself would pick here: the default
98+
# 25% / 25% resolves at replicas: 1 to maxSurge 1 and maxUnavailable 0,
99+
# because the surge rounds up and the unavailability rounds down. Spelling
100+
# it out is not tuning, it is declining to override the default.
101+
# maxUnavailable must stay 0. At replicas: 1 anything higher makes the
102+
# minimum available replica count replicas - maxUnavailable = 0, and the
103+
# Available condition then reports True with no running pod at all, which
104+
# hides a dead operator from every consumer of that status.
105+
# Kubernetes rejects maxUnavailable: 0 together with maxSurge: 0, so the
106+
# surge cannot go the other way either. The surge pod shares the node
107+
# network namespace (hostNetwork below), so nothing in this pod may need
108+
# a port. Declaring one on any container makes the upgrade unschedulable
109+
# on a single-node cluster; binding one at runtime, by pointing
110+
# --metrics-bind-address or --health-probe-bind-address at a real address,
111+
# kills the incoming pod with EADDRINUSE instead. Either way the rollout
112+
# wedges for good, because maxUnavailable: 0 means the outgoing pod is
113+
# never retired.
114+
# This also rules out an httpGet probe of any kind on this Deployment: the
115+
# port it would target is one of the two disabled above. Only an exec
116+
# probe can ever work here.
117+
maxSurge: 1
118+
maxUnavailable: 0
40119
template:
41120
metadata:
42121
labels:
@@ -47,6 +126,10 @@ spec:
47126
- name: cozystack-operator
48127
image: "{{ .Values.cozystackOperator.image }}"
49128
args:
129+
# Must stay on. maxSurge: 1 above means a second manager pod exists on
130+
# every upgrade, and the lease is what keeps the incoming one's
131+
# reconcile loops off the cluster until the outgoing one is gone.
132+
# Turning it off lets two managers reconcile the same cluster at once.
50133
- --leader-elect=true
51134
- --install-flux=true
52135
# The operator applies embedded CRDs via server-side apply on every
Lines changed: 143 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,143 @@
1+
suite: cozystack-operator rollout strategy
2+
3+
# Pins the rollout strategy and the four preconditions the surge pod makes
4+
# load-bearing: no declared port, no bound listener, leader election on, no
5+
# httpGet probe. Each is checked against every variant the chart ships. The
6+
# bound-listener one is pinned only partially, for a reason recorded next to
7+
# the asserts themselves; the other three are airtight.
8+
#
9+
# Why the strategy is what it is, why each precondition matters and how each one
10+
# fails: the maintainer note at the top of templates/cozystack-operator.yaml.
11+
# That note is the single owner of the reasoning; this file only asserts it.
12+
13+
templates:
14+
- templates/cozystack-operator.yaml
15+
16+
tests:
17+
- it: keeps a rollout strategy that lets Available go False
18+
set:
19+
cozystackOperator.variant: talos
20+
asserts:
21+
- equal:
22+
path: spec.replicas
23+
value: 1
24+
documentSelector:
25+
path: kind
26+
value: Deployment
27+
# Pinned alongside the two values below because they are meaningless
28+
# without it: MaxUnavailable() reports 0 for any strategy that is not
29+
# RollingUpdate, so a switch to Recreate would satisfy the intent of this
30+
# test by a different route while leaving the rollingUpdate block behind
31+
# as dead configuration the API server then rejects at apply.
32+
- equal:
33+
path: spec.strategy.type
34+
value: RollingUpdate
35+
documentSelector:
36+
path: kind
37+
value: Deployment
38+
- equal:
39+
path: spec.strategy.rollingUpdate.maxUnavailable
40+
value: 0
41+
documentSelector:
42+
path: kind
43+
value: Deployment
44+
- equal:
45+
path: spec.strategy.rollingUpdate.maxSurge
46+
value: 1
47+
documentSelector:
48+
path: kind
49+
value: Deployment
50+
51+
- it: keeps every surge precondition on the talos variant
52+
set:
53+
cozystackOperator.variant: talos
54+
asserts: &surgeSafe
55+
- equal:
56+
path: spec.template.spec.hostNetwork
57+
value: true
58+
documentSelector:
59+
path: kind
60+
value: Deployment
61+
- notExists:
62+
path: spec.template.spec.containers[*].ports
63+
documentSelector:
64+
path: kind
65+
value: Deployment
66+
- notExists:
67+
path: spec.template.spec.initContainers[*].ports
68+
documentSelector:
69+
path: kind
70+
value: Deployment
71+
# The three argument asserts read containers[0], so pin which container
72+
# that is. A container inserted ahead of the operator would otherwise
73+
# shift the index and leave them asserting about something else.
74+
- equal:
75+
path: spec.template.spec.containers[0].name
76+
value: cozystack-operator
77+
documentSelector:
78+
path: kind
79+
value: Deployment
80+
# contains is a membership test with no notion of position, while Go's
81+
# flag package keeps the last occurrence, so these two alone would stay
82+
# green against a duplicate that binds the port wherever it sits — moving
83+
# the =0 flag around does not help. The notContains pair below closes
84+
# that for the two addresses this binary defaults its own flags to; any
85+
# other address still slips through.
86+
- contains:
87+
path: spec.template.spec.containers[0].args
88+
content: --metrics-bind-address=0
89+
documentSelector:
90+
path: kind
91+
value: Deployment
92+
- notContains:
93+
path: spec.template.spec.containers[0].args
94+
content: --metrics-bind-address=:8080
95+
documentSelector:
96+
path: kind
97+
value: Deployment
98+
- contains:
99+
path: spec.template.spec.containers[0].args
100+
content: --health-probe-bind-address=0
101+
documentSelector:
102+
path: kind
103+
value: Deployment
104+
- notContains:
105+
path: spec.template.spec.containers[0].args
106+
content: --health-probe-bind-address=:8081
107+
documentSelector:
108+
path: kind
109+
value: Deployment
110+
- contains:
111+
path: spec.template.spec.containers[0].args
112+
content: --leader-elect=true
113+
documentSelector:
114+
path: kind
115+
value: Deployment
116+
# Only the httpGet field is asserted absent, not the probes themselves, so
117+
# an exec probe stays legitimate.
118+
- notExists:
119+
path: spec.template.spec.containers[*].readinessProbe.httpGet
120+
documentSelector:
121+
path: kind
122+
value: Deployment
123+
- notExists:
124+
path: spec.template.spec.containers[*].livenessProbe.httpGet
125+
documentSelector:
126+
path: kind
127+
value: Deployment
128+
- notExists:
129+
path: spec.template.spec.containers[*].startupProbe.httpGet
130+
documentSelector:
131+
path: kind
132+
value: Deployment
133+
134+
- it: keeps every surge precondition on the generic variant
135+
set:
136+
cozystackOperator.variant: generic
137+
cozystack.apiServerHost: api.example.com
138+
asserts: *surgeSafe
139+
140+
- it: keeps every surge precondition on the hosted variant
141+
set:
142+
cozystackOperator.variant: hosted
143+
asserts: *surgeSafe

0 commit comments

Comments
 (0)