Skip to content

Commit 115d9bb

Browse files
committed
fix(installer): let the operator Deployment report Available honestly
At replicas: 1 with maxUnavailable: 1 the Deployment controller takes the minimum available replica count as 1 - 1 = 0, so the Available condition reports True with no running pod at all. That reading is what every consumer of the status sees: an alert on Available, a health dashboard, or a person reading kubectl get deploy all show green while the operator is dead. The replacement is not a tuned pair, it is the platform default written down. At replicas: 1 the default 25% / 25% resolves to maxSurge 1 and maxUnavailable 0, because the surge rounds up and the unavailability rounds down. So this change stops overriding the default rather than introducing a configuration of its own, and Kubernetes rejects maxUnavailable: 0 together with maxSurge: 0, so the surge could not have gone the other way regardless. The overridden pair was carried over from the earlier cozystack Deployment, where a darkhttpd sidecar published a host port and hostNetwork left a surge pod unschedulable on a single-node cluster. It arrived here in two steps: by the commit before this file was created the sidecar was already gone and no port was declared, yet the pair stayed, so it had outlived its own reason before it was ever copied. That is why looking for a justification in this file's history finds none. With no port declared anywhere in the pod there is no hostNetwork port conflict left, so the surge pod schedules beside the outgoing one; declaring one reintroduces exactly the scheduling failure the pair was added for. Not exercised on a live cluster in this rework. Stay on RollingUpdate rather than move to Recreate, which would fix the condition too, and by a shorter route: MaxUnavailable() reports 0 for any strategy that is not RollingUpdate, so the minimum available replica count becomes replicas - 0 = 1 and the condition is honest without any surge pod at all. The transition was not attempted, and the failure this project fixed in ouroboros 0.8.1 does not transfer to it. There the rollingUpdate block was the one the apiserver defaults onto a Deployment created without one, so it belonged to no applier and outlived the switch to Recreate. This Deployment has carried an explicit block since the commit that created it, and nothing applies it server-side, so whether Recreate would upgrade cleanly is an open question about the field-ownership history of the live object rather than a known rejection. What keeps RollingUpdate here is availability rather than apply semantics: maxUnavailable: 0 leaves the outgoing operator running when the replacement never starts, where Recreate deletes it first and leaves none. Editing the numbers inside the block is upgrade-safe from every state the field can already be in. The surge introduces new windows, and leader election covers less of them than it looks. The operator runs four privileged install steps before mgr.Start(), and the lease is only acquired inside mgr.Start(): the CRD install, the Flux install, the platform source resource and the platform PackageSource. All four therefore run in the incoming pod outside leader election entirely. What the lease does cover is the reconcile loops, the part that would otherwise conflict continuously instead of once at startup. That leaves three seams, and only the first two happen on a healthy upgrade. The routine one is the incoming pod's install steps against the outgoing pod's live reconcile loops, since the outgoing pod ran its own installs at its startup and has been inside mgr.Start() ever since. Those do not contend for the same fields: the installs write as field manager cozystack-operator, while the reconcilers write as cozystack-packagesource-controller and cozystack-package-controller and go mostly to status subresources. Different owners, different objects, so this seam is safe by separation rather than by any shared manager. The second is version skew rather than field contention, and maxSurge: 1 widens it rather than opening it. Under maxSurge: 0 the surge budget counts desired replicas rather than live pods, so the incoming pod is created while the outgoing one may still be Terminating - the overlap existed. What it did not have was a live counterparty: scale-down sends SIGTERM before the new pod is created, and that cancels the manager context, so the old loops were already unwinding. Here the outgoing pod is not signalled until the incoming one is available, so the overlap runs against a fully live reconciler for a termination grace period instead of a vestigial one. The incoming pod reports Ready the instant its container starts, because there is no readiness probe, so its CRD install runs while the outgoing binary's reconcile loops are still live and now writing against a schema the incoming pod has already moved. Old code against a new schema, which no field manager separates. It is bounded rather than benign: the outgoing pod exits on SIGTERM within seconds, a rejected write is transient and the next reconcile repairs it, and no object is left corrupt. Widening the skew, by adding a probe that delays Ready or by a migration the old code cannot round-trip, is what would make this seam matter. The third seam is install against install, and it needs the outgoing pod to be restarting, since a crashlooping pod re-runs its four steps on every restart, which is exactly the broken operator this gate exists to catch. Three of the four steps use server-side apply under the same field manager, so they never conflict and resolve last-writer-wins whenever the two pods carry different manifests. That ordering matters in one direction, and it is accepted here rather than guarded against: the CRD install and the Flux install both apply with Force: true, so an outgoing pod that writes last silently overwrites what the incoming pod has just applied, with no conflict and no error to exit on. The incoming pod is inside mgr.Start() by then and will not apply again until it restarts, so that revert of the upgrade's install phase persists rather than healing itself. Reaching it needs a restart landing in the seconds between the incoming pod's installs and the outgoing pod's SIGTERM. installPlatformSourceResource is the remaining exception, a read-modify-write of a Get followed by Create or by Update with the observed resourceVersion, so two pods running it at once can collide on AlreadyExists or on a conflict; that failure is loud and self-healing, since any error there exits the process and the pod restarts, and the object is not left corrupt. Pin the strategy and all four preconditions the surge makes load-bearing in a unit test: no container or initContainer declares a port, both listener arguments stay disabled, leader election stays on, and no probe uses httpGet. They fail differently and none is visible to the others, since a declared port stops the scheduler, a bound one only fails once the incoming process starts, and a second manager without a lease fails nothing at all until it writes. The httpGet precondition follows from the two disabled listeners leaving no port to target, and it is the worst of the four to get wrong: the other three wedge an upgrade, while a livenessProbe against a disabled listener crashloops the pod on every cluster, including a fresh install where no surge happens. Only the httpGet field is forbidden rather than the probes themselves, because an exec probe stays legitimate. All four are checked against all three variants, because the template branches on variant inside the container spec and a port added under one branch renders for that variant alone. The strategy type is pinned alongside its two values, since maxSurge and maxUnavailable say nothing on their own once the type is no longer RollingUpdate. The listener preconditions are pinned less tightly than the other two, and the test says so rather than implying coverage it does not have. A membership assert has no notion of position while Go's flag package keeps the last occurrence, so a duplicate after the disabled one wins and the assert still passes, while a duplicate before it loses and the assert passes just the same - the test cannot tell those two apart. Moving the disabled flag last fixes the process without making the assert catch anything. Rejecting the two addresses this binary defaults to closes the shape a values-gated block to enable metrics would take, and nothing available here closes an arbitrary address, since helm-unittest cannot express that a flag appears at most once. Note in hack/helm-unit-tests.sh that the loop covers packages/core, which its comment omitted. The suite added here lives there, so a later edit reconciling the loop with the stale comment would have silently stopped running it. Split the rationale by audience rather than keeping it all in shipped comments. What an operator applying the rendered manifest has to respect stays in plain comments and travels into the install artifacts: that the pair is the default, why maxUnavailable must stay 0, that no port may be declared or bound, that no probe may use httpGet, and that leader election has to stay on. Most of what stays in the template comment is what the chart's own suite pins, so it cannot drift out of step with the chart without a test going red: the four preconditions and which assert covers each, and why the variant loop is needed. Three things there are deliberate exceptions, kept because anyone editing the strategy has to know them and no assert can carry them - that the webhook server configured on 9443 is not a port binder, since controller-runtime adds it to the runnables only inside GetWebhookServer() and this binary never calls it; that two overlaps between the pods sit outside the lease; and why RollingUpdate is kept rather than Recreate. Those name what exists. The seam analysis above, which is how each one works, deliberately stays here instead. It describes another file's internals, nothing pins it, and a commit message is fixed in time, so it cannot quietly fall out of date the way a comment can. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 517dd11 commit 115d9bb

3 files changed

Lines changed: 264 additions & 5 deletions

File tree

‎hack/helm-unit-tests.sh‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,11 @@
22
set -eu
33

44
# Script to run unit tests for all Helm charts.
5-
# It iterates through directories in packages/apps, packages/extra,
6-
# packages/system, and packages/library and runs the 'test' Makefile
7-
# target if it exists.
5+
# It iterates through directories in packages/apps, packages/core,
6+
# packages/extra, packages/system, and packages/library and runs the 'test'
7+
# Makefile target if it exists. Keep this list in step with the loop below:
8+
# packages/core carries suites of its own, so dropping it to match a stale
9+
# comment would silently stop running them.
810

911
FAILED_DIRS_FILE="$(mktemp)"
1012
trap 'rm -f "$FAILED_DIRS_FILE"' EXIT

‎packages/core/installer/templates/cozystack-operator.yaml‎

Lines changed: 115 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,86 @@
1+
{{/*
2+
Maintainer notes on the rollout strategy. The operator-facing constraints it
3+
implies are in plain # comments further down, because those ship to whoever
4+
applies the rendered _out/assets/cozystack-operator-*.yaml. Everything here
5+
is about this repository and stays inside the template.
6+
7+
tests/rollout_strategy_test.yaml pins the strategy values and the four
8+
preconditions below against every variant the chart ships, all on one shared
9+
anchor, so gating any of them on a variant is caught by the variants it was
10+
not gated to. Three preconditions are pinned airtight; precondition 2 is not,
11+
and says where the gap is rather than leaving the reader to assume coverage
12+
that is not there:
13+
14+
1. No container or initContainer declares a port. Asserted with a
15+
containers[*] wildcard rather than containers[0], because the constraint
16+
binds every container in the pod and not only the first: under hostNetwork
17+
a declared containerPort defaults to the same hostPort, and the surge pod
18+
then has nowhere to go on a single-node cluster.
19+
2. Neither listener is bound at runtime. --metrics-bind-address=0 and
20+
--health-probe-bind-address=0 are asserted explicitly, because the
21+
operator's own defaults are :8080 and :8081. The arguments are the only
22+
thing holding this, and the scheduler cannot see it either way. This is
23+
the loosely pinned one, and the reason is worth stating precisely so
24+
nobody trusts the wrong model of it. contains is a membership test with
25+
no notion of position, while Go's flag package keeps the last occurrence,
26+
so a duplicate appearing AFTER the disabled one wins and the assert still
27+
passes; a duplicate before it loses, and the assert passes just the same.
28+
Moving the disabled flag last therefore fixes the process but not the
29+
test, which stays green either way. notContains on :8080 and :8081 closes
30+
that for those two addresses, again regardless of position, and they are
31+
the ones a values-gated block added to let users enable metrics would
32+
reach for, since they are this binary's own flag defaults. Any other
33+
address slips through: helm-unittest cannot express "this flag appears no
34+
more than once". So changing a value in place is caught, and duplicating
35+
one is caught only for those two addresses.
36+
3. --leader-elect stays on. maxSurge: 1 means a second manager process
37+
exists by construction on every upgrade, so the lease is what keeps the
38+
incoming pod's reconcile loops off the cluster until the outgoing one is
39+
gone. Turning it off lets two managers reconcile at once.
40+
4. No probe uses httpGet. This follows from 2: with both listeners disabled
41+
an httpGet probe has no port left to target. Only the httpGet field is
42+
asserted absent, not the probes themselves, so an exec probe stays
43+
legitimate — but under maxUnavailable: 0 an exec probe that cannot pass
44+
while the outgoing pod is still alive wedges the rollout permanently
45+
rather than transiently, since the outgoing pod is not retired until the
46+
incoming one is available. A readiness gate that depends on the old
47+
process being gone deadlocks against the strategy.
48+
49+
Read precondition 3 for exactly what it says and no more. The lease governs
50+
the reconcile loops; it does not govern everything the two pods do at once,
51+
and two overlaps sit outside it. The routine one happens on every healthy
52+
upgrade and is bounded by the outgoing pod's termination grace: for that
53+
moment an old binary keeps reconciling against schema the incoming pod has
54+
already moved. The rare one needs the outgoing pod to be broken and
55+
restarting, and it is the worse of the two, because one ordering of it
56+
silently reverts the incoming pod's install phase and does not heal itself
57+
afterwards. Both are known and accepted, not oversights. Unlike the binders
58+
below, neither is hypothetical.
59+
60+
What pays for carrying them, since Recreate would make Available honest with
61+
no second pod at all: when the replacement never starts — a surge pod that
62+
cannot be scheduled, an image that cannot be pulled — maxUnavailable: 0 keeps
63+
the outgoing operator running while the rollout wedges, where Recreate would
64+
have deleted it first and left none. That is the whole of the gain, and it
65+
covers only that class. A replacement that starts and then dies is already
66+
counted available, so the outgoing pod is gone by then under either strategy.
67+
68+
The variant loop is not ceremony: the template branches on variant inside the
69+
container spec, so a port added under one branch renders for that variant
70+
alone and a single-variant suite would not see it. The container-name assert
71+
pins containers[0] for the three argument asserts, which would otherwise start
72+
describing a different container if one were inserted ahead of the operator.
73+
74+
The manager is also configured with a webhook server on 9443, which reads
75+
like a third binder and is not one: controller-runtime adds that server to
76+
its runnables only inside GetWebhookServer(), and this binary registers no
77+
webhook and never calls it, so nothing listens. Registering one would bind
78+
9443 in the host namespace and wedge the rollout the same way, and no
79+
chart-level test can reach that — it is a Go-side change. PprofBindAddress is
80+
a fourth binder of the same kind: unset today, with no flag exposing it and
81+
no path to it from this chart, so it is inert for the same reason and would
82+
wedge the rollout for the same reason if it were ever set.
83+
*/}}
184
{{- $validVariants := list "talos" "generic" "hosted" -}}
285
{{- if not (has .Values.cozystackOperator.variant $validVariants) -}}
386
{{- fail (printf "Invalid cozystackOperator.variant %q: must be one of talos, generic, hosted" .Values.cozystackOperator.variant) -}}
@@ -35,8 +118,34 @@ spec:
35118
strategy:
36119
type: RollingUpdate
37120
rollingUpdate:
38-
maxSurge: 0
39-
maxUnavailable: 1
121+
# This pair is what Kubernetes itself would pick here: the default
122+
# 25% / 25% resolves at replicas: 1 to maxSurge 1 and maxUnavailable 0,
123+
# because the surge rounds up and the unavailability rounds down. Spelling
124+
# it out is not tuning, it is declining to override the default.
125+
# maxUnavailable must stay 0. At replicas: 1 anything higher makes the
126+
# minimum available replica count replicas - maxUnavailable = 0, and the
127+
# Available condition then reports True with no running pod at all, which
128+
# hides a dead operator from every consumer of that status.
129+
# Kubernetes rejects maxUnavailable: 0 together with maxSurge: 0, so the
130+
# surge cannot go the other way either. The surge pod shares the node
131+
# network namespace (hostNetwork below), so nothing in this pod may need
132+
# a port, and the two ways of needing one fail differently. Declaring a
133+
# containerPort makes the surge pod unschedulable on a single-node
134+
# cluster: it never runs, so maxUnavailable: 0 keeps the outgoing pod
135+
# serving and the rollout simply wedges. Binding a port at runtime, by
136+
# pointing --metrics-bind-address or --health-probe-bind-address at a
137+
# real address, is worse, because this pod has no readiness probe and
138+
# counts as available the moment its container starts. The outgoing pod
139+
# is retired at that point, and the incoming one dies with EADDRINUSE
140+
# afterwards, leaving no operator at all rather than the old one. Do not
141+
# read maxUnavailable: 0 as protection against that second class; what
142+
# catches it is whatever waits on the resources the operator is supposed
143+
# to create.
144+
# This also rules out an httpGet probe of any kind on this Deployment: the
145+
# port it would target is one of the two disabled above. Only an exec
146+
# probe can ever work here.
147+
maxSurge: 1
148+
maxUnavailable: 0
40149
template:
41150
metadata:
42151
labels:
@@ -47,6 +156,10 @@ spec:
47156
- name: cozystack-operator
48157
image: "{{ .Values.cozystackOperator.image }}"
49158
args:
159+
# Must stay on. maxSurge: 1 above means a second manager pod exists on
160+
# every upgrade, and the lease is what keeps the incoming one's
161+
# reconcile loops off the cluster until the outgoing one is gone.
162+
# Turning it off lets two managers reconcile the same cluster at once.
50163
- --leader-elect=true
51164
- --install-flux=true
52165
# The operator applies embedded CRDs via server-side apply on every
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
suite: cozystack-operator rollout strategy
2+
3+
# Pins the rollout strategy and the four preconditions the surge pod makes
4+
# load-bearing: no declared port, no bound listener, leader election on, no
5+
# httpGet probe. Each is checked against every variant the chart ships. The
6+
# bound-listener one is pinned only partially, for a reason recorded next to
7+
# the asserts themselves; the other three are airtight.
8+
#
9+
# Why the strategy is what it is, why each precondition matters and how each one
10+
# fails: the maintainer note at the top of templates/cozystack-operator.yaml.
11+
# That note is the single owner of the reasoning; this file only asserts it.
12+
13+
templates:
14+
- templates/cozystack-operator.yaml
15+
16+
tests:
17+
- it: keeps the strategy and every surge precondition on the talos variant
18+
set:
19+
cozystackOperator.variant: talos
20+
asserts: &surgeSafe
21+
# The strategy rides the same anchor as the preconditions rather than
22+
# sitting in a talos-only case of its own. It renders outside every
23+
# variant branch today, so one case would pass, but that is a property
24+
# of the template rather than a guarantee: move the block inside a
25+
# branch and a single-variant assert stops seeing the other two.
26+
- equal:
27+
path: spec.replicas
28+
value: 1
29+
documentSelector:
30+
path: kind
31+
value: Deployment
32+
# Pinned alongside the two values below because they are meaningless
33+
# without it: MaxUnavailable() reports 0 for any strategy that is not
34+
# RollingUpdate, so a switch to Recreate would satisfy the intent of this
35+
# test by a different route while leaving the rollingUpdate block behind
36+
# as dead configuration the API server then rejects at apply.
37+
- equal:
38+
path: spec.strategy.type
39+
value: RollingUpdate
40+
documentSelector:
41+
path: kind
42+
value: Deployment
43+
- equal:
44+
path: spec.strategy.rollingUpdate.maxUnavailable
45+
value: 0
46+
documentSelector:
47+
path: kind
48+
value: Deployment
49+
- equal:
50+
path: spec.strategy.rollingUpdate.maxSurge
51+
value: 1
52+
documentSelector:
53+
path: kind
54+
value: Deployment
55+
- equal:
56+
path: spec.template.spec.hostNetwork
57+
value: true
58+
documentSelector:
59+
path: kind
60+
value: Deployment
61+
- notExists:
62+
path: spec.template.spec.containers[*].ports
63+
documentSelector:
64+
path: kind
65+
value: Deployment
66+
- notExists:
67+
path: spec.template.spec.initContainers[*].ports
68+
documentSelector:
69+
path: kind
70+
value: Deployment
71+
# The three argument asserts read containers[0], so pin which container
72+
# that is. A container inserted ahead of the operator would otherwise
73+
# shift the index and leave them asserting about something else.
74+
- equal:
75+
path: spec.template.spec.containers[0].name
76+
value: cozystack-operator
77+
documentSelector:
78+
path: kind
79+
value: Deployment
80+
# contains is a membership test with no notion of position, while Go's
81+
# flag package keeps the last occurrence, so these two alone stay green
82+
# whether a duplicate lands before the =0 and loses or after it and binds
83+
# the port — they cannot tell the two apart. Moving the =0 flag last
84+
# would fix the process without making these asserts catch anything. The
85+
# notContains pair below closes that for the two addresses this binary
86+
# defaults its own flags to; any other address still slips through.
87+
- contains:
88+
path: spec.template.spec.containers[0].args
89+
content: --metrics-bind-address=0
90+
documentSelector:
91+
path: kind
92+
value: Deployment
93+
- notContains:
94+
path: spec.template.spec.containers[0].args
95+
content: --metrics-bind-address=:8080
96+
documentSelector:
97+
path: kind
98+
value: Deployment
99+
- contains:
100+
path: spec.template.spec.containers[0].args
101+
content: --health-probe-bind-address=0
102+
documentSelector:
103+
path: kind
104+
value: Deployment
105+
- notContains:
106+
path: spec.template.spec.containers[0].args
107+
content: --health-probe-bind-address=:8081
108+
documentSelector:
109+
path: kind
110+
value: Deployment
111+
- contains:
112+
path: spec.template.spec.containers[0].args
113+
content: --leader-elect=true
114+
documentSelector:
115+
path: kind
116+
value: Deployment
117+
# Only the httpGet field is asserted absent, not the probes themselves, so
118+
# an exec probe stays legitimate.
119+
- notExists:
120+
path: spec.template.spec.containers[*].readinessProbe.httpGet
121+
documentSelector:
122+
path: kind
123+
value: Deployment
124+
- notExists:
125+
path: spec.template.spec.containers[*].livenessProbe.httpGet
126+
documentSelector:
127+
path: kind
128+
value: Deployment
129+
- notExists:
130+
path: spec.template.spec.containers[*].startupProbe.httpGet
131+
documentSelector:
132+
path: kind
133+
value: Deployment
134+
135+
- it: keeps the strategy and every surge precondition on the generic variant
136+
set:
137+
cozystackOperator.variant: generic
138+
cozystack.apiServerHost: api.example.com
139+
asserts: *surgeSafe
140+
141+
- it: keeps the strategy and every surge precondition on the hosted variant
142+
set:
143+
cozystackOperator.variant: hosted
144+
asserts: *surgeSafe

0 commit comments

Comments
 (0)