Repository navigation
Commit 33fa5e5
committed
fix(etcd): fix etcd-operator 1.5->1.6 upgrade (immutable selector + OOM floor)
Two defects that break the etcd-operator on an in-cluster 1.5 -> 1.6 upgrade
(#2859 swapped the upstream chart for the cozystack-authored one):
- #3242: the controller Deployment's spec.selector.matchLabels changed
({instance,name} -> {name,control-plane}). spec.selector is immutable, so
`helm upgrade` cannot patch the existing Deployment and the whole HelmRelease
upgrade fails ("field is immutable") -- the operator and the v1alpha2 CRDs it
serves never come up, which blocks the etcd v1alpha2 adoption. Fresh installs
use the new selector directly, so fresh-install CI does not catch it. Add a
pre-upgrade hook that deletes the Deployment ONLY when its live selector is
the pre-1.6 one (lacks control-plane=controller-manager), so Helm recreates it
cleanly. No-op when the selector already matches (rc.1 -> later) and never
runs on a fresh install (pre-upgrade only). Keeping the selector stable in the
chart is not an option: rc.1 already shipped the new selector, so aligning it
back would merely move the immutable break to rc.1 -> next.
- Raise the manager's cold-start memory limit floor 128Mi -> 256Mi (and the VPA
minAllowed to match). The steady-state working set is ~250Mi (the VPA's own
recommendation); at 128Mi a Pod that starts before the VPA admission webhook
rewrites it (e.g. a Deployment recreated out of band, or the webhook briefly
unavailable during upgrade) OOMKills into a crash loop. Defense in depth so
the operator never depends on VPA timing merely to avoid crashing; the VPA
still scales it further under load up to maxAllowed.
Verified live on a 1.5 -> 1.6 adoption: the delete-Deployment step plus the VPA
re-applying 256Mi+ let the operator come up and the adoption complete.
Refs: #3242, #3243
Signed-off-by: Andrey Kolkov <[email protected]>1 parent 2932ecc commit 33fa5e5
2 files changed
Lines changed: 169 additions & 4 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
23 | | - | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
24 | 33 | | |
25 | 34 | | |
26 | 35 | | |
27 | 36 | | |
28 | 37 | | |
29 | 38 | | |
30 | | - | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
31 | 42 | | |
32 | | - | |
| 43 | + | |
33 | 44 | | |
34 | | - | |
| 45 | + | |
35 | 46 | | |
36 | 47 | | |
37 | 48 | | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
0 commit comments