Skip to content

Commit dc06090

Browse files
mattia-eleuterimattia-eleuteri
committed
fix(platform): give the full recovery from a kept group snapshot CRD
A group snapshot CRD that migration 61 keeps carries no Helm ownership, so the chart leaves it out of the release: snapshot-controller upgrades with that API still at v1alpha1 rather than failing. Once the CRD is cleaned up nothing renders either release again, so the warning now lists every step: force a snapshot-controller upgrade to create the CRD at v1beta2, restart the controller so it sets up its group snapshot watches, then force a linstor upgrade. This also corrects 198d8ac, which said Helm refuses to upgrade over CRDs without ownership metadata. helm-controller enables takeover by default; what keeps foreign CRDs out of the release is the chart patch in 0d2e0ca, and migration 61's adoption keeps the release's own CRDs in it. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]>
1 parent c204b58 commit dc06090

2 files changed

Lines changed: 25 additions & 10 deletions

File tree

‎hack/migration-61-snapshot-crds.bats‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,12 @@ crd_field() {
144144
grep -qxF "DELETE volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io" "$FAKE_CMDLOG"
145145
if grep -qF "CREATE" "$FAKE_CMDLOG"; then echo "a v1beta2 CRD was created next to a kept v1alpha1 one" >&2; return 1; fi
146146
grep -qF "cozystack.io/migration-61-needs-attention=volumegroupsnapshots.groupsnapshot.storage.k8s.io=2" "$FAKE_CMDLOG"
147+
# The kept CRD stays out of the release, so the warning carries the whole
148+
# recovery: recreate it through the release, restart the controller, then
149+
# re-render linstor.
150+
grep -qF "flux reconcile helmrelease --namespace cozy-snapshot-controller snapshot-controller --force" "$WORK/out"
151+
grep -qF "kubectl rollout restart --namespace cozy-snapshot-controller deployment/snapshot-controller" "$WORK/out"
152+
grep -qF "flux reconcile helmrelease --namespace cozy-linstor linstor --force" "$WORK/out"
147153
grep -qxF "STAMP 62" "$FAKE_CMDLOG"
148154
rm -rf "$WORK"
149155
}
@@ -327,6 +333,12 @@ crd_field() {
327333
fi
328334
if grep -qF "CREATE" "$FAKE_CMDLOG"; then echo "a CRD was created next to the recorded one" >&2; return 1; fi
329335
grep -qF "cozystack.io/migration-61-needs-attention=volumegroupsnapshotcontents.groupsnapshot.storage.k8s.io=stored-v1alpha1" "$FAKE_CMDLOG"
336+
# The kept CRD stays out of the release, so the warning carries the whole
337+
# recovery: recreate it through the release, restart the controller, then
338+
# re-render linstor.
339+
grep -qF "flux reconcile helmrelease --namespace cozy-snapshot-controller snapshot-controller --force" "$WORK/out"
340+
grep -qF "kubectl rollout restart --namespace cozy-snapshot-controller deployment/snapshot-controller" "$WORK/out"
341+
grep -qF "flux reconcile helmrelease --namespace cozy-linstor linstor --force" "$WORK/out"
330342
grep -qxF "STAMP 62" "$FAKE_CMDLOG"
331343
rm -rf "$WORK"
332344
}

‎packages/core/platform/images/migrations/migrations/61‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -18,21 +18,22 @@
1818
# in status.storedVersions. The old snapshot-controller ran without the
1919
# CSIVolumeGroupSnapshot gate, so no v1alpha1 object was ever acted on, and an
2020
# empty CRD is deleted here. A CRD that still holds objects is left alone and
21-
# recorded on the cozystack-version ConfigMap: deleting it would delete them,
22-
# and the upgrade of that one release fails with the storedVersions error until
23-
# they are removed by hand, without holding the rest of the platform.
21+
# recorded on the cozystack-version ConfigMap: deleting it would delete them.
22+
# It carries no Helm ownership metadata, so the chart leaves it out of the
23+
# release, which upgrades with that group snapshot API still at v1alpha1.
2424
#
2525
# The deleted CRDs are created again right away from the v1beta2 copies baked
2626
# into the image (see image-migrations in the platform Makefile), already
2727
# carrying the release's ownership metadata. Every HelmRelease waits for this
2828
# hook, so the linstor release, which renders its group snapshot gate and
2929
# VolumeGroupSnapshotClass only when the v1beta2 API is served, renders against
3030
# it even when it upgrades before snapshot-controller. When a CRD was kept, none
31-
# is created: the API stays v1alpha1 and linstor renders the gate off. Recovery
32-
# is by hand, and the warning spells it out: delete the objects, then the CRD,
33-
# which still lists v1alpha1 in storedVersions and would keep failing the
34-
# snapshot-controller upgrade; once snapshot-controller serves the three
35-
# v1beta2 kinds, force a linstor upgrade, since nothing renders it again.
31+
# is created here, the release creates the missing ones, and linstor renders
32+
# the gate off. Recovery is by hand, since nothing renders either release again
33+
# once the CRD is gone, and RECOVERY below spells it out: delete the objects and
34+
# the CRD, force a snapshot-controller upgrade so the chart creates it at
35+
# v1beta2, restart snapshot-controller, which sets up its group snapshot
36+
# watches only when it starts, then force a linstor upgrade.
3637
#
3738
# Only CRDs labelled by helm-controller as installed by
3839
# cozy-snapshot-controller/snapshot-controller are touched, so a cluster whose
@@ -65,6 +66,8 @@ GROUP_SNAPSHOT_CRDS="volumegroupsnapshotclasses volumegroupsnapshotcontents volu
6566

6667
GROUP_CRD_DIR="${GROUP_CRD_DIR:-/snapshot-controller-crds}"
6768

69+
RECOVERY="once its objects are deleted, delete the CRD, run 'flux reconcile helmrelease --namespace $RELEASE_NS $RELEASE_NAME --force' so the chart creates it at v1beta2, 'kubectl rollout restart --namespace $RELEASE_NS deployment/snapshot-controller', then 'flux reconcile helmrelease --namespace cozy-linstor linstor --force' to enable group snapshots"
70+
6871
LEFTOVERS=""
6972
CREATE=""
7073

@@ -132,7 +135,7 @@ for plural in $GROUP_SNAPSHOT_CRDS; do
132135
# while v1alpha1 is still a stored version, so such a CRD is left to an
133136
# operator rather than adopted into an upgrade that cannot apply.
134137
if printf '%s' "$obj" | jq -e '.status.storedVersions // [] | index("v1alpha1")' >/dev/null; then
135-
echo "WARN: CRD $crd serves v1beta2 but still stores v1alpha1; the snapshot-controller upgrade fails until its objects are migrated and storedVersions trimmed, and the linstor HelmRelease needs a forced upgrade after that to enable group snapshots" >&2
138+
echo "WARN: CRD $crd serves v1beta2 but still stores v1alpha1; it stays out of $RELEASE_NS/$RELEASE_NAME and group snapshots stay off: $RECOVERY" >&2
136139
LEFTOVERS="${LEFTOVERS}${LEFTOVERS:+,}${crd}=stored-v1alpha1"
137140
continue
138141
fi
@@ -141,7 +144,7 @@ for plural in $GROUP_SNAPSHOT_CRDS; do
141144
fi
142145
count=$(kubectl get "$crd" --all-namespaces -o json | jq '.items | length')
143146
if [ "$count" -gt 0 ]; then
144-
echo "WARN: CRD $crd still holds $count v1alpha1 object(s); the snapshot-controller upgrade fails until they and then the empty CRD are deleted (the CRD keeps v1alpha1 in storedVersions), after which snapshot-controller recreates it at v1beta2 and the linstor HelmRelease needs a forced upgrade to enable group snapshots" >&2
147+
echo "WARN: CRD $crd still holds $count v1alpha1 object(s); it stays out of $RELEASE_NS/$RELEASE_NAME and group snapshots stay off: $RECOVERY" >&2
145148
LEFTOVERS="${LEFTOVERS}${LEFTOVERS:+,}${crd}=${count}"
146149
continue
147150
fi

0 commit comments

Comments
 (0)