Follow-ups for v1.7.1: readiness waits, FoundationDB pins, upgrade findings #5165

Left for 1.7.1 after the v1.7.0 cut. Every item was checked again at v1.7.0-rc.3, main has the same code for all of them.

• Gateway release reads a TenantGateway with a reconcile error as Current

  From the review of IvanHunters on fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates #5086. markFailed in internal/controller/tenantgateway/reconciler.go writes observedGeneration together with Ready=False/ReconcileError, so current: "true" in packages/apps/tenant/templates/gateway.yaml passes a TenantGateway the controller cannot reconcile and the tenant release goes on. ReconcileError also carries transient API errors, so instead of a failed expression on that reason the controller should mark terminal config errors (failed spec validation, GatewayClassUnsupported) with Stalled=True and the expression should read it. What the gateway release waits for gets written down in the same change.

• Generation gate of the gateway release does not hold on a fresh install

  Same review, CodeRabbit flagged it too. When TenantGateway has no status.observedGeneration Flux skips the generation check, so on a fresh install the release can be Ready before the controller wrote any status. Install behaviour is the same as before fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates #5086 because kstatus also reads a CR without status as Current, but the comment in gateway.yaml says otherwise. Fix the comment, or make the expression wait for the first status write.

• E2E for the gateway release with a pending certificate

  Same review. helm-unittest pins only the rendered spec and the controller side has TestReconcile_StatusObservedGeneration, nothing checks that the tenant gateway HelmRelease gets Ready while its TenantGateway is ListenersNotReady. I checked it by hand on rc.3, it needs a chainsaw step.

• Migration 61: copy healthCheckExprs and waitStrategy onto existing releases before the chart artifacts change

  fix(controller): carry healthCheckExprs and waitStrategy to existing releases #5087 does the copy from cozystack-controller, which comes up only after Cilium, Kube-OVN and the ApplicationDefinition CRD, while tenant releases upgrade as soon as their chart artifact changes. On a test cluster the vm-disk chart changed 12m38s before the copy, so a disk with a running import failed two 10-minute attempts and held a helm-controller slot for 20m33s (pre-upgrade item 15 of the v1.7.0 notes). Migration 61 writes the same fields by the same rules (skip releases labelled helm.toolkit.fluxcd.io/name, poller when there are expressions) from the pre-upgrade hook, which runs before the upgrade re-points cozystack-packages, and changing only these fields starts no Helm action on a healthy release. The migrations image has no charts, so it needs its own copy of the expressions (vm-disk DataVolume and postgres ScaledObject, both current: "true") with a bats check against cozyrds, like migration 60 has for the FoundationDB tags. Only variants with migrations.enabled run it. It helps clusters going from 1.6.x straight to 1.7.1 and any later change of the expressions, on 1.7.0 it is a no-op. Slot 61 is free on main and release-1.7 (last migration is 60, targetVersion: 61 on both), so targetVersion becomes 62. feat(kafka)!: run on Strimzi 1.2.0 and Kafka 4.x #4900, feat(postgres)!: mint tenant passwords through the credentials API #4922 and feat(kubernetes): give each tenant Kubernetes cluster its own etcd #4817 still add a migration 60 and will want 61 after rebase, so this one has to land on main first or main and release-1.7 number it differently.

• FoundationDB: raise the pins to 7.3.79 and 7.4.8

  packages/apps/foundationdb/files/versions.yaml maps v7.3 to 7.3.63 and v7.4 to 7.4.1, and migration 60 moves a cluster to the pinned patch of its line, downwards included. With higher pins the operator moves clusters on 7.3.63 and 7.4.1 up on the 1.7.1 upgrade, and a 1.6.x cluster at or below the new patch no longer steps down. FOUNDATIONDB_TAGS in migration 60 has to change with it, hack/migration-60-foundationdb-version.bats checks that. Both tags exist upstream, and foundationdb/foundationdb, foundationdb/foundationdb-kubernetes-sidecar (-1 tags) and foundationdb/fdb-kubernetes-monitor have them on Docker Hub for linux/amd64 and linux/arm64. The vendored operator v2.30.0 supports FDB 7.1.0 and later and treats patches of one line as protocol compatible, so its 7.3.63 and 7.4.1 client libraries can stay (upstream v2.38.0 ships the same ones). Not settled yet: GitHub marks 7.3.79 as pre-release (newest 7.3 without the mark is 7.3.77) and marks every 7.4 release too, 7.4.1 included, upstream operator e2e runs 7.3.69 and 7.4.5, and I haven't run 7.3.79 or 7.4.8 under this operator.

• vm-instance: platform upgrade unplugs a volume that migration 59 copied into the VM spec

  vm-instance: a change to the disk list unplugs hotplug volumes the chart does not render #4769. Seen on v1.6.5 to v1.7.0-rc.2: migration 59 copied a VMI-only hotplug volume into VM.spec, the vm-instance release upgrade in the same platform upgrade applied the chart's volume list over it (server-side apply, the list is atomic in the CRD) and KubeVirt unplugged the disk from the running VM. packages/apps/vm-instance/templates/vm.yaml and migration 59 did not change since rc.2. The v1.7.0 notes give a manual step (pre-upgrade item 5), not tried on a live VM yet. Fix as proposed in the issue: render back the hotpluggable volumes of the live VM that .Values.disks does not list.

• OpenSearch: instances created before 1.7 keep a securityconfig Job that never reaches the cluster

  On a cluster whose networking.clusterDomain is not cluster.local (default cozy.local included) the <release>-securityconfig-update Job made before 1.7 still targets .svc.cluster.local, so securityadmin never runs against the cluster. Nothing in rc.3 replaces the Job, and the operator recreates it only when the securityconfig checksum changes. On a test cluster I deleted such a Job, created on v1.6.5 and already ended Failed (BackoffLimitExceeded). The operator created a new one 3 seconds later with the host .svc.cozy.local, its curl probe passed and securityadmin connected with the admin certificate, then it waited for YELLOW cluster state and kept timing out for 15 minutes. That instance also had a second fault (bootstrap and node pods held different cluster UUIDs), so it is not confirmed that initialization finishes after the delete. Tracked separately in opensearch: securityconfig Job of an instance created before v1.7 keeps the cluster.local host after upgrade and is never recreated #5164. Fix: a migration or pre-upgrade hook that deletes such a Job.

• OpenSearch: WorkloadsReady counts the bootstrap pod, so an instance that never initialized reads as operational

  The WorkloadMonitor in packages/apps/opensearch/templates/opensearch.yaml:170-182 selects every pod labelled opster.io/opensearch-cluster: <release> with minReplicas: 1, and internal/controller/workloadmonitor_controller.go:908, :924 and :1044 mark it operational once that many selected pods are Ready. The bootstrap pod carries the same label and was Ready while security was not initialized, so on the instance above the monitor saw 2 pods with 1 available (bootstrap Ready, the only node pod nodes-0 at 0/1), and the application showed WorkloadsReady=True "All workloads are operational" (pkg/registry/apps/application/rest.go:1652-1654) for the whole check. Ready copies the HelmRelease (rest.go:1580-1591) and workload health does not override it by design (rest.go:1658-1666), so WorkloadsReady is the only place this can show. Node pool pods also carry opster.io/opensearch-nodepool: nodes and the bootstrap pod does not, so adding that label to the selector would leave the bootstrap pod out. Same class as WorkloadsReady reports all workloads operational for a vm-disk whose PVC never binds #4139 for vm-disk.

• Kafka: node pools created by the migration hook have no Flux labels

  The KafkaNodePools in packages/apps/kafka/templates/migration-hook.yaml are created by a hook Job with no helm.toolkit.fluxcd.io/name label and no owner reference, so lineage walk from broker and controller pods stops at the pool and those pods get no apps.cozystack.io/application.* labels. Pools are created only there, so every Kafka cluster on 1.7 has it. Unchanged at rc.3, and on a test cluster after the rc.3 upgrade broker and controller pods have no application labels while the entity operator pod has them. I did not check what this breaks in WorkloadMonitor or policies. fix(apps): keep Job and CronJob names within the Kubernetes limits #4945 changes the same file.