[Backport release-1.6] fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out - #4918

What this PR does

Backport of #4778 to release-1.6, fixes #3740 on the 1.6 line.

On the kube-ovn variants (isp-full, isp-full-generic) cilium allocates no pod IPs, but it still takes its router IP and the Gateway API ingress IP from Node.spec.podCIDR, a /24 out of the same range kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an address cilium already holds. With this change cilium runs ipam.mode: cluster-pool with a /29 per node out of the last /21 of networking.podCIDR, platform passes the same range to kube-ovn as --default-exclude-ips, and a post-upgrade hook in the cilium chart moves existing clusters. Mechanism and hook steps are the same as in #4778, its description has the details.

This is breaking for a patch release, please read before merging:

• networking.podCIDR has to be /20 or wider on the kube-ovn variants, otherwise platform refuses to render and the upgrade stops before anything changes. Widening the value later doesn't help, the hook refuses a block outside the subnet the cluster was installed with.
• A /21 of /29s caps these variants at 256 nodes.
• Every cilium agent restarts twice, once for the config change and once in the hook. While the cilium release is failed, packages that depend on networking wait for it.
• If kube-ovn already handed out an address from the last /21, the hook names that address and stops before it moves any node. Recreate the pods holding it and the next retry goes through.
• Rolling back to 1.6.4 brings ipam.mode: kubernetes back, agents take their /24 again, and the exclusions stay in ovn-default.

Differences from #4778

• No slim variants on release-1.6, so the cozystack.platform.slim gate is gone and the pool is wired in the isp-full and isp-full-generic branches only. isp-hosted renders noop and is not touched.
• The code around the wiring in system.yaml is release-1.6's (no gateway.http2, no networking.encryption, the older cozystack.platform.package call).
• The wiring test for isp-slim is left out of the cherry-pick, the second commit adds one for isp-hosted with a /24 pod CIDR, which has to keep rendering without the /20 requirement.
• packages/system/cilium has no make test target here, so hack/helm-unit-tests.sh would skip the new suite without a word. The third commit adds it.
• The follow-up from main that loads test_helper in hack/cilium-ipam-migration.bats is not applied. release-1.6 has no hack/test_helper.bash and runs hack/*.bats through hack/cozytest.sh, with that line the file doesn't load.
• release-1.6 ships kube-ovn v1.15.10, fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out #4778 was written against v1.15.26.

Validation

• helm unittest: cilium 11/11, platform 113/113 with [Backport release-1.6] fix(cert-manager): enable the Gateway API solver in the host cert-manager #4808 merged. hack/cilium-ipam-migration.bats passes 27/27 through hack/cozytest.sh.
• Live hook run on a three-node Talos cluster with kube-ovn v1.15.10. I put cilium on the release-1.6 chart first, then upgraded to this branch with helm upgrade (operator stopped, HelmRelease suspended). A pod pinned to 10.244.250.10 made the hook add the block and the old ingress IPs to excludeIps of ovn-default, name that address and stop before moving any node. After deleting the pod the same upgrade moved all three nodes to /29s from 10.244.248.0/21, router and ingress IPs inside. The available range of the subnet ends at 10.244.247.255, no pod IP is duplicated, and pod to pod across nodes, DNS and a ping to a VM pod IP work. A third run of the same upgrade restarted nothing and left the CIDRs as they were.
• Rolling the release back on that cluster returned the agents to their /24s with new ingress IPs, and excludeIps stayed until I patched it by hand.
• Not run: the upgrade through the operator, the generic variant, Gateway API traffic (no gateway on that cluster). kube-ovn still admits a pod with a static ovn.kubernetes.io/ip_address inside the block after the migration, the hook looks at it only while migrating.

Downstream repositories

The website note for this change is cozystack/website#739, merged for docs/next. The v1.6 docs need the same note, not opened yet, so the box stays empty.

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:

Release note

fix(cilium)!: on the kube-ovn variants cilium now takes its router and gateway api ingress ips from the last /21 of networking.podCIDR, which kube-ovn no longer hands out, so pods stop landing on addresses cilium holds. existing clusters move on upgrade through a post-upgrade hook, and each cilium agent restarts twice, once for the config change and once in the hook. networking.podCIDR needs /20 or wider on these variants, and a /21 of /29s caps them at 256 nodes.