Gateway API + LB-IPAM: cilium-gateway Service LoadBalancer IP flaps (release/re-assign ~1/s), breaking BGP service advertisement (v1.5.2) #3307

Summary

On Cozystack v1.5.2 with the Gateway API publishing path enabled (tenant.spec.gateway: true + platform gateway.enabled: true), the LB-IPAM assignment on the generated cilium-gateway-* Service flaps continuously: status.loadBalancer.ingress alternates between the assigned IP and empty every 1–2 seconds, indefinitely. Any consumer of the Service status is broken by this — in our case the Cilium BGP control plane (v2 CRDs) advertises and withdraws the /32 route ~2 times per second (our upstream FRR router logged 520 BGP UPDATE messages in under 4 minutes for a single prefix), making BGP-based LoadBalancer reachability unusable.

This looks like the "known-bad interaction between LB-IPAM and Gateway" mentioned as out-of-scope in #2586 — opening this issue since we could not find a dedicated tracking issue, and we have a fairly complete reproduction + diagnosis.

Environment

• Cozystack v1.5.2 (variant isp-full), installed via cozy-installer Helm chart
• Talos v1.13.0 (Cozystack nocloud image), Kubernetes v1.34.3
• 3 combined control-plane+worker nodes (Proxmox VE, 8 vCPU / 32 GB each, 2 NICs: mgmt + external)
• Cilium: Cozystack build cilium:v1.5.2 (Cilium 1.19.3), operator quay.io/cilium/operator-generic:v1.19.3 (also reproduced with operator v1.19.5)
• Networking: default kubeovn-cilium variant (Kube-OVN primary CNI + Cilium chained, kube-proxy replacement, routing-mode=native)

Configuration

• Tenant root: gateway: true → TenantGateway → Gateway tenant-root/cozystack (class cilium), certMode: existingSecret (wildcard secret)
• Platform package: gateway.enabled: true, attachedNamespaces per docs → dashboard published via HTTPRoute (works)
• CiliumLoadBalancerIPPool with a single /32 block; the generated Service tenant-root/cilium-gateway-cozystack gets that IP
• Cilium BGP control plane enabled with CiliumBGPClusterConfig / CiliumBGPPeerConfig / CiliumBGPAdvertisement (advertisementType: Service, addresses: [LoadBalancerIP]), eBGP peering with an FRR router — sessions establish fine from all 3 nodes
• externalTrafficPolicy of the gateway Service is Cluster (so not Gateway API and BGP control plane cilium/cilium#36804)

Observed behaviour

1. Sampling the Service once per second shows the LB IP flapping (empty ↔ assigned, several transitions per 20 s window):

[1] ip=172.16.255.253
[2] ip=
[3] ip=172.16.255.253
[4] ip=
[5] ip=172.16.255.253
...

2. cilium-operator logs show a tight reconcile loop (~1/s, continuous, no errors):

level=info msg="Reconciling Gateway" ... controller=gateway resource=tenant-root/cozystack
level=info msg="Successfully reconciled Gateway" ...
level=info msg="Reconciling GAMMA service" ... controller=gamma resource=tenant-root/cilium-gateway-cozystack
(repeats forever, ~40 "Reconciling Gateway" per minute)

3. managedFields captured at the moment the status is empty show cilium-operator-lb-ipam is the only writer of status (owning an empty f:loadBalancer.f:ingress), i.e. LB-IPAM itself releases and re-assigns the IP on every Service update performed by the gateway/GAMMA reconcilers.

4. The BGP speaker faithfully follows the Service status → advertise/withdraw storm. FRR peer stats: Updates Rcvd: 520 within a ~4 min session for one prefix. External reachability alternates between working / connection refused / no route to host.

5. Notably, the datapath is unaffected: the BPF service entry for the VIP stays programmed the whole time; with a static route to a node IP, HTTPS requests to the VIP return 302/200 consistently while the status keeps flapping.

What we ruled out / tried

• externalTrafficPolicy is Cluster (not the Gateway API and BGP control plane cilium/cilium#36804 case)
• Restarting cilium agents: no change (datapath resync ok, flap continues)
• Restarting cilium-operator: no change
• Pinning the IP with lbipam.cilium.io/ips on the Service: annotation persists, flap continues
• Adding spec.externalIPs to advertise via ExternalIP instead: the gateway controller's periodic re-apply removes the field within ~40 s (SSA prune), so no stable workaround on the Service object itself
• Bumping only the operator to quay.io/cilium/operator-generic:v1.19.5: reconcile loop and flap unchanged
• k8s client rate limit raised (k8sClientRateLimit: qps 100 / burst 200) and L2-announcement lease timers relaxed earlier in the investigation: unrelated to this loop

Expected behaviour

The cilium-gateway-* Service should keep its LB-IPAM address stably; gateway/GAMMA reconciliation should not cause LB-IPAM to release/re-assign the IP, so that BGP service advertisement (and anything else reading status.loadBalancer) is stable.

Questions

• Is there a dedicated upstream/cozystack tracking issue for this interaction (referenced in Migrate LoadBalancer announcer from MetalLB to Cilium native (LB-IPAM + L2) #2586)? Happy to close this as duplicate if so.
• Is a fixed Cilium version already identified for the Cozystack v1.6 line?

Workaround we settled on

Static route for the VIP /32 towards a node IP on the external network (datapath is stable), keeping the established BGP sessions in place for when the fix lands.