Skip to content

Commit 0642ee0

Browse files
kvapsclaude
andcommitted
revert(ingress)!: remove broken publishing.exposure=loadBalancer mode
Reverts the ingress loadBalancer exposure mode added in #2468. The mode rendered the ingress-nginx Service as type: LoadBalancer and a CiliumLoadBalancerIPPool over publishing.externalIPs, as a migration path off Service.spec.externalIPs (deprecated upstream in Kubernetes v1.36 per KEP-5707). In default Cozystack the implementation produces two failure modes: - publishing.externalIPs is the operator-supplied list of node public IPs (addresses the cluster nodes already hold and listen on), not a virtual-VIP pool. Treating it as one and DNATing those IPs via Cilium LB-IPAM collides with the existing ingress-nginx DNAT on the same IPs (Service.spec.externalIPs path), so traffic is forwarded non-deterministically. - Cilium in default Cozystack runs with ipam.mode: kubernetes and no L2/BGP announcer, so a pool over addresses outside the node range is allocated but never advertised — the Service stays unreachable. The KEP-5707 deprecation is real, but the right migration is a mechanism-agnostic tenant API knob translated by a Cozystack-side controller into whichever backend the cluster admin has configured (MetalLB pool, Cilium LB-IPAM, robotlb, externalIPs pin) — not a chart-side IPPool baked into the tenant chart. That follow-up belongs in its own PR. Removed: - publishing.exposure value (and the inline comment) from core/platform/values.yaml - expose-mode plumbing line in core/platform/templates/apps.yaml - CiliumLoadBalancerIPPool template at apps/tenant/templates/cilium-lb-pool.yaml - type: LoadBalancer + externalTrafficPolicy: Local branch in extra/ingress/templates/nginx-ingress.yaml (Service renders back as ClusterIP + spec.externalIPs when externalIPs are set, falling through to type: LoadBalancer when they are not — pre-#2468 shape) - Exposure mode section from extra/ingress/README.md - exposure_test.yaml fixtures in both charts Operators who explicitly set publishing.exposure=loadBalancer will see their ingress Service revert to ClusterIP + spec.externalIPs on upgrade. Anyone on the default value (externalIPs) is unaffected. Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]>
1 parent 99e7070 commit 0642ee0

7 files changed

Lines changed: 2 additions & 403 deletions

File tree

‎packages/apps/tenant/templates/cilium-lb-pool.yaml‎

Lines changed: 0 additions & 28 deletions
This file was deleted.

‎packages/apps/tenant/tests/exposure_test.yaml‎

Lines changed: 0 additions & 141 deletions
This file was deleted.

‎packages/core/platform/templates/apps.yaml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,6 @@ stringData:
3030
expose-services: {{ .Values.publishing.exposedServices | join "," | quote }}
3131
expose-ingress: {{ .Values.publishing.ingressName | quote }}
3232
expose-external-ips: {{ .Values.publishing.externalIPs | join "," | quote }}
33-
expose-mode: {{ .Values.publishing.exposure | default "externalIPs" | quote }}
3433
cluster-domain: {{ .Values.networking.clusterDomain | quote }}
3534
api-server-endpoint: {{ .Values.publishing.apiServerEndpoint | quote }}
3635
{{- with .Values.branding }}

‎packages/core/platform/values.yaml‎

Lines changed: 0 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -45,41 +45,6 @@ publishing:
4545
- cdi-uploadproxy
4646
apiServerEndpoint: "" # example: "https://api.example.org"
4747
externalIPs: []
48-
# Exposure mode for the ingress-nginx Service. When "externalIPs" (current
49-
# default) is selected, the Service is created as ClusterIP with
50-
# Service.spec.externalIPs set from publishing.externalIPs. When
51-
# "loadBalancer" is selected, the Service is type: LoadBalancer and a
52-
# CiliumLoadBalancerIPPool makes those same addresses allocatable via LB IPAM.
53-
#
54-
# Service.spec.externalIPs is deprecated upstream in Kubernetes v1.36
55-
# (KEP-5707). The AllowServiceExternalIPs feature gate is expected to default
56-
# to false around v1.40 and the implementation removed around v1.43 — switch
57-
# to "loadBalancer" before upgrading past v1.40.
58-
#
59-
# Caveats for the "loadBalancer" mode:
60-
# - publishing.externalIPs must contain at least one non-empty address,
61-
# otherwise the chart render fails with an explicit error (a LoadBalancer
62-
# Service without a pool would sit in <pending> forever).
63-
# - The ingress-nginx Service is created with externalTrafficPolicy: Local
64-
# to preserve the client source IP. Traffic arriving on a node that does
65-
# not host an ingress-nginx pod is dropped, so the external IP must be
66-
# routed to a node that runs the ingress pod (floating IP / keepalived /
67-
# upstream router / podAntiAffinity).
68-
# - Cilium does NOT announce the IP on its own unless L2 announcements or
69-
# BGP are enabled in the Cilium values (disabled by default in Cozystack).
70-
# This mode assumes the operator already routes the externalIPs to a
71-
# cluster node; enabling announcements is out of scope for this setting.
72-
# - Switching this value on a running cluster causes the ingress-nginx
73-
# Service to be recreated (the HelmRelease has upgrade.force: true and
74-
# the Service kind changes between ClusterIP and LoadBalancer). Expect a
75-
# brief interruption of ingress traffic during the flip.
76-
#
77-
# Scope: this setting only controls the ingress-nginx Service. Other
78-
# cozystack components that currently write Service.spec.externalIPs directly
79-
# (e.g. the vpn app at packages/apps/vpn/templates/service.yaml) are NOT
80-
# migrated by flipping this value and must be addressed separately before
81-
# the AllowServiceExternalIPs feature gate flips to off in ~v1.40.
82-
exposure: externalIPs # "externalIPs" or "loadBalancer"
8348
# Enable PROXY-protocol on the host ingress-nginx and auto-deploy
8449
# ouroboros (system/ouroboros) to fix the resulting hairpin-NAT
8550
# problem (intra-cluster traffic to the cluster's own public hostnames

‎packages/extra/ingress/README.md‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -14,17 +14,3 @@
1414
| `resources.memory` | Memory (RAM) available to each replica. | `quantity` | `""` |
1515
| `resourcesPreset` | Default sizing preset used when `resources` is omitted. | `string` | `micro` |
1616

17-
18-
## Exposure mode
19-
20-
The ingress Service type is driven by the cluster-wide `publishing.exposure` value in the platform chart, not by any key in this package. Two modes exist:
21-
22-
- `externalIPs` (default) has three rendered shapes:
23-
- Release namespace matches `publishing.ingressName` AND `publishing.externalIPs` is non-empty → Service is `ClusterIP` with `Service.spec.externalIPs` set from that list and `externalTrafficPolicy: Cluster`.
24-
- Release namespace matches `publishing.ingressName` but `publishing.externalIPs` is empty → Service falls back to `type: LoadBalancer` with `externalTrafficPolicy: Local`.
25-
- Release namespace does not match `publishing.ingressName` (non-root tenants) → Service is `type: LoadBalancer` with `externalTrafficPolicy: Local`.
26-
`Service.spec.externalIPs` is deprecated upstream in Kubernetes v1.36 (KEP-5707); plan migration before v1.40.
27-
- `loadBalancer` — Service is `type: LoadBalancer` with `externalTrafficPolicy: Local`, and a `CiliumLoadBalancerIPPool` makes the addresses in `publishing.externalIPs` allocatable via Cilium LB IPAM. Requires `publishing.externalIPs` to contain at least one non-empty address (render fails otherwise) and assumes the addresses are already routed to a cluster node (floating IP / upstream router). See the inline comment on `publishing.exposure` in the platform chart for full caveats, including the note that switching the value on a running cluster causes the ingress Service to be recreated.
28-
29-
This setting only migrates ingress-nginx away from `Service.spec.externalIPs`. Other cozystack components that use the same deprecated field (e.g. the `vpn` app) must be migrated separately before Kubernetes v1.40 flips the `AllowServiceExternalIPs` feature gate off.
30-

‎packages/extra/ingress/templates/nginx-ingress.yaml‎

Lines changed: 2 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,5 @@
11
{{- $exposeIngress := (index .Values._cluster "expose-ingress") | default "tenant-root" }}
22
{{- $exposeExternalIPs := (index .Values._cluster "expose-external-ips") | default "" | nospace }}
3-
{{- $exposeMode := (index .Values._cluster "expose-mode") | default "externalIPs" }}
4-
{{- $exposeIPsList := list }}
5-
{{- range splitList "," $exposeExternalIPs }}
6-
{{- $ip := . | trim }}
7-
{{- if $ip }}
8-
{{- $exposeIPsList = append $exposeIPsList $ip }}
9-
{{- end }}
10-
{{- end }}
11-
{{- if not (has $exposeMode (list "externalIPs" "loadBalancer")) }}
12-
{{- fail (printf "unknown publishing.exposure mode %q: must be \"externalIPs\" or \"loadBalancer\"" $exposeMode) }}
13-
{{- end }}
14-
{{- if and (eq $exposeMode "loadBalancer") (eq $exposeIngress .Release.Namespace) (not $exposeIPsList) }}
15-
{{- fail "publishing.exposure=loadBalancer requires publishing.externalIPs to contain at least one non-empty address: the CiliumLoadBalancerIPPool has nothing to advertise and the Service would stay in <pending>." }}
16-
{{- end }}
173
apiVersion: helm.toolkit.fluxcd.io/v2
184
kind: HelmRelease
195
metadata:
@@ -55,12 +41,9 @@ spec:
5541
enabled: false
5642
{{- end }}
5743
service:
58-
{{- if and (eq $exposeIngress .Release.Namespace) (eq $exposeMode "loadBalancer") }}
59-
type: LoadBalancer
60-
externalTrafficPolicy: Local
61-
{{- else if and (eq $exposeIngress .Release.Namespace) $exposeIPsList }}
44+
{{- if and (eq $exposeIngress .Release.Namespace) $exposeExternalIPs }}
6245
externalIPs:
63-
{{- toYaml $exposeIPsList | nindent 12 }}
46+
{{- toYaml (splitList "," $exposeExternalIPs) | nindent 12 }}
6447
type: ClusterIP
6548
externalTrafficPolicy: Cluster
6649
{{- else }}

0 commit comments

Comments
 (0)