Skip to content

Commit 31c15b2

Browse files
authored
feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081)
## What this PR does Lays the foundation for migrating tenant LoadBalancer exposure off the deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off MetalLB onto Cilium-native announcement — the mechanism-agnostic shape prescribed when the earlier chart-side approach (#2468) was reverted in #2603. It adds a new `network.cozystack.io/v1alpha1` API group and a controller that translates intent into whichever LoadBalancer backend the cluster admin configured: - `ExposureClass` (cluster-scoped, admin): binds a logical class name to a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP address scope — the StorageClass/IngressClass pattern. - `ServiceExposure` (namespaced, tenant intent): "expose this Service via class X"; status reports the resolved backend and assigned IPs. - A `serviceexposure` controller renders the backend's pool + announcer once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium `CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to the namespaces that opted in. Backends are pluggable behind a small Go interface, so adding BGP or another cloud LB is one implementation, not an API change. The controller never mutates the target Service — the chart still owns its shape. Platform wiring: `publishing.exposureClass.*` renders the admin `ExposureClass` and the host ingress publishes through it. The default (no class set, `externalIPs`) leaves every existing cluster's behaviour byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes a single `ExposureClass.spec.backend` change. This is the foundation only — Part of #2586. Converting per-app exposure, flipping Cilium `l2announcements` on, changing the default, and removing MetalLB are deferred to later phases. ### Release note ```release-note feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new `ExposureClass` and `ServiceExposure` custom resources with support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB), including optional L2 and address configuration. * Added Helm wiring and templates to enable service/ingress publishing via `ServiceExposure`, selecting a configured (or default) `ExposureClass`. * **Bug Fixes** * Improved reconciliation, finalizer-based cleanup, and status updates for readiness and assigned IP reporting, including safer handling when backends or managed resources change. * **Tests** * Added end-to-end and unit test coverage for backend rendering, readiness, and cleanup flows. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 50748fe + 511f2a3 commit 31c15b2

31 files changed

Lines changed: 3795 additions & 2 deletions
Lines changed: 135 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,135 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package v1alpha1
18+
19+
import (
20+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
21+
)
22+
23+
// IsDefaultExposureClassAnnotation marks an ExposureClass as the
24+
// cluster default, mirroring the storageclass.kubernetes.io/is-default-class
25+
// convention. A ServiceExposure with an empty exposureClassName resolves
26+
// to the ExposureClass carrying this annotation set to "true".
27+
const IsDefaultExposureClassAnnotation = "exposureclass.network.cozystack.io/is-default-class"
28+
29+
// ExposureBackend names the cluster-side mechanism that allocates a
30+
// LoadBalancer IP and announces it on the network. A backend covers BOTH
31+
// allocation and announcement: for bare-metal the VIP pool and the L2
32+
// announcer are inseparable.
33+
// +kubebuilder:validation:Enum=externalIPs;metallb;cilium;robotlb
34+
type ExposureBackend string
35+
36+
const (
37+
// BackendExternalIPs pins the Service to operator-supplied addresses
38+
// via Service.spec.externalIPs (typically node public IPs). No pool
39+
// and no announcer are rendered — the node already owns the address.
40+
// This is the default and reproduces the historical behaviour.
41+
BackendExternalIPs ExposureBackend = "externalIPs"
42+
43+
// BackendMetalLB allocates from a metallb.io IPAddressPool and
44+
// announces via an L2Advertisement, both rendered by the controller
45+
// in the cozy-metallb namespace.
46+
BackendMetalLB ExposureBackend = "metallb"
47+
48+
// BackendCilium allocates from a CiliumLoadBalancerIPPool and
49+
// announces via a CiliumL2AnnouncementPolicy, both rendered by the
50+
// controller (cluster-scoped).
51+
BackendCilium ExposureBackend = "cilium"
52+
53+
// BackendRobotLB delegates allocation and announcement to a cloud
54+
// load balancer (e.g. Hetzner robotlb). The controller renders no
55+
// in-cluster pool or announcer; it only reports the assigned IP.
56+
BackendRobotLB ExposureBackend = "robotlb"
57+
)
58+
59+
// ExposureClassSpec is the cluster-admin backend configuration. It maps a
60+
// logical exposure class to a concrete backend and its address scope,
61+
// following the StorageClass / IngressClass / GatewayClass pattern: the
62+
// admin owns the backend choice, tenants reference the class by name.
63+
type ExposureClassSpec struct {
64+
// Backend selects the concrete allocation+announcement mechanism.
65+
// +required
66+
Backend ExposureBackend `json:"backend"`
67+
68+
// Addresses is the VIP address scope for pool-based backends (metallb,
69+
// cilium), as CIDRs or start-end ranges. These are virtual VIPs, NOT
70+
// node public IPs. Ignored by the externalIPs and robotlb backends.
71+
// +optional
72+
Addresses []string `json:"addresses,omitempty"`
73+
74+
// L2 enables L2 (ARP/NDP) announcement for bare-metal backends
75+
// (metallb, cilium). Defaults to true. Set false when announcement is
76+
// handled out of band (e.g. BGP added later, or a routed network).
77+
// +kubebuilder:default=true
78+
// +optional
79+
L2 *bool `json:"l2,omitempty"`
80+
81+
// Interfaces optionally scopes L2 announcement to the named host
82+
// network interfaces (cilium CiliumL2AnnouncementPolicy.spec.interfaces;
83+
// metallb L2Advertisement.spec.interfaces). Empty announces on all.
84+
// +optional
85+
Interfaces []string `json:"interfaces,omitempty"`
86+
}
87+
88+
// ExposureClassStatus reports the observed state of the ExposureClass.
89+
type ExposureClassStatus struct {
90+
// ObservedGeneration mirrors the .metadata.generation reflected in the
91+
// latest reconciled state.
92+
// +optional
93+
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
94+
95+
// Conditions describes the current state of the ExposureClass.
96+
// Standard condition type: Ready.
97+
// +optional
98+
// +patchMergeKey=type
99+
// +patchStrategy=merge
100+
// +listType=map
101+
// +listMapKey=type
102+
Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
103+
}
104+
105+
// +kubebuilder:object:root=true
106+
// +kubebuilder:subresource:status
107+
// +kubebuilder:resource:scope=Cluster,shortName=expcls
108+
// +kubebuilder:printcolumn:name="Backend",type="string",JSONPath=".spec.backend"
109+
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
110+
111+
// ExposureClass is the Schema for the exposureclasses API. It is the
112+
// cluster-admin-facing configuration that binds a logical class name to a
113+
// LoadBalancer backend and its address scope. Tenants reference it by name
114+
// from a ServiceExposure; the cozystack-controller renders the backend's
115+
// pool and announcer resources from this CR.
116+
type ExposureClass struct {
117+
metav1.TypeMeta `json:",inline"`
118+
metav1.ObjectMeta `json:"metadata,omitempty"`
119+
120+
Spec ExposureClassSpec `json:"spec,omitempty"`
121+
Status ExposureClassStatus `json:"status,omitempty"`
122+
}
123+
124+
// +kubebuilder:object:root=true
125+
126+
// ExposureClassList contains a list of ExposureClass.
127+
type ExposureClassList struct {
128+
metav1.TypeMeta `json:",inline"`
129+
metav1.ListMeta `json:"metadata,omitempty"`
130+
Items []ExposureClass `json:"items"`
131+
}
132+
133+
func init() {
134+
SchemeBuilder.Register(&ExposureClass{}, &ExposureClassList{})
135+
}
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
// Package v1alpha1 contains API Schema definitions for the
18+
// network.cozystack.io v1alpha1 API group.
19+
// +kubebuilder:object:generate=true
20+
// +groupName=network.cozystack.io
21+
package v1alpha1
22+
23+
import (
24+
"k8s.io/apimachinery/pkg/runtime/schema"
25+
"sigs.k8s.io/controller-runtime/pkg/scheme"
26+
)
27+
28+
var (
29+
// GroupVersion is group version used to register these objects.
30+
GroupVersion = schema.GroupVersion{Group: "network.cozystack.io", Version: "v1alpha1"}
31+
32+
// SchemeBuilder is used to add go types to the GroupVersionKind scheme.
33+
SchemeBuilder = &scheme.Builder{GroupVersion: GroupVersion}
34+
35+
// AddToScheme adds the types in this group-version to the given scheme.
36+
AddToScheme = SchemeBuilder.AddToScheme
37+
)
Lines changed: 104 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,104 @@
1+
/*
2+
Copyright 2026 The Cozystack Authors.
3+
4+
Licensed under the Apache License, Version 2.0 (the "License");
5+
you may not use this file except in compliance with the License.
6+
You may obtain a copy of the License at
7+
8+
http://www.apache.org/licenses/LICENSE-2.0
9+
10+
Unless required by applicable law or agreed to in writing, software
11+
distributed under the License is distributed on an "AS IS" BASIS,
12+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13+
See the License for the specific language governing permissions and
14+
limitations under the License.
15+
*/
16+
17+
package v1alpha1
18+
19+
import (
20+
corev1 "k8s.io/api/core/v1"
21+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
22+
)
23+
24+
// ServiceExposureSpec is the mechanism-agnostic tenant intent: "expose
25+
// this Service externally via the named ExposureClass". It never names a
26+
// backend; the controller translates it into whichever backend the cluster
27+
// admin wired behind the referenced class.
28+
type ServiceExposureSpec struct {
29+
// ServiceRef names the Service in this namespace to expose. The
30+
// controller scopes the backend pool/announcer to that Service and
31+
// reads its assigned IP back into status. The Service's own type is
32+
// owned by the chart that renders it, not by this controller.
33+
// +required
34+
ServiceRef corev1.LocalObjectReference `json:"serviceRef"`
35+
36+
// ExposureClassName references a cluster-scoped ExposureClass the
37+
// admin configured. Empty selects the cluster default ExposureClass
38+
// (the one annotated exposureclass.network.cozystack.io/is-default-class=true).
39+
// +optional
40+
ExposureClassName string `json:"exposureClassName,omitempty"`
41+
}
42+
43+
// ServiceExposureStatus reports the observed state of the exposure.
44+
type ServiceExposureStatus struct {
45+
// ObservedGeneration mirrors the .metadata.generation reflected in the
46+
// latest reconciled state.
47+
// +optional
48+
ObservedGeneration int64 `json:"observedGeneration,omitempty"`
49+
50+
// ResolvedBackend records which backend the referenced class resolved
51+
// to, so operators can see where an exposure was routed.
52+
// +optional
53+
ResolvedBackend string `json:"resolvedBackend,omitempty"`
54+
55+
// AssignedIPs reports the external addresses the backend assigned,
56+
// read back from the Service (status.loadBalancer.ingress or the
57+
// externalIPs pin).
58+
// +optional
59+
AssignedIPs []string `json:"assignedIPs,omitempty"`
60+
61+
// Conditions describes the current state of the ServiceExposure.
62+
// Standard condition type: Ready.
63+
// +optional
64+
// +patchMergeKey=type
65+
// +patchStrategy=merge
66+
// +listType=map
67+
// +listMapKey=type
68+
Conditions []metav1.Condition `json:"conditions,omitempty" patchStrategy:"merge" patchMergeKey:"type"`
69+
}
70+
71+
// +kubebuilder:object:root=true
72+
// +kubebuilder:subresource:status
73+
// +kubebuilder:resource:scope=Namespaced,shortName=svcexp
74+
// +kubebuilder:printcolumn:name="Service",type="string",JSONPath=".spec.serviceRef.name"
75+
// +kubebuilder:printcolumn:name="Class",type="string",JSONPath=".spec.exposureClassName"
76+
// +kubebuilder:printcolumn:name="Backend",type="string",JSONPath=".status.resolvedBackend"
77+
// +kubebuilder:printcolumn:name="IPs",type="string",JSONPath=".status.assignedIPs"
78+
// +kubebuilder:printcolumn:name="Ready",type="string",JSONPath=`.status.conditions[?(@.type=="Ready")].status`
79+
// +kubebuilder:printcolumn:name="Age",type="date",JSONPath=".metadata.creationTimestamp"
80+
81+
// ServiceExposure is the Schema for the serviceexposures API. It expresses
82+
// the mechanism-agnostic tenant intent to expose a Service externally; the
83+
// cozystack-controller reconciles the backend-specific pool and announcer
84+
// resources from this CR and reports the assigned IPs in status.
85+
type ServiceExposure struct {
86+
metav1.TypeMeta `json:",inline"`
87+
metav1.ObjectMeta `json:"metadata,omitempty"`
88+
89+
Spec ServiceExposureSpec `json:"spec,omitempty"`
90+
Status ServiceExposureStatus `json:"status,omitempty"`
91+
}
92+
93+
// +kubebuilder:object:root=true
94+
95+
// ServiceExposureList contains a list of ServiceExposure.
96+
type ServiceExposureList struct {
97+
metav1.TypeMeta `json:",inline"`
98+
metav1.ListMeta `json:"metadata,omitempty"`
99+
Items []ServiceExposure `json:"items"`
100+
}
101+
102+
func init() {
103+
SchemeBuilder.Register(&ServiceExposure{}, &ServiceExposureList{})
104+
}

0 commit comments

Comments
 (0)