Skip to content

Commit 23ffd6d

Browse files
kvapsclaude
andcommitted
feat: label-only selector and port-filter default
Tighten the management contract for cozy-proxy: - The service.kubernetes.io/service-proxy-name=cozy-proxy label is now the sole selector. Services without the label are no longer managed even if they carry the networking.cozystack.io/wholeIP annotation — the annotation alone has no selecting effect. - Whole-IP passthrough becomes opt-in: wholeIPPassthrough returns true only for explicit wholeIP="true". Absent or any other value keeps the service in port-filter mode (consistent with wholeIP="false"), so declared Service.spec.ports are the only ports forwarded by default. Drop the now-unused hasWholeIPAnnotation helper. Tests updated to cover the new isCozyProxyService selector and the new wholeIPPassthrough default. README rewritten: drop the dual-selector narrative and the "Which selector should I use?" section, update the ingress-mode table to mark port-filter as the default, and add wholeIP="true" to the Usage example so the bogus port-65535 demonstration of whole-IP behaviour keeps working. BREAKING CHANGE: Services that previously relied on the wholeIP annotation alone for selection must add the service.kubernetes.io/service-proxy-name: cozy-proxy label. Services that previously relied on the absent-annotation passthrough default must add networking.cozystack.io/wholeIP: "true" to keep that behaviour. Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]>
1 parent 1b459f6 commit 23ffd6d

3 files changed

Lines changed: 41 additions & 57 deletions

File tree

‎README.md‎

Lines changed: 13 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -17,37 +17,29 @@ The last option is the simplest and most flexible, but it has a limitation: Kube
1717
but only traffic on specific ports (see: [Kubernetes Issue #23864](https://github.com/kubernetes/kubernetes/issues/23864)).
1818
Additionally, kube-proxy does not perform SNAT, which causes outgoing traffic from the pod to use the default gateway of the host where it is running.
1919

20-
To address these issues, we have added an additional controller that performs 1:1 NAT for services selected by either the `service.kubernetes.io/service-proxy-name: cozy-proxy` label or the `networking.cozystack.io/wholeIP` annotation.
20+
To address these issues, we have added an additional controller that performs NAT for services carrying the `service.kubernetes.io/service-proxy-name: cozy-proxy` label.
2121

2222
## How It Works
2323

24-
cozy-proxy is a simple Kubernetes controller that watches for services selected by either of the following:
25-
26-
- **`service.kubernetes.io/service-proxy-name: cozy-proxy`** label (recommended) — the standard Kubernetes mechanism for delegating a service to a non-default proxy. kube-proxy skips services carrying this label, so cozy-proxy becomes the sole handler and no rules collide.
27-
- **`networking.cozystack.io/wholeIP`** annotation — also selects the service for management. The annotation value additionally drives the ingress mode (see below).
24+
cozy-proxy is a simple Kubernetes controller that watches for services labeled with `service.kubernetes.io/service-proxy-name: cozy-proxy` — the standard Kubernetes mechanism for delegating a service to a non-default proxy. kube-proxy skips services carrying this label, so cozy-proxy becomes the sole handler and no rules collide.
2825

2926
When it finds such a service, it creates NFT rules that forward traffic from the service's external IP to the pod's IP and vice versa, performing source-IP preservation for egress traffic.
3027

3128
This controller can be used together with kube-proxy and Cilium in kube-proxy replacement mode.
3229

33-
### Which selector should I use?
34-
35-
- If your cluster runs **plain kube-proxy** (iptables or IPVS mode) — for example, a default RKE2/kubeadm install with Calico or Flannel — use the `service.kubernetes.io/service-proxy-name: cozy-proxy` label. Without it, kube-proxy installs its own LoadBalancer rules that conflict with cozy-proxy's NAT and break outbound SNAT.
36-
- If your cluster runs **Cilium in kube-proxy replacement mode** (as in the reference Cozystack environment), either selector works.
37-
38-
You can safely set both on the same service.
30+
> The label is the **only** selector. Services that do not carry it are not managed by cozy-proxy, regardless of any annotations they may have.
3931
4032
## Ingress mode
4133

42-
The `networking.cozystack.io/wholeIP` annotation value selects the ingress mode:
34+
The optional `networking.cozystack.io/wholeIP` annotation selects the ingress mode for a managed service:
4335

44-
| Value | Behavior |
45-
|-----------|-----------------------------------------------------------------------------------------------------------------|
46-
| `"true"` | **Whole-IP passthrough.** All TCP/UDP traffic to the LoadBalancer IP is forwarded to the backend pod. |
47-
| `"false"` | **Per-port filtering.** Only TCP/UDP traffic to ports listed in `Service.spec.ports` is forwarded; rest dropped.|
48-
| absent | Defaults to **passthrough** (services selected by label only behave the same as `wholeIP: "true"`). |
36+
| Value | Behavior |
37+
|-----------|-----------------------------------------------------------------------------------------------------------------------|
38+
| `"true"` | **Whole-IP passthrough.** All TCP/UDP traffic to the LoadBalancer IP is forwarded to the backend pod. |
39+
| `"false"` | **Per-port filtering** (default). Only TCP/UDP traffic to ports listed in `Service.spec.ports` is forwarded; rest dropped. |
40+
| absent | Same as `"false"` — per-port filtering. |
4941

50-
In both managed modes, egress traffic from the backend pod is SNATed to the
42+
In both modes, egress traffic from the backend pod is SNATed to the
5143
LoadBalancer IP for source-IP preservation.
5244

5345
The optional `networking.cozystack.io/allowICMP: "true"` annotation, only
@@ -88,7 +80,7 @@ helm install cozy-proxy charts/cozy-proxy -n kube-system
8880

8981
## Usage
9082

91-
Create a LoadBalancer service with the `service.kubernetes.io/service-proxy-name: cozy-proxy` label. This also tells kube-proxy to stay away from the service:
83+
Create a LoadBalancer service with the `service.kubernetes.io/service-proxy-name: cozy-proxy` label. The default mode is per-port filtering, so to forward every port to the backend pod (whole-IP passthrough) add the `networking.cozystack.io/wholeIP: "true"` annotation:
9284

9385
```yaml
9486
apiVersion: v1
@@ -97,6 +89,8 @@ metadata:
9789
name: example-service
9890
labels:
9991
service.kubernetes.io/service-proxy-name: cozy-proxy
92+
annotations:
93+
networking.cozystack.io/wholeIP: "true"
10094
spec:
10195
allocateLoadBalancerNodePorts: false
10296
externalTrafficPolicy: Local

‎pkg/controllers/services_controller.go‎

Lines changed: 11 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -475,42 +475,27 @@ const (
475475
serviceProxyName = "cozy-proxy"
476476
)
477477

478-
// isCozyProxyService reports whether the service should be managed by cozy-proxy.
479-
// A service is selected if it carries the
480-
// service.kubernetes.io/service-proxy-name=cozy-proxy label (standard
481-
// Kubernetes mechanism, also tells kube-proxy to ignore the service) or the
482-
// networking.cozystack.io/wholeIP annotation (any value — the value drives
483-
// passthrough vs port-filter mode, see wholeIPPassthrough).
478+
// isCozyProxyService reports whether the service should be managed by
479+
// cozy-proxy. The sole selector is the standard Kubernetes
480+
// service.kubernetes.io/service-proxy-name=cozy-proxy label, which also
481+
// tells kube-proxy to ignore the service. Services without the label are
482+
// not managed regardless of any cozy-proxy annotations they may carry.
484483
func isCozyProxyService(svc *v1.Service) bool {
485484
if svc == nil {
486485
return false
487486
}
488-
if svc.Labels[serviceProxyNameLabel] == serviceProxyName {
489-
return true
490-
}
491-
return hasWholeIPAnnotation(svc)
492-
}
493-
494-
// hasWholeIPAnnotation reports whether the service carries the wholeIP
495-
// annotation. Any value is accepted; the value's meaning (passthrough vs
496-
// port-filter) is determined by wholeIPPassthrough.
497-
func hasWholeIPAnnotation(svc *v1.Service) bool {
498-
if svc == nil {
499-
return false
500-
}
501-
_, ok := svc.Annotations[wholeIPAnnotation]
502-
return ok
487+
return svc.Labels[serviceProxyNameLabel] == serviceProxyName
503488
}
504489

505490
// wholeIPPassthrough reports whether ingress traffic should bypass port
506-
// filtering. Defaults to true for any value other than the explicit string
507-
// "false", so services selected by label alone (no annotation) and services
508-
// with annotation: "true" both behave as passthrough.
491+
// filtering. Opt-in: requires an explicit networking.cozystack.io/wholeIP=true
492+
// annotation. Any other value, or no annotation at all, keeps the service in
493+
// the default port-filter mode.
509494
func wholeIPPassthrough(svc *v1.Service) bool {
510495
if svc == nil || svc.Annotations == nil {
511-
return true
496+
return false
512497
}
513-
return svc.Annotations[wholeIPAnnotation] != "false"
498+
return svc.Annotations[wholeIPAnnotation] == "true"
514499
}
515500

516501
// allowICMP reports whether ICMP traffic should bypass the port_filter drop

‎pkg/controllers/services_controller_test.go‎

Lines changed: 17 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -11,22 +11,26 @@ func svcWith(annot map[string]string) *v1.Service {
1111
return &v1.Service{ObjectMeta: metav1.ObjectMeta{Annotations: annot}}
1212
}
1313

14-
func TestHasWholeIPAnnotation(t *testing.T) {
14+
func svcWithLabels(labels map[string]string) *v1.Service {
15+
return &v1.Service{ObjectMeta: metav1.ObjectMeta{Labels: labels}}
16+
}
17+
18+
func TestIsCozyProxyService(t *testing.T) {
1519
cases := []struct {
1620
name string
1721
svc *v1.Service
1822
expect bool
1923
}{
20-
{"true value", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "true"}), true},
21-
{"false value", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "false"}), true},
22-
{"absent annotation", svcWith(map[string]string{}), false},
23-
{"nil annotations", &v1.Service{}, false},
24-
{"unrelated annotation", svcWith(map[string]string{"foo": "bar"}), false},
24+
{"label with correct value", svcWithLabels(map[string]string{"service.kubernetes.io/service-proxy-name": "cozy-proxy"}), true},
25+
{"label with other value", svcWithLabels(map[string]string{"service.kubernetes.io/service-proxy-name": "kube-router"}), false},
26+
{"label absent", svcWithLabels(map[string]string{}), false},
27+
{"nil service", nil, false},
28+
{"only wholeIP annotation no label", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "true"}), false},
2529
}
2630
for _, c := range cases {
2731
t.Run(c.name, func(t *testing.T) {
28-
if got := hasWholeIPAnnotation(c.svc); got != c.expect {
29-
t.Errorf("hasWholeIPAnnotation = %v, want %v", got, c.expect)
32+
if got := isCozyProxyService(c.svc); got != c.expect {
33+
t.Errorf("isCozyProxyService = %v, want %v", got, c.expect)
3034
}
3135
})
3236
}
@@ -38,10 +42,11 @@ func TestWholeIPPassthrough(t *testing.T) {
3842
svc *v1.Service
3943
expect bool
4044
}{
41-
{"true value", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "true"}), true},
42-
{"false value", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "false"}), false},
43-
{"absent annotation defaults to passthrough", svcWith(map[string]string{}), true},
44-
{"nil annotations defaults to passthrough", &v1.Service{}, true},
45+
{"explicit true", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "true"}), true},
46+
{"explicit false", svcWith(map[string]string{"networking.cozystack.io/wholeIP": "false"}), false},
47+
{"empty value", svcWith(map[string]string{"networking.cozystack.io/wholeIP": ""}), false},
48+
{"absent annotation defaults to port-filter", svcWith(map[string]string{}), false},
49+
{"nil annotations defaults to port-filter", &v1.Service{}, false},
4550
}
4651
for _, c := range cases {
4752
t.Run(c.name, func(t *testing.T) {

0 commit comments

Comments
 (0)