Skip to content

Commit 8bffa34

Browse files
lexfreilexfrei
authored
feat(platform): add opt-in in-cluster east-west encryption toggle (#2984)
## What this PR does Adds `networking.encryption.enabled` to the platform chart — a single operator-facing opt-in, off by default, that turns on in-cluster (east-west) transport encryption for the active datapath. When enabled, pod-to-pod traffic across nodes is encrypted at the network layer regardless of whether the workload implements TLS: defense-in-depth for the whole mesh without per-application changes. The default `isp-full` bundle selects the `kubeovn-cilium` datapath (and `isp-full-generic` selects `kubeovn-cilium-generic`), where KubeOVN owns the datapath and the overlay is a Geneve tunnel. The toggle therefore maps to KubeOVN's overlay IPsec (`kube-ovn` `ENABLE_OVN_IPSEC`). KubeOVN's own controller provisions the IPsec CA and per-node certificates through the Kubernetes CSR API, so no cert-manager wiring is needed. This complements, and does not replace, application-level / end-to-end TLS: in-cluster encryption never reaches an external client and gives a tenant no verifiable endpoint certificate. The values comments document the per-datapath scope — including that Kilo's WireGuard mesh (`meshGranularity: location`) covers only cross-location traffic and leaves the intra-location gap this toggle targets, and that the Cilium-native WireGuard/IPsec path on the `cilium` / `cilium-generic` variants is not reached because the system bundle does not select those variants — plus the MTU/throughput overhead. Scope: only the default KubeOVN datapath is wired through the platform toggle, which is what the system bundle actually selects. Wiring the Cilium-native path would require the bundle to expose cilium-only variant selection; that is out of scope here and called out in the values comments. Validation: `helm unittest` on the platform chart passes (66 tests, 7 new) covering on / off / absent / null and verifying the IPsec flag deep-merges without dropping the existing `kube-ovn` overrides, for both `isp-full` and `isp-full-generic`. Closes #2977 Part of #2811 ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(platform): add opt-in `networking.encryption.enabled` (off by default) that enables in-cluster east-west transport encryption for the default KubeOVN datapath via overlay IPsec; complements, and does not replace, application-level TLS. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced opt-in in-cluster network encryption using KubeOVN IPsec for east-west traffic protection. * Added `networking.encryption` configuration toggle (disabled by default) with full backward compatibility. * **Tests** * Added comprehensive test coverage for network encryption configurations across system variants and edge cases. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 66c96b4 + aced011 commit 8bffa34

3 files changed

Lines changed: 190 additions & 0 deletions

File tree

‎packages/core/platform/templates/bundles/system.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,15 @@
1313
{{- if .Values.networking.kubeovn.MASTER_NODES -}}
1414
{{- $_ := set $kubeovnDict "MASTER_NODES" .Values.networking.kubeovn.MASTER_NODES -}}
1515
{{- end -}}
16+
{{- /* Opt-in east-west encryption (networking.encryption.enabled): on the
17+
default kubeovn-cilium datapath the overlay is KubeOVN's Geneve
18+
tunnel, so encrypting it means turning on KubeOVN IPsec. KubeOVN's
19+
own controller provisions the IPsec CA and per-node certs through
20+
the Kubernetes CSR API — no cert-manager wiring needed here. Off by
21+
default; complements (does not replace) application-level TLS. */ -}}
22+
{{- if dig "encryption" "enabled" false .Values.networking -}}
23+
{{- $_ := set $kubeovnDict "func" (dict "ENABLE_OVN_IPSEC" true) -}}
24+
{{- end -}}
1625
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
1726
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
1827
{{- /* For Talos (isp-full): use KubePrism endpoint and disable cgroup autoMount */ -}}
@@ -83,6 +92,12 @@
8392
{{- end -}}
8493
{{- /* For generic k8s (k3s), control-plane label has value "true" */ -}}
8594
{{- $_ := set $kubeovnDict "MASTER_NODES_LABEL" "node-role.kubernetes.io/control-plane=true" -}}
95+
{{- /* Opt-in east-west encryption (networking.encryption.enabled): same
96+
KubeOVN Geneve-overlay IPsec path as isp-full. See the isp-full
97+
branch above for the mechanism notes. */ -}}
98+
{{- if dig "encryption" "enabled" false .Values.networking -}}
99+
{{- $_ := set $kubeovnDict "func" (dict "ENABLE_OVN_IPSEC" true) -}}
100+
{{- end -}}
86101
{{- $kubeovnValues := dict "kube-ovn" $kubeovnDict -}}
87102
{{- $_ := set $networkingComponents "kubeovn" (dict "values" $kubeovnValues) -}}
88103
{{- /* Cilium configuration - for generic k8s, always enable cgroup autoMount */ -}}

‎packages/core/platform/tests/bundles_networking_encryption_test.yaml‎

Lines changed: 137 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,137 @@
1+
suite: networking.encryption bundle wiring
2+
templates:
3+
- templates/bundles/system.yaml
4+
release:
5+
name: cozystack
6+
namespace: cozy-system
7+
tests:
8+
- it: leaves KubeOVN overlay IPsec unset on the default isp-full datapath when encryption is off
9+
set:
10+
bundles:
11+
system:
12+
enabled: true
13+
variant: isp-full
14+
networking:
15+
encryption:
16+
enabled: false
17+
documentSelector:
18+
path: metadata.name
19+
value: cozystack.networking
20+
asserts:
21+
- equal:
22+
path: spec.variant
23+
value: kubeovn-cilium
24+
- notExists:
25+
path: spec.components.kubeovn.values.kube-ovn.func.ENABLE_OVN_IPSEC
26+
27+
- it: leaves KubeOVN overlay IPsec unset on isp-full when networking.encryption is absent (backward compatible, no nil-pointer)
28+
set:
29+
bundles:
30+
system:
31+
enabled: true
32+
variant: isp-full
33+
documentSelector:
34+
path: metadata.name
35+
value: cozystack.networking
36+
asserts:
37+
- equal:
38+
path: spec.variant
39+
value: kubeovn-cilium
40+
- notExists:
41+
path: spec.components.kubeovn.values.kube-ovn.func.ENABLE_OVN_IPSEC
42+
43+
- it: enables KubeOVN Geneve-overlay IPsec on the default isp-full datapath when encryption is on
44+
set:
45+
bundles:
46+
system:
47+
enabled: true
48+
variant: isp-full
49+
networking:
50+
encryption:
51+
enabled: true
52+
documentSelector:
53+
path: metadata.name
54+
value: cozystack.networking
55+
asserts:
56+
- equal:
57+
path: spec.variant
58+
value: kubeovn-cilium
59+
- equal:
60+
path: spec.components.kubeovn.values.kube-ovn.func.ENABLE_OVN_IPSEC
61+
value: true
62+
63+
- it: enables KubeOVN overlay IPsec on the generic kubeovn-cilium-generic datapath when encryption is on
64+
set:
65+
bundles:
66+
system:
67+
enabled: true
68+
variant: isp-full-generic
69+
networking:
70+
encryption:
71+
enabled: true
72+
documentSelector:
73+
path: metadata.name
74+
value: cozystack.networking
75+
asserts:
76+
- equal:
77+
path: spec.variant
78+
value: kubeovn-cilium-generic
79+
- equal:
80+
path: spec.components.kubeovn.values.kube-ovn.func.ENABLE_OVN_IPSEC
81+
value: true
82+
83+
- it: leaves the generic datapath IPsec unset when encryption is off
84+
set:
85+
bundles:
86+
system:
87+
enabled: true
88+
variant: isp-full-generic
89+
networking:
90+
encryption:
91+
enabled: false
92+
documentSelector:
93+
path: metadata.name
94+
value: cozystack.networking
95+
asserts:
96+
- equal:
97+
path: spec.variant
98+
value: kubeovn-cilium-generic
99+
- notExists:
100+
path: spec.components.kubeovn.values.kube-ovn.func.ENABLE_OVN_IPSEC
101+
102+
- it: keeps the existing kube-ovn ipv4 component overrides alongside the IPsec toggle (deep merge, not replace)
103+
set:
104+
bundles:
105+
system:
106+
enabled: true
107+
variant: isp-full
108+
networking:
109+
encryption:
110+
enabled: true
111+
documentSelector:
112+
path: metadata.name
113+
value: cozystack.networking
114+
asserts:
115+
- equal:
116+
path: spec.components.kubeovn.values.kube-ovn.ipv4.POD_CIDR
117+
value: 10.244.0.0/16
118+
- equal:
119+
path: spec.components.kubeovn.values.kube-ovn.func.ENABLE_OVN_IPSEC
120+
value: true
121+
122+
- it: renders without a nil-pointer error when networking is set to null and encryption is therefore unreachable
123+
set:
124+
bundles:
125+
system:
126+
enabled: true
127+
variant: isp-full
128+
networking: null
129+
documentSelector:
130+
path: metadata.name
131+
value: cozystack.networking
132+
asserts:
133+
- equal:
134+
path: spec.variant
135+
value: kubeovn-cilium
136+
- notExists:
137+
path: spec.components

‎packages/core/platform/values.yaml‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,44 @@ networking:
7171
# Set this to comma-separated list of master node IPs to override.
7272
kubeovn:
7373
MASTER_NODES: ""
74+
# In-cluster (east-west) transport encryption. Opt-in, OFF by default.
75+
#
76+
# When enabled, pod-to-pod traffic that crosses nodes is encrypted at the
77+
# network layer by the ACTIVE datapath, regardless of whether the workload
78+
# speaks TLS — defense-in-depth for the whole mesh without per-application
79+
# changes. This COMPLEMENTS, and does NOT replace, application-level /
80+
# end-to-end TLS: it never reaches an external client and gives a tenant no
81+
# verifiable endpoint certificate (see the per-app certificate series). Use
82+
# it alongside, not instead of, TLS on exposed endpoints.
83+
#
84+
# The mechanism is datapath-specific (selected by bundles.system.variant):
85+
#
86+
# * kubeovn-cilium (isp-full, the default) and kubeovn-cilium-generic
87+
# (isp-full-generic): KubeOVN owns the datapath; its Geneve overlay is
88+
# encrypted with IPsec (kube-ovn ENABLE_OVN_IPSEC). KubeOVN's own
89+
# controller provisions the IPsec CA and per-node certificates through
90+
# the Kubernetes CSR API — no cert-manager wiring is required. This is
91+
# the path this toggle drives today.
92+
# * cilium / cilium-generic (Cilium owns the datapath): Cilium-native
93+
# WireGuard/IPsec is the right mechanism there, but the platform system
94+
# bundle does not select those variants, so this toggle does not reach
95+
# them yet.
96+
# * cilium-kilo: Kilo already establishes a WireGuard mesh, but only at
97+
# meshGranularity=location — it encrypts CROSS-LOCATION traffic and
98+
# leaves intra-location pod-to-pod in cleartext by design. This toggle
99+
# targets exactly that intra-location gap and composes with Kilo rather
100+
# than conflicting with it.
101+
#
102+
# CAVEATS: encrypting the overlay adds per-packet ESP overhead and lowers
103+
# the effective MTU, so east-west throughput drops and latency rises; the
104+
# cost is highest for high-bandwidth flows. IPsec here protects the
105+
# inter-node overlay tunnels — same-node pod-to-pod traffic never leaves the
106+
# host and is not tunneled or encrypted. Flipping this on an existing
107+
# cluster triggers a rollout of the CNI components. On variants without a
108+
# cozystack-managed encryptable datapath (e.g. isp-hosted / noop) the toggle
109+
# has no effect.
110+
encryption:
111+
enabled: false
74112
# Service publishing and ingress configuration
75113
publishing:
76114
host: "example.org"

0 commit comments

Comments
 (0)