Skip to content

Commit cb76b4e

Browse files
committed
feat(networking): add cozyplane eBPF CNI + cozyplane-kpr networking variant
A "cozyplane" networking variant that replaces BOTH kube-ovn and Cilium (and kube-proxy) with cozyplane's eBPF CNI plus cozyplane-kpr for in-cluster Services (socket-LB, importing Cilium's LB control plane — no Cilium agent). - packages/system/cozyplane: the CNI/agent/apiserver/controller (image bumped to the build with the off-VPC-transit fix; the old 0.3.17 pin had that regression, which broke grouped-pod TCP egress via the gateway). - packages/system/cozyplane-kpr: NEW — the kube-proxy replacement DaemonSet (hostNetwork, privileged, host bpffs+cgroup2 mounts, an init container that mounts bpffs; on Talos it reaches the apiserver via KubePrism https://localhost:7445 since the kubernetes.default ClusterIP is unserved until kpr runs). Scoped RBAC (services/endpointslices/nodes read + cilium.io). - networking.yaml: the "cozyplane" variant = cozyplane + cozyplane-kpr, no Cilium; drop the old cozyplane-cilium variant and system/cilium/values-cozyplane.yaml. - platform system bundle + values: accept networkingVariant "cozyplane". Single IPAM authority (cozyplane) over the podCIDR — removes the Cilium standalone-IPAM coexistence hazard (lllamnyp/cozyplane#12). Known gaps carried as follow-ups: host-firewall + default-network NetworkPolicy (were Cilium's), and external NodePort + VM-guest ClusterIP (cozyplane-kpr increment 3). Assisted-by: LLM Signed-off-by: Timofei Larkin <[email protected]>
1 parent cd91627 commit cb76b4e

22 files changed

Lines changed: 2956 additions & 1 deletion

‎packages/core/platform/sources/networking.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,3 +146,37 @@ spec:
146146
releaseName: kubeovn
147147
dependsOn:
148148
- cilium
149+
# cozyplane eBPF CNI + cozyplane-kpr (kube-proxy replacement) for Talos
150+
# clusters — NO Cilium and NO kube-proxy. cozyplane owns the datapath and the
151+
# CNI conflist; cozyplane-kpr provides in-cluster Services via socket-LB
152+
# (importing Cilium's LB control plane, no Cilium agent). One IPAM authority
153+
# (cozyplane) over the podCIDR — the coexistence hazard that Cilium's
154+
# standalone IPAM created is gone. See docs/kube-proxy-replacement.md.
155+
# NOTE: host-firewall and default-network NetworkPolicy that Cilium provided
156+
# are not yet replaced (follow-ups); external NodePort + VM-guest ClusterIP
157+
# need the increment-3 per-packet fallback.
158+
- name: cozyplane
159+
dependsOn:
160+
- cozystack.gateway-api-crds
161+
components:
162+
- name: cozyplane
163+
path: system/cozyplane
164+
valuesFiles:
165+
- values.yaml
166+
- values-talos.yaml
167+
install:
168+
privileged: true
169+
namespace: cozy-cozyplane
170+
releaseName: cozyplane
171+
dependsOn: []
172+
- name: cozyplane-kpr
173+
path: system/cozyplane-kpr
174+
valuesFiles:
175+
- values.yaml
176+
- values-talos.yaml
177+
install:
178+
privileged: true
179+
namespace: cozy-cozyplane
180+
releaseName: cozyplane-kpr
181+
dependsOn:
182+
- cozyplane

‎packages/core/platform/templates/bundles/system.yaml‎

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,14 @@
3535
"gatewayAPI" (dict "enableAlpn" .Values.gateway.http2)) -}}
3636
{{- $_ := set $networkingComponents "cilium" (dict "values" $ciliumValues) -}}
3737
{{- end -}}
38-
{{include "cozystack.platform.package" (list "cozystack.networking" "kubeovn-cilium" $ $networkingComponents) }}
38+
{{- /* Networking variant selector. Defaults to kubeovn-cilium (backward compatible);
39+
cozyplane swaps kube-ovn AND Cilium for the cozyplane eBPF CNI +
40+
cozyplane-kpr (socket-LB kube-proxy replacement) — no Cilium, no kube-proxy. */ -}}
41+
{{- $networkingVariant := .Values.bundles.system.networkingVariant | default "kubeovn-cilium" -}}
42+
{{- if not (or (eq $networkingVariant "kubeovn-cilium") (eq $networkingVariant "cozyplane")) -}}
43+
{{- fail (printf "bundles.system.networkingVariant must be \"kubeovn-cilium\" or \"cozyplane\", got %q" $networkingVariant) -}}
44+
{{- end -}}
45+
{{include "cozystack.platform.package" (list "cozystack.networking" $networkingVariant $ $networkingComponents) }}
3946
{{include "cozystack.platform.system.common-packages" $ }}
4047
{{- /* Talos bundles only bridge, firewall, flannel, host-local, loopback and
4148
portmap in /opt/cni/bin from its own node image; the other ten this

‎packages/core/platform/values.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,11 @@ bundles:
3838
system:
3939
enabled: false
4040
variant: "isp-full" # Options: "isp-full", "isp-full-generic", "isp-hosted", "distro-full"
41+
# CNI/networking selector for the "isp-full" (Talos) variant:
42+
# "kubeovn-cilium" — KubeOVN fabric + Cilium (default)
43+
# "cozyplane" — cozyplane eBPF CNI + cozyplane-kpr socket-LB KPR
44+
# (no Cilium, no kube-proxy)
45+
networkingVariant: "kubeovn-cilium"
4146
iaas:
4247
enabled: false
4348
# gpu-operator variant selector — applies when cozystack.gpu-operator is
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
.DS_Store
2+
*.swp
3+
*.bak
4+
*.tmp
5+
*~
6+
.git
7+
.gitignore
8+
tests/
Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
apiVersion: v2
2+
name: cozyplane-kpr
3+
description: cozyplane kube-proxy replacement — socket-LB service load balancing (replaces Cilium/kube-proxy KPR)
4+
type: application
5+
# Placeholder; the real version is stamped by the build/release pipeline.
6+
version: 0.1.0
7+
appVersion: "0.1.0"
8+
kubeVersion: ">=1.30.0-0"
9+
keywords:
10+
- kube-proxy
11+
- service
12+
- load-balancer
13+
- ebpf
14+
- cozystack
15+
home: https://github.com/lllamnyp/cozyplane
16+
sources:
17+
- https://github.com/lllamnyp/cozyplane
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
export NAME=cozyplane-kpr
2+
export NAMESPACE=cozy-cozyplane
3+
4+
include ../../../hack/common-envs.mk
5+
include ../../../hack/package.mk
6+
7+
# cozyplane-kpr builds and publishes its own container image from
8+
# github.com/lllamnyp/cozyplane (kpr/); this package only vendors the Helm chart
9+
# and references the published image in values.yaml. No image build target here.
10+
11+
test:
12+
helm lint .
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
apiVersion: v1
2+
kind: ServiceAccount
3+
metadata:
4+
name: cozyplane-kpr
5+
namespace: {{ .Release.Namespace }}
6+
---
7+
apiVersion: rbac.authorization.k8s.io/v1
8+
kind: ClusterRole
9+
metadata:
10+
name: cozyplane-kpr
11+
rules:
12+
# The LB control plane watches Services + their endpoints and node/topology.
13+
- apiGroups: [""]
14+
resources: [services, endpoints, nodes, namespaces, pods]
15+
verbs: [get, list, watch]
16+
- apiGroups: [discovery.k8s.io]
17+
resources: [endpointslices]
18+
verbs: [get, list, watch]
19+
# Leader election.
20+
- apiGroups: [coordination.k8s.io]
21+
resources: [leases]
22+
verbs: [get, list, watch, create, update, patch, delete]
23+
- apiGroups: [""]
24+
resources: [events]
25+
verbs: [create, patch]
26+
# Cilium's imported cells reconcile their own CRDs (CiliumNode, LB IP pools, …).
27+
- apiGroups: [cilium.io]
28+
resources: ["*"]
29+
verbs: [get, list, watch, create, update, patch, delete]
30+
---
31+
apiVersion: rbac.authorization.k8s.io/v1
32+
kind: ClusterRoleBinding
33+
metadata:
34+
name: cozyplane-kpr
35+
roleRef:
36+
apiGroup: rbac.authorization.k8s.io
37+
kind: ClusterRole
38+
name: cozyplane-kpr
39+
subjects:
40+
- kind: ServiceAccount
41+
name: cozyplane-kpr
42+
namespace: {{ .Release.Namespace }}
43+
---
44+
apiVersion: apps/v1
45+
kind: DaemonSet
46+
metadata:
47+
name: cozyplane-kpr
48+
namespace: {{ .Release.Namespace }}
49+
labels: {app: cozyplane-kpr}
50+
spec:
51+
selector:
52+
matchLabels: {app: cozyplane-kpr}
53+
updateStrategy:
54+
type: RollingUpdate
55+
template:
56+
metadata:
57+
labels: {app: cozyplane-kpr}
58+
spec:
59+
serviceAccountName: cozyplane-kpr
60+
hostNetwork: true
61+
dnsPolicy: ClusterFirstWithHostNet
62+
priorityClassName: system-node-critical
63+
tolerations:
64+
- operator: Exists
65+
{{- with .Values.imagePullSecrets }}
66+
imagePullSecrets:
67+
{{- toYaml . | nindent 8 }}
68+
{{- end }}
69+
initContainers:
70+
# Ensure bpffs is mounted on the host so the LB map pins are a real
71+
# bpffs and outlive the process; Bidirectional propagation pushes it to
72+
# the node. Cilium's DaemonSet does the equivalent.
73+
- name: mount-bpffs
74+
image: busybox:1.36
75+
command: ["sh", "-c", "mount | grep -q 'bpf on {{ .Values.bpffsRoot }} ' || mount -t bpf bpf {{ .Values.bpffsRoot }}"]
76+
securityContext:
77+
privileged: true
78+
volumeMounts:
79+
- {name: bpffs, mountPath: {{ .Values.bpffsRoot }}, mountPropagation: Bidirectional}
80+
containers:
81+
- name: kpr
82+
image: {{ .Values.image }}
83+
imagePullPolicy: {{ .Values.imagePullPolicy }}
84+
args:
85+
{{- if .Values.apiServerURL }}
86+
- --k8s-api-server-urls={{ .Values.apiServerURL }}
87+
{{- end }}
88+
env:
89+
- {name: KPR_CGROUP_ROOT, value: {{ .Values.cgroupRoot | quote }}}
90+
- {name: KPR_BPFFS_ROOT, value: {{ .Values.bpffsRoot | quote }}}
91+
securityContext:
92+
privileged: true
93+
volumeMounts:
94+
- {name: bpffs, mountPath: {{ .Values.bpffsRoot }}, mountPropagation: Bidirectional}
95+
- {name: cgroup2, mountPath: {{ .Values.cgroupRoot }}}
96+
volumes:
97+
- name: bpffs
98+
hostPath: {path: {{ .Values.bpffsRoot }}, type: DirectoryOrCreate}
99+
- name: cgroup2
100+
hostPath: {path: /sys/fs/cgroup, type: Directory}
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Talos: reach the apiserver through KubePrism (the node-local load balancer
2+
# Talos runs on localhost:7445), exactly as the Cilium KPR variant did
3+
# (k8sServiceHost: localhost / k8sServicePort: 7445). A hostNetwork pod hits it
4+
# on localhost, so it works before any ClusterIP proxy exists.
5+
apiServerURL: "https://localhost:7445"
Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
# cozyplane-kpr container image. Built and published from
2+
# github.com/lllamnyp/cozyplane (kpr/); digest-pinned (no floating tag).
3+
# It imports Cilium's LB control plane, so it is a large image and a SEPARATE
4+
# component from the cozyplane agent — never co-mingled.
5+
image: ghcr.io/lllamnyp/cozyplane-kpr@sha256:34b54bb3fddc11e0ef4165733552c90ee469f09d205c86ab80e087532dc6ae51
6+
imagePullPolicy: IfNotPresent
7+
imagePullSecrets: []
8+
9+
# The apiserver endpoint cozyplane-kpr connects to. It must NOT be the
10+
# kubernetes.default ClusterIP: with kube-proxy gone, that VIP is unserved until
11+
# kpr itself is running (chicken-and-egg). Point it at a real endpoint. Empty
12+
# uses the in-cluster config (only valid where some other proxy already serves
13+
# the ClusterIP); on Talos, values-talos.yaml sets the KubePrism local endpoint.
14+
apiServerURL: ""
15+
16+
# The host cgroup2 root (bind-mounted into the pod) where socket-LB attaches,
17+
# and the host bpffs root where the LB service/backend maps are pinned.
18+
cgroupRoot: /host/cgroup2
19+
bpffsRoot: /sys/fs/bpf
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
.DS_Store
2+
*.swp
3+
*.bak
4+
*.tmp
5+
*~
6+
.git
7+
.gitignore
8+
tests/

0 commit comments

Comments
 (0)