Skip to content

Commit 077f04f

Browse files
committed
cozyplane: re-vendor charts from upstream main
Brings in the chart comment trim (lllamnyp/cozyplane#41): the etcd-cluster header now describes the PVC-backed default, the takeover comments in apiserver.yaml are scoped to the pre-split upgrade path, the tenant-rbac narrative is gone, and the KubePrism bootstrap argument keeps one home in each chart's values.yaml. Also carries the bugfix stream up to upstream main: kpr now ensures bpffs in-process by statfs rather than a busybox initContainer matching the mount's source name (Talos names its bpffs source "none", so the old guard stacked an empty bpffs over the real one and hid the agent's pins), kpr programs Service.spec.externalIPs, the agent discovery-gates its sdn.cozystack.io informers, and the apiserver reconciles its APIService instead of asserting it once. Image pins move to the multi-arch CI builds of that main (the kpr image is CI-built now, amd64+arm64). The agent memory request rises to 192Mi (the preallocated eBPF maps are charged to its cgroup), and the plugin SA gains deletecollection on FabricIPs (lllamnyp/cozyplane#36). Chart.yaml keeps the Cozystack-side version stanzas. The upstream README still describes the pre-group-split bootstrap-CRD takeover; vendored as-is, to be fixed upstream. Assisted-by: LLM Signed-off-by: Timofei Larkin <[email protected]>
1 parent d8d23cc commit 077f04f

12 files changed

Lines changed: 96 additions & 79 deletions

File tree

‎packages/system/cozyplane-apiserver/Chart.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,16 @@ apiVersion: v2
22
name: cozyplane-apiserver
33
description: Aggregated API server for cozyplane's sdn.cozystack.io group (VPC tenancy), with its dedicated etcd
44
type: application
5+
# Placeholder; the real version is stamped by the build/release pipeline.
56
version: 0.0.1
67
appVersion: "0.1.0"
78
kubeVersion: ">=1.30.0-0"
9+
keywords:
10+
- cni
11+
- networking
12+
- api-aggregation
13+
- cozystack
14+
- vpc
815
home: https://github.com/lllamnyp/cozyplane
916
sources:
1017
- https://github.com/lllamnyp/cozyplane

‎packages/system/cozyplane-apiserver/templates/apiserver.yaml‎

Lines changed: 8 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -77,17 +77,15 @@ rules:
7777
- apiGroups: ["flowcontrol.apiserver.k8s.io"]
7878
resources: ["flowschemas", "prioritylevelconfigurations"]
7979
verbs: ["get", "list", "watch"]
80-
# The server registers (or takes over from CRD autoregistration) its own
81-
# APIService at startup — a chart manifest can't: the kube-apiserver has
82-
# already auto-created the object for the bootstrap CRDs, and Helm refuses
83-
# to adopt an object it does not own.
80+
# The server registers its own APIService at startup
81+
# (--ensure-apiservice-service), rather than the chart shipping the object.
8482
- apiGroups: ["apiregistration.k8s.io"]
8583
resources: ["apiservices"]
8684
verbs: ["get", "create", "patch"]
87-
# ...and then removes the bootstrap CRDs it superseded. They are shadowed for
88-
# routing but NOT for OpenAPI: while they exist, the group's spec fails to
89-
# merge (duplicated paths) and `kubectl apply` of any object in it dies with
90-
# "failed to download openapi".
85+
# For upgrades from a pre-split install only: --remove-bootstrap-crds (off by
86+
# default) deletes the old single-group CRDs, which would otherwise keep the
87+
# group's OpenAPI from merging. A fresh install has no CRDs in this group —
88+
# see docs/api-groups.md §"Why two groups".
9189
- apiGroups: ["apiextensions.k8s.io"]
9290
resources: ["customresourcedefinitions"]
9391
verbs: ["get", "list", "delete"]
@@ -282,8 +280,8 @@ spec:
282280
args:
283281
- --serve-sdn
284282
# Register the group's APIService. Nothing else creates one: the
285-
# tenant kinds have no CRDs, so there is no CRD autoregistration to
286-
# fight with — the takeover is gone (docs/api-groups.md).
283+
# tenant kinds have no CRDs, so no CRD autoregistration competes for
284+
# the group (docs/api-groups.md).
287285
- --ensure-apiservice-service={{ $ns }}/cozyplane-apiserver
288286
- --ensure-apiservice-ca-injection={{ $ns }}/cozyplane-apiserver-serving
289287
{{- if .Values.etcd.operator.enabled }}

‎packages/system/cozyplane-apiserver/templates/etcd-cluster.yaml‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,6 @@
11
{{- /* Operator-managed etcd for the aggregated apiserver (etcd-operator.cozystack.io
2-
v1alpha2). MEMORY-BACKED: no PVC, no storageClass — so the aggregated API's
3-
etcd comes up without a storage layer and does not create the
4-
CNI -> apiserver -> etcd -> storage -> CNI bootstrap cycle. The CNI installs
5-
first; a persistent (storageClassName) mode is a follow-up once storage is
6-
sequenced after the CNI. Data is not durable across a full-cluster restart.
2+
v1alpha2). PVC-backed by default; `etcd.operator.persistent: false` gives a
3+
tmpfs member instead, for an install with no storage layer.
74
TLS uses the operator's cert-manager integration: the operator emits the
85
server / operator-client / peer Certificates itself; we only supply the CA
96
issuers and the apiserver's own client cert. Requires cert-manager. The
@@ -114,16 +111,13 @@ spec:
114111
replicas: {{ $replicas }}
115112
storage:
116113
{{- if .Values.etcd.operator.persistent }}
117-
# PVC-backed: the aggregated API's objects (VPCs, Ports, policies) are
118-
# cluster state. A memory-backed etcd loses every one of them on a
119-
# full-cluster restart — which is exactly what a node reboot is.
114+
# PVC-backed (the default; see values.yaml `etcd.operator.persistent`).
120115
medium: ""
121116
{{- with .Values.etcd.operator.storageClass }}
122117
storageClassName: {{ . | quote }}
123118
{{- end }}
124119
{{- else }}
125-
# tmpfs: no storage layer needed, but NOT durable across a full-cluster
126-
# restart. `medium` is immutable post-create; size is the tmpfs cap.
120+
# tmpfs. `medium` is immutable post-create; size is the tmpfs cap.
127121
medium: Memory
128122
{{- end }}
129123
size: {{ .Values.etcd.operator.size }}

‎packages/system/cozyplane-apiserver/values.yaml‎

Lines changed: 11 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Container image (shared with the cozyplane chart — the apiserver binary lives
2-
# in the same image). Digest-pinned (no floating tag); update on each published
3-
# build.
4-
image: ghcr.io/lllamnyp/cozyplane@sha256:79866d6807d6b20f06f421435fe5e2ea4f0d6b484a0fde0411b7f3da1e326fa1
2+
# in the same image). Digest-pinned (no floating tag); keep it equal to
3+
# chart/cozyplane/values.yaml's pin. See docs/packaging.md.
4+
image: ghcr.io/lllamnyp/cozyplane@sha256:133c1e1c95e28f4a857288c3a58aca592de7e2a0d91c689cd99382f61ec1805c
55
imagePullPolicy: IfNotPresent
66
imagePullSecrets: []
77

@@ -15,10 +15,12 @@ resources:
1515

1616
etcd:
1717
# Operator-managed etcd (the aenix-io etcd-operator; Cozystack ships it):
18-
# replicated, PVC-backed, TLS end to end. The production path — without it
19-
# the built-in single-pod etcd below is used, and with its default
20-
# emptyDir storage a single pod reschedule erases every sdn.cozystack.io
21-
# object in the cluster.
18+
# replicated, PVC-backed, TLS end to end. The production path, and the default
19+
# — the Cozystack install this chart is packaged for always has the operator
20+
# (packages/packagesources/apiserver.yaml dependsOn cozystack.etcd-operator).
21+
# Turn it off for a standalone/kind install with no operator: the chart then
22+
# falls back to the built-in single-pod etcd below, which is one pod holding
23+
# every sdn.cozystack.io object in the cluster.
2224
operator:
2325
enabled: true
2426
replicas: 3
@@ -31,12 +33,11 @@ etcd:
3133
# layer, so the component must be sequenced after it.
3234
persistent: true
3335
# StorageClass for the members' PVCs ("" = cluster default). Replicated, so
34-
# a member's data survives losing the node it was scheduled on.
36+
# a member's data survives losing the node it was scheduled on. Cozystack's
37+
# storage layers (linstor, blockstor) both provide a "replicated" class.
3538
storageClass: "replicated"
3639
# Built-in single-pod etcd (dev/evaluation only).
3740
image: registry.k8s.io/etcd:3.5.16-0
38-
# Storage for the built-in etcd. The default is ephemeral (emptyDir); use a
39-
# persistentVolumeClaim volume source to survive a pod restart.
4041
# Storage for the built-in single-pod etcd. Default is a PVC — the aggregated
4142
# API's objects (VPCs, Ports, policies) are cluster state, and an emptyDir
4243
# loses them on a pod move. Set `emptyDir: {}` explicitly for a throwaway

‎packages/system/cozyplane-kpr/templates/daemonset.yaml‎

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -66,17 +66,6 @@ spec:
6666
imagePullSecrets:
6767
{{- toYaml . | nindent 8 }}
6868
{{- end }}
69-
initContainers:
70-
# Ensure bpffs is mounted on the host so the LB map pins are a real
71-
# bpffs and outlive the process; Bidirectional propagation pushes it to
72-
# the node. Cilium's DaemonSet does the equivalent.
73-
- name: mount-bpffs
74-
image: busybox:1.36
75-
command: ["sh", "-c", "mount | grep -q 'bpf on {{ .Values.bpffsRoot }} ' || mount -t bpf bpf {{ .Values.bpffsRoot }}"]
76-
securityContext:
77-
privileged: true
78-
volumeMounts:
79-
- {name: bpffs, mountPath: {{ .Values.bpffsRoot }}, mountPropagation: Bidirectional}
8069
containers:
8170
- name: kpr
8271
image: {{ .Values.image }}
@@ -91,6 +80,12 @@ spec:
9180
securityContext:
9281
privileged: true
9382
volumeMounts:
83+
# kpr mounts bpffs here itself at startup if the node has none, and
84+
# Bidirectional propagates that to the host so the pins outlive the
85+
# process. The check is statfs on the mount point, not the mount's
86+
# source name: Talos names its bpffs source "none", and a guard that
87+
# matched "bpf" stacked an empty bpffs over the real one, hiding the
88+
# agent's pins (kpr/bpffs.go).
9489
- {name: bpffs, mountPath: {{ .Values.bpffsRoot }}, mountPropagation: Bidirectional}
9590
- {name: cgroup2, mountPath: {{ .Values.cgroupRoot }}}
9691
volumes:
Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
1-
# Talos: reach the apiserver through KubePrism (the node-local load balancer
2-
# Talos runs on localhost:7445), exactly as the Cilium KPR variant did
3-
# (k8sServiceHost: localhost / k8sServicePort: 7445). A hostNetwork pod hits it
4-
# on localhost, so it works before any ClusterIP proxy exists.
1+
# Talos runs KubePrism (a node-local apiserver load balancer) on localhost:7445 —
2+
# the real endpoint `apiServerURL` asks for; see values.yaml. A hostNetwork pod
3+
# reaches it on localhost, before any ClusterIP proxy exists.
54
apiServerURL: "https://localhost:7445"
5+
6+
# Applied as the SECOND valuesFile of the `cozyplane-kpr` component
7+
# (packages/packagesources/networking.yaml): the Cozystack operator applies the
8+
# first with strategy Overwrite and every later one with Merge, so this overlay
9+
# only works as a file distinct from values.yaml.

‎packages/system/cozyplane-kpr/values.yaml‎

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,10 @@
1-
# cozyplane-kpr container image. Built and published from
2-
# github.com/lllamnyp/cozyplane (kpr/); digest-pinned (no floating tag).
1+
# cozyplane-kpr container image, built from kpr/ in this repository.
2+
# Digest-pinned (no floating tag); built and pushed by release.yml's kpr-image
3+
# job for amd64 and arm64, like the cozyplane image. docs/packaging.md has the
4+
# procedure and how the pin is refreshed.
35
# It imports Cilium's LB control plane, so it is a large image and a SEPARATE
46
# component from the cozyplane agent — never co-mingled.
5-
image: ghcr.io/lllamnyp/cozyplane-kpr@sha256:425693919d8a4f24daea9629548180ef5cef72dece4d31bf5b6505a671147766
7+
image: ghcr.io/lllamnyp/cozyplane-kpr@sha256:93353087eb57f1cb0488e5fe503a376173703a5c626a9c074f0b527a57e4b4b6
68
imagePullPolicy: IfNotPresent
79
imagePullSecrets: []
810

‎packages/system/cozyplane/README.md‎

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,10 @@
22

33
Packages cozyplane — a multi-tenant eBPF CNI (flat default network + VPC
44
tenancy + VPC peering) — as the node agent DaemonSet, the controller
5-
Deployment, the `sdn.cozystack.io` API (CRDs or the aggregated API server),
6-
RBAC, and the VPCBinding `export` admission policy.
5+
Deployment, the `sdn.cozystack.io` API (as CRDs — the bootstrap surface),
6+
RBAC, and the VPCBinding `export` admission policy. The aggregated API server
7+
for the same group is the separate
8+
[cozyplane-apiserver](../cozyplane-apiserver) chart.
79

810
cozyplane is the **primary CNI**. Install it on a cluster with no other CNI
911
(or in place of one being removed), and keep a Service implementation — stock
@@ -17,18 +19,21 @@ for the tenancy model.
1719
- Kubernetes >= 1.30 (the `export` ValidatingAdmissionPolicy; set
1820
`exportPolicy.enabled=false` for older clusters).
1921
- A Linux kernel with BTF (`/sys/kernel/btf/vmlinux`), 5.10+.
20-
- Each node has `spec.podCIDR` set.
22+
- No per-node `spec.podCIDR` requirement: the fabric pool is flat and cluster-wide
23+
(`--cluster-cidr`), so cozyplane works with or without the node-ipam controller.
2124

2225
## Install
2326

2427
```bash
2528
helm install cozyplane ./chart/cozyplane --namespace cozy-cozyplane --create-namespace
2629
```
2730

28-
The `sdn.cozystack.io` API is served as CRDs by default. With
29-
`apiserver.enabled=true` the chart instead deploys the aggregated API server
30-
(plus a dedicated etcd and a cert-manager serving certificate) and installs no
31-
CRDs — same group/version/kinds, transparent to clients.
31+
The `sdn.cozystack.io` API is served as CRDs, so tenancy works the moment the
32+
CNI lands — no cert-manager required. Installing the separate
33+
[cozyplane-apiserver](../cozyplane-apiserver) chart later switches the group to
34+
the aggregated API server (same group/version/kinds, transparent to clients):
35+
its explicit APIService atomically takes over the serving, and these CRDs stay
36+
installed, shadowed and inert.
3237

3338
## Configuration
3439

@@ -42,11 +47,11 @@ most likely to set:
4247
`00-cozyplane.conflist` to sort ahead of a co-installed CNI (e.g. Cilium).
4348
- `genevePort` — override only to avoid a clash with another overlay on 6081.
4449
- `exportPolicy.enabled` — the VPCBinding export admission gate (needs k8s 1.30+).
45-
- `apiserver.enabled` — serve the API from the aggregated API server instead of
46-
CRDs (needs cert-manager).
50+
- `crds.enabled` — the CRD serving of the group (default true; disable only to
51+
keep the group unserved until the cozyplane-apiserver chart installs).
4752
- `egress.*` — cluster networking facts (pod/service CIDRs, cluster DNS) that
48-
drive node masquerade and the per-VPC egress gateways
49-
(`VPC.spec.egress.natGateway`); add node/management networks to
53+
drive node masquerade and the pool-less per-VPC egress gateway pod (a
54+
`VPCGateway` *with* a pool needs none -- its NAT is eBPF); add node/management networks to
5055
`egress.internalCIDRs`.
5156

5257
## What gets installed
@@ -56,6 +61,7 @@ most likely to set:
5661
- `cozyplane-controller` (Deployment): assigns VNIs, reaps Ports on VPCBinding
5762
revocation, and maintains VPCPeering status.
5863
- The `sdn.cozystack.io` API: `vpcs`, `vpcbindings`, `vpcpeerings`, `ports` — as
59-
CRDs, or served by the aggregated API server when `apiserver.enabled=true`.
60-
- RBAC for both components, the `cozyplane-vpc-owner` sample role, and the export
64+
CRDs (the aggregated server is [cozyplane-apiserver](../cozyplane-apiserver)).
65+
- RBAC for both components, the aggregated tenant roles (`cozyplane-tenant-edit` /
66+
`-view`, docs/multitenancy.md), and the export
6167
ValidatingAdmissionPolicy.

‎packages/system/cozyplane/templates/agent.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,11 @@ rules:
7171
verbs: ["get", "list", "watch"]
7272
# FabricIPs: the CNI plugin claims and releases underlay addresses through
7373
# this SA (the agent writes its kubeconfig), and the agent watches them.
74+
# deletecollection lets a failed ADD's rollback return the pod's claims in one
75+
# round trip; it reaches nothing the list+delete above could not.
7476
- apiGroups: ["local.sdn.cozystack.io"]
7577
resources: ["fabricips"]
76-
verbs: ["get", "list", "watch", "create", "delete"]
78+
verbs: ["get", "list", "watch", "create", "delete", "deletecollection"]
7779
# HostFirewalls: the agent compiles the objects selecting its node into the
7880
# hf_allow map and arms enforcement (docs/host-firewall.md).
7981
- apiGroups: ["sdn.cozystack.io"]

‎packages/system/cozyplane/templates/tenant-rbac.yaml‎

Lines changed: 1 addition & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,11 @@
11
---
22
# The TENANT persona (docs/multitenancy.md).
33
#
4-
# Until now cozyplane had no tenant in its API — only component roles. A tenant was
5-
# something the datapath enforced, not something the API knew about. These are the
6-
# roles a tenant holds.
7-
#
84
# R2 ("a tenant enumerates nothing it does not own") is satisfied STRUCTURALLY, not
95
# by vigilance: these roles list only NAMESPACED kinds, and a RoleBinding cannot
106
# grant access to a cluster-scoped resource. So a tenant bound to these in its own
117
# namespace *cannot* be given `list ports`, `list servicevips` or
12-
# `list hostfirewalls` — not by accident, not by a later careless edit here.
8+
# `list hostfirewalls`.
139
# Those are cluster-scoped precisely because allocation needs a global keyspace, and
1410
# that allocation detail must never become a tenant-visible surface: one `list ports`
1511
# would hand over every other tenant's pods, addresses, MACs and node placement.

0 commit comments

Comments
 (0)