feat(vpc,kubernetes): back VPC subnets with Proxmox VLANs - #4813

What this PR does

Part of #3481. Design proposal: cozystack/community#93 (on hold).

Status: reference implementation, not for merge in this form. The first step is the smaller design in cozystack/community#104, implemented in #5047: static zones, namespaced IP pools and admission, with no controller and no writes to the OVN database. This PR stays open as a draft so the controller, the egress gateway and the OVN gateway handling can be read next to it. It is rebased on main of 2026-11-06 (cb3fcd335).

This is the networking piece for Proxmox-backed tenant clusters, on top of the provisioning layer that #4087, #4483, #4088 and #4089 put in main. It is opened as a draft so that the proposal can be discussed first; the decision on Proxmox as a supported platform stays in #3481.

A tenant can extend a subnet of its VPC onto Proxmox VE by adding proxmox: {} to the subnet, and put Proxmox VMs on it, the workers of a Proxmox-backed Kubernetes cluster included. The subnet becomes a VLAN-backed logical switch of the tenant's own VPC router: VMs and pods on it are L2 neighbours, the router port is the VMs' gateway, and the VPC's isolation covers the VMs too. The platform picks the VLAN, the bridge and the IP pool, and the tenant cannot set them. Until now a Proxmox-backed cluster needed a raw bridge, VLAN tag and address range from the admin, outside any VPC, and admission had no way to tell a tenant's own VLAN from someone else's.

The owner of every object, lifecycle, failure handling, the security model and operations are in docs/proxmox-networking.md.

• API proxmox.cozystack.io/v1alpha1: ProxmoxNetworkZone (cluster-scoped, owned by the admin: bridge, Kube-OVN ProviderNetwork, VLAN range, transit network for egress) and ProxmoxNetwork (one VLAN request for one subnet, rendered by the vpc chart).
• Controller proxmox-network-controller: allocates a Kube-OVN Vlan from the zone's range and names it after the Subnet, so a restart recovers every allocation instead of allocating again; holds the Vlan, the Subnet and the IP pool with finalizers until no VM holds an address; puts each Proxmox-backed router port into an OVN HA_Chassis_Group (see below); annotates each namespace with its ready networks for admission; collects orphaned Vlans; exports cozy_proxmox_* metrics.
• Package packages/system/proxmox-network (and proxmox-network-crds, installed first): opt-in in the iaas bundle with bundles.enabledPackages: [cozystack.proxmox-network], which also enables the in-cluster IPAM provider. It ships the controller, its RBAC, the trunk ProviderNetwork, the zones, a transit network whose ACL keeps the egress gateways of different VPCs apart, ValidatingAdmissionPolicies, alerts and a read-only role that lets a tenant see its own networks.
• vpc: subnets[].proxmox (zone, vmRange) and egress (enabled, replicas, internalCidr): a VpcEgressGateway that SNATs traffic leaving the VPC onto the zone's transit network. VM addresses come from one namespaced InClusterIPPool over vmRange, which is excluded from Kube-OVN, so a pod and a VM never get the same address.
• kubernetes: proxmox.network.{vpc,subnet} instead of proxmox.ipv4Config. kubernetes-nodes: proxmox.network.{vpc,subnet} for net0 and proxmox.additionalNetworks[] for more NICs. The charts read the subnet's ProxmoxNetwork and give capmox the bridge, the VLAN and an ipv4PoolRef, so every worker address comes from the subnet's pool. A raw bridge/vlan next to a subnet is refused.
• E2E, two Chainsaw suites:
  • hack/e2e-chainsaw/proxmox-network runs in the regular lanes and needs no hypervisor. The e2e nodes have one NIC, so its provider network selects no node and every network stops at NoTrunkNodes, which still shows that the controller reached the OVN NB and found the router port. It covers the package and CRDs, VLAN allocation, exhaustion and release, the Kube-OVN objects and the localnet tag, a network held while an address is in use, and admission of pods and KubeVirt VMs, each refusal next to an admitted control. hack/select-e2e.sh maps the package to the suite, and runs that do not select it disable the package again.
  • hack/e2e-chainsaw/proxmox-network-env needs Proxmox VE with trunk NICs, a zone bridge and a transit router. It ships as chainsaw-test.yaml.disabled, like kubernetes-proxmox, and is run by hand (see its README.md); every environment detail comes from COZY_PXNET_* variables.

The commits can be reviewed one by one. The first is an independent fix, listed separately in the proposal; it can move to a separate PR if that is preferred:

• fix(vpc): each end of a VPC peering gets a /30. Kube-OVN parses localConnectIP as a CIDR and fails the whole Vpc reconcile, routes included, on a bare address.

Changes that reach existing installations

• vpc, every VPC: a static route's nextHopIP must lie inside one of the VPC's subnets, or in 169.254.0.0/16 when peers are declared. A VPC with a next hop elsewhere stops rendering after the upgrade with route to <cidr>: next hop <ip> is outside every subnet of this VPC. A next hop on the shared transit network would otherwise hand a VPC's traffic to another VPC's egress gateway.
• vpc, every peering: localConnectIP now carries /30 (the fix(vpc) commit).
• Only with the package enabled: the attachment policies hold in every namespace outside cozy-*, kube-system, the package namespace, the egress gateway namespaces and admission.exemptNamespaces. A pod or a KubeVirt VM may attach only NetworkAttachmentDefinitions and Kube-OVN subnets of its own namespace, and a *.kubernetes.io/logical_switch pin other than the namespace's default switch is refused on CREATE, so a copy of a running pod that keeps Kube-OVN's annotations (kubectl debug --copy-to, a restored backup) is refused as well. admission.vmNetworkNamespace and admission.podNetworkNamespace switch them off.

The controller writes to the OVN northbound database

Kube-OVN drops ARP that arrives from a localnet port for the IP of a distributed router port, so a VM outside the cluster never resolves its gateway. Making the router port a gateway port with an HA_Chassis_Group of the zone's trunk nodes removes that drop. Kube-OVN has no field for it, so the controller manages one group per Proxmox-backed router port, px-<port>, with external_ids owner=proxmox-network and network=<namespace>/<name>. It refuses a group another owner has marked, removes the group before it frees the VLAN, and reports GatewayReady and cozy_proxmox_network_gateway_*. It connects with Kube-OVN's client certificate, read through the API with a Role limited by resourceNames, and writes through the Service that selects the RAFT leader. The groups are keyed by VPC, so every Proxmox-backed port of one VPC is active on the same chassis: with the ports of one VPC on different chassis, the path to the egress gateway crossed Geneve, where Cilium's kube-proxy replacement rewrote traffic to a LoadBalancer IP. --manage-gateway-chassis=false turns it off. This is meant to go away once Kube-OVN sets the group for such subnets itself; that change has not been proposed to Kube-OVN yet.

Known limits

• Between two VPCs' Proxmox-backed subnets there is no drop inside OVN yet. Isolation holds because each VPC has its own router and VLAN, and the transit router the admin runs outside the cluster forwards only to the LoadBalancer range, DNS, NTP and the Internet. That router alone carries the L3 boundary between VPCs (docs/proxmox-networking.md §8, §10, §12 question 9; an open question of the proposal).
• VMs run at the zone MTU next to overlay pods at 1400 with no gateway_mtu, so a full-size DF packet from a VM to an overlay pod is dropped silently.
• A reply from an overlay pod on a node without the trunk reaches the VLAN switch on a chassis with no bridge mapping. It was predicted to fail and passed in the run below, without an explanation yet, so the suite still reports it as an open check.
• Deleting a VPC may delete its router while the held Subnets stay.
• Chassis failover and VPC peering with Proxmox-backed subnets have not been exercised on a live cluster.
• IPv4 only. No inbound path (EIP/DNAT) to a VM and no ClusterIP access from VMs. capmox and the CCM still use one Proxmox API token for every tenant.
• Neither Chainsaw suite has run on a cluster yet. The fork e2e lane cannot run the CI suite for this PR: Plan build refuses a fork PR that adds an image (hack/fork-new-images.sh, docs/agents/e2e-testing.md section 10), and this PR adds proxmox-network-controller. The environment suite is a port of the shell suite that produced the results below.

How this was tested

On this branch, rebased on main (cb3fcd335). The Proxmox CCM fix that used to be the first commit reached main through #4917 and has left the branch.

• make unit-tests test-controllers: the Go targets (go-unit-tests, go-module-tests, test-controllers, test-check-readiness, test-backport-audit, rd-presets-check, migrations-target-check), helm-unit-tests and bats-posix-compat-tests pass; go-module-tests with GOTOOLCHAIN=go1.27.1, since the host's own Go is older than the nested modules require. bats-unit-tests did not run: bats-core is not installed on this host.
• helm-unittest for the touched charts: vpc 32, kubernetes 371, kubernetes-nodes 253, proxmox-network 34, proxmox-network-crds 1, platform 289, all passing.
• go test -race for ./internal/proxmoxnetworkcontroller/..., ./api/proxmox/..., ./cmd/proxmox-network-controller/... and the api/apps/v1alpha1 module, against a fake OVN NB for the gateway. go vet, gofmt and go build ./... are clean.
• CRDs, deepcopy and cozyvalues-gen output are in sync. The one diff a local cozyvalues-gen leaves, x-cozystack-name in the kubernetes schema, is the same on main.
• chainsaw lint test on both suites; bash -n and shellcheck on the suite helper; the guard suites chainsaw-leak-guard_test, chainsaw-op-budgets, bats-runner-coverage, platform-packages_test (with a new test for the opt-in package), run-kubernetes-install-wait_test, select-e2e_test and select-install_test; hack/check-commit-trailers.sh on the range.

End to end, the shell form of the environment suite ran on a two-host Proxmox VE 9 cluster (a VLAN-aware bridge whose tenant VLANs ride inside one outer VLAN between the hosts, MTU 9000) next to a Talos management cluster running Kube-OVN v1.15.10 with Cilium's kube-proxy replacement, capmox v0.7.7 and in-cluster IPAM v1.0.3. Two tenants, each with a VPC of two Proxmox-backed subnets and one overlay subnet, and a Proxmox-backed Kubernetes cluster per tenant whose worker has net0 on one subnet (and, for tenant A, net1 on the other):

The open checks are the known limits above; the skipped ones are cells that topology could not exercise, such as the gateways of two VPCs on different chassis. The resilience phase restarts the controller and kube-ovn-cni and stops capmox during scaling with no change to any allocation; the lifecycle phase deletes a VPC under live VMs (the Vlan, the Subnet and the pool stay held) and then the cluster, after which VLANs, pools, subnets, addresses, OVN groups and the namespace annotation are all gone. main now ships Kube-OVN v1.15.26, so the gateway checks have to be repeated on it.

Screenshots

No UI change.

Downstream repositories

Walking the trigger map against this diff, three repositories are reached: the proposal, the website and the Terraform provider. The website follow-up is a draft PR until this one is decided; the provider pins released API types, so its follow-up is an issue until a tagged release carries these fields.

• No downstream repository is affected by this change
• cozystack/website - follow-up: docs(networking): describe Proxmox-backed VPC subnets website#741 (draft). A new platform component (content/en/docs/next/guides/platform-stack/_index.md, operations/configuration/licenses.md), the iaas bundle's new opt-in package (operations/configuration/variants.md), and an operator guide for zones and the trunk.
• cozystack/terraform-provider-cozystack - follow-up: VPC/Kubernetes/KubernetesNodes: model the Proxmox-backed VPC subnet fields (upstream cozystack#4813) terraform-provider-cozystack#66 (issue; a PR follows once a tagged release carries these fields). New fields in the vpc (subnets[].proxmox, egress), kubernetes (proxmox.network) and kubernetes-nodes (proxmox.network, proxmox.additionalNetworks) schemas.
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up: docs(design-proposals): add Proxmox-backed VPC subnets (follow-up, on hold) community#93

Release note

feat(vpc,kubernetes): a VPC subnet can be backed by a Proxmox VLAN (`subnets[].proxmox`), and Proxmox-backed Kubernetes clusters and node pools can place their workers on such a subnet (`proxmox.network.vpc`/`subnet`, `proxmox.additionalNetworks`) instead of a raw bridge and address range. VLANs, bridges and IP pools come from the new opt-in `cozystack.proxmox-network` package (`ProxmoxNetworkZone`, `ProxmoxNetwork`). VPC egress for these subnets goes through a VpcEgressGateway (`egress.enabled`). Every VPC now refuses a static route whose next hop lies outside its subnets, and VPC peering link addresses carry their /30.

Assisted-by: LLM