|
| 1 | +--- |
| 2 | +title: "Network Isolation Between Tenants on Bare Metal: Cilium eBPF Policies, Plus Kube-OVN VPC" |
| 3 | +slug: network-isolation-between-tenants-cilium-ebpf-kube-ovn-vpc |
| 4 | +date: 2026-07-14 |
| 5 | +author: "Timur Tukaev" |
| 6 | +description: "How Cozystack isolates tenants on bare metal by default with Cilium eBPF policies, plus optional Kube-OVN VPCs for workloads that need dedicated subnets." |
| 7 | +article_types: |
| 8 | + - tech-article |
| 9 | + - networking |
| 10 | +topics: |
| 11 | + - networking |
| 12 | + - security |
| 13 | + - cilium |
| 14 | + - kubernetes |
| 15 | + - platform |
| 16 | + |
| 17 | +--- |
| 18 | + |
| 19 | +Multi-tenant infrastructure without network isolation is a ticking time bomb. One misconfigured service in Tenant A can scan, attack, or accidentally connect to Tenant B's databases. In AWS you get this by default — VPCs and security groups. On bare metal? You're usually left stitching together CNI policies, overlays, and hoping nobody makes a mistake. |
| 20 | + |
| 21 | +Cozystack solves this out of the box with two complementary layers: **tenant isolation enforced by Cilium eBPF** (always on, zero config), and **optional Kube-OVN VPCs** for workloads that need their own dedicated subnets. |
| 22 | + |
| 23 | +## Layer 1: Tenant isolation with Cilium eBPF (default, always on) |
| 24 | + |
| 25 | +This is the part most people get wrong about Cozystack. Tenant-to-tenant isolation is **not** done by network segmentation. All tenants share a single Pod CIDR (`10.244.0.0/16`), and Kube-OVN allocates IPs centrally from that shared pool. Isolation is enforced by **Cilium eBPF network policies at the kernel level**. |
| 26 | + |
| 27 | +How it works: |
| 28 | + |
| 29 | +- Kube-OVN is the CNI — it handles pod networking and IPAM (centralized, no per-node CIDR splitting, GENEVE overlay). Its own network policy engine is disabled (`ENABLE_NP: false`). |
| 30 | +- Cilium runs as a chained CNI (`generic-veth` mode) and owns all policy enforcement plus service load balancing (kube-proxy replacement). |
| 31 | +- Cilium assigns each pod a **security identity** derived from its labels, then enforces policy on identities — not IPs — entirely in kernel space (no userspace bypass). |
| 32 | + |
| 33 | +When you create a tenant, Cozystack **automatically applies** `CiliumNetworkPolicy` and `CiliumClusterwideNetworkPolicy` resources. These enforce namespace-level isolation and restrict access to system ports (etcd, kubelet, controllers). Same-tenant traffic is allowed; cross-tenant traffic is dropped. Policies match on hierarchical `tenant.cozystack.io/*` namespace labels, so a parent tenant can include its sub-tenant namespaces: |
| 34 | + |
| 35 | +```yaml |
| 36 | +apiVersion: cilium.io/v2 |
| 37 | +kind: CiliumNetworkPolicy |
| 38 | +metadata: |
| 39 | + name: allow-internal-communication |
| 40 | + namespace: tenant-example |
| 41 | +spec: |
| 42 | + endpointSelector: {} |
| 43 | + ingress: |
| 44 | + - fromEndpoints: |
| 45 | + - matchLabels: |
| 46 | + k8s:io.cilium.k8s.namespace.labels.tenant.cozystack.io/tenant-example: "" |
| 47 | + egress: |
| 48 | + - toEndpoints: |
| 49 | + - matchLabels: |
| 50 | + k8s:io.cilium.k8s.namespace.labels.tenant.cozystack.io/tenant-example: "" |
| 51 | + - toEntities: |
| 52 | + - kube-apiserver |
| 53 | + - cluster |
| 54 | +``` |
| 55 | +
|
| 56 | +You don't write these by hand — they come with the tenant. This is the isolation boundary between tenants. |
| 57 | +
|
| 58 | +## Layer 2: Dedicated subnets with Kube-OVN VPC (optional) |
| 59 | +
|
| 60 | +On top of tenant isolation, some workloads need their own dedicated networking space — separate subnets, multiple NICs on a VM. That's what VPC is for. |
| 61 | +
|
| 62 | +A VPC provides a set of dedicated subnets backed by Kube-OVN (OVN virtual routers and switches, with DHCP and IPAM). Multus provides multi-NIC capability, so a pod or VM can have two or more interfaces. Note the current scope: every workload still connects to a default management network with the default gateway, through which the majority of traffic flows — **VPC subnets are additional, dedicated networking spaces**. More isolation capabilities are planned as the feature evolves. |
| 63 | +
|
| 64 | +**Prerequisites:** VPC requires `kube-ovn` and `multus` CNI, so by default it only works on the **`paas-full`** bundle. |
| 65 | + |
| 66 | +Create a VPC via kubectl: |
| 67 | + |
| 68 | +```yaml |
| 69 | +apiVersion: apps.cozystack.io/v1alpha1 |
| 70 | +kind: VirtualPrivateCloud |
| 71 | +metadata: |
| 72 | + name: production |
| 73 | + namespace: tenant-team1 |
| 74 | +spec: |
| 75 | + subnets: |
| 76 | + app-subnet: |
| 77 | + cidr: 172.16.0.0/24 |
| 78 | + db-subnet: |
| 79 | + cidr: 172.16.1.0/24 |
| 80 | +``` |
| 81 | + |
| 82 | +```bash |
| 83 | +kubectl apply -f vpc-production.yaml |
| 84 | +``` |
| 85 | + |
| 86 | +`subnets` is a map keyed by subnet name, not a list. |
| 87 | + |
| 88 | +A VM or a pod may be connected to one or more of these subnets at once. Each connection shows up as an additional network interface, in addition to the default management interface. |
| 89 | + |
| 90 | +Deployment notes (from the docs): |
| 91 | + |
| 92 | +- A VPC name must be unique within its namespace; subnet name and CIDR must be unique within a VPC. |
| 93 | +- Subnet CIDRs must not overlap with the default management network — subsets of `172.16.0.0/12` are recommended. |
| 94 | +- Different VPCs may use overlapping subnet CIDRs. |
| 95 | +- There are currently no fail-safe validation checks on these constraints (planned for the future), so pick ranges carefully. |
| 96 | + |
| 97 | +## The bigger picture |
| 98 | + |
| 99 | +Cozystack's default networking stack (`kubeovn-cilium` variant on Talos) layers MetalLB (external load balancing), Cilium eBPF (service load balancing + network policies), and Kube-OVN (pod networking + IPAM). Tenant isolation is a property of the platform you get for free; VPC is an opt-in tool for dedicated subnets on top of it. |
| 100 | + |
| 101 | +## Documentation |
| 102 | + |
| 103 | +- [Networking](https://cozystack.io/docs/networking/) |
| 104 | +- [VPC](https://cozystack.io/docs/networking/vpc/) |
| 105 | +- [VPN](https://cozystack.io/docs/networking/vpn/) |
| 106 | + |
| 107 | +## Join the community |
| 108 | + |
| 109 | +- [Cozystack on GitHub](https://github.com/cozystack/cozystack) |
| 110 | +- Telegram [group](https://t.me/cozystack) |
| 111 | +- Slack [group](https://kubernetes.slack.com/archives/C06L3CPRVN1) (Get invite at [https://slack.kubernetes.io](https://slack.kubernetes.io)) |
| 112 | +- [Community Meeting Calendar](https://calendar.google.com/calendar?cid=ZTQzZDIxZTVjOWI0NWE5NWYyOGM1ZDY0OWMyY2IxZTFmNDMzZTJlNjUzYjU2ZGJiZGE3NGNhMzA2ZjBkMGY2OEBncm91cC5jYWxlbmRhci5nb29nbGUuY29t) |
0 commit comments