- Title:
IP addresses as a first-class resource: IPAddress, IPAddressClaim, IPAddressClass, and a provisioner contract - Author(s):
@lllamnyp - Date:
2026-07-15 - Status: Draft
The shape is settled and implemented; the contract details are not. The problem statement, the primitives survey, and the object model are worked out, and phase 1 of Rollout now exists in code (see Implementation status). Deliberately still open: the provisioner contract's own API (capabilities are not modelled yet), the admission policy of Security, the sequencing beyond phase 1, and the open questions below.
A tenant cannot own an address. They can only cause one to appear as a side effect
of creating a Service type: LoadBalancer, and it evaporates when that Service does.
There is no object to hold, keep, quota, hand to another workload, or point DNS
at with confidence.
This proposal makes the address a resource. Applying the PersistentVolume pattern
to addresses:
| storage | addresses | role |
|---|---|---|
StorageClass |
IPAddressClass |
which pool, which provisioner, which announcer |
PersistentVolume |
IPAddress (cluster-scoped) |
the address itself, with a claimRef and a reclaim policy |
PersistentVolumeClaim |
IPAddressClaim (namespaced) |
"give me one" — what a tenant creates |
| CSI driver | a provisioner contract | how a class's backend allocates, adopts, and pins |
The driving use case is the public address — the AWS Elastic IP experience:
allocate an address, keep it, attach it to something, detach it, attach it to
something else, release it when done. Not: "create a Service and hope you get the
same IP back." But the resource is not public-specific. A class names a pool, and a
platform can hand out more than one kind of pool — public ranges, a private internal
LB range, a provider's static addresses. So the kinds are named for the general thing
(IPAddress), and "public" is a property of the class's pool, not of the API.
The kinds IPAddress and IPAddressClaim are deliberate echoes of an existing
pattern, and the overlap is worth naming up front so reviewers are not surprised:
- Kubernetes core ships an
IPAddresskind innetworking.k8s.io(KEP-1880). It is not this: it is a low-level record where the object name is the IP, with a genericparentRef, and its authors named "any generalization onto something like an IPAM API" an explicit non-goal (see Context). Ours lives in a different API group, carries a reclaim policy, a claim binding, and a class, and is the IPAM object core declined to build. - Cluster API ships
IPAddressClaim/IPAddressinipam.cluster.x-k8s.iowith a documented third-party provider contract — the same claim-plus-concrete-address shape, but only ever wired to Machine addressing, never Services, and with no reclaim policy or association. Whether we adopt that group and contract or mint our own is an open question (below); the name choice keeps that door open.
Kinds are qualified by API group, so the short-name overlap is not a collision. The
kinds live in local.sdn.cozystack.io (see Positioning).
- Adjacent, not a dependency:
design-proposals/structured-external-exposure(community #29) restructures how a managed application requests exposure. This proposal is about what an address is. The two meet at one point — an exposure entry should be able to name anIPAddressClaiminstead of implicitly minting an address — but neither needs the other to land first, and this proposal does not assume #29's shape. - Related:
Service.spec.loadBalancerClass, the upstream field selecting which LB implementation serves a Service. AnIPAddressClassnames a provisioner and a pool, which determines the implementation that will announce the address. AnIPAddressClasstherefore fixes theloadBalancerClassa consuming Service must carry (§5). (AnExposureClasskind was proposed alongside community #29 and never shipped; there is nonetwork.cozystack.iogroup. That direction went toloadBalancerClassinstead — cozystack/cozystack#3164, cozystack/cozystack#3218.) - Depended on by:
design-proposals/endpoint-attachments(community #45), which attaches one endpoint of an application to an external address. It consumes anIPAddressClaimand the association annotation of §5, and itsEndpointAttachmentkind lives in this samelocal.sdn.cozystack.iofamily — the exposure half beside the allocation half, another CRD-served resident relying on thelocal.prefix's quota guarantee (Positioning). This proposal does not wait on it and lands independently. - Deferred to a sibling: the datapath for whole-IP 1:1 NAT (all ports in, and the workload egressing as that address) — today provided by cozy-proxy's nftables rules. That is a CNI concern, tracked separately. This proposal is the allocation half; that is the forwarding half. They are independent: allocation is useful with a stock LoadBalancer Service and no 1:1 NAT at all.
This should stand on its own, and also ship as part of the Cozystack API surface —
both, not either. A controller that reserves an address, binds it to a workload
through whatever pin mechanism the backend exposes, and gates that binding behind an
admission policy is useful to any Kubernetes platform running MetalLB, Cilium LB-IPAM,
or a cloud LB — not only Cozystack. So it is designed to be valuable standalone: the
tenant contract is a plain namespaced CRD, the quota is a stock ResourceQuota, the
enforcement is a ValidatingAdmissionPolicy — all vanilla Kubernetes, with no
Cozystack-only machinery required to run it.
Standalone value does not mean it lives outside the Cozystack API group. Cozystack
already ships components that are fully standalone products yet live under
*.cozystack.io — etcd-operator.cozystack.io is the precedent. This belongs there the
same way, and the component is deliberately shaped so Cozystack integrates it cleanly (a
default class, and an exposure path able to name a claim). Standalone-valuable and
in-the-Cozystack-group are not in tension, and designing for easy Cozystack integration
is a goal, not a compromise.
It also belongs in the networking group rather than a new one of its own. Cozystack
networking is sdn.cozystack.io today: SecurityGroup lives there, served by the
Cozystack API as a tenant-facing interface over Cilium network policies. Cozyplane serves
its own SecurityGroup in that same group, plus local.sdn.cozystack.io for the kinds it
serves as CRDs. Addresses are the third networking feature to want a home, and the
argument here is only that networking APIs should not accumulate several different group
names: one sdn family, with the local. prefix marking CRD-served kinds and the
bare group the aggregated ones. So the kinds live in local.sdn.cozystack.io.
ipam.cozystack.io is not wrong on its own terms — it adds a fourth name to a subject
area that should have one.
To be clear that this is a proposal and not an appeal to precedent:
local.sdn.cozystack.io is new, nothing serves it today, and this would be among the
first components to introduce it. Nor does the group
string create a Cozystack dependency for a standalone user — the same three CRDs install
under any name.
The obvious objection is that ipam. describes the object better than sdn. does: an
address ledger is not a datapath concern, and its consumers include Gateways and cloud
providers that are not SDN either. That is a fair reading, and the reason it does not win
here is that this is not all of IPAM and cannot become it — allocation stays with the
allocator, since AWS and MetalLB both keep it — so an ipam.cozystack.io group would
promise a generality the component deliberately does not have. If upstream moves and that
generality becomes real, it is a v2 question. Until then the kinds sit with the rest of
networking.
Whether the kinds are greenfield or an adoption of Cluster API's IPAM kinds is an open question (below).
Phase 1 of Rollout exists:
- address-controller — the class-agnostic core: the three kinds, claim–address binding, status, reclaim, and the contract per-class drivers plug into. It never touches a Service.
- metallb-iad — the reference per-class driver, for addresses announced by MetalLB. A reservation is held as a placeholder Service in a driver-owned namespace, so MetalLB's own accounting refuses to double-assign a held address.
One backend proves little on its own, so the load-bearing evidence is a second consumer. Cozyplane wears reserved addresses through this contract with no module import, no CRD dependency, and no informer on the claim kinds: it copies a claim name into the association annotation (§5) on a Service it already owns, and the driver does the rest. With the mechanism absent entirely it behaves identically, auto-assigning instead — reserved and dynamic are one code path. Validated end to end on a three-node cluster running address-controller + metallb-iad + MetalLB + cozyplane: an address survived delete-and-rebind and stayed externally reachable throughout.
Note precisely which seam that tests. Rollout §2 — a second provisioner — is the allocation seam, and it remains outstanding. What is demonstrated is the association seam: that a backend with an entirely different datapath can wear a reserved address while knowing nothing about the ledger beyond one annotation. The coupling turned out looser than this document assumed it would need to be.
The ValidatingAdmissionPolicy of Security is implemented in neither repository. It is
the outstanding half of phase 1, not a phase 2 item.
Cozystack installs MetalLB but renders no IPAddressPool/L2Advertisement itself;
the admin configures an allocator to suit the environment (MetalLB L2/BGP, Cilium
LB-IPAM, a cloud LB, or Service.spec.externalIPs pinning).
Across every one of those, the model is the same: an address is allocated to a Service, at the moment the Service is created, and freed when it is deleted.
"I gave my customer this IP address. It is in their firewall allow-list. I need to rebuild the VM behind it, and I need the address to still be mine afterwards."
"I want to move this address from the old VM to the new one during a cutover."
"I want to reserve four addresses now, because the range is nearly full, and attach them over the next month."
None of these are expressible. The address has no independent existence, so:
- It cannot be held. Delete the Service, lose the address. Someone else may get it.
- It cannot be moved. There is no detach/attach; there is only "hope the allocator hands you the same one."
- It cannot be quota'd. There is no object to count, so nothing bounds how many addresses a tenant consumes. The only lever is pre-carving a pool per tenant.
- It cannot be gated. Anyone who can create a
Servicein a namespace can cause an address to be allocated. RBAC authorizes verbs on resources, not fields. - It cannot be shown. No inventory. "Which addresses do we own, and who has them?" has no answer short of listing every Service in the cluster.
This is a new capability, argued on its own merits: hold, move, quota, enumerate. Cozystack has no prior reserve-then-associate mechanism to preserve — the platform uses kube-ovn only for subnets and VPCs, not for its EIP objects — so there is nothing here being restored, only something being added.
- A tenant can request an address and receive one, without an admin pre-creating a pool object for them.
- An address survives the deletion of the workload it was attached to (reclaim
policy
Retain), and can be attached to a different workload afterwards. - An operator can enumerate every address the cluster owns, and see who holds it.
- The number of addresses a tenant may hold is bounded by a plain
ResourceQuota(count/ipaddressclaims.<group>), with no new quota machinery. - An admin configures the address source once per class, not once per tenant and
not once per address. No pool-per-tenant, no
/32pools. - The tenant-facing API does not name the backend. A tenant asks for an address; it does not learn whether MetalLB, Cilium, or a cloud API produced it.
- The design admits at least: MetalLB, Cilium LB-IPAM, and a cloud provider whose addresses are allocated outside the cluster.
- A reserved address is never silently lost to a plain
Service type: LoadBalancer: reservations are reconciled against live Service assignments, so a collision is a detected, surfaced state rather than a quiet theft (see Design §8 and Security).
- Announcing addresses. Attraction (ARP/NDP, BGP, a cloud VNIC assignment) stays with the LB implementation. This proposal allocates and binds; it never puts a packet on a wire.
- The 1:1 NAT datapath. See Scope.
- Several owned addresses on one consumer. One rule is settled and enforced: a claim
is consumed by one consumer at a time, so two consumers cannot quietly share a
reservation. The converse is not settled, and this proposal does not pretend it is —
how many claims one consumer may hold is open. No limit on a workload is claimed:
nothing stops an application owning several Services, each with its own claim, and a
managed service reachable on a public address and an internal routable one at once is
exactly that — two Services, two claims, no new concept. But a single consumer wearing
several addresses is a real case rather than a hypothetical one, and Gateway API states
it outright:
Gateway.spec.addressesis a list, so one Gateway legitimately wears several addresses, and nothing in the model below says how several claims attach to one consumer or how a provisioner renders that. Acknowledged and deferred to future work. Two neighbouring cases stay out with it: N owned addresses on a workload whose extra addresses live on a tunnel interface the platform neither owns nor sees (a VM acting as a router or VPN concentrator), which is orthogonal to reservation; and the platform-owned variant of it, which would need an internal address and in-guest configuration per address. Dual-stack (one v4 + one v6) is a separate, narrower question, treated below rather than folded into this. - Replacing the LB implementation. MetalLB/Cilium/cloud stay exactly where they are; this sits above them.
- IPv4-only thinking. The model is family-agnostic; a claim may request v4, v6, or
both. (Whether one
IPAddresscarries both families or aDualclaim binds two is an open question below.) - Solving field-level authorization in general. It is used here (see Security) but the general problem is bigger than this proposal.
Load-bearing findings, because the design is shaped by them:
-
Nothing has a standing reservation. Not MetalLB, not Cilium LB-IPAM, not kube-vip, not PureLB, not OpenELB — and not Kubernetes core. KEP-1880 shipped cluster-scoped
ServiceCIDR/IPAddresskinds (the name is the IP, with a genericspec.parentRef), and then explicitly named "any generalization onto something like an IPAM API" a non-goal. Nothing upstream is coming to provide this. It is ours to build. -
Every backend has a "pin this exact address" hook, and it is always an annotation — because
Service.spec.loadBalancerIPwas deprecated in Kubernetes 1.24 with no core replacement, so every implementation invented its own:backend pin mechanism MetalLB metallb.io/loadBalancerIPsCilium LB-IPAM lbipam.cilium.io/ipskube-vip kube-vip.io/loadbalancerIPsOpenELB eip.openelb.kubesphere.io/v1alpha1AWS LB Controller service.beta.kubernetes.io/aws-load-balancer-eip-allocationsGCP / GKE networking.gke.io/load-balancer-ip-addressesPureLB none found — see Open questions Different spelling, identical semantics. This is the seam the whole design rests on: if we can pin, we can bind a reserved address to a workload.
-
A reserved-but-unattached address is inert, everywhere. MetalLB never announces an address that is not the live
status.loadBalancer.ingressof a Service it allocated. Cilium LB-IPAM only assigns; something else announces. A cloud EIP routes nowhere until associated. This is not an obstacle — it is correct semantics, and it is exactly why AWS bills for an idle EIP. -
One pool can serve many tenants. MetalLB's
IPAddressPool.spec.serviceAllocation.{namespaces,namespaceSelectors,serviceSelectors}and Cilium'sCiliumLoadBalancerIPPool.spec.serviceSelectorboth scope a single pool by namespace or label. AndL2Advertisement/BGPAdvertisementselect pools, not addresses — so one advertisement covers arbitrarily many addresses. The pool-per-tenant and/32-pool patterns were never necessary. -
Service.spec.loadBalancerClass(GA in 1.24) is the standard way for multiple LB implementations to coexist; MetalLB honours it via--lb-class. It is how a cluster can run more than one backend without them fighting.
# Admin, once per address source.
apiVersion: local.sdn.cozystack.io/v1alpha1
kind: IPAddressClass
metadata:
name: public
annotations:
ipaddressclass.local.sdn.cozystack.io/is-default-class: "true"
spec:
provisioner: metallb.drivers.local.sdn.cozystack.io # who fulfils claims of this class
loadBalancerClass: metallb.io/public # the implementation that announces it;
# empty => the cluster's default implementation
reclaimPolicy: Retain # Retain | Delete (default Retain)
parameters: # opaque to the core controller
addresses: ["203.0.113.0/24"]
---
# Tenant. This is the whole tenant-facing API.
apiVersion: local.sdn.cozystack.io/v1alpha1
kind: IPAddressClaim
metadata: {name: web, namespace: tenant-a}
spec:
className: public # empty => the default class
family: IPv4 # IPv4 | IPv6 | Dual
# addressName: ip-203-0-113-7 # optional: bind this specific address (see below)
status:
phase: Bound
addresses: # a list, so a Dual claim can report v4 + v6
- name: ip-203-0-113-7 # the IPAddress object
address: 203.0.113.7 # what the tenant reads, and puts in DNS
---
# Cluster-scoped. The inventory. Created by the provisioner, not the tenant.
apiVersion: local.sdn.cozystack.io/v1alpha1
kind: IPAddress
metadata: {name: ip-203-0-113-7}
spec:
className: public
address: 203.0.113.7
reclaimPolicy: Retain
claimRef: {namespace: tenant-a, name: web}
source: # a union — see below
fromClass: {} # this controller allocated it
# providerRef: {id: eipalloc-0a1b} # ...or it wraps a provider-side reservation
status:
phase: Bound
associatedTo: # nil => reserved but inert
kind: Service
namespace: tenant-a
name: web-lbspec.addressName — asking for a specific address. The
PersistentVolumeClaim.spec.volumeName analogue: a claim may name one IPAddress rather
than take whatever its class yields. It is meaningful only for a single-family claim, and
the controller honours it only if every ordinary condition already holds — the address is
Available, carries no claimRef, is not being deleted, and its class and family match
the ones the claim resolved. A name that matches nothing eligible is not an error: the
claim stays Pending and keeps waiting, exactly as it would with no addressName at all.
The field therefore narrows a candidate set and never widens one. It cannot pull an
address out of another class, take one away from another claim, or reclaim a Released
address before an admin has cleared its claimRef.
What it deliberately does not do is authorize. IPAddress is cluster-scoped, so a claim
naming one names an object outside its own namespace. Neither this field nor
spec.className — equally free-text, and equally unable to be gated by namespace RBAC —
decides which classes a namespace may draw from. That gap is identical with or without
addressName, and closing it belongs above this controller; see Security.
IPAddressClaim.status.addresses is a list deliberately: a Dual claim binds a
v4 and a v6 IPAddress and must report both, and a scalar address field could not.
For a single-family claim the list has one entry. (Whether Dual is even one claim
binding two objects, or two claims, is an open question — but the status shape must not
foreclose the list case, so it is a list from the start.)
There are two structurally different worlds, and the model must admit both:
- Self-allocating backends (MetalLB, Cilium, kube-vip, OpenELB): nothing anywhere
holds a reservation. Our provisioner is the IPAM of record.
IPAddressis authoritative, and its address is carved from the class's range (source.fromClass). - Provider-reservation backends (AWS, GCP, Hetzner): the reservation is already a
real object with a stable handle (
eipalloc-…, a named static address), guarded by the provider's own IAM. Our provisioner must adopt, not allocate (source.providerRef).
This is precisely the PersistentVolume volume-source union (spec.csi /
spec.nfs / …), and it is the strongest evidence the shape is not bent around any one
backend: it survives a backend where allocation is not ours at all. The tenant's
IPAddressClaim never learns the difference.
IPAddressClass.spec.provisioner names a controller, exactly as StorageClass names a
CSI driver. Each provisioner declares capabilities, and the core controller refuses
what a backend cannot do rather than half-supporting it:
| capability | meaning | consequence if absent |
|---|---|---|
Allocate |
can carve an address from a class range | class must use providerRef adoption |
Adopt |
can wrap a provider-side reservation | providerRef claims rejected |
Pin |
can bind a specific address to a Service | the class cannot serve claims at all |
Pin is not optional decoration — it is the capability the entire model depends on.
A backend that cannot be told which address to use can never attach a reserved one,
and a class over such a backend must reject IPAddressClaims outright instead of
allocating an address it can never bind. (PureLB may be exactly this case; see Open
questions.)
Whether provisioners are in-tree (a switch in one controller) or out-of-tree (separate deployments, a real CRD contract, as CSI did) is an open question. In-tree is right for three backends; the CSI lesson is that it stops being right at about five.
Association is a separate, reversible act — that is the entire point, and the thing the current model cannot express.
A workload references the claim, never the raw address:
kind: Service
metadata:
annotations:
local.sdn.cozystack.io/ip-address-claim: web # a claim in THIS namespace
spec: {type: LoadBalancer, ...}The controller resolves the claim → address, verifies the claim is in the Service's own namespace, and writes the backend's pin annotation (§1.2). MetalLB/Cilium/the cloud then allocates and announces exactly that address.
The class fixes the loadBalancerClass. An IPAddressClass names the pool and the
provisioner that carves it, which determines the LB implementation that will announce
anything drawn from it; the class states that implementation in
spec.loadBalancerClass, and a consuming Service must match it — empty matching empty,
meaning the cluster's default implementation (§1.5). This is not a preference: a Service
whose loadBalancerClass selects one implementation while its claim's address is held by
another is asking two backends for one address, and both will be wrong about who owns it.
The provisioner's own holder Services (§6) carry the class's value for the same reason.
The check can only ever be a refusal, never a repair: loadBalancerClass is immutable
on an existing Service, so a mismatch surfaces as a refused association naming both
values, and whoever creates the Service — a chart, a controller minting one on a tenant's
behalf, or a user — is responsible for setting it at creation. A controller that creates
the Service and names the claim has no excuse for getting it wrong, since it can read
the class first.
Remove the annotation and the announcement is withdrawn, but the IPAddress stays
Bound to its claim and simply becomes inert — a reserved address, held, attached to
nothing. Which is what an unassociated EIP is.
The annotated object is not always a Service. Some consumers own the Service they
are announced on rather than being one: under Gateway API the implementation renders the
data-plane Service from the Gateway, so there is no user-authored Service to annotate,
and the object the user does write has no way to name an address. The contract is therefore
an annotated object that resolves to an address consumer — Service is the first
member and Gateway the obvious second. Which object a given backend annotates, and where
it lands the pin (Gateway.spec.addresses, infrastructure annotations propagated onto the
generated Service, or the generated Service itself), is the provisioner's business (§4).
The consequence that is not the provisioner's business is in Security:
Gateway.spec.addresses names an address by value, so it is the same escalation
surface as a backend pin annotation and the admission policy has to cover it.
This is the claim-first path: the address exists before the Service names it. §6 covers the opposite order — when an eager allocator hands a Service an address before any claim exists.
An eager allocator does not wait to be asked. MetalLB assigns an address the moment it
sees a Service type: LoadBalancer, before any IPAddressClaim exists and whether or not
one ever will. Two things follow.
A reservation is held in the allocator's own books, not merely recorded in ours. For
every address that is reserved but not currently attached, the provisioner keeps a
holder: for MetalLB, a selectorless type: LoadBalancer Service in a provisioner-owned
namespace, which MetalLB assigns the address to and, having no endpoints, never announces —
held in the backend's accounting, silent on the wire. Neither MetalLB nor Cilium will give
one address to a second Service without an explicit sharing key, so for as long as the
holder exists the reservation is enforced by the allocator itself rather than by our
detection. This is the primary guarantee, and it is what makes a reserved-but-unattached
address safe to leave lying around.
The inventory still works backwards. The ledger cannot assume every address originates
from a claim: the controller also observes status.loadBalancer on Services and reconciles
what it finds into IPAddress objects after the fact, so the inventory converges on what is
actually in use rather than only on what was asked for. Its authority is scoped to ranges
a class manages; addresses from pools no class owns are observed, not adjudicated —
otherwise every foreign auto-assign pool in the cluster becomes a source of Conflict
noise.
This is loosely the shape of a StatefulSet volumeClaimTemplate — a workload causing a
first-class object to exist without anyone hand-writing it — but the analogy is weak and not
worth leaning on. An address is non-fungible: any 20 GiB PV substitutes for any other, a
specific IP does not, and that scarce identity is the whole reason an inventory is needed.
And the allocator does not wait for us, so claim-first and allocator-first allocations
can race and the inventory can transiently disagree with the live status.loadBalancer set.
The design commits to eventual consistency there, and to Service → inventory
reconciliation as a first-class direction alongside claim → address.
What remains once holders are in place is narrower than a general theft window, and worth naming precisely rather than waving at:
- The handoff. Releasing the holder and pinning the workload's own Service is not atomic. For that window the address is unheld and an eager allocator could hand it to whichever Service happens to be asking. Narrow, but real — and one of the intervals where layer 1 of Security earns its place.
- Annotation writers. With a reserved pool set
autoAssign: falsea plain Service cannot automatically draw a reserved address; it can only get one by naming the pool or pinning the address explicitly. That is precisely the write layer 1 gates and layer 2 detects.
For MetalLB, forever, for the whole cluster:
- one
IPAddressPoolcovering the reserved range, withautoAssign: false, and - one
L2Advertisement(orBGPAdvertisement) selecting it.
That is all. No pool per tenant, no /32s. The IPAddressClass carries the range; the
provisioner renders the backend objects. autoAssign: false keeps MetalLB from handing a
reserved address out automatically — but it does not make the pool unreachable,
because a Service can still name the pool explicitly (metallb.io/address-pool) or pin an
address (metallb.io/loadBalancerIPs). Closing that path is a security concern, not a
pool-config one; see §8 and Security.
No static pool layout can guarantee a reservation. On MetalLB, autoAssign: false
stops only automatic selection — a Service can still name the reserved pool
(metallb.io/address-pool) or pin a specific address (metallb.io/loadBalancerIPs) and
be handed a reserved address. And any principal who can impersonate the controller's
ServiceAccount — a cluster-admin can — sails past an admission policy keyed on identity.
So the design does not rest on an invariant it cannot enforce. It uses two layers,
and is explicit about which one covers which threat:
- Admission reduces the blast radius for the parties multi-tenancy is about —
tenants and namespaced principals. A
ValidatingAdmissionPolicyforbids them from writing any backend annotation that selects a pool or pins an address (MetalLB'smetallb.io/address-poolandmetallb.io/loadBalancerIPs, plus the §1.2 equivalents), leaving the claim reference as their only route to the reserved range. - Reconciliation makes a collision a detected, correctable state — however it arose.
The
IPAddressledger is authoritative. The controller watches Services and their assignedstatus.loadBalancer.ingress; if a Service holds an address belonging to anIPAddresswhoseclaimRef/associatedTodoes not name that Service, it setsIPAddress.status.phase: Conflict, raises a condition and a Service event, and does not let the rightful claim silently lose its address. This is what "dealt with" means for the cases admission cannot prevent (see the trust boundary in Security).
A Service arriving without a claim reference is not by itself the thing being
defended against — an eager allocator handing a Service an address is the normal case §6
reconciles after the fact. The conflict is narrower and specific: a Service ending up with
an address already Bound to a different claim — by an explicit pin, a race, or an
impersonated write. Layer 2 keys on exactly that — a claimRef/associatedTo mismatch —
not on the mere absence of an annotation. What separates legitimate allocation from theft
is whose reservation the address belongs to, not whether a claim annotation was present.
MetalLB. The class's IPAddressPool is autoAssign: false, so plain Services never
draw from it automatically; they draw from the admin's other pools. Explicit selection
(address-pool/loadBalancerIPs) is blocked for tenants by layer 1 and detected by layer
2 for anyone else. Scoping the pool further with IPAddressPool.spec.serviceAllocation
(limit to a controller-set label or namespace) is available defense-in-depth — not a
guarantee.
"A cluster-admin runs
kubectl create -f bad-service.yaml --as system:serviceaccount:cozy-address-controller:address-controller, referencing the reserved pool and maybe pinning a specific address." — Admission cannot stop this: the request carries the controller's identity, and a cluster-admin is the cluster's trust root anyway (they can rewrite the policy or edit MetalLB objects directly). Layer 2 is what applies. The Service is assigned a reserved address; the controller sees an assignment no claim authorizes; theIPAddressgoesConflictwith the offending Service named. The reservation is not silently overwritten — the conflict is surfaced for an operator (or an automated policy) to resolve. That is the honest boundary: the model defends tenants against each other, and turns admin/impersonator collisions into visible faults rather than pretending to prevent them.
Cilium LB-IPAM. Cilium has no autoAssign flag; the equivalent is serviceSelector.
The reserved CiliumLoadBalancerIPPool carries a serviceSelector matching a label the
controller stamps on associated Services only, so a plain Service lacks it and draws from
an unselected default pool. The same two layers apply: admission stops a tenant from
setting that label directly, and reconciliation catches an assignment no claim authorizes.
Cloud (adopt). The provider holds the reservation (eipalloc-…) under its own IAM. A
plain LB Service with no annotation is handed an ephemeral address from the provider's
general pool; the provider's API refuses to double-assign a reserved EIP. Here the
separation is enforced by the provider, not by us — which is exactly why
source.providerRef (adopt, don't allocate) is the right model for this world, and why
this is the one backend where the reservation genuinely cannot be raided from inside the
cluster.
- Tenants gain one kind:
IPAddressClaim. Create it, readstatus.addresses[].address, put that in DNS, reference the claim by name from a Service. They never seeIPAddress,IPAddressClass, MetalLB, or a pin annotation. - Admins gain
IPAddressClass(one per address source) andIPAddress(a read-mostly inventory:kubectl get ipaddressesfinally answers "what do we own and who has it"). - Quota works with no new machinery:
count/ipaddressclaims.<group>in a stockResourceQuota.
- Additive. Nothing about today's
Service type: LoadBalancerpath changes; a Service with no claim annotation behaves exactly as now. - Adoption path. A cluster with existing LoadBalancer Services can import their
addresses as
IPAddressobjects withsource.providerRef/fromClassand aclaimRefto a generated claim — an offline, reversible migration. - Rollback. Deleting the CRDs with
reclaimPolicy: Retainleaves the backend objects and the live Services untouched; addresses stay where they are. WithDelete, it does not — flagged as the irreversible case.
Selecting a pool or pinning an address is a privilege-escalation surface, and this is the sharp edge of the proposal.
RBAC authorizes verbs on resources, not fields or annotations. Today, anyone who
can create a Service in a namespace can write metallb.io/loadBalancerIPs: <any address> — or name the reserved pool with metallb.io/address-pool — and because a
self-allocating backend has no concept of a reservation, it will happily hand over an
address another tenant reserved but has not yet attached. That is a theft window, on
MetalLB and Cilium alike. It is not a cloud problem: there, IAM already guards
eipalloc-….
No single mechanism closes this, so the response has two layers (see Design §8), and the design is explicit about which threat each one covers.
Layer 1 — admission, against tenants. Turn the ungatable annotation into a reference to an RBAC-gated object:
- A tenant may only write
local.sdn.cozystack.io/ip-address-claim, naming an object in their own namespace, which RBAC can gate. - The controller writes the backend's raw pool/pin annotations.
- A
ValidatingAdmissionPolicyrejects a non-allowlisted principal writing any backend annotation that selects a pool or pins an address — MetalLB'smetallb.io/address-poolandmetallb.io/loadBalancerIPs, plus the §1.2 equivalents for other backends — on a Service.
Identifying the controller, and not breaking the platform. The policy must not rely on
a hardcoded username. It matches the controller's ServiceAccount by name and
namespace (system:serviceaccount:<ns>:<sa>), configured at install time. And it must
carry an allowlist: real clusters have non-tenant principals that legitimately write
these annotations — GitOps reconcilers (Flux/Argo SAs) and system controllers (system
Istio Services ship with pool annotations already set). A naive "reject everyone but the
controller" breaks reconciliation on day one, so the policy denies the write only for
principals outside an explicit, admin-configurable allowlist.
The allowlist's hard case: trusted principals applying untrusted content. An identity allowlist assumes a trusted principal writes content it authored, and that assumption fails wherever a privileged controller applies material some user supplied. Two shapes of this are ordinary rather than exotic: a GitOps reconciler that applies user-authored releases under its own identity instead of impersonating the user, and a cloud-controller-manager that copies Service annotations from a user-controlled cluster into an infrastructure Service verbatim. Allowlist such a deputy and every user behind it inherits the ability to write a pin annotation; deny it and legitimate reconciliation breaks. So annotation filtering at the deputy is a precondition of layer 1, not a hardening extra: a strip-list for the gated keys in any controller that forwards user-supplied annotations, and impersonation rather than ambient authority in any reconciler applying user-authored content. Until those hold for a given deputy, layer 1 is partial for the users behind it and layer 2 is what covers them. The design says so rather than assuming a trusted identity implies trusted content.
Layer 2 — reconciliation, for everyone admission cannot bind. Layer 1 is keyed on
identity, so it is defeated by anyone who can impersonate an allowlisted ServiceAccount —
which a cluster-admin can (--as system:serviceaccount:cozy-address-controller:address-controller), along with rewriting
the policy or editing MetalLB objects directly. This is a real trust
boundary, not a bug to be closed: a cluster-admin is the cluster's trust root, and
handing out a reserved address is within their authority, not an escalation. What the
design owes here is not prevention but detection. Because the IPAddress ledger is
authoritative and continuously reconciled, any collision — impersonation, a direct
backend edit, a misconfigured pool, MetalLB assigning a reserved address to a Service with
no matching claim — surfaces as IPAddress.status.phase: Conflict with the offending
Service named, and the rightful claim is never silently overwritten. Admission defends
tenants against each other; reconciliation turns the cases admission cannot reach into
visible faults instead of quiet theft.
Landing layer 1 is worth doing on its own, independent of the rest of the model: the theft window it closes is live in any cluster running a shared auto-assign pool today.
Where this component stops. The two layers above cover the surface a reservation can
be stolen through. They do not close field-level authorization in general: a tenant who
can create an IPAddressClaim can consume an address, may name any spec.className, and
may name a specific IPAddress through spec.addressName (§2). ResourceQuota bounds how
many addresses they hold; nothing here decides which class a namespace is entitled to
draw from.
That is deliberate rather than unfinished. This component's job is the capability —
reserve an address, hold it, attach it, detach it, release it — and a capability that also
tried to be its own policy engine would have to encode a tenancy model it does not have and
cannot see. The gating belongs where the tenancy model lives: a ValidatingAdmissionPolicy
restricting which classes a namespace may name (the same mechanism as layer 1, with a
different subject), or platform machinery that mints claims on a tenant's behalf instead of
letting tenants write claims directly. spec.className and spec.addressName are the two
fields such a policy must cover — naming them here is the point, so that a deployment
hardening this does not have to discover them.
- Class range exhausted → claim stays
Pendingwith a reason; no partial binding. - Claim deleted,
reclaimPolicy: Retain→IPAddressgoesReleased, keeps the address, is not reusable until an admin clears theclaimRef. (PV semantics, deliberately.) - Claim deleted,
Delete→ address returned to the range; the backend object is torn down. - Service references a claim in another namespace → rejected. Cross-namespace address sharing is not a thing.
- Two Services reference one claim → resolved by reconciliation, not at admission. A
ValidatingAdmissionPolicycannot see the claim — CEL is given the request object and the policy's params, not arbitrary cluster state — so admission-time rejection would need a webhook. Instead one association wins by a deterministic rule, the loser reportsConflicted, and the address is never silently moved. A 1:1 binding is 1:1; letting the second win quietly is how an address goes dead. - Plain Service pulls a reserved address (explicit
address-pool/pin, or a request made under an impersonated controller identity) → not prevented in general; detected. TheIPAddressgoesConflict, the offending Service is named, and the reservation is not silently overwritten. See §8 and Security. spec.addressNamenames an address that is absent,Bound,Released, or of another class or family → nothing binds; the claim staysPendingwith the ordinary waiting reason. Naming an address filters, it never seizes.- A
Dualclaim associated with a single-stack Service → the Service carries the one family it has; the other address staysBoundto the claim and inert. The claim is notLost, and the unattached family is never silently released. - Backend lacks
Pin→ claims against that class are refused loudly, rather than allocating an address that can never be attached. With no capability registry there is nothing for admission to consult, so this is the provisioner refusing and reporting it on the claim, not an admission-time rejection. - Address adopted from a provider, then released provider-side →
IPAddressgoesLost. It must not silently re-allocate.
- Unit: the allocator (range carving, exhaustion, reclaim transitions, the source union).
- Integration: claim → bind → associate → delete the Service → assert the address is still held → associate it to a different workload → assert the same address comes back. That single test is the proposal. It is precisely what cannot pass today.
- e2e (per backend): MetalLB and Cilium, on a real cluster: an external client reaches the workload on the reserved address, and still reaches it after the workload is rebuilt behind the same claim.
- Admission: a tenant writing a raw pool-select or pin annotation is rejected; a tenant reserving an address and a second tenant attempting to pin it directly is rejected; an allowlisted GitOps/system principal writing a pool annotation is admitted (the allowlist does not over-block).
- Reconciliation: a Service assigned a reserved address with no authorizing claim
(e.g. created under an impersonated controller identity) drives the
IPAddresstoConflictand emits an event; the rightful claim's binding is not lost.
Sketch — sequencing is an open question:
- CRDs + core controller + the MetalLB provisioner + the admission policy. (The admission policy is not phase 2. See Security.) The controllers and the driver exist; the admission policy does not — see Implementation status.
- The Cilium provisioner. Proves the abstraction is not a MetalLB adapter — this is the phase that either validates or falsifies the design, and it should come early. A second consumer has since exercised the association half (Implementation status); a second provisioner is what remains.
- A cloud provisioner (adoption path,
source.providerRef). Proves the source union.
- Provisioners: in-tree or out-of-tree? In-tree for three backends; CSI's history says that stops scaling. Where is the line, and do we want the CRD contract on day one?
- Reuse Cluster API's IPAM?
ipam.cluster.x-k8s.ioIPAddressClaim/IPAddressis literally this pattern — namespaced claim, concrete address object, a documented third-party provider contract — and shares even the kind names we chose. It has only ever been wired to Machine addressing, never Services, and has no reclaim policy or association. Do we adopt those kinds (and that group), or mint our own and merely copy the contract? (Leaning: our own — we need reclaim policy and association, which it has no concept of. But the shape is not novel and review should know that.) This is the greenfield-vs-adopt question Positioning leaves open. - Dual-stack. Does one
IPAddresscarry a v4 and a v6 address, or does aDualclaim bind twoIPAddresses? (PV has no precedent. Leaning: two objects, one claim — the status list already admits it.) - PureLB. Does it have any "request this exact IP" mechanism? If not, it cannot be a backend, and that should be stated rather than discovered.
- Sharing one address across Services (different ports — GCP and Cilium both permit it). Today one consumer at a time holds a claim and a second reference is refused (§5). Should that relax to several consumers on disjoint ports? It is the mirror of the deferred several-addresses-on-one-consumer case in Non-goals, and the two probably want answering together, since both are really "what is the cardinality between a claim and the things that wear it".
Bind the address to a Service — the shape community #29 proposed as
ServiceExposure. A namespaced object naming a serviceRef, which allocates an address
and reports it in status. Rejected as the
model for this problem: it fuses allocation with association, so the address's
lifetime is the Service's lifetime — which is the exact thing being fixed. It cannot
express "keep this address, the workload is gone", which is the entire user request.
The class half of that idea is right and is kept here; the exposure half is not a
reservation and cannot be made into one.
Per-tenant pools. Give each tenant an IPAddressPool (or a /32 per address) and let
MetalLB do the rest. Rejected: it is the status quo's failure mode, it makes the admin
the allocator, it scales as O(tenants) or O(addresses) in operator-managed objects, and it
still provides no reservation — the address is released the moment the Service goes away.
Let the tenant write the backend's pin annotation directly. Rejected: unauthorizable (RBAC does not gate fields), and it hands every tenant the ability to steal any reserved address that is not currently attached. See Security.
Wait for upstream. Rejected: KEP-1880 lists generalizing to an IPAM API as an
explicit non-goal, and loadBalancerIP was deprecated with no replacement. There is
nothing to wait for.
Do nothing. Rejected: the capability simply does not exist today — an address cannot be held, moved, quota'd, or enumerated — and the pin-annotation theft window in Security is live in any cluster running a shared auto-assign pool. Doing nothing leaves both the missing capability and the open security hole in place.