Repository navigation
Description
Is there an existing issue for this?
- I have searched the existing issues
Version
equal or higher than v1.19.6 and lower than v1.20.0
What happened?
L2 announcements derive the per-Service leader-election Lease name by gluing the namespace and the Service name onto a prefix with a hyphen, and a hyphen is legal inside both halves. Two unrelated Services can therefore map onto the same Lease name and end up in one leader election.
const leasePrefix = "cilium-l2announce"
Name: fmt.Sprintf("%s-%s-%s", leasePrefix, svc.Name.Namespace(), svc.Name.Name()),cilium/pkg/l2announcer/l2announcer.go
Lines 847 to 860 in ee26050
| const leasePrefix = "cilium-l2announce" | |
| func (l2a *L2Announcer) newLeaseLock(svc *loadbalancer.Service) *resourcelock.LeaseLock { | |
| return &resourcelock.LeaseLock{ | |
| LeaseMeta: metav1.ObjectMeta{ | |
| Namespace: l2a.leaseNamespace(), | |
| Name: fmt.Sprintf("%s-%s-%s", leasePrefix, svc.Name.Namespace(), svc.Name.Name()), | |
| }, | |
| Client: l2a.params.Clientset.CoordinationV1(), | |
| LockConfig: resourcelock.ResourceLockConfig{ | |
| Identity: l2a.localNode.Name, | |
| }, | |
| } | |
| } |
Namespace foo + Service bar-baz, and namespace foo-bar + Service baz, both produce cilium-l2announce-foo-bar-baz. There is no sanitization, escaping or hashing anywhere on that path; the only other place the name is touched is the GC scan, which just prefix-matches (strings.HasPrefix(lease.Name, leasePrefix)) and does not disambiguate either. Same code on v1.19:
cilium/pkg/l2announcer/l2announcer.go
Lines 820 to 826 in 44d55ea
| const leasePrefix = "cilium-l2announce" | |
| func (l2a *L2Announcer) newLeaseLock(svc *loadbalancer.Service) *resourcelock.LeaseLock { | |
| return &resourcelock.LeaseLock{ | |
| LeaseMeta: metav1.ObjectMeta{ | |
| Namespace: l2a.leaseNamespace(), | |
| Name: fmt.Sprintf("%s-%s-%s", leasePrefix, svc.Name.Namespace(), svc.Name.Name()), |
Expected: one Lease per Service.
Mechanism
Impact
Needs a hyphen in the right place, so it is not going to be common, but it is silent when it happens and the symptom points at the L2 policy rather than at a name collision. Multi-tenant clusters where namespaces are generated with a prefix (team-a, team-a-staging) and Service names are equally hyphenated are where I would expect it.
Suggested fix
Use a separator that cannot appear in either half. Neither a namespace (DNS-1123 label) nor a Service name (DNS-1035 label) may contain ., and . is legal in an object name (DNS-1123 subdomain), so fmt.Sprintf("%s-%s.%s", leasePrefix, ns, name) is unambiguous and stays readable in kubectl get leases. A hash suffix would also work but costs readability.
Either way the name changes on upgrade, which orphans the existing Leases. Those are released on agent shutdown (ReleaseOnCancel), which empties the holder identity, and leaseGC already deletes prefix-matching Leases with an empty holder, so the old objects should clean themselves up. I have not confirmed that.
How can we reproduce the issue?
- Enable L2 announcements and a
CiliumLoadBalancerIPPool. kubectl create namespace foo && kubectl create namespace foo-bar- Create Service
bar-bazof type LoadBalancer infoo, and Servicebazof type LoadBalancer infoo-bar. - Create
CiliumL2AnnouncementPolicyobjects selecting both, with disjointnodeSelectors so the two Services are announced from different nodes. kubectl get leases -n kube-system | grep cilium-l2announceshows one Lease namedcilium-l2announce-foo-bar-bazinstead of two.
Cilium Version
N/A — found by source inspection, no live cluster involved.
Kernel Version
N/A — found by source inspection, no live cluster involved.
Kubernetes Version
N/A — found by source inspection, no live cluster involved.
Regression
No response
Sysdump
No response
Relevant log output
Anything else?
No response
Cilium Users Document
- Are you a user of Cilium? Please add yourself to the Users doc
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
lemon0333 commentedon Aug 22, 2026
Hi, I'd like to work on this.
Proposed approach: make the leader-election Lease name injective by appending a short deterministic hash of the canonical
namespace/namekey to the existingcilium-l2announce-<namespace>-<name>name. Since neither a namespace nor a Service name can contain/, that key is unambiguous, so the hash differs for any distinct pair even when the readable part collides (e.g.foo/bar-bazvsfoo-bar/baz). Keeping thecilium-l2announceprefix means the existingleaseGCprefix scan continues to match both new Leases and orphaned pre-upgrade Leases, so old objects still clean themselves up.I'll open a PR shortly with a unit test covering the collision case.
lemon0333 commentedon Aug 22, 2026
Update: I'm no longer working on this and have closed my PR (#48135). Un-claiming so it's free for anyone else to take.
fix: use unambiguous separator in l2announcer Lease name
fix: use unambiguous separator in l2announcer Lease name
fix: use unambiguous separator in l2announcer Lease name
fix: use unambiguous separator in l2announcer Lease name