Skip to content

l2announcer: leader-election Lease name is a plain namespace+name concatenation, so two Services can end up sharing one Lease #48084

Closed

Description

@lexfrei

Is there an existing issue for this?

  • I have searched the existing issues

Version

equal or higher than v1.19.6 and lower than v1.20.0

What happened?

L2 announcements derive the per-Service leader-election Lease name by gluing the namespace and the Service name onto a prefix with a hyphen, and a hyphen is legal inside both halves. Two unrelated Services can therefore map onto the same Lease name and end up in one leader election.

const leasePrefix = "cilium-l2announce"
Name: fmt.Sprintf("%s-%s-%s", leasePrefix, svc.Name.Namespace(), svc.Name.Name()),

const leasePrefix = "cilium-l2announce"
func (l2a *L2Announcer) newLeaseLock(svc *loadbalancer.Service) *resourcelock.LeaseLock {
return &resourcelock.LeaseLock{
LeaseMeta: metav1.ObjectMeta{
Namespace: l2a.leaseNamespace(),
Name: fmt.Sprintf("%s-%s-%s", leasePrefix, svc.Name.Namespace(), svc.Name.Name()),
},
Client: l2a.params.Clientset.CoordinationV1(),
LockConfig: resourcelock.ResourceLockConfig{
Identity: l2a.localNode.Name,
},
}
}

Namespace foo + Service bar-baz, and namespace foo-bar + Service baz, both produce cilium-l2announce-foo-bar-baz. There is no sanitization, escaping or hashing anywhere on that path; the only other place the name is touched is the GC scan, which just prefix-matches (strings.HasPrefix(lease.Name, leasePrefix)) and does not disambiguate either. Same code on v1.19:

const leasePrefix = "cilium-l2announce"
func (l2a *L2Announcer) newLeaseLock(svc *loadbalancer.Service) *resourcelock.LeaseLock {
return &resourcelock.LeaseLock{
LeaseMeta: metav1.ObjectMeta{
Namespace: l2a.leaseNamespace(),
Name: fmt.Sprintf("%s-%s-%s", leasePrefix, svc.Name.Namespace(), svc.Name.Name()),

Expected: one Lease per Service.

Mechanism

Impact

Needs a hyphen in the right place, so it is not going to be common, but it is silent when it happens and the symptom points at the L2 policy rather than at a name collision. Multi-tenant clusters where namespaces are generated with a prefix (team-a, team-a-staging) and Service names are equally hyphenated are where I would expect it.

Suggested fix

Use a separator that cannot appear in either half. Neither a namespace (DNS-1123 label) nor a Service name (DNS-1035 label) may contain ., and . is legal in an object name (DNS-1123 subdomain), so fmt.Sprintf("%s-%s.%s", leasePrefix, ns, name) is unambiguous and stays readable in kubectl get leases. A hash suffix would also work but costs readability.

Either way the name changes on upgrade, which orphans the existing Leases. Those are released on agent shutdown (ReleaseOnCancel), which empties the holder identity, and leaseGC already deletes prefix-matching Leases with an empty holder, so the old objects should clean themselves up. I have not confirmed that.

How can we reproduce the issue?

  1. Enable L2 announcements and a CiliumLoadBalancerIPPool.
  2. kubectl create namespace foo && kubectl create namespace foo-bar
  3. Create Service bar-baz of type LoadBalancer in foo, and Service baz of type LoadBalancer in foo-bar.
  4. Create CiliumL2AnnouncementPolicy objects selecting both, with disjoint nodeSelectors so the two Services are announced from different nodes.
  5. kubectl get leases -n kube-system | grep cilium-l2announce shows one Lease named cilium-l2announce-foo-bar-baz instead of two.

Cilium Version

N/A — found by source inspection, no live cluster involved.

Kernel Version

N/A — found by source inspection, no live cluster involved.

Kubernetes Version

N/A — found by source inspection, no live cluster involved.

Regression

No response

Sysdump

No response

Relevant log output

Anything else?

No response

Cilium Users Document

  • Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

  • I agree to follow this project's Code of Conduct

Activity

  1. added theissue type on Aug 19, 2026
  2. added
    kind/bugThis is a bug in the Cilium logic.
    needs/triageThis issue requires triaging to establish severity and affected area.
    kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.
    on Aug 19, 2026
  3. lemon0333 commented on Aug 22, 2026

    @lemon0333

    Hi, I'd like to work on this.

    Proposed approach: make the leader-election Lease name injective by appending a short deterministic hash of the canonical namespace/name key to the existing cilium-l2announce-<namespace>-<name> name. Since neither a namespace nor a Service name can contain /, that key is unambiguous, so the hash differs for any distinct pair even when the readable part collides (e.g. foo/bar-baz vs foo-bar/baz). Keeping the cilium-l2announce prefix means the existing leaseGC prefix scan continues to match both new Leases and orphaned pre-upgrade Leases, so old objects still clean themselves up.

    I'll open a PR shortly with a unit test covering the collision case.

  4. lemon0333 commented on Aug 22, 2026

    @lemon0333

    Update: I'm no longer working on this and have closed my PR (#48135). Un-claiming so it's free for anyone else to take.

  5. added a commit that references this issue on Aug 22, 2026
    36da26d
  6. added a commit that references this issue on Aug 27, 2026
    2b9120f
  7. added a commit that references this issue on Sep 25, 2026
    c2502c4
  8. added a commit that references this issue on Nov 9, 2026
    ec392f3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/agentCilium agent related.feature/l2-announcementkind/bugThis is a bug in the Cilium logic.kind/community-reportThis was reported by a user in the Cilium community, eg via Slack.needs/triageThis issue requires triaging to establish severity and affected area.

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions