Skip to content

The LB IP address assigned to a service changes when another LB service is deleted #2723

Closed

Description

@jcaamano

MetalLB Version

v0.14.9

Deployment method

Not relevant

Main CNI

OVN-Kubernetes

Kubernetes Version

v1.32.2

Cluster Distribution

kind

Describe the bug

A service gets .3 address assigned

{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-1914/udn-service","ts":"2025-03-17T11:20:08Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.3","fc00:f853:ccd:e799::3"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:08Z"}

A different service gets deleted

{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-7056/udn-service","ts":"2025-03-17T11:20:13Z"}
{"caller":"main.go:58","event":"serviceDeleted","level":"info","msg":"service deleted","ts":"2025-03-17T11:20:13Z"}

The first service is updated and assigned .2 address

re-assigned .2
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-1914/udn-service","ts":"2025-03-17T11:20:14Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.2","fc00:f853:ccd:e799::2"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:14Z"}

Full default logs of the full time window

{"caller":"pool_controller.go:48","controller":"PoolReconciler","level":"info","start reconcile":"/udn-services-1914","ts":"2025-03-17T11:20:08Z"}
{"caller":"pool_controller.go:99","controller":"PoolReconciler","event":"force service reload","level":"info","ts":"2025-03-17T11:20:08Z"}
{"caller":"pool_controller.go:112","controller":"PoolReconciler","event":"config reloaded","level":"info","ts":"2025-03-17T11:20:08Z"}
{"caller":"pool_controller.go:113","controller":"PoolReconciler","end reconcile":"/udn-services-1914","level":"info","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller_reload.go:63","controller":"ServiceReconciler - reprocessAll","level":"info","start reconcile":"metallbreload/reload","ts":"2025-03-17T11:20:08Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.0","fc00:f853:ccd:e799::"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:08Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:08Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.1","fc00:f853:ccd:e799::1"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:08Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:08Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.2","fc00:f853:ccd:e799::2"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:08Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller_reload.go:119","controller":"ServiceReconciler - reprocessAll","end reconcile":"metallbreload/reload","level":"info","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-1914/udn-service","ts":"2025-03-17T11:20:08Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.3","fc00:f853:ccd:e799::3"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:08Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"udn-services-1914/udn-service","level":"info","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-1914/udn-service","ts":"2025-03-17T11:20:08Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.3","fc00:f853:ccd:e799::3"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:08Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"udn-services-1914/udn-service","level":"info","ts":"2025-03-17T11:20:08Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-7056/udn-service","ts":"2025-03-17T11:20:13Z"}
{"caller":"main.go:58","event":"serviceDeleted","level":"info","msg":"service deleted","ts":"2025-03-17T11:20:13Z"}
{"caller":"service_controller.go:107","controller":"ServiceReconciler","event":"force service reload","level":"info","ts":"2025-03-17T11:20:13Z"}
{"caller":"service_controller.go:109","controller":"ServiceReconciler","end reconcile":"udn-services-7056/udn-service","level":"info","ts":"2025-03-17T11:20:13Z"}
{"caller":"service_controller_reload.go:63","controller":"ServiceReconciler - reprocessAll","level":"info","start reconcile":"metallbreload/reload","ts":"2025-03-17T11:20:13Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.1","fc00:f853:ccd:e799::1"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:13Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:13Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.0","fc00:f853:ccd:e799::"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:13Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:14Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.2","fc00:f853:ccd:e799::2"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:14Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller_reload.go:119","controller":"ServiceReconciler - reprocessAll","end reconcile":"metallbreload/reload","level":"info","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-1914/udn-service","ts":"2025-03-17T11:20:14Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.2","fc00:f853:ccd:e799::2"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:14Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"udn-services-1914/udn-service","level":"info","ts":"2025-03-17T11:20:14Z"}
{"caller":"pool_controller.go:48","controller":"PoolReconciler","level":"info","start reconcile":"/udn-services-7056","ts":"2025-03-17T11:20:14Z"}
{"caller":"pool_controller.go:99","controller":"PoolReconciler","event":"force service reload","level":"info","ts":"2025-03-17T11:20:14Z"}
{"caller":"pool_controller.go:112","controller":"PoolReconciler","event":"config reloaded","level":"info","ts":"2025-03-17T11:20:14Z"}
{"caller":"pool_controller.go:113","controller":"PoolReconciler","end reconcile":"/udn-services-7056","level":"info","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller_reload.go:63","controller":"ServiceReconciler - reprocessAll","level":"info","start reconcile":"metallbreload/reload","ts":"2025-03-17T11:20:14Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.0","fc00:f853:ccd:e799::"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:14Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:14Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.2","fc00:f853:ccd:e799::2"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:14Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:14Z"}
{"caller":"service.go:180","event":"ipAllocated","ip":["192.168.10.1","fc00:f853:ccd:e799::1"],"level":"info","msg":"IP address assigned by controller","ts":"2025-03-17T11:20:14Z"}
{"caller":"main.go:127","event":"serviceUpdated","level":"info","msg":"updated service object","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller_reload.go:119","controller":"ServiceReconciler - reprocessAll","end reconcile":"metallbreload/reload","level":"info","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller.go:64","controller":"ServiceReconciler","level":"info","start reconcile":"udn-services-7056-default/service-default","ts":"2025-03-17T11:20:14Z"}
{"caller":"service_controller.go:115","controller":"ServiceReconciler","end reconcile":"udn-services-7056-default/service-default","level":"info","ts":"2025-03-17T11:20:14Z"}

To Reproduce

Not available for now

Expected Behavior

Once assigned, the LB IP address assigned to a service should not change.

Additional Context

Not available for now

I've read and agree with the following

  • I've checked all open and closed issues and my request is not there.
  • I've checked all open and closed pull requests and my request is not there.

I've read and agree with the following

  • I've checked all open and closed issues and my issue is not there.
  • This bug is reproducible when deploying MetalLB from the main branch
  • I have read the troubleshooting guide and I am still not able to make it work
  • I checked the logs and MetalLB is not discarding the configuration as not valid
  • I enabled the debug logs, collected the information required from the cluster using the collect script and will attach them to the issue
  • I will provide the definition of my service and the related endpoint slices and attach them to this issue

Activity

  1. lllamnyp commented on May 2, 2025

    @lllamnyp

    We encountered the same problem recently. During an update of a k8s cluster, which was done somewhat carelessly -- all nodes, workers and controlplane were updated and rebooted at once, -- all load balancer services had their IPs reassigned.

  2. fedepaol commented on May 2, 2025

    @fedepaol
    Member

    This was fixed by #2726 (altough not yet released). We are planning to do a release next week.

    Closing the issue as fixed by #2726

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions