Skip to content

Speaker - error log - ServiceL2Status.metallb.io XXX is invalid: status.node: Invalid value: \"string\": Value is immutable #2930

Closed

Description

@pavel-jancik

MetalLB Version

0.15.3

Deployment method

Charts

Main CNI

flannel

Kubernetes Version

1.29.15

Cluster Distribution

vanilla, kubeadm

Describe the bug

The speak logs contains following error.

2026-01-23T11:54:45.217332072Z {"level":"error","ts":"2026-01-23T11:54:45Z","msg":"Reconciler error","controller":"servicel2status","namespace":"ceph","name":"rook-ceph-zabbix-exporter","reconcileID":"17715dea-de1d-48f5-b18f-e4e993484ae5","error":"ServiceL2Status.metallb.io \"l2-4mf7b\" is invalid: status.node: Invalid value: \"string\": Value is immutable","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:296"}

The above log is from pod metallb-speaker-w6rp5.

List of ServiceL2Statuses:

kubectl get ServiceL2Status.metallb.io -A
NAMESPACE   NAME       ALLOCATED NODE        SERVICE NAME                SERVICE NAMESPACE
metallb     l2-4mf7b   ch03-infra-worker01   rook-ceph-zabbix-exporter   ceph
metallb     l2-tpx7z   ch03-infra-worker02   test-app                    pjancik

Describe of the problematic status:

kubectl describe  ServiceL2Status.metallb.io -n metallb l2-4mf7b
Name:         l2-4mf7b
Namespace:    metallb
Labels:       metallb.io/node=ch03-infra-worker02
              metallb.io/service-name=rook-ceph-zabbix-exporter
              metallb.io/service-namespace=ceph
Annotations:  <none>
API Version:  metallb.io/v1beta1
Kind:         ServiceL2Status
Metadata:
  Creation Timestamp:  2026-01-23T11:54:44Z
  Generate Name:       l2-
  Generation:          1
  Owner References:
    API Version:     v1
    Kind:            Pod
    Name:            metallb-speaker-45qmz
    UID:             73ae1fa5-5844-4939-93cc-9b5d4c02a188
  Resource Version:  735357838
  UID:               7a21548d-3a49-4d20-810b-d9c4fb96384a
Spec:
Status:
  Node:               ch03-infra-worker01
  Service Name:       rook-ceph-zabbix-exporter
  Service Namespace:  ceph
Events:               <none>

Notice serviceL2Status has annotation metallb.io/node for -worker02 not but .status.node is for different node -worker01.
The pod metallb-speaker-45qmz (which owning the ServiceL2Status) is on worker-02
I have no clue how this happened.

kubectl get pods --output wide
NAME                                  READY   STATUS    RESTARTS        AGE     IP             NODE                  NOMINATED NODE   READINESS GATES
metallb-controller-84bc95f5b8-kb7sg   1/1     Running   0               75m     10.247.4.128   ch03-infra-worker04   <none>           <none>
metallb-speaker-45qmz                 1/1     Running   1 (29m ago)     29m     10.4.21.34     ch03-infra-worker02   <none>           <none>
metallb-speaker-lgsnl                 1/1     Running   1 (3m11s ago)   3m12s   10.4.21.33     ch03-infra-worker01   <none>           <none>

I discovered this issue after upgrade from 0.15.2 to 0.15.3, but the issue is not 0.15.3 specific, the further logs analysis found the same type of error log even in 0.15.2 (I am sadly do not have pre 0.15.2 logs any more)

The load balancer worked (all services), so no disruption.

I tried to enable debug logs, but it rolled out the speaker --> new ServiceL2Status.metallb.io created (correctly -> --> node in annotation match the one in status), and error log disappeared (for now)

To Reproduce

Not sure, but I expect this is what happened.

  1. Install metalLB with at least 2 speakers/
  2. Create k8s service of LB type.
  3. Cause somehow that the service is load balanced by other speaker (guessing earlier may be cordoning was enough). In mine case the metallb upgrade was the trigger.

I can experiment a bit, but at the moment I do not have exact steps to recreate problematic configuration. Any hint how move service to be balance by other nodes may help me.

Expected Behavior

Well, no errors are logged.

Additional Context

I found very similar issue #2626 which is closed, but it seems it does does not solve it correctly / completely. I do not want to mess up so I opening new one.

Some related notices.

  1. CRD spec for ServiceL2Status says the status.node is immutable (essentially all 3 entries of the status are immutable)

    description: Node indicates the node that receives the directed traffic
    type: string
    x-kubernetes-validations:
    - message: Value is immutable
    rule: self == oldSelf
    serviceName:

  2. However in the code, it seems you are trying to use createOrPatch, i.e., patch it, when it is needed to update the status.

    result, err = controllerutil.CreateOrPatch(ctx, r.Client, state, func() error {
    state.Labels = map[string]string{
    LabelAnnounceNode: r.NodeName,
    LabelServiceName: serviceName,
    LabelServiceNamespace: serviceNamespace,
    }
    state.Status = desiredStatus
    err = controllerutil.SetOwnerReference(r.SpeakerPod, state, r.Scheme())
    if err != nil {
    return err
    }
    return nil
    })

Notices:

  • not tested if from deploy on main
  • Not collected the debug logs, I can try later if I recreate the problematic state.

I've read and agree with the following

  • I've checked all open and closed issues and my request is not there.
  • I've checked all open and closed pull requests and my request is not there.

I've read and agree with the following

  • I've checked all open and closed issues and my issue is not there.
  • This bug is reproducible when deploying MetalLB from the main branch
  • I have read the troubleshooting guide and I am still not able to make it work
  • I checked the logs and MetalLB is not discarding the configuration as not valid
  • I enabled the debug logs, collected the information required from the cluster using the collect script and will attach them to the issue
  • I will provide the definition of my service and the related endpoint slices and attach them to this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions