Repository navigation
Description
MetalLB Version
0.15.3
Deployment method
Charts
Main CNI
flannel
Kubernetes Version
1.29.15
Cluster Distribution
vanilla, kubeadm
Describe the bug
The speak logs contains following error.
2026-01-23T11:54:45.217332072Z {"level":"error","ts":"2026-01-23T11:54:45Z","msg":"Reconciler error","controller":"servicel2status","namespace":"ceph","name":"rook-ceph-zabbix-exporter","reconcileID":"17715dea-de1d-48f5-b18f-e4e993484ae5","error":"ServiceL2Status.metallb.io \"l2-4mf7b\" is invalid: status.node: Invalid value: \"string\": Value is immutable","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:474\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:296"}
The above log is from pod metallb-speaker-w6rp5.
List of ServiceL2Statuses:
kubectl get ServiceL2Status.metallb.io -A
NAMESPACE NAME ALLOCATED NODE SERVICE NAME SERVICE NAMESPACE
metallb l2-4mf7b ch03-infra-worker01 rook-ceph-zabbix-exporter ceph
metallb l2-tpx7z ch03-infra-worker02 test-app pjancik
Describe of the problematic status:
kubectl describe ServiceL2Status.metallb.io -n metallb l2-4mf7b
Name: l2-4mf7b
Namespace: metallb
Labels: metallb.io/node=ch03-infra-worker02
metallb.io/service-name=rook-ceph-zabbix-exporter
metallb.io/service-namespace=ceph
Annotations: <none>
API Version: metallb.io/v1beta1
Kind: ServiceL2Status
Metadata:
Creation Timestamp: 2026-01-23T11:54:44Z
Generate Name: l2-
Generation: 1
Owner References:
API Version: v1
Kind: Pod
Name: metallb-speaker-45qmz
UID: 73ae1fa5-5844-4939-93cc-9b5d4c02a188
Resource Version: 735357838
UID: 7a21548d-3a49-4d20-810b-d9c4fb96384a
Spec:
Status:
Node: ch03-infra-worker01
Service Name: rook-ceph-zabbix-exporter
Service Namespace: ceph
Events: <none>
Notice serviceL2Status has annotation metallb.io/node for -worker02 not but .status.node is for different node -worker01.
The pod metallb-speaker-45qmz (which owning the ServiceL2Status) is on worker-02
I have no clue how this happened.
kubectl get pods --output wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
metallb-controller-84bc95f5b8-kb7sg 1/1 Running 0 75m 10.247.4.128 ch03-infra-worker04 <none> <none>
metallb-speaker-45qmz 1/1 Running 1 (29m ago) 29m 10.4.21.34 ch03-infra-worker02 <none> <none>
metallb-speaker-lgsnl 1/1 Running 1 (3m11s ago) 3m12s 10.4.21.33 ch03-infra-worker01 <none> <none>
I discovered this issue after upgrade from 0.15.2 to 0.15.3, but the issue is not 0.15.3 specific, the further logs analysis found the same type of error log even in 0.15.2 (I am sadly do not have pre 0.15.2 logs any more)
The load balancer worked (all services), so no disruption.
I tried to enable debug logs, but it rolled out the speaker --> new ServiceL2Status.metallb.io created (correctly -> --> node in annotation match the one in status), and error log disappeared (for now)
To Reproduce
Not sure, but I expect this is what happened.
- Install metalLB with at least 2 speakers/
- Create k8s service of LB type.
- Cause somehow that the service is load balanced by other speaker (guessing earlier may be cordoning was enough). In mine case the metallb upgrade was the trigger.
I can experiment a bit, but at the moment I do not have exact steps to recreate problematic configuration. Any hint how move service to be balance by other nodes may help me.
Expected Behavior
Well, no errors are logged.
Additional Context
I found very similar issue #2626 which is closed, but it seems it does does not solve it correctly / completely. I do not want to mess up so I opening new one.
Some related notices.
-
CRD spec for ServiceL2Status says the status.node is immutable (essentially all 3 entries of the status are immutable)
metallb/charts/metallb/charts/crds/templates/crds.yaml
Lines 1450 to 1455 in 4c931df
description: Node indicates the node that receives the directed traffic type: string x-kubernetes-validations: - message: Value is immutable rule: self == oldSelf serviceName: -
However in the code, it seems you are trying to use
createOrPatch, i.e., patch it, when it is needed to update the status.
metallb/internal/k8s/controllers/layer2_status_controller.go
Lines 130 to 142 in 85bfb4f
result, err = controllerutil.CreateOrPatch(ctx, r.Client, state, func() error { state.Labels = map[string]string{ LabelAnnounceNode: r.NodeName, LabelServiceName: serviceName, LabelServiceNamespace: serviceNamespace, } state.Status = desiredStatus err = controllerutil.SetOwnerReference(r.SpeakerPod, state, r.Scheme()) if err != nil { return err } return nil })
Notices:
- not tested if from deploy on main
- Not collected the debug logs, I can try later if I recreate the problematic state.
I've read and agree with the following
- I've checked all open and closed issues and my request is not there.
- I've checked all open and closed pull requests and my request is not there.
I've read and agree with the following
- I've checked all open and closed issues and my issue is not there.
- This bug is reproducible when deploying MetalLB from the main branch
- I have read the troubleshooting guide and I am still not able to make it work
- I checked the logs and MetalLB is not discarding the configuration as not valid
- I enabled the debug logs, collected the information required from the cluster using the collect script and will attach them to the issue
- I will provide the definition of my service and the related endpoint slices and attach them to this issue
Activity