linkerd fails to find Lease component for service-mirror #11509
Description
Discussed in #11503
Originally posted by lboutin-mwm October 18, 2023
Hello,
I use the linkerd multicluster plugin to ease communications between some of my clusters and deploy it using its helm chart.
I recently upgraded the stack to stable-2.14.1 (from stable-2.14.0) and one of my clusters is unable to find the lease created when linking two clusters:
This is the result of linkerd multicluster check on my central cluster:
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
* ml-demos-us
* ml-staging-us
√ remote cluster access credentials are valid
* ml-demos-us
* ml-staging-us
√ clusters share trust anchors
* ml-demos-us
* ml-staging-us
√ service mirror controller has required permissions
* ml-demos-us
* ml-staging-us
√ service mirror controllers are running
* ml-demos-us
* ml-staging-us
√ probe services able to communicate with all gateway mirrors
* ml-demos-us
* ml-staging-us
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match
Status check results are √
and this is the result of the same command on my faulty cluster:
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
* tools-eu
√ remote cluster access credentials are valid
* tools-eu
√ clusters share trust anchors
* tools-eu
√ service mirror controller has required permissions
* tools-eu
√ service mirror controllers are running
* tools-eu
× probe services able to communicate with all gateway mirrors
failed to get the service-mirror component Lease for target cluster tools-eu: leases.coordination.k8s.io "service-mirror-write-tools-eu" not found
see https://linkerd.io/2.14/checks/#l5d-multicluster-gateways-endpoints for hints
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
√ multicluster extension proxies are up-to-date
√ multicluster extension proxies and cli versions match
Status check results are ×
For some reason it seems to work one way and not the other.
The lease does in fact exist:
kctl get leases -n linkerd-multicluster
NAME HOLDER AGE
service-mirror-write-tools-eu linkerd-service-mirror-tools-eu-7854857ffc-j6c8k 44m
and I can get its description just fine:
Name: service-mirror-write-tools-eu
Namespace: linkerd-multicluster
Labels: <none>
Annotations: <none>
API Version: coordination.k8s.io/v1
Kind: Lease
Metadata:
Creation Timestamp: 2023-10-18T15:28:29Z
Managed Fields:
API Version: coordination.k8s.io/v1
Fields Type: FieldsV1
fieldsV1:
f:spec:
f:acquireTime:
f:holderIdentity:
f:leaseDurationSeconds:
f:leaseTransitions:
f:renewTime:
Manager: controller
Operation: Update
Time: 2023-10-18T16:14:20Z
Resource Version: 178043801
UID: 99550f50-0d88-4076-a7fb-1abcc07c7386
Spec:
Acquire Time: 2023-10-18T15:28:29.396070Z
Holder Identity: linkerd-service-mirror-tools-eu-7854857ffc-j6c8k
Lease Duration Seconds: 30
Lease Transitions: 0
Renew Time: 2023-10-18T16:14:20.345628Z
Events: <none>
All of the aforementioned clusters are running in GKE with version 1.25.12-gke.500 of k8s
These are the commands I used to link both clusters:
linkerd --context=tools-eu multicluster link --cluster-name tools-eu | kubectl --context=ml-demos-us apply -f -
linkerd --context=ml-demos-us multicluster link --cluster-name ml-demos-us | kubectl --context=tools-eu apply -f -
Only one of three clusters connected to the central one is affected and there was no issue with the earlier 2.14.0 version of linkerd.
It also does seem like the mirroring and connection between clusters is working, this might just only be an error with the check command