Noisy log for Multi-Cluster communication #10299
Description
Two options: reduce the log level for that line to DEBUG or improve the docs with the suggestion given.
Discussed in #8148
Originally posted by mbaykara March 25, 2022
Following steps I have performed:
- I have created 2 AKS clusters
eastandwest. - Create certs via
//cert
step certificate create root.linkerd.cluster.local root.crt root.key --profile root-ca --no-password --insecure
//issuer
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key --profile intermediate-ca --not-after 8760h --no-password --insecure --ca root.crt --ca-key root.key
- Installed Linkerd for each cluster
linkerd install \
--identity-trust-anchors-file root.crt \
--identity-issuer-certificate-file issuer.crt \
--identity-issuer-key-file issuer.key \
| kubectl apply -f -
- Verified the installation, then installed multi-cluster part for each cluster
linkerd multicluster install | kubectl apply -f -
- Then run
k -n linkerd-multicluster logs --context west linkerd-gateway-57988667cd-tkt2w -c linkerd-proxythe output
0.000442s] ERROR ThreadId(01) linkerd_app::env: No inbound ports specified via LINKERD2_PROXY_INBOUND_PORTS
[ 0.000611s] INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[ 0.001147s] INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[ 0.001157s] INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[ 0.001159s] INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[ 0.001161s] INFO ThreadId(01) linkerd2_proxy: Tap DISABLED
[ 0.001163s] INFO ThreadId(01) linkerd2_proxy: Local identity is linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 0.001167s] INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.001169s] INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[ 0.047778s] INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local
[ 4.938778s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[ 9.947119s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[ 10.949054s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[ 15.950121s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[ 16.952876s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[ 21.959351s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[ 22.961091s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[ 27.964759s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[ 28.967106s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
[ 33.982771s] INFO ThreadId(01) inbound: linkerd_app_core::serve: Connection closed error=direct connections must be mutually authenticated
[ 34.983523s] INFO ThreadId(02) daemon:admin{listen.addr=0.0.0.0:4191}: linkerd_app_core::serve: Connection closed error=TLS detection timed out
Where do I do wrong? Why are these errors?
environment:
aks cluster
version: 1.22.6
Linkerd version
Client version: stable-2.11.1
Server version: stable-2.11.1
- Link the
westcluster toeast
linkerd --context=east multicluster link --set enableHeadlessServices=true --cluster-name east | kubectl --context=west apply -f -
- Create namespace for cluster
eastandwest
apiVersion: v1
kind: Namespace
metadata:
annotations:
linkerd.io/inject: enabled
name: monitoring
- Lastly I installed Prometheus both cluster and Thanos in cluster west
- Expose it headless svc from east to west
kubectl --context=east label svc prometheus-kube-prometheus-thanos-discovery mirror.linkerd.io/exported=true
Now I have same service in west cluster with east suffix but the new created mirror svc is not headless, it has an ClusterIP address.
And the Thanos query is not getting the metrics from east cluster.
Both clusters monitoring namespace has linkerd injected. So I do not understand why is not working