Multicluster check failed with custom Prometheus #4943
Description
Bug Report
What is the issue?
I linked 2 clusters and exported a service in the target cluster, but the gateway is not accepting inbound requests.
How can it be reproduced?
$ linkerd install --identity-trust-anchors-file root.crt \
--identity-issuer-certificate-file issuer.crt \
--identity-issuer-key-file issuer.key \
--addon-config addon-config.yaml \
--context tools \
| kubectl apply -f - --context tools
$ linkerd install --identity-trust-anchors-file root.crt \
--identity-issuer-certificate-file issuer.crt \
--identity-issuer-key-file issuer.key \
--addon-config addon-config.yaml \
--context shop \
| kubectl apply -f - --context shop
$ linkerd mc install --context tools | kubectl apply -f - --context tools
$ linkerd mc install --context shop | kubectl apply -f - --context shop
$ linkerd mc link --context shop --cluster-name shop | kubectl apply -f - --context tools
$ kubectl label svc/monitoring-thanos-query-grpc -n monitoring --context shop mirror.linkerd.io/exported=trueLogs, error output, etc
shop gateway proxy logs:
[ 267.153402552s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"
[ 267.153443644s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[ 267.153472107s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}:http1{name=probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181 port=4181 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.28.9.129:4181 self.port=4143
[ 267.153676466s] DEBUG ThreadId(02) inbound:request{method=GET uri=http://probe-gateway-shop:4181//health version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "probe-gateway-shop:4181", "user-agent": "Go-http-client/1.1", "accept-encoding": "gzip", "l5d-dst-canonical": "probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181"}
[ 267.241061373s] DEBUG ThreadId(06) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.28.9.1:48268}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=119
[ 267.241248433s] DEBUG ThreadId(06) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.28.9.1:48268}: linkerd2_app_core::serve: Connection closed
[ 270.241107507s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"
[ 270.241149076s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[ 270.241171877s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}:http1{name=probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181 port=4181 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.28.9.129:4181 self.port=4143
[ 270.241490915s] DEBUG ThreadId(05) inbound:request{method=GET uri=http://probe-gateway-shop:4181//health version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "probe-gateway-shop:4181", "user-agent": "Go-http-client/1.1", "accept-encoding": "gzip", "l5d-dst-canonical": "probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181"}
[ 271.871296725s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.132.0.51:63404}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=64
[ 271.871342679s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.132.0.51:63404}: linkerd2_app_inbound::require_identity_for_ports: port=4143 peer.id=None(NoTlsFromRemote) id_required=true
[ 271.871429404s] INFO ThreadId(03) inbound:accept{peer.addr=10.132.0.51:63404}: linkerd2_app_core::serve: Connection closed error=identity required
[ 271.906598692s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.28.9.1:49856}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=126
[ 271.906638379s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.28.9.1:49856}: linkerd2_app_inbound::require_identity_for_ports: port=8888 peer.id=None(NoTlsFromRemote) id_required=false
[ 271.906656152s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.28.9.1:49856}: linkerd2_app_inbound::prevent_loop: port=8888 self.port=4143
[ 271.906964636s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.28.9.1:49856}:source{target.addr=10.28.9.129:8888}:http1{port=8888 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.28.9.129:8888 self.port=4143
[ 271.907171081s] DEBUG ThreadId(02) inbound:request{method=GET uri=http://10.28.9.129:8888/health-local version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "10.28.9.129:8888", "user-agent": "kube-probe/1.16+", "accept-encoding": "gzip"}
[ 271.907687389s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.28.9.1:49856}: linkerd2_app_core::serve: Connection closed
[ 272.124994902s] DEBUG ThreadId(06) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.28.9.1:48316}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=118
[ 272.125234862s] DEBUG ThreadId(06) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.28.9.1:48316}: linkerd2_app_core::serve: Connection closed
[ 272.244564723s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.28.9.1:49862}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=126
[ 272.244611119s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.28.9.1:49862}: linkerd2_app_inbound::require_identity_for_ports: port=8888 peer.id=None(NoTlsFromRemote) id_required=false
[ 272.244640354s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.28.9.1:49862}: linkerd2_app_inbound::prevent_loop: port=8888 self.port=4143
[ 272.244809339s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.28.9.1:49862}:source{target.addr=10.28.9.129:8888}:http1{port=8888 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.28.9.129:8888 self.port=4143
[ 272.245012942s] DEBUG ThreadId(03) inbound:request{method=GET uri=http://10.28.9.129:8888/health-local version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "10.28.9.129:8888", "user-agent": "kube-probe/1.16+", "accept-encoding": "gzip"}
[ 272.245524398s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.28.9.1:49862}: linkerd2_app_core::serve: Connection closed
[ 273.332871149s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"
[ 273.332929208s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[ 273.332952880s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}:http1{name=probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181 port=4181 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.28.9.129:4181 self.port=4143
[ 273.333285741s] DEBUG ThreadId(02) inbound:request{method=GET uri=http://probe-gateway-shop:4181//health version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "probe-gateway-shop:4181", "user-agent": "Go-http-client/1.1", "accept-encoding": "gzip", "l5d-dst-canonical": "probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181"}
[ 276.577624278s] DEBUG ThreadId(05) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_proxy_http::orig_proto: translating HTTP2 to orig-proto: "HTTP/1.1"
[ 276.577667440s] DEBUG ThreadId(05) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[ 276.577689909s] DEBUG ThreadId(05) inbound:accept{peer.addr=10.132.0.51:53418}:source{target.addr=10.28.9.129:4181}:http1{name=probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181 port=4181 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.28.9.129:4181 self.port=4143
[ 276.577999965s] DEBUG ThreadId(02) inbound:request{method=GET uri=http://probe-gateway-shop:4181//health version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "probe-gateway-shop:4181", "user-agent": "Go-http-client/1.1", "accept-encoding": "gzip", "l5d-dst-canonical": "probe-gateway-shop.linkerd-multicluster.svc.cluster.local:4181"}
[ 276.757512984s] DEBUG ThreadId(05) inbound:accept{peer.addr=10.132.0.52:7886}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=297
[ 276.758433756s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.52:7886}: linkerd2_app_inbound::require_identity_for_ports: port=4143 peer.id=Some("default.monitoring.serviceaccount.identity.linkerd.cluster.local") id_required=true
[ 276.758483603s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.132.0.52:7886}: linkerd2_app_inbound::prevent_loop: port=4143 self.port=4143
[ 276.758538056s] INFO ThreadId(02) inbound:accept{peer.addr=10.132.0.52:7886}: linkerd2_app_core::serve: Connection closed error=inbound requests must not target localhost:4143
the proxy from my pod trying to use the mirrored service:
[ 1594.898286938s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_reconnect::service: Recovering
[ 1595.600811s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_transport::tls::client: peer.identity=Some("linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local")
[ 1595.656683s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_transport::connect: Connecting peer.addr=GATEWAY_EXTERNAL_IP:4143
[ 1595.3109102s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:h2: linkerd2_proxy_transport::connect: Connected local.addr=10.52.7.238:40694 keepalive=Some(10s)
[ 1595.4219393s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:h2: linkerd2_proxy_transport::metrics: client connection open
[ 1595.4364732s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_http::override_authority: Stripped header header=l5d-dst-canonical value="monitoring-thanos-query-grpc-shop.monitoring.svc.cluster.local:10901"
[ 1595.4383775s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_http::override_authority: Overriding authority=monitoring-thanos-query-grpc.monitoring.svc.cluster.local:10901
[ 1595.4410566s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:request{method=POST uri=http://monitoring-thanos-query-grpc.monitoring.svc.cluster.local:10901/thanos.Store/Info version=HTTP/2.0}: linkerd2_proxy_http::client: client request headers={"content-type": "application/grpc", "user-agent": "grpc-go/1.29.1", "te": "trailers", "grpc-timeout": "4995136u"}
[ 1595.4923555s] WARN ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42186}:source{target.addr=10.119.12.165:10901}: linkerd2_app_core::errors: Failed to proxy request: connection error: broken pipe
[ 1595.4953254s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42186}:source{target.addr=10.119.12.165:10901}: linkerd2_app_core::errors: Handling error with gRPC status code=Unavailable
[ 1595.5082062s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:h2: linkerd2_proxy_http::h2: failed error=connection error: broken pipe
[ 1595.5832596s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42186}: linkerd2_app_core::serve: Connection closed
[ 1597.874777133s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.5.144:60480}:source{target.addr=10.52.7.238:10901}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[ 1597.874830074s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.5.144:60480}:source{target.addr=10.52.7.238:10901}:http2{port=10901}: linkerd2_app_inbound::prevent_loop: addr=10.52.7.238:10901 self.port=4143
[ 1597.875012393s] DEBUG ThreadId(02) inbound:request{method=POST uri=http://10.52.7.238:10901/thanos.Store/Info version=HTTP/2.0}: linkerd2_proxy_http::client: client request headers={"content-type": "application/grpc", "user-agent": "grpc-go/1.29.1", "te": "trailers", "grpc-timeout": "4999944u", "l5d-dst-canonical": "10.52.7.238:10901"}
[ 1598.430207653s] DEBUG ThreadId(04) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.52.7.1:45600}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=119
[ 1598.430341950s] DEBUG ThreadId(04) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.52.7.1:45600}: linkerd2_app_core::serve: Connection closed
[ 1599.897289480s] DEBUG ThreadId(03) outbound:accept{peer.addr=10.52.7.238:42258}: linkerd2_app_outbound::prevent_loop: addr=10.119.12.165:10901 self.port=4140
[ 1599.898884118s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42258}:source{target.addr=10.119.12.165:10901}: linkerd2_app_outbound::endpoint: using authority addr=monitoring-thanos-query-grpc-shop.monitoring:10901
[ 1599.898914259s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42258}:source{target.addr=10.119.12.165:10901}: linkerd2_app_outbound::endpoint: Setting target for request headers={"content-type": "application/grpc", "user-agent": "grpc-go/1.29.1", "te": "trailers", "grpc-timeout": "4993809u"} uri=http://monitoring-thanos-query-grpc-shop.monitoring:10901/thanos.Store/Info target.addr=monitoring-thanos-query-grpc-shop.monitoring:10901 http.settings=Http2
[ 1599.899326293s] WARN ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_reconnect::service: Service failed error=channel closed
[ 1599.899354702s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_reconnect::service: Recovering
[ 1599.984177407s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36428}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=124
[ 1599.984214464s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36428}: linkerd2_app_inbound::require_identity_for_ports: port=10902 peer.id=None(NoTlsFromRemote) id_required=false
[ 1599.984243190s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36428}: linkerd2_app_inbound::prevent_loop: port=10902 self.port=4143
[ 1599.984383269s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36428}:source{target.addr=10.52.7.238:10902}:http1{port=10902 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.52.7.238:10902 self.port=4143
[ 1599.984625538s] DEBUG ThreadId(02) inbound:request{method=GET uri=http://10.52.7.238:10902/-/healthy version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "10.52.7.238:10902", "user-agent": "kube-probe/1.16+", "accept-encoding": "gzip"}
[ 1599.986369492s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.52.7.1:36428}: linkerd2_app_core::serve: Connection closed
[ 1600.9595572s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_transport::tls::client: peer.identity=Some("linkerd-gateway.linkerd-multicluster.serviceaccount.identity.linkerd.cluster.local")
[ 1600.9644927s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_transport::connect: Connecting peer.addr=GATEWAY_EXTERNAL_IP:4143
[ 1600.11987854s] DEBUG ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:h2: linkerd2_proxy_transport::connect: Connected local.addr=10.52.7.238:40766 keepalive=Some(10s)
[ 1600.14199493s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:h2: linkerd2_proxy_transport::metrics: client connection open
[ 1600.14471732s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_http::override_authority: Stripped header header=l5d-dst-canonical value="monitoring-thanos-query-grpc-shop.monitoring.svc.cluster.local:10901"
[ 1600.14627224s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_proxy_http::override_authority: Overriding authority=monitoring-thanos-query-grpc.monitoring.svc.cluster.local:10901
[ 1600.14708077s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:request{method=POST uri=http://monitoring-thanos-query-grpc.monitoring.svc.cluster.local:10901/thanos.Store/Info version=HTTP/2.0}: linkerd2_proxy_http::client: client request headers={"content-type": "application/grpc", "user-agent": "grpc-go/1.29.1", "te": "trailers", "grpc-timeout": "4993809u"}
[ 1600.15003289s] DEBUG ThreadId(02) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}:h2: linkerd2_proxy_http::h2: failed error=connection error: broken pipe
[ 1600.15079349s] WARN ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42258}:source{target.addr=10.119.12.165:10901}: linkerd2_app_core::errors: Failed to proxy request: operation was canceled: connection closed
[ 1600.15113408s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42258}:source{target.addr=10.119.12.165:10901}: linkerd2_app_core::errors: Handling error with gRPC status code=Internal
[ 1600.16629872s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42258}: linkerd2_app_core::serve: Connection closed
[ 1602.24315279s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36468}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=122
[ 1602.24354229s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36468}: linkerd2_app_inbound::require_identity_for_ports: port=10902 peer.id=None(NoTlsFromRemote) id_required=false
[ 1602.24371042s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36468}: linkerd2_app_inbound::prevent_loop: port=10902 self.port=4143
[ 1602.24720001s] DEBUG ThreadId(02) inbound:accept{peer.addr=10.52.7.1:36468}:source{target.addr=10.52.7.238:10902}:http1{port=10902 keep_alive=true wants_h1_upgrade=false was_absolute_form=false}: linkerd2_app_inbound::prevent_loop: addr=10.52.7.238:10902 self.port=4143
[ 1602.25228273s] DEBUG ThreadId(02) inbound:request{method=GET uri=http://10.52.7.238:10902/-/ready version=HTTP/1.1}: linkerd2_proxy_http::client: client request headers={"host": "10.52.7.238:10902", "user-agent": "kube-probe/1.16+", "accept-encoding": "gzip"}
[ 1602.26021738s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.52.7.1:36468}: linkerd2_app_core::serve: Connection closed
[ 1602.875090644s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.52.5.144:60480}:source{target.addr=10.52.7.238:10901}: linkerd2_app_inbound::endpoint: using l5d-dst-canonical
[ 1602.875139015s] DEBUG ThreadId(03) inbound:accept{peer.addr=10.52.5.144:60480}:source{target.addr=10.52.7.238:10901}:http2{port=10901}: linkerd2_app_inbound::prevent_loop: addr=10.52.7.238:10901 self.port=4143
[ 1602.875935542s] DEBUG ThreadId(03) inbound:request{method=POST uri=http://10.52.7.238:10901/thanos.Store/Info version=HTTP/2.0}: linkerd2_proxy_http::client: client request headers={"content-type": "application/grpc", "user-agent": "grpc-go/1.29.1", "te": "trailers", "grpc-timeout": "4999925u", "l5d-dst-canonical": "10.52.7.238:10901"}
[ 1604.865075023s] DEBUG ThreadId(04) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.52.7.1:45708}: linkerd2_proxy_transport::tls::accept: Peeked bytes from TCP stream sz=118
[ 1604.866039559s] DEBUG ThreadId(04) daemon:admin{listen.addr=0.0.0.0:4191}:accept{peer.addr=10.52.7.1:45708}: linkerd2_app_core::serve: Connection closed
[ 1604.894915118s] DEBUG ThreadId(02) outbound:accept{peer.addr=10.52.7.238:42354}: linkerd2_app_outbound::prevent_loop: addr=10.119.12.165:10901 self.port=4140
[ 1604.897480168s] DEBUG ThreadId(03) outbound:accept{peer.addr=10.52.7.238:42354}:source{target.addr=10.119.12.165:10901}: linkerd2_app_outbound::endpoint: using authority addr=monitoring-thanos-query-grpc-shop.monitoring:10901
[ 1604.897581872s] DEBUG ThreadId(03) outbound:accept{peer.addr=10.52.7.238:42354}:source{target.addr=10.119.12.165:10901}: linkerd2_app_outbound::endpoint: Setting target for request headers={"content-type": "application/grpc", "user-agent": "grpc-go/1.29.1", "te": "trailers", "grpc-timeout": "4997190u"} uri=http://monitoring-thanos-query-grpc-shop.monitoring:10901/thanos.Store/Info target.addr=monitoring-thanos-query-grpc-shop.monitoring:10901 http.settings=Http2
[ 1604.897944048s] WARN ThreadId(03) endpoint{peer.addr=GATEWAY_EXTERNAL_IP:4143}: linkerd2_reconnect::service: Service failed error=channel closed
linkerd check output
tools cluster:
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist
linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor
linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match
linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
* shop
√ remote cluster access credentials are valid
* shop
√ clusters share trust anchors
* shop
√ service mirror controller has required permissions
* shop
√ service mirror controllers are running
* shop
× all gateway mirrors are healthy
wrong number of (0) gateway metrics entries for probe-gateway-shop.linkerd-multicluster
see https://linkerd.io/checks/#l5d-multicluster-gateways-endpoints for hints
√ all mirror services have endpoints
√ all mirror services are part of a Link
Status check results are ×
I think the failure here is because I use a custom prometheus (it's disabled in linkerd)
shop cluster:
kubernetes-api
--------------
√ can initialize the client
√ can query the Kubernetes API
kubernetes-version
------------------
√ is running the minimum Kubernetes API version
√ is running the minimum kubectl version
linkerd-existence
-----------------
√ 'linkerd-config' config map exists
√ heartbeat ServiceAccount exist
√ control plane replica sets are ready
√ no unschedulable pods
√ controller pod is running
√ can initialize the client
√ can query the control plane API
linkerd-config
--------------
√ control plane Namespace exists
√ control plane ClusterRoles exist
√ control plane ClusterRoleBindings exist
√ control plane ServiceAccounts exist
√ control plane CustomResourceDefinitions exist
√ control plane MutatingWebhookConfigurations exist
√ control plane ValidatingWebhookConfigurations exist
√ control plane PodSecurityPolicies exist
linkerd-identity
----------------
√ certificate config is valid
√ trust anchors are using supported crypto algorithm
√ trust anchors are within their validity period
√ trust anchors are valid for at least 60 days
√ issuer cert is using supported crypto algorithm
√ issuer cert is within its validity period
√ issuer cert is valid for at least 60 days
√ issuer cert is issued by the trust anchor
linkerd-api
-----------
√ control plane pods are ready
√ control plane self-check
√ [kubernetes] control plane can talk to Kubernetes
√ [prometheus] control plane can talk to Prometheus
√ tap api service is running
linkerd-version
---------------
√ can determine the latest version
√ cli is up-to-date
control-plane-version
---------------------
√ control plane is up-to-date
√ control plane and cli versions match
linkerd-addons
--------------
√ 'linkerd-config-addons' config map exists
linkerd-multicluster
--------------------
√ Link CRD exists
Status check results are √
Environment
- Kubernetes Version: v1.16.13-gke.1
- Cluster Environment: GKE
- Host OS: COS
- Linkerd version: edge-20.9.1