What is the issue?

My services are running, but I don't see any metrics for them.

How can it be reproduced?

Leave linkerd running for awhile. Any changes to the cluster (new deployment for example) won't get metrics.

Logs, error output, etc

Everything's healthy.

linkerd check output

Check passes 100%.

Environment

• Kubernetes Version: 1.11.5
• Cluster Environment: AKS
• Linkerd version: stable-2.1.0

Possible solution

It appears that watches become stale on AKS. You can fix this temporarily by restarting the linkerd control pods (effectively refreshing the state).

kubectl -n linkerd delete pod --all

From a code perspective, there might be an update that makes linkerd more resilient to these types of problems, kubernetes/kubernetes#67817. Unfortunately, that won't fix the prometheus side of things.

Additional context

There's a bunch of possible issues:

• Connection timeouts in case of pod-to-pod communication when both pods are on the same host node Azure/AKS#790
• streamwatcher.go:109] Unable to decode an event from the watch stream: stream error: stream ID 1; INTERNAL_ERROR Azure/AKS#676