Skip to content

Stop using the proxy's /metrics endpoint as a livenessProbe #3984

@siggy

Description

@siggy

Background

Linkerd proxies are configured with a livenessProbe and a readinessProbe. For the livenessProbe we are using the proxy's /metrics endpoint. This endpoint is unnecessarily expensive, as we just require an endpoint to return a 200, similar to the readinessProbe's /ready endpoint.

Proposal

Modify the proxy's livenessProbe to point to a different endpoint, either a new /liveness endpoint, or re-use the /ready endpoint.

Additional context

liveness/readiness k8s config for proxy

livenessProbe:
  httpGet:
    path: /metrics
    port: 4191
  initialDelaySeconds: 10
readinessProbe:
  httpGet:
    path: /ready
    port: 4191
  initialDelaySeconds: 2

/metrics payload size

A typical control-plane component's proxy emits 60kb of data on the /metrics endpoint:

$ linkerd metrics -n linkerd $(
    kubectl --namespace linkerd get pod \
      --selector linkerd.io/control-plane-component=controller \
      --output name \
    | head -n1
  ) | wc -c
61828

Kubernetes probing code

Kubernetes reads up to 10kb when probing:
https://github.com/kubernetes/kubernetes/blob/866ea74326fba45241cbccd3ca4624b68ba7022f/pkg/probe/http/http.go#L36

...and then logs a non-fatal error:
https://github.com/kubernetes/kubernetes/blob/866ea74326fba45241cbccd3ca4624b68ba7022f/pkg/probe/http/http.go#L114-L118

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions