Feature Request

The prometheus instance that comes with Linkerd is nice but for large clusters this can become a bottleneck. By allowing the prometheus instance to be configurable, we can give users the ability to re-use their existing monitoring infra with Linkerd and also use longer metric retention.

There have also been projects like cortex, thanos which have been getting some traction recently to make prometheus scalabale. This would also allow users to use managed monitoring offerings like Grafana Cloud, etc

The following components use prometheus for queries right now:

• public-api : To serve metrics for CLI and dashboard.
• heart-beat
  Both the components take the --prometheus-url flag during startup which points to http://linkerd-prometheus.{{.Namespace}}.svc.{{.ClusterDomain}}:9090 right now.
• Grafana: This can be configured, by tweaking the data-sources field of linkerd-grafana-configmap

Users will then have to take the prometheus config and apply it to their monitoring infra i.e prometheus, cortex, etc.

For prom instances running outside their k8s cluster i.e managed, etc, A prometheus instace (with the linkerd-prometheus config) with remote-write configured to the outside instance. Assuming that the outside instace is scalable, by having the linkerd components still use the outside instance can help in faster metric retrieval and longer retention.

Solution

Having a global config to set the prometheus-url through helm and linkerd install command should be enough. In that case, the default prometheus installation should be disabled.

How would users interact with this feature?

linkerd install --prometheus-url

Out of Scope

• How should the outside prometheus instance be installed and configured?

related #3406

@grampelberg @ihcsim @siggy