Proposal

We are monitoring hundreds of Kubernetes clusters from some central observability clusters. To decide whether a cluster is operating fine, relying on current service discovery information seems a reasonable idea -- if we cannot update service discovery information, we definitely have issues with this cluster.

Currently, there are no reasonably usable metrics for this. As we're monitoring across the network, some HTTP errors during service discovery are kind of expected, and just indicate issues after all -- for Prometheus service discovery, we don't mind some errors, but care about whether service discovery works often enough to have a current state. So monitoring based on prometheus_sd_kubernetes_http_request_total doesn't work out too well.

There is also prometheus_sd_updates_total, but on one hand this is aggregated over all service discovery targets (and a rate still going up even if API server is down), on the other hand it's also just a kind of indirect observation.

I could well imagine all kinds of service discovery should be able to provide a timestamp of the last service discovery update, so a generic alert on failing service discovery for 15 minutes is as easy as time() - prometheus_sd_last_update > 300. This metric should be per service discovery configuration block. From a very first peak at the code, this might easily be implemented centrally in discovery/manager.go in the update handling code.

^{Jens Erat [email protected], Mercedes-Benz Tech Innovation GmbH, imprint}