Motivation

Controller-runtime's has built in duration metrics but none of them track e2e object processing latency qualified by success or failure:

1. workqueue_queue_duration_seconds_bucket: events can be requeued and rate limited—duration spent waiting due to rate limits are not measured
2. workqueue_work_duration_seconds_bucket: is not qualified with processing success or failure

We want a metric that closely approximates what matters to end users or consuming systems—the amount of time it takes for a spec change to be successfully processed.

Implementation

Instruments a new histogram metric achilles_processing_duration_seconds tracking the time from when the controller receives an update to an object's spec to when that change is processed.

The processing time span for a given spec change is correlated using metadata.generation. A single reconciliation can process multiple generations, which is handled in the instrumentation.

Details

1. To avoid double counting failed processing, the metric will only observe a failed (name, namespace, generation) a single time. Once observed, the data entry is marked with a boolean indicating that it's failure has already been observed, allowing subsequent observations to skip this one.
2. To avoid infinitely accumulating memory, data points are deleted as soon as a successful reconciliation is observed.
3. A b-tree is used as the backing data structure, allow ordered range queries and efficient storage and retrieval.

Appendix

A more accurate metric would mark start time at admission time. But achieving this would require that consuming controllers deploy a webhook and further more, that the webhook share memory with the main controller binary.

I think this requirement is too onerous for the added accuracy. This metric doesn't measure latency incurred by the Kubernetes control plane itself (i.e. the time taken from when Kubernetes receives the object update to when it reflects that update back to WATCHers).

This latency can't be affected by the controller anyways + we have separate metrics for understanding k8s control plane performance.

✅ Checks

• CI tests (if present) are passing
• Adheres to code style for repo
• Contributor License Agreement (CLA) completed if not a Reddit employee