-
Notifications
You must be signed in to change notification settings - Fork 852
Track sample validation errors with synthetic metrics #577
Copy link
Copy link
Closed
Labels
help wantedtype/observabilityTo help know what is going on inside CortexTo help know what is going on inside Cortex
Description
Samples can be rejected for various reasons:
- Timestamp too old
- Out of order timestamps
- Repeated timestamps
- Invalid labels or metric names
- Too many series
- ...
Our approach to these is rather ad-hoc at the moment - some are silently dropped, some return errors, and some cause (voluminous) logging. Instead we could record them as a set of synthetic metrics in cortex itself on a per-instance basis under an appropriate namespace (e.g. weaveworks_cortex_timestamp_too_old_count) so that they're a) visible to those who're sending them to us and b) available for the generation of alerts or notifications.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
help wantedtype/observabilityTo help know what is going on inside CortexTo help know what is going on inside Cortex