Skip to content

Track sample validation errors with synthetic metrics #577

@awh

Description

@awh

Samples can be rejected for various reasons:

  • Timestamp too old
  • Out of order timestamps
  • Repeated timestamps
  • Invalid labels or metric names
  • Too many series
  • ...

Our approach to these is rather ad-hoc at the moment - some are silently dropped, some return errors, and some cause (voluminous) logging. Instead we could record them as a set of synthetic metrics in cortex itself on a per-instance basis under an appropriate namespace (e.g. weaveworks_cortex_timestamp_too_old_count) so that they're a) visible to those who're sending them to us and b) available for the generation of alerts or notifications.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions