Skip to content

Stats histogram support#824

Merged
sodomelle merged 22 commits intoaxoflow:mainfrom
bazsi:stats-histogram-support
Nov 10, 2025
Merged

Stats histogram support#824
sodomelle merged 22 commits intoaxoflow:mainfrom
bazsi:stats-histogram-support

Conversation

@bazsi
Copy link
Member

@bazsi bazsi commented Oct 13, 2025

This adds prometheus style histogram support for 3 metrics in LogThreadedDest based drivers (http, otel, snmp, sql and a few others):

batch_size, message_size, request_latency

It also forces registration of all aggregated stats that LogThreadedDest supports, these were previously "excplitily" requested by the driver implementation, so they were only added to http and grpc. Now all such drivers get these.

  • afamqp
  • afmongodb
  • afsmtp
  • afsnmp
  • afsql
  • afstomp
  • grpc
  • http
  • java
  • kafka
  • mqtt
  • python
  • redis
  • riemann

It builds on top of #823 which needs to go in first.

@bazsi bazsi force-pushed the stats-histogram-support branch 2 times, most recently from c47b697 to 7a1a0d6 Compare October 18, 2025 21:13
@bazsi bazsi force-pushed the stats-histogram-support branch 2 times, most recently from 50c65a9 to 4f9a71f Compare October 27, 2025 05:24
@bazsi bazsi changed the title WIP: Stats histogram support Stats histogram support Oct 27, 2025
@bazsi bazsi force-pushed the stats-histogram-support branch from 4f9a71f to 44c44f6 Compare October 27, 2025 05:29
@alltilla
Copy link
Member

FYI

There is a so-called "native histogram" feature in Prometheus, which uses one complex metric, instead of multiple simple ones: https://prometheus.io/docs/specs/native_histograms/

However, this is currently only supported in the Prometheus Protobuf representation, not with the text one, which we use.

Native histogram is planned to replace the use of the classic histograms eventually, but until then we are good with the classic one.

bazsi added 22 commits November 9, 2025 15:19
To allow more counters in a single cluster, we need more bits, as
used counters are tracked using these bits.

Signed-off-by: Balazs Scheidler <[email protected]>
This is not always const, StatsClusterSingle for example allocates this
value and then expects it to pass to g_free().

Signed-off-by: Balazs Scheidler <[email protected]>
stats_cluster_key_add_unit() and stats_cluster_key_add_frame_of_reference()
were part of the StatsClusterSingle API, but in reality these are
generic functions, so move them to the right place.

Signed-off-by: Balazs Scheidler <[email protected]>
This is simpler, especially if we have multiple counters in the same
cluster, which the histogram support is going to use.  Although that has a
specific free function, I find it easier to follow code that uses similar
solutions for the same problem.

Signed-off-by: Balazs Scheidler <[email protected]>
…o __name__

The metric name was so far constant, independent of the "type" within a
specific StatsCluster. Change this so that we have a callback function that
can return an alternative, type-specific suffix.

This is to be used by the histogram support.

Signed-off-by: Balazs Scheidler <[email protected]>
…() from StatsClusterKey

Signed-off-by: Balazs Scheidler <[email protected]>
…nterGroup

This has the potential to override the key specific unit/for values to
be type specific. Used by the histogram support to get a different
formatting for the "sum" field over "buckets".

Signed-off-by: Balazs Scheidler <[email protected]>
This patch adds the histogram specific counter layout (e.g. StatsClusterHist).

Signed-off-by: Balazs Scheidler <[email protected]>
Previously this was an optional step by the threaded driver implementation,
now they are always registered.

Signed-off-by: Balazs Scheidler <[email protected]>
This is too long and is also too specific. "kb" is just neater (and can
be the same everywhere).

Signed-off-by: Balazs Scheidler <[email protected]>
Instead of constructing another StatsClusterKey, just use what was already
created.

Signed-off-by: Balazs Scheidler <[email protected]>
Signed-off-by: Balazs Scheidler <[email protected]>
@bazsi bazsi force-pushed the stats-histogram-support branch from d3f532c to cb5ba0d Compare November 9, 2025 14:19
@sodomelle sodomelle merged commit b256fce into axoflow:main Nov 10, 2025
22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants