Skip to content

persist: export bounded aggregates of per-shard metrics - #39357

Merged
mtabebe merged 1 commit into
MaterializeInc:mainfrom
mtabebe:michaelabebe/per-101-persist-bounded-shard-metric-aggregates
Oct 6, 2026
Merged

mtabebe merged 1 commit into
MaterializeInc:mainfrom
mtabebe:michaelabebe/per-101-persist-bounded-shard-metric-aggregates

Conversation

@mtabebe

@mtabebe mtabebe commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Persist exports 28 metric families with one series per shard, in every process that holds a handle to that shard. At around 14k shards that's roughly 390k series from environmentd alone, and the federated /metrics endpoint stacks each replica's copy on top. The scrape body grows with the shard count until it goes over the scraper's 50 MiB limit, which breaks the scraper.

Solution

Add bounded aggregates of the per-shard families, whose cardinality doesn't depend on the number of shards. The per-shard families are still exported unchanged, so existing dashboards and alerts keep working, for now.

The generic infrastructure lives in mz_ore::metrics::aggregation. A collector pushes one value per object and family into a snapshot, and each family's values are sorted once.

From that one sort it emits _percentile{percentile=...} with min, p25, p50, p75, p90, p99, p99.9, p99.99 and max, and _topk with the k=10 largest non-zero objects. Each carries the help of the family it summarizes, so the aggregate and the family read the same.

Note that these are built at runtime rather than with metric!, so bin/gen-metrics-catalog misses them and they aren't in doc/user/data/metrics.yml.

In persist, the existing ShardsAggregateMetrics walk of the shards map now feeds all of this, along with mz_persist_shard_count and mz_persist_stale_shard_count, so a scrape still walks the map once and the hot path doesn't change. The walk holds the shards lock, which new shard handles wait on, only long enough to collect the live handles. It reads their values after releasing it:

Note: For now, since the per-shard families are still exported, this makes the scrape a bit bigger. The aggregates add about 385 series and 45 KB of text per process (constant overhead). For comparison, the per-shard families are 28 series and about 2.9 KB per shard, so it is roughly equivalent to adding ~15 shards, for now.

Testing:

  • New mz-ore tests check that percentiles are exact (10,000 values, 100 values, one value, none), that top-K picks the largest objects and skips zeros, that an object leaving the top-K disappears instead of going stale
  • New persist tests check that PER_SHARD_FAMILIES matches exactly the registered per-shard families, that each aggregate's help matches the family's, and names real process counters

Closes: PER-101

Problem

Persist exports 28 metric families with one series per shard, in every
process that holds a handle to that shard. At around 14k shards that's
roughly 390k series from environmentd alone, and the federated
/metrics endpoint stacks each replica's copy on top. The scrape body
grows with the shard count until it goes over the scraper's 50 MiB
limit, which breaks the scraper.

Solution

Add bounded aggregates of the per-shard families, whose cardinality
doesn't depend on the number of shards. The per-shard families are
still exported unchanged, so existing dashboards and alerts keep
working, for now.

The generic infrastructure lives in mz_ore::metrics::aggregation.
A collector pushes one value per object and family into a snapshot, and each
family's values are sorted once.

From that one sort it emits <name>_percentile{percentile=...} with
min, p25, p50, p75, p90, p99, p99.9, p99.99 and max, and <name>_topk with
the k=10 largest non-zero objects. Each carries the help of the family
it summarizes, so the aggregate and the family read the same.

Note that these are built at runtime rather than with metric!, so
bin/gen-metrics-catalog misses them and they aren't in
doc/user/data/metrics.yml.

In persist, the existing ShardsAggregateMetrics walk of the shards map
now feeds all of this, along with mz_persist_shard_count and
mz_persist_stale_shard_count, so a scrape still walks the map once and
the hot path doesn't change. The walk holds the shards lock, which new
shard handles wait on, only long enough to collect the live handles.
It reads their values after releasing it:

Note: For now, since the per-shard families are still exported,
this makes the scrape a bit bigger. The aggregates add about 385 series
and 45 KB of text per process (constant overhead). For comparison,
the per-shard families are 28 series and about 2.9 KB per shard, so it
is roughly equivalent to adding ~15 shards, for now.

Testing:

- New mz-ore tests check that percentiles are exact (10,000 values, 100
values, one value, none), that top-K picks the largest objects and
skips zeros, that an object leaving the top-K disappears instead of
going stale
- New persist tests check that PER_SHARD_FAMILIES matches
exactly the registered per-shard families, that each aggregate's help
matches the family's, and names real process counters

Closes: PER-101
}

/// Folds the snapshot into the aggregated metric families.
pub fn finish(self) -> Vec<MetricFamily> {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the actual aggregation function


/// Every per-shard family. Must list exactly the families with a `shard`
/// label that `ShardsMetrics::new` registers.
const PER_SHARD_FAMILIES: &[PerShardFamily] = &[

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what turns all the per shard metrics into these aggregated ones

@mtabebe
mtabebe marked this pull request as ready for review September 30, 2026 01:05
@mtabebe
mtabebe requested review from a team as code owners September 30, 2026 01:05
@mtabebe

mtabebe commented Sep 30, 2026

Copy link
Copy Markdown
Contributor Author

Full stack is here: #39354

Also I'm OOO until next Tuesday so low priority until then 😄

@mtabebe
mtabebe merged commit 57daf7a into MaterializeInc:main Oct 6, 2026
91 checks passed
@bosconi bosconi mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants