Repository navigation
Conversation
Problem
Persist exports 28 metric families with one series per shard, in every
process that holds a handle to that shard. At around 14k shards that's
roughly 390k series from environmentd alone, and the federated
/metrics endpoint stacks each replica's copy on top. The scrape body
grows with the shard count until it goes over the scraper's 50 MiB
limit, which breaks the scraper.
Solution
Add bounded aggregates of the per-shard families, whose cardinality
doesn't depend on the number of shards. The per-shard families are
still exported unchanged, so existing dashboards and alerts keep
working, for now.
The generic infrastructure lives in mz_ore::metrics::aggregation.
A collector pushes one value per object and family into a snapshot, and each
family's values are sorted once.
From that one sort it emits <name>_percentile{percentile=...} with
min, p25, p50, p75, p90, p99, p99.9, p99.99 and max, and <name>_topk with
the k=10 largest non-zero objects. Each carries the help of the family
it summarizes, so the aggregate and the family read the same.
Note that these are built at runtime rather than with metric!, so
bin/gen-metrics-catalog misses them and they aren't in
doc/user/data/metrics.yml.
In persist, the existing ShardsAggregateMetrics walk of the shards map
now feeds all of this, along with mz_persist_shard_count and
mz_persist_stale_shard_count, so a scrape still walks the map once and
the hot path doesn't change. The walk holds the shards lock, which new
shard handles wait on, only long enough to collect the live handles.
It reads their values after releasing it:
Note: For now, since the per-shard families are still exported,
this makes the scrape a bit bigger. The aggregates add about 385 series
and 45 KB of text per process (constant overhead). For comparison,
the per-shard families are 28 series and about 2.9 KB per shard, so it
is roughly equivalent to adding ~15 shards, for now.
Testing:
- New mz-ore tests check that percentiles are exact (10,000 values, 100
values, one value, none), that top-K picks the largest objects and
skips zeros, that an object leaving the top-K disappears instead of
going stale
- New persist tests check that PER_SHARD_FAMILIES matches
exactly the registered per-shard families, that each aggregate's help
matches the family's, and names real process counters
Closes: PER-101
mtabebe
commented
Sep 30, 2026
| } | ||
|
|
||
| /// Folds the snapshot into the aggregated metric families. | ||
| pub fn finish(self) -> Vec<MetricFamily> { |
Contributor
Author
There was a problem hiding this comment.
This is the actual aggregation function
|
|
||
| /// Every per-shard family. Must list exactly the families with a `shard` | ||
| /// label that `ShardsMetrics::new` registers. | ||
| const PER_SHARD_FAMILIES: &[PerShardFamily] = &[ |
Contributor
Author
There was a problem hiding this comment.
This is what turns all the per shard metrics into these aggregated ones
Contributor
Author
|
Full stack is here: #39354 Also I'm OOO until next Tuesday so low priority until then 😄 |
Alphadelta14
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Persist exports 28 metric families with one series per shard, in every process that holds a handle to that shard. At around 14k shards that's roughly 390k series from environmentd alone, and the federated /metrics endpoint stacks each replica's copy on top. The scrape body grows with the shard count until it goes over the scraper's 50 MiB limit, which breaks the scraper.
Solution
Add bounded aggregates of the per-shard families, whose cardinality doesn't depend on the number of shards. The per-shard families are still exported unchanged, so existing dashboards and alerts keep working, for now.
The generic infrastructure lives in mz_ore::metrics::aggregation. A collector pushes one value per object and family into a snapshot, and each family's values are sorted once.
From that one sort it emits _percentile{percentile=...} with min, p25, p50, p75, p90, p99, p99.9, p99.99 and max, and _topk with the k=10 largest non-zero objects. Each carries the help of the family it summarizes, so the aggregate and the family read the same.
Note that these are built at runtime rather than with metric!, so bin/gen-metrics-catalog misses them and they aren't in doc/user/data/metrics.yml.
In persist, the existing ShardsAggregateMetrics walk of the shards map now feeds all of this, along with mz_persist_shard_count and mz_persist_stale_shard_count, so a scrape still walks the map once and the hot path doesn't change. The walk holds the shards lock, which new shard handles wait on, only long enough to collect the live handles. It reads their values after releasing it:
Note: For now, since the per-shard families are still exported, this makes the scrape a bit bigger. The aggregates add about 385 series and 45 KB of text per process (constant overhead). For comparison, the per-shard families are 28 series and about 2.9 KB per shard, so it is roughly equivalent to adding ~15 shards, for now.
Testing:
Closes: PER-101