Repository navigation
Conversation
Problem: Persist exports 28 metric families with one series per shard, so the scrape body grows with the shard count until it goes over the scraper's size limit. Dropping those series on the scraper side doesn't help, because the limit applies to the body the process sends, before the scraper looks at any series. The process has to stop sending them. Solution Add persist_shard_metrics, which picks what a process exports: - per_shard: only the per-shard families (what we had before) - summary: only their bounded aggregates (the _percentile and _topk families and mz_persist_batch_part_version_count) (what we want to do long term) - both: everything, which is the default (for now) and matches what's exported without this setting. - none: neither Shard counts and the process-level counters are exported in every mode. An unrecognized value exports the default. The parse itself is silent, since both readers run per scrape and would otherwise log forever. The postprocessor reports it once per change instead, and mz_persist_shard_metrics_mode_invalid counts the scrapes it affected. Production keeps both (for now), because dashboards that read the per-shard families stop getting data without them. Tests default to summary for now Testing New tests switch one Metrics through all four modes and check that each exports exactly the families it should, that the process-level counters hold their value in every mode, and that an unrecognized value exports the default and counts the scrapes it hit Closes: PER-102
Contributor
Author
|
This is the second in the stack of 3: github.com//pull/39354 |
Alphadelta14
approved these changes
Oct 6, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem:
Persist exports 28 metric families with one series per shard, so the scrape body grows with the shard count until it goes over the scraper's size limit.
Dropping those series on the scraper side doesn't help, because the limit applies to the body the process sends, before the scraper looks at any series. The process has to stop sending them.
Solution
Add persist_shard_metrics, which picks what a process exports:
Shard counts and the process-level counters are exported in every mode.
An unrecognized value exports the default. The parse itself is silent, since both readers run per scrape and would otherwise log forever. The postprocessor reports it once per change instead, and mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.
Production keeps both (for now), because dashboards that read the per-shard families stop getting data without them. Tests default to summary for now
Testing
New tests switch one Metrics through all four modes and check that each exports exactly the families it should, that the process-level counters hold their value in every mode, and that an unrecognized value exports the default and counts the scrapes it hit
Closes: PER-102