Skip to content

persist: choose which shard metrics to export - #39591

Merged
mtabebe merged 1 commit into
MaterializeInc:mainfrom
mtabebe:michaelabebe/per-102-persist-choose-shard-metrics
Oct 6, 2026
Merged

mtabebe merged 1 commit into
MaterializeInc:mainfrom
mtabebe:michaelabebe/per-102-persist-choose-shard-metrics

Conversation

@mtabebe

@mtabebe mtabebe commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem:

Persist exports 28 metric families with one series per shard, so the scrape body grows with the shard count until it goes over the scraper's size limit.

Dropping those series on the scraper side doesn't help, because the limit applies to the body the process sends, before the scraper looks at any series. The process has to stop sending them.

Solution

Add persist_shard_metrics, which picks what a process exports:

  • per_shard: only the per-shard families (what we had before)
  • summary: only their bounded aggregates (the _percentile and _topk families and mz_persist_batch_part_version_count) (what we want to do long term)
  • both: everything, which is the default (for now) and matches what's exported without this setting.
  • none: neither

Shard counts and the process-level counters are exported in every mode.

An unrecognized value exports the default. The parse itself is silent, since both readers run per scrape and would otherwise log forever. The postprocessor reports it once per change instead, and mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.

Production keeps both (for now), because dashboards that read the per-shard families stop getting data without them. Tests default to summary for now

Testing

New tests switch one Metrics through all four modes and check that each exports exactly the families it should, that the process-level counters hold their value in every mode, and that an unrecognized value exports the default and counts the scrapes it hit

Closes: PER-102

Problem:

Persist exports 28 metric families with one series per shard, so the
scrape body grows with the shard count until it goes over the
scraper's size limit.

Dropping those series on the scraper side doesn't help, because the
limit applies to the body the process sends, before the scraper looks
at any series. The process has to stop sending them.

Solution

Add persist_shard_metrics, which picks what a process exports:

- per_shard: only the per-shard families (what we had before)
- summary: only their bounded aggregates (the _percentile and _topk
  families and mz_persist_batch_part_version_count) (what we want to do
  long term)
- both: everything, which is the default (for now) and matches what's exported
  without this setting.
- none: neither

Shard counts and the process-level counters are exported in every
mode.

An unrecognized value exports the default. The parse itself is silent,
since both readers run per scrape and would otherwise log forever. The
postprocessor reports it once per change instead, and
mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.

Production keeps both (for now), because dashboards that read the
per-shard families stop getting data without them.
Tests default to summary for now

Testing

New tests switch one Metrics through all four modes and check that
each exports exactly the families it should, that the process-level
counters hold their value in every mode, and that an unrecognized value
exports the default and counts the scrapes it hit

Closes: PER-102
@mtabebe

mtabebe commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author

This is the second in the stack of 3: github.com//pull/39354

@mtabebe
mtabebe requested a review from Alphadelta14 October 6, 2026 15:48
@mtabebe
mtabebe marked this pull request as ready for review October 6, 2026 15:48
@mtabebe
mtabebe requested review from a team as code owners October 6, 2026 15:48
@mtabebe
mtabebe merged commit 988f054 into MaterializeInc:main Oct 6, 2026
88 checks passed
@bosconi bosconi mentioned this pull request Oct 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants