DNM: stacked per shard metric cardinality - #39354

Problem

Persist exports 28 metric families with one series per shard, in every
process that holds a handle to that shard. At around 14k shards that's
roughly 390k series from environmentd alone, and the federated
/metrics endpoint stacks each replica's copy on top. The scrape body
grows with the shard count until it goes over the scraper's 50 MiB
limit, which breaks the scraper.

Solution

Add bounded aggregates of the per-shard families, whose cardinality
doesn't depend on the number of shards. The per-shard families are
still exported unchanged, so existing dashboards and alerts keep
working, for now.

The generic infrastructure lives in mz_ore::metrics::aggregation.
A collector pushes one value per object and family into a snapshot, and each
family's values are sorted once.

From that one sort it emits <name>_percentile{percentile=...} with
min, p25, p50, p75, p90, p99, p99.9, p99.99 and max, and <name>_topk with
the k=10 largest non-zero objects. Each carries the help of the family
it summarizes, so the aggregate and the family read the same.

Note that these are built at runtime rather than with metric!, so
bin/gen-metrics-catalog misses them and they aren't in
doc/user/data/metrics.yml.

In persist, the existing ShardsAggregateMetrics walk of the shards map
now feeds all of this, along with mz_persist_shard_count and
mz_persist_stale_shard_count, so a scrape still walks the map once and
the hot path doesn't change. The walk holds the shards lock, which new
shard handles wait on, only long enough to collect the live handles.
It reads their values after releasing it:

Note: For now, since the per-shard families are still exported,
this makes the scrape a bit bigger. The aggregates add about 385 series
and 45 KB of text per process (constant overhead). For comparison,
the per-shard families are 28 series and about 2.9 KB per shard, so it
is roughly equivalent to adding ~15 shards, for now.

Testing:

- New mz-ore tests check that percentiles are exact (10,000 values, 100
values, one value, none), that top-K picks the largest objects and
skips zeros, that an object leaving the top-K disappears instead of
going stale
- New persist tests check that PER_SHARD_FAMILIES matches
exactly the registered per-shard families, that each aggregate's help
matches the family's, and names real process counters

Closes: PER-101

Problem:

Persist exports 28 metric families with one series per shard, so the
scrape body grows with the shard count until it goes over the
scraper's size limit.

Dropping those series on the scraper side doesn't help, because the
limit applies to the body the process sends, before the scraper looks
at any series. The process has to stop sending them.

Solution

Add persist_shard_metrics, which picks what a process exports:

- per_shard: only the per-shard families (what we had before)
- summary: only their bounded aggregates (the _percentile and _topk
  families and mz_persist_batch_part_version_count) (what we want to do
  long term)
- both: everything, which is the default (for now) and matches what's exported
  without this setting.
- none: neither

Shard counts and the process-level counters are exported in every
mode.

An unrecognized value exports the default. The parse itself is silent,
since both readers run per scrape and would otherwise log forever. The
postprocessor reports it once per change instead, and
mz_persist_shard_metrics_mode_invalid counts the scrapes it affected.

Production keeps both (for now), because dashboards that read the
per-shard families stop getting data without them.
Tests default to summary for now

Testing

New tests switch one Metrics through all four modes and check that
each exports exactly the families it should, that the process-level
counters hold their value in every mode, and that an unrecognized value
exports the default and counts the scrapes it hit

Closes: PER-102