Repository navigation
adapter: stop the arrangement-sizes metric sink from measuring itself - #39332
Conversation
Problem: The curated `mz_metric_arrangement_sizes` sink folded operators of dataflows without a catalog export (its own dataflow, the introspection subscribes, the logging dataflows) into an `unattributable` series. That put the sink's own arrangement sizes on its input: every change to its arrangements changed their logged size, which it read on the next logging tick, so the whole dataflow re-ran every tick for the replica's lifetime. Across the 41 replicas that the kafka-matrix creates, it pushed over capacity, and after a restart the catalog server's indexes took minutes to hydrate, so the status checks timed out. Solution: Count only dataflows with a non-transient export. Map each operator to its export before joining the raw logs, and join each raw log through its existing index, so no arrangement the sink owns ever holds rows about its own operators. The attributed series are unchanged. The `unattributable` series is gone. Testing: Re-enable the curated sinks in kafka-matrix. Fixes SQL-730.
| // their own worker fall through to an `unattributable` sentinel, so their bytes still count | ||
| // without a churning `t<N>` label growing series without bound. | ||
| // NOTE: only dataflows with a non-transient export are counted, and the join against `oe` | ||
| // drops every other operator before any arrangement this sink owns. Counting this sink's |
There was a problem hiding this comment.
This drops the unattributable series (e.g., peek/subscribe or logging).
We could keep those, only the sink measuring its own arrangements causes the feedback loop. But it gets to be quite messy SQL: we have to do a join to filter the name, and then we need to make sure we never create an arrangement that includes the metric sink data.
The feedback on the other PR was already to try and avoid the left join where possible.
There are a few other places that effectively do this filtering by having object level roll up (e.g., mz_object_arrangement_sizes) so I think this is a reasonable trade-off to make.
antiguru
left a comment
antiguru
left a comment
There was a problem hiding this comment.
Posted by Claude Code on behalf of @antiguru.
Fixing the feedback loop in the query is the right move, but one path remains open.
Blocking
1. Logging dataflows are still counted when INTROSPECTION DEBUGGING = true, so the loop is still reachable.
- The
exfilterexport_id NOT LIKE 't%'only drops transient ids. Log indexes are exported asGlobalId::IntrospectionSourceIndex, which prints assi<N>(src/repr/src/global_id.rs). They are registered as real exports throughCollectionLogging::newinComputeState::initialize_logging(src/compute/src/compute_state.rs), so they pass the filter. - Today this works by accident. With
log_logging = false, the loggers are registered after the logging dataflow is built (src/compute/src/logging/initialize.rs), so its operators have no address rows andoedrops them. INTROSPECTION DEBUGGING = truesetslog_logging, and the loggers are registered before the logging dataflow is built. Its address and heap-size rows then appear, and itssi…export passesex. The cycle: the sink's reduce changes, the resulting log events change the logging index arrangements, their sizes are logged undersi…, and the sink's reduce changes again.- The new
NOTE:says the logging dataflows' arrangements are excluded, which does not hold in this configuration. - Fix: also exclude
si%inex, or exclude the logging dataflow directly. A testdrive case on anINTROSPECTION DEBUGGING = truereplica would pin it.
I derived this from the code and did not reproduce it.
Strong suggestions
2. The unattributable assertion cannot fail. The new query cannot produce that label, so count(*) = 0 for labels -> 'id' = 'unattributable' is always true. The regression is the sink's dataflow re-running on every tick. A test that catches it would check that the sink's own dataflow stops accruing mz_scheduling_elapsed once idle. The kafka-matrix change covers this only indirectly, through a timeout.
3. User-visible metric change. The description says the unattributable series is gone. It should also say that arrangement bytes from transient dataflows (subscribes, peeks, the sinks themselves) are no longer reported anywhere, so sums over these series come out lower than before.
Nits
- In the
helpCASE, theELSEbranch catchesarrangement_batchesimplicitly. UseWHEN 'arrangement_batches'so a fourth family added tofdoes not silently get the batches help text. - The new
NOTE:packs the reason, the SQL-730 cause, and the no-union rule into one paragraph. Move the union-vs-per-index point next tof, where that decision is made.
Checked and fine
- The three raw logs have no key, so they are indexed on
(operator_id, worker_id)(LogVariant::index_by). The joins infcan reuse those indexes rather than arranging the logs inside the sink. - The final
GROUP BYarranges only rows that survived the join withoe. - The testdrive oracle still joins
mz_dataflow_operator_dataflowswithout matching onworker_id. That predates this PR.
QA LLM Review1. MEDIUM -- Logging dataflow is still counted under
|
Release notes for the v26.45 train, published as **v26.45.1**. v26.45.0 did not ship, so (as with v26.44) its notes are folded in here and [#39477](#39477) is closed unmerged. - **v26.45.0 content**: 8 Improvements, 2 Agent Skills, 8 Bug Fixes, copied verbatim from #39477 with anchors renamed to `{#v26.45.1-…}`. - **v26.45.1 content**: Alerting for Self-Managed and the external dependency monitoring Improvement (editor-added). The release's only PR, [#39659](#39659), is omitted. - **Dates**: Cloud 2026-10-08, Self-Managed 2026-10-09. The operator compatibility row is `v26.45.1`. `v26.45.md` (`patch: 1`, no `rc`) is already on `main`.⚠️ **Open questions** - **Group-sync bullet**: `oidc_group_role_sync_enabled` defaults to `false` at the tag. Is it on in Cloud? If not, cut the "Quieter logins" bullet. - **Certificate SAN requirements** ([#39316](#39316), [#39415](https://github.com/MaterializeInc/materialize/pull/39415)/[#39459](https://github.com/MaterializeInc/materialize/pull/39459)) need action at upgrade time but sit in Improvements. Should they be called out more loudly? Also, does [#39351](#39351) (S3 on rustls) impose the same SAN rule on custom S3 endpoints? - **[#39659](#39659) (omitted)**: is `enable_upsert_paged_spill` on in production? A draft bullet is in [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.45.1/final/omitted.md#borderline). - **`EXPLAIN ANALYZE ... CPU`**: [agent-skills#90](MaterializeInc/agent-skills#90) says it overstates joins, reductions, and TopK by 1.5x–3x before v26.46. Not published. Is that a known issue? - **Metric sinks**: [#39332](#39332), [#38910](#38910), and [#39337](#39337) are withheld because `enable_metric_sink` is `false`. #39337 fixes a panic any superuser can reach, so consider publishing it regardless. - **Possible cuts**: [#39362](#39362) was observed only in CI. The latency figures from [#39179](#39179) were measured on one noisy machine. - **Possible adds**: [#38894](https://github.com/MaterializeInc/materialize/pull/38894)–[#38896](https://github.com/MaterializeInc/materialize/pull/38896) (MV correction buffer) need an end-to-end number. [#39149](#39149) fixes the kind install guide, which crash-loops today. [#39549](#39549) (jemalloc page) is omitted because no released version had the regression. Snapshot drafts in mz-skills: [v26.45.0](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.0) (rc.1–rc.4, final), [v26.45.1](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.1/final). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>
Problem:
The curated
mz_metric_arrangement_sizessink folded operators of dataflows without a catalog export (its own dataflow, the introspection subscribes, the logging dataflows) into anunattributableseries. That put the sink's own arrangement sizes on its input: every change to its arrangements changed their logged size, which it read on the next logging tick, so the whole dataflow re-ran every tick for the replica's lifetime.Across the 41 replicas that the kafka-matrix creates, it pushed over capacity, and after a restart the catalog server's indexes took minutes to hydrate, so the status checks timed out.
Solution:
Count only dataflows with a non-transient export. Map each operator to its export before joining the raw logs, and join each raw log through its existing index, so no arrangement the sink owns ever holds rows about its own operators. The attributed series are unchanged. The
unattributableseries is gone.Testing:
Re-enable the curated sinks in kafka-matrix.
Fixes SQL-730.