Repository navigation
compute: raise the correction buffer's staging size to 2 MiB - #38894
Merged
Merged
Conversation
antiguru
changed the title
antiguru
force-pushed
the
moritz/correction-chunk-size
branch
2 times, most recently
from
September 23, 2026 13:23
7bff8bc to
64c48d4
Compare
antiguru
marked this pull request as ready for review
September 30, 2026 11:50
antiguru
commented
Sep 30, 2026
antiguru
left a comment
antiguru
left a comment
Member
Author
There was a problem hiding this comment.
Self-review.
- Description: the 3443 ms to 1860 ms figure was measured before the stage became byte-sized. The description lands on
mainwith the squash, so it needs a number measured on this code, or no number. - Tests: byte-paced shipping is a behavior change with only a bench behind it. A unit test should feed rows of two widths at one flag value and assert that the wider rows ship after fewer updates.
🤖 Posted by Claude Code
antiguru
force-pushed
the
moritz/correction-chunk-size
branch
from
September 30, 2026 13:12
64c48d4 to
c2fd569
Compare
Member
Author
|
Addressed in the rebased push. The figure was re-measured at 3503 ms to 1897 ms, with drain and peak RSS unchanged. 🤖 Posted by Claude Code |
antiguru
force-pushed
the
moritz/correction-chunk-size
branch
from
September 30, 2026 15:44
c2fd569 to
31c2f43
Compare
`compute_correction_v2_chunk_size` sizes `CorrectionV2`'s staging area, not its chunks: chunk bodies are fixed at the columnar ship size. The staging size sets how many updates accumulate before a chain is minted, and every chain minted is a chain some later read has to merge. Raising the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append workload from 3503 ms to 1897 ms, with drain and peak RSS unchanged (medians of three `correction_mem` runs with the rest of the stack applied and no pool installed). The curve is monotone across the range, so the default is bounded only by the per-sink, per-worker cost of the staging vector, and the `correction` bench gains a `bench_chunk_size` sweep to show it. The flag's description now says what it sizes. The stage counted its fill in tuples of `size_of::<(D, Timestamp, Diff)>()`, which is 40 bytes for a `Row` however wide the row is, so a byte-sized flag bounded neither the staging memory nor the reported size once rows spill out of line. A new `DataBytes` trait on the buffer's data types reports each datum's bytes with the heap included, the stage ships when the staged bytes reach the flag, and its size metrics carry the same figure. `Row` answers with `byte_len`, `DataflowErrorSer` with its serialized length. The staging vector was allocated eagerly at the configured capacity and re-allocated at that capacity on every flush. At 2 MiB per sink and worker that is no longer free, so it grows on demand instead and gives its allocation back on flush. The mzcompose and parallel-workload flag tables follow the new default, since CI runs the value listed there rather than the compiled default. Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
antiguru
force-pushed
the
moritz/correction-chunk-size
branch
from
September 30, 2026 18:07
31c2f43 to
cc11def
Compare
This was referenced Oct 2, 2026
val-materialize
added a commit
that referenced
this pull request
Oct 9, 2026
Release notes for the v26.45 train, published as **v26.45.1**. v26.45.0 did not ship, so (as with v26.44) its notes are folded in here and [#39477](#39477) is closed unmerged. - **v26.45.0 content**: 8 Improvements, 2 Agent Skills, 8 Bug Fixes, copied verbatim from #39477 with anchors renamed to `{#v26.45.1-…}`. - **v26.45.1 content**: Alerting for Self-Managed and the external dependency monitoring Improvement (editor-added). The release's only PR, [#39659](#39659), is omitted. - **Dates**: Cloud 2026-10-08, Self-Managed 2026-10-09. The operator compatibility row is `v26.45.1`. `v26.45.md` (`patch: 1`, no `rc`) is already on `main`.⚠️ **Open questions** - **Group-sync bullet**: `oidc_group_role_sync_enabled` defaults to `false` at the tag. Is it on in Cloud? If not, cut the "Quieter logins" bullet. - **Certificate SAN requirements** ([#39316](#39316), [#39415](https://github.com/MaterializeInc/materialize/pull/39415)/[#39459](https://github.com/MaterializeInc/materialize/pull/39459)) need action at upgrade time but sit in Improvements. Should they be called out more loudly? Also, does [#39351](#39351) (S3 on rustls) impose the same SAN rule on custom S3 endpoints? - **[#39659](#39659) (omitted)**: is `enable_upsert_paged_spill` on in production? A draft bullet is in [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.45.1/final/omitted.md#borderline). - **`EXPLAIN ANALYZE ... CPU`**: [agent-skills#90](MaterializeInc/agent-skills#90) says it overstates joins, reductions, and TopK by 1.5x–3x before v26.46. Not published. Is that a known issue? - **Metric sinks**: [#39332](#39332), [#38910](#38910), and [#39337](#39337) are withheld because `enable_metric_sink` is `false`. #39337 fixes a panic any superuser can reach, so consider publishing it regardless. - **Possible cuts**: [#39362](#39362) was observed only in CI. The latency figures from [#39179](#39179) were measured on one noisy machine. - **Possible adds**: [#38894](https://github.com/MaterializeInc/materialize/pull/38894)–[#38896](https://github.com/MaterializeInc/materialize/pull/38896) (MV correction buffer) need an end-to-end number. [#39149](#39149) fixes the kind install guide, which crash-loops today. [#39549](#39549) (jemalloc page) is omitted because no released version had the regression. Snapshot drafts in mz-skills: [v26.45.0](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.0) (rc.1–rc.4, final), [v26.45.1](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.1/final). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
compute_correction_v2_chunk_sizesizesCorrectionV2's staging area, not its chunks: chunk bodies are fixed at the columnar ship size. The staging size sets how many updates accumulate before a chain is minted, and every chain minted is a chain some later read has to merge. Raising the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append workload from 3503 ms to 1897 ms, with drain and peak RSS unchanged. These are medians of threecorrection_memruns with the rest of the stack applied and no pool installed. The curve is monotone across the range (the newbench_chunk_sizesweep in thecorrectionbench shows it), so the default is bounded only by the per-sink, per-worker cost of the staging vector.The stage counted its fill in tuples of
size_of::<(D, Timestamp, Diff)>(), which is 40 bytes for aRowhowever wide the row is, so a byte-sized flag bounded neither the staging memory nor the reported size once rows spill out of line. A newDataBytestrait on the buffer's data types reports each datum's bytes with the heap included (Rowviabyte_len,DataflowErrorServia its serialized length), the stage ships when the staged bytes reach the flag, and its size metrics carry the same figure.stage_ships_by_byteschecks that rows of two widths ship after the number of updates their bytes predict.That vector was allocated eagerly at the configured capacity and re-allocated at that capacity on every flush. At 2 MiB per sink and worker that is no longer free, so it now grows on demand and gives its allocation back on flush.
The mzcompose and parallel-workload flag tables follow the new default, since CI runs the value listed there rather than the compiled default. The flag's description now says what it sizes.
First of four, split out of #38645: this, the O(1) metrics (#38895), the per-chunk borrow hoist (#38896), and the pool substrate (#38897) land in that order. Supersedes the fork PR #38647.
🤖 Generated with Claude Code