Skip to content

compute: raise the correction buffer's staging size to 2 MiB - #38647

Closed
antiguru wants to merge 1 commit into
MaterializeInc:mainfrom
antiguru:moritz/correction-chunk-size
Closed

antiguru wants to merge 1 commit into
MaterializeInc:mainfrom
antiguru:moritz/correction-chunk-size

Conversation

@antiguru

@antiguru antiguru commented Sep 4, 2026 •

Copy link
Copy Markdown
Member

compute_correction_v2_chunk_size sizes CorrectionV2's staging area, not its chunks: chunk bodies are fixed at the columnar ship size. The staging size sets how many updates accumulate before a chain is minted, and every chain minted is a chain some later read has to merge. Raising the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append workload from 3443 ms to 1860 ms at an unchanged peak RSS. The curve is monotone across the range (the new bench_chunk_size sweep in the correction bench shows it), so the default is bounded only by the per-sink, per-worker cost of the staging vector.

The stage counted its fill in tuples of size_of::<(D, Timestamp, Diff)>(), which is 40 bytes for a Row however wide the row is, so a byte-sized flag bounded neither the staging memory nor the reported size once rows spill out of line. A new DataBytes trait on the buffer's data types reports each datum's bytes with the heap included (Row via byte_len, DataflowErrorSer via its serialized length), the stage ships when the staged bytes reach the flag, and its size metrics carry the same figure.

That vector was allocated eagerly at the configured capacity and re-allocated at that capacity on every flush. At 2 MiB per sink and worker that is no longer free, so it now grows on demand and gives its allocation back on flush.

The mzcompose and parallel-workload flag tables follow the new default, since CI runs the value listed there rather than the compiled default. The flag's description now says what it sizes.

First of four, split out of #38645: this, the O(1) metrics (#38648), the per-chunk borrow hoist (#38649), and the pool substrate (#38645) land in that order.

🤖 Generated with Claude Code

`compute_correction_v2_chunk_size` sizes `CorrectionV2`'s staging area, not
its chunks: chunk bodies are fixed at the columnar ship size. The staging size
sets how many updates accumulate before a chain is minted, and every chain
minted is a chain some later read has to merge. Raising the default from 8 KiB
to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append
workload from 3443 ms to 1860 ms at an unchanged peak RSS. The curve is
monotone across the range, so the default is bounded only by the per-sink,
per-worker cost of the staging vector, and the `correction` bench gains a
`bench_chunk_size` sweep to show it. The flag's description now says what it
sizes.

The stage counted its fill in tuples of `size_of::<(D, Timestamp, Diff)>()`,
which is 40 bytes for a `Row` however wide the row is, so a byte-sized flag
bounded neither the staging memory nor the reported size once rows spill out of
line. A new `DataBytes` trait on the buffer's data types reports each datum's
bytes with the heap included, the stage ships when the staged bytes reach the
flag, and its size metrics carry the same figure. `Row` answers with
`byte_len`, `DataflowErrorSer` with its serialized length.

The staging vector was allocated eagerly at the configured capacity and
re-allocated at that capacity on every flush. At 2 MiB per sink and worker that
is no longer free, so it grows on demand instead and gives its allocation back
on flush.

The mzcompose and parallel-workload flag tables follow the new default, since CI
runs the value listed there rather than the compiled default.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
@antiguru

Copy link
Copy Markdown
Member Author

Closed in favor of #38894, the same branch as a native stacked PR in this repository (stack #38898). Fork PRs cannot join GitHub stacks.

🤖 Posted by Claude Code

antiguru added a commit that referenced this pull request Sep 30, 2026
`compute_correction_v2_chunk_size` sizes `CorrectionV2`'s staging area,
not its chunks: chunk bodies are fixed at the columnar ship size. The
staging size sets how many updates accumulate before a chain is minted,
and every chain minted is a chain some later read has to merge. Raising
the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on
a 4.19M-update append workload from 3503 ms to 1897 ms, with drain and
peak RSS unchanged. These are medians of three `correction_mem` runs
with the rest of the stack applied and no pool installed. The curve is
monotone across the range (the new `bench_chunk_size` sweep in the
`correction` bench shows it), so the default is bounded only by the
per-sink, per-worker cost of the staging vector.

The stage counted its fill in tuples of `size_of::<(D, Timestamp,
Diff)>()`, which is 40 bytes for a `Row` however wide the row is, so a
byte-sized flag bounded neither the staging memory nor the reported size
once rows spill out of line. A new `DataBytes` trait on the buffer's
data types reports each datum's bytes with the heap included (`Row` via
`byte_len`, `DataflowErrorSer` via its serialized length), the stage
ships when the staged bytes reach the flag, and its size metrics carry
the same figure.

`stage_ships_by_bytes` checks that rows of two widths ship after the
number of updates their bytes predict.

That vector was allocated eagerly at the configured capacity and
re-allocated at that capacity on every flush. At 2 MiB per sink and
worker that is no longer free, so it now grows on demand and gives its
allocation back on flush.

The mzcompose and parallel-workload flag tables follow the new default,
since CI runs the value listed there rather than the compiled default.
The flag's description now says what it sizes.

First of four, split out of #38645: this, the O(1) metrics (#38895), the
per-chunk borrow hoist (#38896), and the pool substrate (#38897) land in
that order. Supersedes the fork PR #38647.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Fable 5.1 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant