Repository navigation
Conversation
This was referenced Sep 4, 2026
antiguru
force-pushed
the
moritz/correction-chunk-size
branch
from
September 14, 2026 20:17
f1db9a6 to
4db88b5
Compare
`compute_correction_v2_chunk_size` sizes `CorrectionV2`'s staging area, not its chunks: chunk bodies are fixed at the columnar ship size. The staging size sets how many updates accumulate before a chain is minted, and every chain minted is a chain some later read has to merge. Raising the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append workload from 3443 ms to 1860 ms at an unchanged peak RSS. The curve is monotone across the range, so the default is bounded only by the per-sink, per-worker cost of the staging vector, and the `correction` bench gains a `bench_chunk_size` sweep to show it. The flag's description now says what it sizes. The stage counted its fill in tuples of `size_of::<(D, Timestamp, Diff)>()`, which is 40 bytes for a `Row` however wide the row is, so a byte-sized flag bounded neither the staging memory nor the reported size once rows spill out of line. A new `DataBytes` trait on the buffer's data types reports each datum's bytes with the heap included, the stage ships when the staged bytes reach the flag, and its size metrics carry the same figure. `Row` answers with `byte_len`, `DataflowErrorSer` with its serialized length. The staging vector was allocated eagerly at the configured capacity and re-allocated at that capacity on every flush. At 2 MiB per sink and worker that is no longer free, so it grows on demand instead and gives its allocation back on flush. The mzcompose and parallel-workload flag tables follow the new default, since CI runs the value listed there rather than the compiled default. Co-Authored-By: Claude Fable 5.1 <[email protected]>
antiguru
force-pushed
the
moritz/correction-chunk-size
branch
from
September 14, 2026 20:25
4db88b5 to
ba7ffc6
Compare
Member
Author
|
Closed in favor of #38894, the same branch as a native stacked PR in this repository (stack #38898). Fork PRs cannot join GitHub stacks. 🤖 Posted by Claude Code |
antiguru
added a commit
that referenced
this pull request
Sep 30, 2026
`compute_correction_v2_chunk_size` sizes `CorrectionV2`'s staging area, not its chunks: chunk bodies are fixed at the columnar ship size. The staging size sets how many updates accumulate before a chain is minted, and every chain minted is a chain some later read has to merge. Raising the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append workload from 3503 ms to 1897 ms, with drain and peak RSS unchanged. These are medians of three `correction_mem` runs with the rest of the stack applied and no pool installed. The curve is monotone across the range (the new `bench_chunk_size` sweep in the `correction` bench shows it), so the default is bounded only by the per-sink, per-worker cost of the staging vector. The stage counted its fill in tuples of `size_of::<(D, Timestamp, Diff)>()`, which is 40 bytes for a `Row` however wide the row is, so a byte-sized flag bounded neither the staging memory nor the reported size once rows spill out of line. A new `DataBytes` trait on the buffer's data types reports each datum's bytes with the heap included (`Row` via `byte_len`, `DataflowErrorSer` via its serialized length), the stage ships when the staged bytes reach the flag, and its size metrics carry the same figure. `stage_ships_by_bytes` checks that rows of two widths ship after the number of updates their bytes predict. That vector was allocated eagerly at the configured capacity and re-allocated at that capacity on every flush. At 2 MiB per sink and worker that is no longer free, so it now grows on demand and gives its allocation back on flush. The mzcompose and parallel-workload flag tables follow the new default, since CI runs the value listed there rather than the compiled default. The flag's description now says what it sizes. First of four, split out of #38645: this, the O(1) metrics (#38895), the per-chunk borrow hoist (#38896), and the pool substrate (#38897) land in that order. Supersedes the fork PR #38647. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Fable 5.1 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
compute_correction_v2_chunk_sizesizesCorrectionV2's staging area, not its chunks: chunk bodies are fixed at the columnar ship size. The staging size sets how many updates accumulate before a chain is minted, and every chain minted is a chain some later read has to merge. Raising the default from 8 KiB to 2 MiB, one chunk body's worth, takes insert on a 4.19M-update append workload from 3443 ms to 1860 ms at an unchanged peak RSS. The curve is monotone across the range (the newbench_chunk_sizesweep in thecorrectionbench shows it), so the default is bounded only by the per-sink, per-worker cost of the staging vector.The stage counted its fill in tuples of
size_of::<(D, Timestamp, Diff)>(), which is 40 bytes for aRowhowever wide the row is, so a byte-sized flag bounded neither the staging memory nor the reported size once rows spill out of line. A newDataBytestrait on the buffer's data types reports each datum's bytes with the heap included (Rowviabyte_len,DataflowErrorServia its serialized length), the stage ships when the staged bytes reach the flag, and its size metrics carry the same figure.That vector was allocated eagerly at the configured capacity and re-allocated at that capacity on every flush. At 2 MiB per sink and worker that is no longer free, so it now grows on demand and gives its allocation back on flush.
The mzcompose and parallel-workload flag tables follow the new default, since CI runs the value listed there rather than the compiled default. The flag's description now says what it sizes.
First of four, split out of #38645: this, the O(1) metrics (#38648), the per-chunk borrow hoist (#38649), and the pool substrate (#38645) land in that order.
🤖 Generated with Claude Code