Repository navigation
timely-util: split the chunk body off the edge column - #39159
Merged
Merged
Conversation
antiguru
changed the title
antiguru
marked this pull request as ready for review
September 30, 2026 12:02
antiguru
commented
Sep 30, 2026
antiguru
left a comment
antiguru
left a comment
Member
Author
There was a problem hiding this comment.
Self-review, read against the chunk-native arrangements design in #39072.
- CI: both rust-build jobs failed at
docker login ghcr.iowith "denied". That is an infrastructure failure, not the code. - Fit with #39072:
ColumnBodyis the seam that design needs. An encoded chunk payload becomes a body form, andColumnremains the edge container.MergeChunkshould stay minimal, because the design's range builder replaces it. - Naming:
ColumnBody::Wordsis specifically the flatindexedencoding. The doc should say so, so that nobody reads it as the general serialized case once a second encoding exists. - Copy:
into_bodyon a shared resident body that is alreadyWordsdecodes it into typed containers. Cloning the words is cheaper and keeps the form. - Transitional code: the duplicated
empty_chunkandrecycle_chunkinmerge_batcher.rsshould say at their definitions that they exist until the pager is retired. - Not a pattern to copy: correction
Chunk::bodytakes a body out of the pool permanently on first read. That suits the correction buffer, but the design's bounded read windows avoid exactly this, so arrangements should not copy it.
🤖 Posted by Claude Code
antiguru
force-pushed
the
moritz/column-body-split
branch
from
September 30, 2026 13:12
b4afc94 to
31c2f68
Compare
Member
Author
|
Addressed and rebased onto #38900, whose chunked consolidation now hands bodies back onto the edge through 🤖 Posted by Claude Code |
antiguru
force-pushed
the
moritz/column-body-split
branch
from
September 30, 2026 15:44
31c2f68 to
0b673aa
Compare
antiguru
force-pushed
the
moritz/column-body-split
branch
from
September 30, 2026 18:07
0b673aa to
55a8442
Compare
`Column` served two roles. On a dataflow edge it is built by pushing, crosses an exchange as channel bytes, and is read on arrival, which is what its `Typed`, `Bytes`, and `Align` variants are for. Behind an edge, in a merge batcher's chains, a spill-backed `ColumnChunk`, and a batch builder's seal, the same type held sorted, consolidated bodies that were only ever typed or, when read back from a store, serialized words. That code never constructed `Bytes`, and it asserted the variant it expected at seven sites with `unreachable!`. `ColumnBody<C>` is the type at rest: `Typed(C::Container)` while a body is written, `Words(Vec<u64>)` once a store handed it back, and nothing else. Writing to a serialized body materializes it by a bulk per-leaf copy (`typed_mut`), so the merge, extract, advance, and settle bodies address the typed containers without matching on a variant. `ColumnChunk`, `ColumnChunker`, `ColumnMerger`, `UnchunkBuilder`, `RowRowColPagedBuilder`, and the correction buffer's chunks and builder now hold bodies. The edge container keeps its name and every operator that names it is untouched. The boundaries are moves. A chunker takes whatever the edge delivered and emits bodies. A body goes back onto an edge through `From`, typed staying typed and words becoming `Align`, which is how the temporal bucket's peeled chunks, the chunked consolidation's output, and the pager batcher's input travel. The column pager's batcher and `ValRowColPagedBuilder` keep `Column` chains until the pager is retired, so the merge and extract bodies are generic over a `MergeChunk` trait both types implement, and `PagedChunker` wraps the chunker for that batcher. A shared resident body is copied in its current form when a chunk is unwrapped, so a serialized body is cloned as words rather than decoded into typed containers. Co-Authored-By: Claude Fable 5.1 <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]>
antiguru
force-pushed
the
moritz/column-body-split
branch
from
September 30, 2026 19:45
55a8442 to
6cd76a7
Compare
Member
Author
|
Staging results for the stack, which includes this PR, are posted on #38897: #38897 (comment). They cover V1 against V2 and spill off against spill on, from 200cc down to 50cc. 🤖 Posted by Claude Code |
DAlperin
approved these changes
Oct 1, 2026
DAlperin
left a comment
DAlperin
left a comment
Member
There was a problem hiding this comment.
LGTM, but could we have some tests covering merge and extract in the Words variant? Seems uncovered
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Member
Author
antiguru
added a commit
that referenced
this pull request
Oct 1, 2026
The correction buffer spilled every chunk at depth 0, which put all of them in the buffer pool's hottest eviction band and kept them out of compression. The merge batcher instead tags chunks with a generational depth that grows with each merge they survive, and the pool evicts deeper chunks first and compresses them past `column_chunk_compress_min_depth`. This change gives correction chunks the same depth. A bucket's chain merge writes its output one generation deeper than its deepest input. The bucket's geometric chain invariant is the same structure the merge batcher's depth describes: a merged chain waits about `chain_proportionality` times longer for its next merge. Chunks that a read writes into `emitted` stay at depth 0, because the caller iterates them next and the next consolidation merges them with the feedback retractions. Chunks built from staged updates also start at depth 0, and a split keeps the depth of the chunk it copies. On a CPU-bound M.1-nano, compressing every spilled correction chunk was about 12% slower than not compressing ([CPU-279](https://linear.app/materializeinc/issue/CPU-279)). With the default floor of 1, this change compresses the chain-merge generations of a hydration snapshot, so it trades CPU for memory on such replicas. The floor stays tunable per replica. On top of #39159. Part of [CPU-279](https://linear.app/materializeinc/issue/CPU-279). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Columnserved two roles. On a dataflow edge it is built by pushing, crosses an exchange as channel bytes, and is read on arrival, which is what itsTyped,Bytes, andAlignvariants are for. Behind an edge, in a merge batcher's chains, a spill-backedColumnChunk, and a batch builder's seal, the same type held sorted, consolidated bodies that were only ever typed or, when read back from a store, serialized words. That code never constructedBytes, and it asserted the variant it expected at seven sites withunreachable!.ColumnBody<C>is the type at rest:Typed(C::Container)while a body is written,Words(Vec<u64>)once a store handed it back, and nothing else. Writing to a serialized body materializes it by a bulk per-leaf copy, so the merge, extract, advance, and settle bodies address the typed containers without matching on a variant.ColumnChunk,ColumnChunker,ColumnMerger,UnchunkBuilder,RowRowColPagedBuilder, and the correction buffer's chunks and builder now hold bodies. The edge container keeps its name and every operator that names it is untouched.The boundaries are moves. A chunker takes whatever the edge delivered and emits bodies, and a body goes back onto an edge through
From, typed staying typed and words becomingAlign. The column pager's batcher andValRowColPagedBuilderkeepColumnchains until the pager is retired, so the merge and extract bodies are generic over aMergeChunktrait both types implement, andPagedChunkerwraps the chunker for that batcher.Fifth in the stack, on top of #38897. The chunked consolidation from #38900 hands its bodies back onto the edge through
From. #38334 touches the same files and will need rebasing onto this, since edge paging lands onColumnalone, which is the point of the split.Part of CPU-279.
🤖 Generated with Claude Code