Repository navigation
timely-util: give each subsystem its own chunk spill gate - #39659
Merged
Merged
Conversation
ColumnChunk read one process-wide spill gate, the OR of a compute leg (enable_column_paged_batcher_spill) and a storage leg (enable_upsert_paged_spill). Compute and storage dataflows share a clusterd process and its worker threads, so either flag spilled every ColumnChunk in the process, whichever subsystem owned it. Once columnar consolidation moved onto ColumnChunk, enable_upsert_paged_spill alone spilled union and recursive consolidation through the buffer pool, leaving freed slots on the pool's warm list. The gate is now a zero-sized type parameter, ColumnChunk<D, T, R, G> with G: SpillGate. ComputeSpill reads compute's flag and StorageSpill reads storage's. Every consumer names its gate explicitly: compute's arrange, consolidate and temporal-bucket batchers use ComputeSpill, and the upsert-v2 chunked stash and feedback arrangement use StorageSpill. The gate is resolved at compile time, so commit reads one relaxed atomic instead of two. The MV sink correction buffer's gate is unchanged. The spill_gates_compose test becomes spill_gates_are_independent, which checks that each gate spills only its own chunks.
DAlperin
enabled auto-merge (squash)
October 7, 2026 18:00
antiguru
approved these changes
Oct 7, 2026
antiguru
left a comment
antiguru
left a comment
Member
There was a problem hiding this comment.
Thanks, this is fine. I hope we can remove this soon again!
This was referenced Oct 7, 2026
val-materialize
added a commit
that referenced
this pull request
Oct 9, 2026
Release notes for the v26.45 train, published as **v26.45.1**. v26.45.0 did not ship, so (as with v26.44) its notes are folded in here and [#39477](#39477) is closed unmerged. - **v26.45.0 content**: 8 Improvements, 2 Agent Skills, 8 Bug Fixes, copied verbatim from #39477 with anchors renamed to `{#v26.45.1-…}`. - **v26.45.1 content**: Alerting for Self-Managed and the external dependency monitoring Improvement (editor-added). The release's only PR, [#39659](#39659), is omitted. - **Dates**: Cloud 2026-10-08, Self-Managed 2026-10-09. The operator compatibility row is `v26.45.1`. `v26.45.md` (`patch: 1`, no `rc`) is already on `main`.⚠️ **Open questions** - **Group-sync bullet**: `oidc_group_role_sync_enabled` defaults to `false` at the tag. Is it on in Cloud? If not, cut the "Quieter logins" bullet. - **Certificate SAN requirements** ([#39316](#39316), [#39415](https://github.com/MaterializeInc/materialize/pull/39415)/[#39459](https://github.com/MaterializeInc/materialize/pull/39459)) need action at upgrade time but sit in Improvements. Should they be called out more loudly? Also, does [#39351](#39351) (S3 on rustls) impose the same SAN rule on custom S3 endpoints? - **[#39659](#39659) (omitted)**: is `enable_upsert_paged_spill` on in production? A draft bullet is in [omitted.md](https://github.com/MaterializeInc/mz-skills/blob/main/data/mz-release-notes/v26.45.1/final/omitted.md#borderline). - **`EXPLAIN ANALYZE ... CPU`**: [agent-skills#90](MaterializeInc/agent-skills#90) says it overstates joins, reductions, and TopK by 1.5x–3x before v26.46. Not published. Is that a known issue? - **Metric sinks**: [#39332](#39332), [#38910](#38910), and [#39337](#39337) are withheld because `enable_metric_sink` is `false`. #39337 fixes a panic any superuser can reach, so consider publishing it regardless. - **Possible cuts**: [#39362](#39362) was observed only in CI. The latency figures from [#39179](#39179) were measured on one noisy machine. - **Possible adds**: [#38894](https://github.com/MaterializeInc/materialize/pull/38894)–[#38896](https://github.com/MaterializeInc/materialize/pull/38896) (MV correction buffer) need an end-to-end number. [#39149](#39149) fixes the kind install guide, which crash-loops today. [#39549](#39549) (jemalloc page) is omitted because no released version had the regression. Snapshot drafts in mz-skills: [v26.45.0](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.0) (rc.1–rc.4, final), [v26.45.1](https://github.com/MaterializeInc/mz-skills/tree/main/data/mz-release-notes/v26.45.1/final). 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Valerie Chiang <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
ColumnChunkread one process-wide spill gate: the OR of a compute leg (enable_column_paged_batcher_spill) and a storage leg (enable_upsert_paged_spill). Compute and storage dataflows share aclusterdprocess and its worker threads, so either flag spilled everyColumnChunkin the process, whichever subsystem owned it.That became visible when columnar consolidation (
UnionConsolidation,LetRecConsolidation) moved ontoColumnChunk(#38900). In production,enable_upsert_paged_spillis on by default, so union and recursive consolidation started spilling through the buffer pool even though compute's spill flag is off. The freed slots stay on the pool's warm list, and that shows up asmz_column_pool_warm_bytesgrowth on the v26.45.0 canaries (CPU-280).It also means compute and storage spilling cannot be turned on or tested independently.
Change
The gate is now a zero-sized type parameter:
ColumnChunk<D, T, R, G>withG: SpillGate.ComputeSpillreads compute's flag, andStorageSpillreads storage's.ComputeSpill. The upsert-v2 chunked stash and feedback arrangement useStorageSpill.SpilledBody, so theResidentvariant and its match arms are unchanged.G::enabled()is resolved at compile time, so a commit reads one relaxed atomic where it used to read two.try_spill_ref) is unchanged.enable_upsert_paged_spillstill installs the buffer pool, but it now only spills upsert's chunks.Tests
spill_gates_composeinmz_timely_util::columnar::chunkis replaced byspill_gates_are_independent, which checks that each of the compute, storage, and sink gates spills only its own chunks. It fails ifComputeSpillreads the storage gate.Release notes
None. No user-visible change.
🤖 Generated with Claude Code