Repository navigation
pool: bound insertion debt during concurrent enforcement - #39374
Merged
Merged
Conversation
Reserve resident bytes before filling pool slots. If enforcement cannot make room within the bounded allowance, write the insertion directly to an extent and expose that fallback in pool metrics. This prevents an occupied single-flight enforcer from allowing an unbounded queue of resident insertions. The allowance, an eighth of the budget, also counts against the RSS target, so slots overshooting by it cannot push total RSS past the target. Add deterministic tests for a stalled enforcer and concurrent fills, including payload round trips and accounting after drops.
DAlperin
force-pushed
the
dov/pool-insertion-debt
branch
from
September 30, 2026 16:07
c227ba8 to
52adfc4
Compare
DAlperin
added a commit
that referenced
this pull request
Oct 1, 2026
…9042) ### Motivation Merged column chunks were emitted once they crossed 2 MiB rather than bounded to it, and the check ran every 1024 records, so wide rows routinely produced chunks well above the 2 MiB size class. Those chunks take 4 MiB huge-page slots in the buffer pool, halving the effective pool budget and doubling the bytes every eviction and read-back touches. This showed up on a production upsert hydration with about 2 KB rows and 30x more state than pool budget. ### Description - Settle and the chunker bound serialized column chunks to 2 MiB, splitting oversized output while preserving order and generational depth. Pieces are cut at the column's average record width, so uniform rows fill their size class and leave one short remainder to coalesce. A piece that still exceeds the bound, from uneven widths, is halved until it fits, which bounds the recursion depth by log2 of the record count. A single indivisible update may still exceed the bound. - Settle fills an open carry toward the bound with a prefix of the next chunk and returns the rest to the input, so a stream of chunks just over half the bound does not commit each into a half-empty slot. An oversized shared body is cut by borrow, copied once. - Merges check their output size at a cadence set by observed row width instead of every 1024 records, which keeps uniform rows within the ship size, and reserve output capacity only when both inputs fit within two ship-sized chunks. The pool's bound on insertion debt, previously part of this PR, is split out into #39374, below this one in the stack. ### Verification New unit tests cover the chunker, merge output before settle, settle byte bounds, depth preservation when a shared body is split and its last piece coalesces with a shallower neighbor, carry filling, indivisible oversized updates, and uneven record widths. Release note: None.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
When pool enforcement could not make room, concurrent insertions queued unbounded resident bytes behind a single-flight enforcer. On a production upsert hydration with about 2 KB rows and 30x more state than pool budget, the pool ledger reached 7.3x its budget and drove the container to its working-set limit.
Description
Reserve resident bytes before filling pool slots. If enforcement cannot make room within the bounded allowance, an eighth of the budget, encode the insertion directly to an extent on the inserting thread and expose that fallback as
mz_column_pool_direct_extent_inserts_total.Verification
New unit tests cover a stalled enforcer with concurrent fills, payload round trips through the fallback, accounting after drops, and a compressed tier left intact by unused slack. Benchmark results for this change and the chunk bounds in #39042, which were measured together, are in this comment.
Release note: None.
Generated with Claude Code