Repository navigation
Commit 7791e93
Ravel Fleet Executor
fix(sql): size aggregate output slices before a repartition charges them
DataFusion 55.2's migrated aggregate streams materialize every group into
one output batch and hand it downstream as batch_size slices of it. A slice
shares the whole batch's buffers and get_array_memory_size reports the
buffers, not the rows, so RepartitionExec reserves the size of the entire
aggregate output for every slice it buffers, and SortExec sizes its input
the same way (apache/datafusion#24704). On ClickBench q29 over 100M log
rows the query-pool peak at 2, 4 and 8 partitions was 4x to 10x the
977,218,657-byte output, and at 32 partitions the statement was refused
with "the query memory pool is full and spill is disabled" at every pool
size up to 26 GB, where DataFusion 54.1 answered it at a 1.63 GB peak.
Add the CompactSlices physical optimizer rule. It inserts a
CompactSlicesExec directly above every AggregateExec, of any mode, whose
output reaches a RepartitionExec, a SortExec or a SortPreservingMergeExec,
looking through a CoalescePartitionsExec, CoalesceBatchesExec,
ProjectionExec or FilterExec between them, and above nothing else. The exec
is a one-child pass-through that keeps its input's partitioning, ordering
and distribution. Its stream copies every column whose buffers exceed
twice its rows' bytes into buffers sized to the batch, with take over the
row range or gc for a string view array, and hands every other column on
untouched, so an unsliced batch costs no copy. arrow::compute::concat over
a single array returns a slice of it in Arrow 59, so it is not used. The
exec reports copied_bytes, copied_batches and passed_through_batches
metrics per partition.
The rule is registered last in build_session, after ConfirmTopKThreshold:
BoundedTopKAggregate, AttrsPerKeyProjection and MetadataOnlyAggregate each
walk the aggregate-to-sort or aggregate-to-scan chain through an allowlist
of pass-through nodes the new exec is not on, so it runs after they have
seen the shapes they were written against.
Tests: the module pins the 55.2 q29 plan shape getting exactly one exec
between AggregateExec: mode=Final and RepartitionExec:
partitioning=RoundRobinBatch(32), the sort edge through HAVING getting one,
a plan without an aggregate and an aggregate under a root projection
staying unchanged, a 10,000-row batch sliced into 100 pieces coming out
with each piece's buffers within twice its rows' bytes, an unsliced batch
passing through with the same column Arcs, and a real plan's answer
matching the unrewritten one with the metrics reporting the copies.
tests/compact_slices.rs runs the q29 shape over a 20,000-group metrics
corpus at four SQL partitions and a 256-row batch_size, asserts the
round-robin repartition sits above the final aggregate, and asserts that
fanning the output out through the repartition adds less than one output's
worth to the peak of the same aggregate with nothing above it when the
exec is in the plan (measured: nothing) and at least one output's worth
with the exec stripped (measured: more than five), with the same answer
either way. The brief asked for a bound of three times the output; that
bound is not reachable here because a two-stage aggregate over string keys
peaks at about four times its output with no repartition in the plan at
all, so the test pins the repartition's own contribution instead. A second
test drives the statement through SqlExecutor and asserts the exec's
placement in its plan.
Refs: #2720
Signed-off-by: Ravel Fleet Executor <[email protected]>1 parent 432940c commit 7791e93
5 files changed
Lines changed: 1370 additions & 0 deletions
0 commit comments