Skip to content

Commit 351ef09

Browse files
author
Ravel Fleet Executor
committed
fix(sql): size aggregate output slices before a repartition charges them
DataFusion 55.2's migrated aggregate streams materialize every group into one output batch and hand it downstream as batch_size slices of it. A slice shares the whole batch's buffers and get_array_memory_size reports the buffers, not the rows, so RepartitionExec reserves the size of the entire aggregate output for every slice it buffers, and SortExec sizes its input the same way (apache/datafusion#24704). On ClickBench q29 over 100M log rows the query-pool peak at 2, 4 and 8 partitions was 4x to 10x the 977,218,657-byte output, and at 32 partitions the statement was refused with "the query memory pool is full and spill is disabled" at every pool size up to 26 GB, where DataFusion 54.1 answered it at a 1.63 GB peak. Add the CompactSlices physical optimizer rule. It inserts a CompactSlicesExec directly above every AggregateExec, of any mode, whose output reaches a RepartitionExec, a SortExec or a SortPreservingMergeExec, looking through a CoalescePartitionsExec, CoalesceBatchesExec, ProjectionExec or FilterExec between them, and above nothing else. The exec is a one-child pass-through that keeps its input's partitioning, ordering and distribution. Its stream copies every column whose buffers exceed twice its rows' bytes into buffers sized to the batch, with take over the row range or gc for a string view array, and hands every other column on untouched, so an unsliced batch costs no copy. arrow::compute::concat over a single array returns a slice of it in Arrow 59, so it is not used. The exec reports copied_bytes, copied_batches and passed_through_batches metrics per partition. The rule is registered last in build_session, after ConfirmTopKThreshold: BoundedTopKAggregate, AttrsPerKeyProjection and MetadataOnlyAggregate each walk the aggregate-to-sort or aggregate-to-scan chain through an allowlist of pass-through nodes the new exec is not on, so it runs after they have seen the shapes they were written against. Tests: the module pins the 55.2 q29 plan shape getting exactly one exec between AggregateExec: mode=Final and RepartitionExec: partitioning=RoundRobinBatch(32), the sort edge through HAVING getting one, a plan without an aggregate and an aggregate under a root projection staying unchanged, a 10,000-row batch sliced into 100 pieces coming out with each piece's buffers within twice its rows' bytes, an unsliced batch passing through with the same column Arcs, and a real plan's answer matching the unrewritten one with the metrics reporting the copies. tests/compact_slices.rs runs the q29 shape over a 20,000-group metrics corpus at four SQL partitions and a 256-row batch_size, asserts the round-robin repartition sits above the final aggregate, and asserts that fanning the output out through the repartition adds less than one output's worth to the peak of the same aggregate with nothing above it when the exec is in the plan (measured: nothing) and at least one output's worth with the exec stripped (measured: more than five), with the same answer either way. The brief asked for a bound of three times the output; that bound is not reachable here because a two-stage aggregate over string keys peaks at about four times its output with no repartition in the plan at all, so the test pins the repartition's own contribution instead. A second test drives the statement through SqlExecutor and asserts the exec's placement in its plan. Refs: #2720 Signed-off-by: Ravel Fleet Executor <[email protected]>
1 parent 432940c commit 351ef09

5 files changed

Lines changed: 1310 additions & 0 deletions

File tree

0 commit comments

Comments
 (0)