Skip to content

Commit 975ff81

Browse files
author
Ravel Fleet Executor
committed
docs(sql): describe CompactSlicesExec and the slice charge it removes
docs/query-engine.md gains a paragraph after the one on DataFusion 55.2's migrated aggregate streams: a slice of an aggregate's materialized output reports the whole batch's buffers, RepartitionExec and SortExec reserve that figure per buffered slice (apache/datafusion#24704), and the CompactSlices rule inserts a CompactSlicesExec above every aggregate whose output reaches one of them. It states where the exec sits on the 55.2 q29 plan, what its stream copies and what it passes through, its cost of one copy of each such aggregate output, the metrics that report it, which edge the integration test measures and which is pinned by placement alone, and that there is no configuration knob. changelog.d/2720.fixed.md records the operator-facing change: a grouped aggregate with many groups over more than one SQL partition no longer refuses with "the query memory pool is full and spill is disabled" on DataFusion 55.2, with the ClickBench q29 figures that found it. Refs: #2720 Signed-off-by: Ravel Fleet Executor <[email protected]>
1 parent 351ef09 commit 975ff81

2 files changed

Lines changed: 50 additions & 0 deletions

File tree

‎changelog.d/2720.fixed.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
- **A grouped aggregate with many groups over more than one SQL partition no
2+
longer refuses with "the query memory pool is full and spill is disabled" on
3+
DataFusion 55.2.** DataFusion 55.2 hands an aggregate's output out as slices
4+
of one materialized batch, and the repartition that fans that output back out
5+
to the SQL partitions charged every slice it buffered the size of the whole
6+
output (apache/datafusion#24704). ClickBench q29 over 100M log rows, which
7+
DataFusion 54.1 answered at a 1.63 GB query-pool peak at 32 partitions, was
8+
refused at every pool size up to 26 GB while the process held 12 GB; at 2, 4
9+
and 8 partitions its peak was 4x to 10x the 977 MB output. A new
10+
`CompactSlicesExec` now sits above every aggregate whose output feeds a
11+
repartition or a sort and gives each slice buffers of its own size, so the
12+
pool is charged the rows that are actually in flight. It costs one copy of
13+
that aggregate output, reported in `EXPLAIN ANALYZE` as the exec's
14+
`copied_bytes`, `copied_batches` and `passed_through_batches` metrics, and
15+
there is no flag to turn it off.

‎docs/query-engine.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1339,6 +1339,41 @@ Each keeps its materialized output reserved in the SQL query memory pool
13391339
it, and the pool forwards their shrinks to the query, tenant and process
13401340
budgets at once (`crates/ravel-sql/tests/aggregate_emit_reservation.rs`).
13411341

1342+
Those slices are charged a second time by whatever buffers them, and at the
1343+
wrong size. A slice shares the whole materialized batch's buffers, and
1344+
`get_array_memory_size` reports the buffers, not the rows, so every slice
1345+
reports the size of the entire aggregate output; `RepartitionExec` reserves
1346+
that figure for each slice it has buffered and a `SortExec` sizes its input
1347+
the same way (apache/datafusion#24704, open as of DataFusion 55.2). On
1348+
ClickBench q29 the charge grew with the number of slices in flight to an
1349+
exact multiple of the one output, and the repartition's fallback is to
1350+
spill, which a session without `--sql-spill-dir` refuses. Ravel's
1351+
`CompactSlices` physical optimizer rule (`crates/ravel-sql/src/compact_slices.rs`)
1352+
inserts a `CompactSlicesExec` directly above every `AggregateExec`, of any
1353+
mode, whose output reaches a `RepartitionExec`, a `SortExec` (a TopK
1354+
included) or a `SortPreservingMergeExec`, looking through a
1355+
`CoalescePartitionsExec`, `CoalesceBatchesExec`, `ProjectionExec` or
1356+
`FilterExec` between them; an aggregate under anything else, another
1357+
aggregate or the plan root included, gets nothing. On the 55.2 q29 plan that
1358+
is one exec, between `AggregateExec: mode=Final` and `RepartitionExec:
1359+
partitioning=RoundRobinBatch(32)`. The exec is a pass-through that keeps its
1360+
input's partitioning, ordering and distribution; its stream gives every
1361+
column whose buffers exceed twice its rows' bytes a copy sized to the batch
1362+
(`take` over the rows, or `gc` for a string view array) and hands every
1363+
other column on as it is, so an unsliced batch is not copied. The cost is
1364+
one copy of each aggregate output that feeds a repartition or a sort, made
1365+
in `batch_size` pieces as the slices are handed out, and reported per
1366+
partition in `EXPLAIN ANALYZE` as the exec's `copied_bytes`,
1367+
`copied_batches` and `passed_through_batches` metrics. The repartition edge
1368+
is the one measured: `crates/ravel-sql/tests/compact_slices.rs` runs the q29
1369+
shape over four partitions and asserts that, with the exec in the plan,
1370+
fanning the output out through the repartition adds less than one output's
1371+
worth to the query-pool peak of the same aggregate with nothing above it
1372+
(measured: nothing at all), and that with the exec stripped it adds at least
1373+
one output's worth (measured: more than five). The sort edge is pinned by
1374+
placement only (the module's own tests), not by a measured peak. The rule is on in
1375+
every session and has no configuration knob.
1376+
13421377
`AggregateExec::execute_typed` still falls back to the legacy
13431378
`GroupedHashAggregateStream` for every shape it has not migrated: a
13441379
single-stage aggregate that carries a limit, such as `SELECT DISTINCT ...

0 commit comments

Comments
 (0)