Repository navigation
Commit a8eb3c8
committed
fix(sql): keep the aggregate hold for DataFusion 55.2's legacy stream
The previous commit dropped the pool's aggregate hold on the premise that
DataFusion 55 reserves an aggregate's emitted output itself. That holds for
the migrated aggregate streams only. AggregateExec::execute_typed in 55.2
still falls back to the legacy GroupedHashAggregateStream for a
single-stage aggregate with a limit, a single-stage aggregate over ordered
input, the partial stage of a grouping set, and a partial-reduce stage
whenever the pool is finite, which Ravel's always is. The legacy stream
still shrinks its reservation at emit while the emitted batch holds the
bytes, so the hold is restored as it was before that commit.
A second acceptance test in aggregate_emit_reservation.rs runs a
single-stage SELECT DISTINCT ... LIMIT with one SQL partition, asserts the
plan is AggregateExec mode=Single with a limit, and reads the pool between
the first and second next(). With the hold it read 4,609,312 bytes reserved
against a first batch of 3,014,768, 4,543,472 of them held by the pool.
With HASH_AGGREGATE_CONSUMER_PREFIX set to a name no consumer carries, it
read 65,840 reserved against the same batch and failed. The existing test
on the default GROUP BY keeps passing with nothing held (3,539,120 reserved
against 3,539,120).
The memory_accounting hold tests are back, moved to the legacy
single-stage DISTINCT LIMIT with sql_partition_count = 1: at the default
eight partitions the statement plans Partial plus Final, both migrated,
and the pool holds nothing.
A ROLLUP is not held on 55.2 at any partition count: its partial stage runs
on the legacy stream but emits early and is spillable, and its final stage
is migrated. Measured once, 3,940,656 bytes reserved against a 3,875,024
byte first batch, nothing held. The docs say so rather than list grouping
sets among the held shapes.
docs/query-engine.md, the ADR-0102 and ADR-1170 amendments, the hold's doc
comments and both changelog fragments now say the hold covers only the
legacy stream and becomes removable once execute_typed no longer falls
back (apache/datafusion#25902, under apache/datafusion#22710). The
group_by_reservation_lag figures are marked as DataFusion 54.1
measurements; that matrix was not re-run on 55.2.
Refs: #2720
Signed-off-by: Panagiotis Moustafellos <[email protected]>1 parent eef3d62 commit a8eb3c8
10 files changed
Lines changed: 875 additions & 145 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
2 | | - | |
3 | | - | |
4 | | - | |
5 | | - | |
6 | | - | |
7 | | - | |
8 | | - | |
9 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
2 | 2 | | |
3 | 3 | | |
4 | 4 | | |
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
14 | | - | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
180 | | - | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
181 | 184 | | |
182 | 185 | | |
183 | 186 | | |
| |||
0 commit comments