Skip to content

Commit f0c8c0c

Browse files
author
Ravel Fleet Executor
committed
fix(sql): keep the aggregate hold for DataFusion 55.2's legacy stream
The previous commit dropped the pool's aggregate hold on the premise that DataFusion 55 reserves an aggregate's emitted output itself. That holds for the migrated aggregate streams only. AggregateExec::execute_typed in 55.2 still falls back to the legacy GroupedHashAggregateStream for a single-stage aggregate with a limit, a single-stage aggregate over ordered input, the partial stage of a grouping set, and a partial-reduce stage whenever the pool is finite, which Ravel's always is. The legacy stream still shrinks its reservation at emit while the emitted batch holds the bytes, so the hold is restored as it was before that commit. A second acceptance test in aggregate_emit_reservation.rs runs a single-stage SELECT DISTINCT ... LIMIT with one SQL partition, asserts the plan is AggregateExec mode=Single with a limit, and reads the pool between the first and second next(). With the hold it read 4,609,312 bytes reserved against a first batch of 3,014,768, 4,543,472 of them held by the pool. With HASH_AGGREGATE_CONSUMER_PREFIX set to a name no consumer carries, it read 65,840 reserved against the same batch and failed. The existing test on the default GROUP BY keeps passing with nothing held (3,539,120 reserved against 3,539,120). The memory_accounting hold tests are back, moved to the legacy single-stage DISTINCT LIMIT with sql_partition_count = 1: at the default eight partitions the statement plans Partial plus Final, both migrated, and the pool holds nothing. A ROLLUP is not held on 55.2 at any partition count: its partial stage runs on the legacy stream but emits early and is spillable, and its final stage is migrated. Measured once, 3,940,656 bytes reserved against a 3,875,024 byte first batch, nothing held. The docs say so rather than list grouping sets among the held shapes. docs/query-engine.md, the ADR-0102 and ADR-1170 amendments, the hold's doc comments and both changelog fragments now say the hold covers only the legacy stream and becomes removable once execute_typed no longer falls back (apache/datafusion#25902, under apache/datafusion#22710). The group_by_reservation_lag figures are marked as DataFusion 54.1 measurements; that matrix was not re-run on 55.2. Refs: #2720 Signed-off-by: Ravel Fleet Executor <[email protected]>
1 parent 22d83a7 commit f0c8c0c

10 files changed

Lines changed: 875 additions & 145 deletions

‎changelog.d/2633-3.fixed.md‎

Lines changed: 22 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,22 @@
1-
- **A grouped aggregate's emitted output stays charged to the SQL memory
2-
pool while it is handed out** (issues #2633, #2720). DataFusion 54.1 shrank
3-
a grouped hash aggregate's reservation when it emitted, while the emitted
4-
batch still held those bytes, and the pool handed them to other queries at
5-
once. DataFusion 55.2's aggregate keeps the emitted batch reserved until
6-
its last slice is handed out, so those bytes count toward
7-
`--sql-max-query-bytes` and the tenant and process budgets until then. A
8-
shape DataFusion 55 still runs on its legacy aggregate, such as a
9-
single-stage `SELECT DISTINCT ... LIMIT`, still releases them early.
1+
- **The SQL memory pool keeps a legacy grouped aggregate's released bytes
2+
charged until its stream ends** (issues #2633, #2720). DataFusion's legacy
3+
grouped hash aggregate shrinks its reservation when it emits, while the
4+
emitted batch still holds those bytes, and the pool used to hand them to
5+
other queries at once. The pool now holds them on the query, tenant and
6+
process budgets until the aggregate's stream is dropped. DataFusion 55.2
7+
runs that legacy aggregate only for the shapes it has not moved to its new
8+
aggregate streams, for example a single-stage aggregate with a limit (such
9+
as `SELECT DISTINCT ... LIMIT` under `--sql-partition-count 1`) or over
10+
ordered input. The new streams keep their emitted output reserved
11+
themselves and are not held. The
12+
hold is chosen per stream: it applies to a legacy aggregate DataFusion
13+
marks as unable to spill, and never to a partial aggregate or to a final
14+
aggregate that can spill to disk. While a downstream operator, for example
15+
a Sort, reserves the emitted batches it is handed, the same bytes are
16+
charged twice, so a statement that fits under `--sql-max-query-bytes`
17+
without the hold can be refused with 422 `ResourcesExhausted`. The extra
18+
charge is bounded by the aggregate's own peak reservation. The sizing
19+
guidance for `--sql-max-query-bytes` is unchanged: a `GROUP BY` matrix
20+
measured on DataFusion 54.1 showed no peak change beyond run-to-run spread,
21+
though a statement that reserves the aggregate's output downstream was not
22+
measured.

‎changelog.d/2720.changed.md‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
- **SQL runs on DataFusion 55.2 and Arrow/Parquet 59.3** (issue #2720). Five
1+
- **SQL runs on DataFusion 55.2 and Arrow/Parquet 59.3** (issue #2720). Six
22
behaviors change. `GROUP BY` puts `-0.0` and `0.0` in one group and
33
`count(DISTINCT)` counts them once, because DataFusion 55 canonicalizes
44
negative zero in group keys; NaN payloads still group apart. The new
@@ -11,4 +11,9 @@
1111
the spill scratch quota names the bytes the spill files held when the
1212
refusal arrived, which is now below the quota and can be zero, because
1313
DataFusion 55 refuses the write before it lands. Flight SQL `GetSqlInfo`
14-
reports Arrow version 59.
14+
reports Arrow version 59. Most grouped aggregates now run on DataFusion
15+
55's new aggregate streams, which keep their emitted output reserved until
16+
the last slice of it is handed out and release it then, so the SQL memory
17+
pool's aggregate hold (issue #2633) no longer applies to them; it still
18+
covers the shapes DataFusion 55.2 runs on its legacy aggregate, such as a
19+
single-stage `SELECT DISTINCT ... LIMIT`.

‎crates/ravel-sql/src/lib.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -177,7 +177,10 @@ pub use logs_schema::{
177177
logs_schema, logs_schema_with_declared,
178178
};
179179
pub use logs_udf::{HAS_WORD_UDF, has_word_udf};
180-
pub use memory::{CeilingBreach, TenantDelegatingPool, TenantMemoryAccountant};
180+
pub use memory::{
181+
CeilingBreach, HASH_AGGREGATE_CONSUMER_PREFIX, TenantDelegatingPool, TenantMemoryAccountant,
182+
sql_memory_held_bytes,
183+
};
181184
pub use metadata_agg::{METADATA_ONLY_AGGREGATE_RULE, MetadataOnlyAggregate, MetadataOnlyExec};
182185
pub use output::QueryOutput;
183186
pub use page_plan::{

0 commit comments

Comments
 (0)