Skip to content

Commit 523abfd

Browse files
committed
feat(optimizer): coalesce peer first_value / last_value into a single struct aggregate
Add the CoalesceFirstLast logical optimizer rule, gated behind optimizer.enable_coalesce_first_last (default off). Peer first_value / last_value aggregates that share an ORDER BY key are rewritten into one struct-valued aggregate plus a get_field projection, so the input is scanned once instead of once per expression and one per-group state slot is kept instead of N. Skips DISTINCT / FILTER / IGNORE NULLS, expressions without an ORDER BY, and grouping-set aggregates, and no-ops when named_struct / get_field are not registered. Closes apache#23602. Part of apache#23600; the struct fast path depends on apache#23628.
1 parent f151c10 commit 523abfd

7 files changed

Lines changed: 614 additions & 0 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/common/src/config.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1390,6 +1390,14 @@ config_namespace! {
13901390
/// cause regressions in both memory usage and runtime.
13911391
pub enable_window_topn: bool, default = false
13921392

1393+
/// When set to true, coalesce peer `first_value` / `last_value`
1394+
/// expressions that share the same `ORDER BY` key — e.g.
1395+
/// `first_value(a ORDER BY o), first_value(b ORDER BY o), ... GROUP BY p`
1396+
/// — into a single `first_value(named_struct(a, b, ...) ORDER BY o)`,
1397+
/// cutting N argmax scans to one pass over the input.
1398+
/// Particularly beneficial for wide payloads.
1399+
pub enable_coalesce_first_last: bool, default = false
1400+
13931401
/// When set to true, the optimizer will push TopK (Sort with fetch)
13941402
/// below hash repartition when the partition key is a prefix of the
13951403
/// sort key, reducing data volume before the shuffle.

‎datafusion/optimizer/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ regex-syntax = "0.8.9"
6969
async-trait = { workspace = true }
7070
criterion = { workspace = true }
7171
ctor = { workspace = true }
72+
datafusion-functions = { workspace = true }
7273
datafusion-functions-aggregate = { workspace = true }
7374
datafusion-functions-window = { workspace = true }
7475
datafusion-functions-window-common = { workspace = true }

0 commit comments

Comments
 (0)