Skip to content

Commit f9e2eb2

Browse files
committed
feat(optimizer): coalesce peer first_value / last_value into a single struct aggregate
Add the CoalesceFirstLast logical optimizer rule, gated behind optimizer.enable_coalesce_first_last (default off). Peer first_value / last_value aggregates that share an ORDER BY key are rewritten into one struct-valued aggregate plus a get_field projection, so the input is scanned once instead of once per expression and one per-group state slot is kept instead of N. Skips DISTINCT / FILTER / IGNORE NULLS, expressions without an ORDER BY, and grouping-set aggregates, and no-ops when named_struct / get_field are not registered. Registers the config and rule in the generated artifacts that enumerate every setting and optimizer rule: configs.md, information_schema.slt, optimizer_rule_reference.md, and the EXPLAIN VERBOSE rule list in explain.slt. Closes apache#23602. Part of apache#23600; the struct fast path depends on apache#23628.
1 parent 18b1e35 commit f9e2eb2

11 files changed

Lines changed: 626 additions & 4 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion/common/src/config.rs‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1390,6 +1390,14 @@ config_namespace! {
13901390
/// cause regressions in both memory usage and runtime.
13911391
pub enable_window_topn: bool, default = false
13921392

1393+
/// When set to true, coalesce peer `first_value` / `last_value`
1394+
/// expressions that share the same `ORDER BY` key — e.g.
1395+
/// `first_value(a ORDER BY o), first_value(b ORDER BY o), ... GROUP BY p`
1396+
/// — into a single `first_value(named_struct(a, b, ...) ORDER BY o)`,
1397+
/// cutting N argmax scans to one pass over the input.
1398+
/// Particularly beneficial for wide payloads.
1399+
pub enable_coalesce_first_last: bool, default = false
1400+
13931401
/// When set to true, the optimizer will push TopK (Sort with fetch)
13941402
/// below hash repartition when the partition key is a prefix of the
13951403
/// sort key, reducing data volume before the shuffle.

‎datafusion/core/src/optimizer_rule_reference.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -58,10 +58,11 @@ Rule order matters. The default pipeline may change between releases.
5858
| 19 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |
5959
| 20 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |
6060
| 21 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |
61-
| 22 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
62-
| 23 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
63-
| 24 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
64-
| 25 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |
61+
| 22 | `coalesce_first_last` | Coalesces peer `first_value` / `last_value` aggregates that share an `ORDER BY` into a single struct aggregate. |
62+
| 23 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |
63+
| 24 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |
64+
| 25 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |
65+
| 26 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |
6566

6667
### Physical Optimizer Rules
6768

‎datafusion/optimizer/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@ regex-syntax = "0.8.9"
6969
async-trait = { workspace = true }
7070
criterion = { workspace = true }
7171
ctor = { workspace = true }
72+
datafusion-functions = { workspace = true }
7273
datafusion-functions-aggregate = { workspace = true }
7374
datafusion-functions-window = { workspace = true }
7475
datafusion-functions-window-common = { workspace = true }

0 commit comments

Comments
 (0)