Skip to content

Commit 6574a8c

Browse files
ShayanGhoAI assistants
andauthored
fix(substrait): consume chained window functions whose default names collide (#25181)
## Which issue does this PR close? - Closes #23007. ## Rationale for this change A logical plan that contains two window aggregates with the same default schema name, distinguished only by aliases, executes fine in DataFusion but cannot be consumed back after a Substrait round trip. The consumer fails with: ``` SchemaError(DuplicateUnqualifiedField { name: "avg(data.b) PARTITION BY [data.a] ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW" }) ``` DataFusion's Substrait producer uses positional projections and does not preserve intermediate window aliases: a `ProjectRel` carries expressions and an output mapping. The aliases that kept the two columns apart in DataFusion (`AS avg1`, `AS avg2`) are dropped on the wire. When the consumer rebuilds the outer projection, the inherited window column and the new window expression both come back under the same default name, and `Window::try_new` rejects the plan. The SQL from the issue reproduces it, and so does a minimal plan built with `LogicalPlanBuilder` that stacks two `row_number()` windows. ## What changes are included in this PR? Two small changes, one per crate: - **`datafusion-expr`**: `group_window_expr_by_sort_keys` (used by `LogicalPlanBuilder::window_plan`) now looks through a single `Expr::Alias` to derive the sort key and keeps the aliased expression in its group, so `window_plan` can build a `Window` node whose output field carries an alias. `LogicalPlanBuilder::window`, filter pushdown, and the physical planner already handle aliased window expressions, so this makes `window_plan` consistent with them. Nested aliases are still rejected, matching what filter pushdown tolerates. - **`datafusion-substrait` consumer**: `from_project_rel` reserves the input schema's field names in a `NameTracker` (new `reserve_schema` helper), aliases any new window expression whose default name would collide, and rewrites the projection to reference the alias while preserving the projection's own output names via `NamePreserver`. Plans without a collision are built exactly as before. ## What is the testing strategy for this PR? - `datafusion-expr`: a unit test (`test_group_window_expr_by_sort_keys_aliased_window_expr`) asserting that an aliased window expression is grouped by the inner function's sort key and kept aliased. It failed on `main` with `Impossibly got non-window expr`. - `datafusion-substrait`: two round-trip tests in `roundtrip_logical_plan.rs`, `stacked_windows_with_same_default_name_via_builder` (minimal `LogicalPlanBuilder` form) and `chained_windows_with_same_default_name` (the SQL from the issue). Both failed before this change with the `DuplicateUnqualifiedField` error above. They assert Arrow schema equality (names, types, nullability) and execute the consumed plan. They do not compare plan text, because the consumer must synthesize an alias for the colliding window, so the text legitimately differs. They compare Arrow schemas rather than full `DFSchema`s because qualifiers and inferred functional dependencies can change during reconstruction: DataFusion's Substrait producer does not preserve subquery aliases, and the rebuilt `Window` nodes infer dependencies the original plan did not record. - The full `datafusion-substrait` test run passes unchanged (273 passed, 6 ignored across the lib, integration, and doc tests), confirming the non-colliding path is unaffected. This change was developed with AI assistance. I have reviewed and understand every line and stand behind it. ## Are there any user-facing changes? No API changes. Plans consumed from Substrait that previously failed with `DuplicateUnqualifiedField` now consume successfully. Synthesized aliases of the form `<default name>__temp__N` may appear in the decoded logical plan; final output column names are preserved. Plans without such a collision are unchanged. --------- Co-authored-by: AI assistants <[email protected]>
1 parent 1e09a2a commit 6574a8c

4 files changed

Lines changed: 213 additions & 17 deletions

File tree

‎datafusion/expr/src/utils.rs‎

Lines changed: 58 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -616,28 +616,45 @@ pub fn compare_sort_expr(
616616
Ordering::Equal
617617
}
618618

619-
/// Group a slice of window expression expr by their order by expressions
619+
/// Group window expressions by their sort keys, preserving any outer alias.
620620
pub fn group_window_expr_by_sort_keys(
621621
window_expr: impl IntoIterator<Item = Expr>,
622622
) -> Result<Vec<(WindowSortKey, Vec<Expr>)>> {
623623
let mut result = vec![];
624-
window_expr.into_iter().try_for_each(|expr| match &expr {
625-
Expr::WindowFunction(window_fun) => {
626-
let WindowFunctionParams{ partition_by, order_by, ..} = &window_fun.as_ref().params;
627-
let sort_key = generate_sort_key(partition_by, order_by)?;
628-
if let Some((_, values)) = result.iter_mut().find(
629-
|group: &&mut (WindowSortKey, Vec<Expr>)| matches!(group, (key, _) if *key == sort_key),
630-
) {
631-
values.push(expr);
632-
} else {
633-
result.push((sort_key, vec![expr]))
634-
}
635-
Ok(())
624+
625+
window_expr.into_iter().try_for_each(|expr| {
626+
// Read the window's settings through one alias.
627+
// Keep `expr` intact so its output name is preserved.
628+
let inner = match &expr {
629+
Expr::Alias(alias) => alias.expr.as_ref(),
630+
_ => &expr,
631+
};
632+
633+
let Expr::WindowFunction(window_fun) = inner else {
634+
return internal_err!("Impossibly got non-window expr {expr:?}");
635+
};
636+
637+
let WindowFunctionParams {
638+
partition_by,
639+
order_by,
640+
..
641+
} = &window_fun.as_ref().params;
642+
643+
let sort_key = generate_sort_key(partition_by, order_by)?;
644+
645+
if let Some((_, values)) = result.iter_mut().find(
646+
|group: &&mut (WindowSortKey, Vec<Expr>)| {
647+
matches!(group, (key, _) if *key == sort_key)
648+
},
649+
) {
650+
values.push(expr);
651+
} else {
652+
result.push((sort_key, vec![expr]));
636653
}
637-
other => internal_err!(
638-
"Impossibly got non-window expr {other:?}"
639-
),
654+
655+
Ok(())
640656
})?;
657+
641658
Ok(result)
642659
}
643660

@@ -1567,6 +1584,31 @@ mod tests {
15671584
Ok(())
15681585
}
15691586

1587+
#[test]
1588+
fn test_group_window_expr_by_sort_keys_aliased_window_expr() -> Result<()> {
1589+
let age_asc = Sort::new(col("age"), true, true);
1590+
let max1 = Expr::from(WindowFunction::new(
1591+
WindowFunctionDefinition::AggregateUDF(max_udaf()),
1592+
vec![col("name")],
1593+
))
1594+
.order_by(vec![age_asc.clone()])
1595+
.build()
1596+
.unwrap();
1597+
// The same window function under an alias, as the Substrait consumer
1598+
// produces when a window column's default name collides with an
1599+
// input column. It must be grouped by the inner function's sort key
1600+
// and kept aliased.
1601+
let max1_aliased = max1.clone().alias("max_name");
1602+
1603+
let result =
1604+
group_window_expr_by_sort_keys(vec![max1.clone(), max1_aliased.clone()])?;
1605+
1606+
let key = vec![(age_asc, false)];
1607+
let expected: Vec<(WindowSortKey, Vec<Expr>)> =
1608+
vec![(key, vec![max1, max1_aliased])];
1609+
assert_eq!(expected, result);
1610+
Ok(())
1611+
}
15701612
#[test]
15711613
fn test_group_window_expr_by_sort_keys() -> Result<()> {
15721614
let age_asc = Sort::new(col("age"), true, true);

‎datafusion/substrait/src/logical_plan/consumer/rel/project_rel.rs‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,8 +18,10 @@
1818
use crate::logical_plan::consumer::SubstraitConsumer;
1919
use crate::logical_plan::consumer::utils::NameTracker;
2020
use async_recursion::async_recursion;
21+
use datafusion::common::tree_node::{Transformed, TreeNode, TreeNodeRecursion};
2122
use datafusion::common::{Column, not_impl_err};
2223
use datafusion::logical_expr::builder::project;
24+
use datafusion::logical_expr::expr_rewriter::NamePreserver;
2325
use datafusion::logical_expr::utils::find_window_exprs;
2426
use datafusion::logical_expr::{Expr, LogicalPlan, LogicalPlanBuilder};
2527
use std::collections::HashSet;
@@ -65,6 +67,37 @@ pub async fn from_project_rel(
6567
}
6668

6769
let input = if !window_exprs.is_empty() {
70+
// Window outputs must have unique names across the input schema
71+
// and the new window expressions.
72+
let mut window_names = NameTracker::new();
73+
window_names.reserve_schema(&original_schema);
74+
75+
let mut aliased_columns: Vec<(Expr, Expr)> = vec![];
76+
let window_exprs = window_exprs
77+
.into_iter()
78+
.map(|window_expr| {
79+
let named =
80+
window_names.get_uniquely_named_expr(window_expr.clone())?;
81+
82+
if let Expr::Alias(alias) = &named {
83+
aliased_columns.push((
84+
window_expr,
85+
Expr::Column(Column::from_name(&alias.name)),
86+
));
87+
}
88+
89+
Ok(named)
90+
})
91+
.collect::<datafusion::common::Result<Vec<_>>>()?;
92+
93+
// References to renamed windows must point to their new output columns.
94+
if !aliased_columns.is_empty() {
95+
explicit_exprs = explicit_exprs
96+
.into_iter()
97+
.map(|expr| reference_aliased_windows(expr, &aliased_columns))
98+
.collect::<datafusion::common::Result<Vec<_>>>()?;
99+
}
100+
68101
LogicalPlanBuilder::window_plan(input, window_exprs)?
69102
} else {
70103
input
@@ -81,3 +114,30 @@ pub async fn from_project_rel(
81114
not_impl_err!("Projection without an input is not supported")
82115
}
83116
}
117+
118+
/// Reference renamed window outputs while preserving the projection's
119+
/// original output name.
120+
fn reference_aliased_windows(
121+
expr: Expr,
122+
aliased_columns: &[(Expr, Expr)],
123+
) -> datafusion::common::Result<Expr> {
124+
let saved_name = NamePreserver::new_for_projection().save(&expr);
125+
126+
let rewritten = expr
127+
.transform_down(|node| {
128+
match aliased_columns
129+
.iter()
130+
.find(|(window_expr, _)| *window_expr == node)
131+
{
132+
Some((_, column)) => Ok(Transformed::new(
133+
column.clone(),
134+
true,
135+
TreeNodeRecursion::Jump,
136+
)),
137+
None => Ok(Transformed::no(node)),
138+
}
139+
})?
140+
.data;
141+
142+
Ok(saved_name.restore(rewritten))
143+
}

‎datafusion/substrait/src/logical_plan/consumer/utils.rs‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
use crate::logical_plan::consumer::SubstraitConsumer;
1919
use datafusion::arrow::datatypes::{DataType, Field, Schema, TimeUnit, UnionFields};
2020
use datafusion::common::{
21-
DFSchema, DFSchemaRef, TableReference, exec_err, not_impl_err,
21+
Column, DFSchema, DFSchemaRef, TableReference, exec_err, not_impl_err,
2222
substrait_datafusion_err, substrait_err,
2323
};
2424
use datafusion::logical_expr::expr::Sort;
@@ -424,6 +424,12 @@ impl NameTracker {
424424
}
425425
}
426426

427+
pub(super) fn reserve_schema(&mut self, schema: &DFSchema) {
428+
for (qualifier, field) in schema.iter() {
429+
self.insert(&Expr::Column(Column::from((qualifier, field))));
430+
}
431+
}
432+
427433
/// Check if the expression would cause a conflict either in:
428434
/// 1. validate_unique_names (duplicate schema_name)
429435
/// 2. DFSchema::check_names (ambiguous reference)

‎datafusion/substrait/tests/cases/roundtrip_logical_plan.rs‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1684,6 +1684,94 @@ async fn simple_window_function() -> Result<()> {
16841684
roundtrip("SELECT RANK() OVER (PARTITION BY a ORDER BY b), d, sum(b) OVER (PARTITION BY a) FROM data;").await
16851685
}
16861686

1687+
#[tokio::test]
1688+
async fn stacked_windows_with_same_default_name_via_builder() -> Result<()> {
1689+
// Substrait projections are positional and drop DataFusion's aliases, so the
1690+
// inherited `rn1` column and the new window both come back under the same
1691+
// default name. The consumer must keep them apart.
1692+
// See https://github.com/apache/datafusion/issues/23007
1693+
use datafusion::arrow::util::pretty::pretty_format_batches;
1694+
use datafusion::functions_window::expr_fn::row_number;
1695+
use datafusion::logical_expr::ExprFunctionExt;
1696+
1697+
let ctx = create_context().await?;
1698+
let scan = ctx.table("data").await?.into_optimized_plan()?;
1699+
let row_number_by_a = || {
1700+
row_number()
1701+
.order_by(vec![col("a").sort(true, false)])
1702+
.build()
1703+
};
1704+
// The filter between the two windows makes `rn1` and `rn2` differ for the
1705+
// surviving row. A consumer that resolved `rn2` to the inherited window
1706+
// column would then return wrong values instead of passing unnoticed.
1707+
let plan = LogicalPlanBuilder::from(scan)
1708+
.window(vec![row_number_by_a()?.alias("rn1")])?
1709+
.filter(col("a").gt(lit(1i64)))?
1710+
.window(vec![row_number_by_a()?.alias("rn2")])?
1711+
.build()?;
1712+
1713+
let plan2 = substrait_roundtrip(&plan, &ctx).await?;
1714+
// Compare output fields; qualifiers and functional dependencies
1715+
// can differ after a Substrait round trip.
1716+
assert_eq!(plan.schema().as_arrow(), plan2.schema().as_arrow());
1717+
1718+
let expected = DataFrame::new(ctx.state(), plan)
1719+
.select_columns(&["a", "rn1", "rn2"])?
1720+
.collect()
1721+
.await?;
1722+
let actual = DataFrame::new(ctx.state(), plan2)
1723+
.select_columns(&["a", "rn1", "rn2"])?
1724+
.collect()
1725+
.await?;
1726+
let actual = pretty_format_batches(&actual)?.to_string();
1727+
assert_eq!(pretty_format_batches(&expected)?.to_string(), actual);
1728+
assert_snapshot!(
1729+
actual,
1730+
@r"
1731+
+---+-----+-----+
1732+
| a | rn1 | rn2 |
1733+
+---+-----+-----+
1734+
| 3 | 2 | 1 |
1735+
+---+-----+-----+
1736+
"
1737+
);
1738+
Ok(())
1739+
}
1740+
1741+
#[tokio::test]
1742+
async fn chained_windows_with_same_default_name() -> Result<()> {
1743+
// SQL form from https://github.com/apache/datafusion/issues/23007. The two
1744+
// `avg` windows differ only by aliases that Substrait does not carry. As in
1745+
// roundtrip_self_join, the consumer must synthesize an alias, so the plan
1746+
// text differs; verify schema and executability instead.
1747+
let ctx = create_context().await?;
1748+
let plan = ctx
1749+
.sql(
1750+
"SELECT a, b, avg1, avg2 FROM (
1751+
SELECT a, b, avg1,
1752+
row_number() OVER () AS seq2,
1753+
avg(b) OVER (PARTITION BY a ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS avg2
1754+
FROM (
1755+
SELECT a, b, avg1 FROM (
1756+
SELECT a, b,
1757+
row_number() OVER () AS seq1,
1758+
avg(b) OVER (PARTITION BY a ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS avg1
1759+
FROM data
1760+
) t1 ORDER BY seq1
1761+
) t2
1762+
) t3 ORDER BY seq2",
1763+
)
1764+
.await?
1765+
.into_optimized_plan()?;
1766+
1767+
let plan2 = substrait_roundtrip(&plan, &ctx).await?;
1768+
// Compare output fields; qualifiers and functional dependencies
1769+
// can differ after a Substrait round trip.
1770+
assert_eq!(plan.schema().as_arrow(), plan2.schema().as_arrow());
1771+
DataFrame::new(ctx.state(), plan2).show().await?;
1772+
Ok(())
1773+
}
1774+
16871775
#[tokio::test]
16881776
async fn window_with_rows() -> Result<()> {
16891777
roundtrip("SELECT sum(b) OVER (PARTITION BY a ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) FROM data;").await?;

0 commit comments

Comments
 (0)