You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Window expressions lose FILTER, IGNORE NULLS and reversal state in physical plan protobuf serialization #27206
Window queries can return different results after a physical plan protobuf round trip. PhysicalWindowExprNodehas no field for some of the state a window expression carries, sodatafusion-proto` drops it:
IGNORE NULLS on window functions such as lag, lead, first_value and nth_value (the encoder always writes false)
the reversed flag that the optimizer sets when it evaluates lag / lead / nth_value over reversed input
the ordering and reversed flag of an aggregate used as a window function (array_agg(x ORDER BY y) OVER () built through the API)
To Reproduce
use datafusion::arrow::util::pretty::pretty_format_batches;use datafusion::physical_plan::collect;use datafusion::prelude::SessionContext;use datafusion_proto::bytes::{physical_plan_from_bytes, physical_plan_to_bytes};#[tokio::main]asyncfnmain() -> datafusion::error::Result<()>{let ctx = SessionContext::new();let sql = "SELECT a, \ count(*) FILTER (WHERE a > 1) OVER () AS c, \ lag(b) IGNORE NULLS OVER (ORDER BY a DESC) AS l, \ lag(a, 1) OVER (ORDER BY a) AS r \ FROM (VALUES (1, 10), (2, NULL), (3, 30), (4, NULL), (5, 50)) AS t(a, b) \ ORDER BY a DESC";let plan = ctx.sql(sql).await?.create_physical_plan().await?;let bytes = physical_plan_to_bytes(plan)?;let plan = physical_plan_from_bytes(&bytes,&ctx.task_ctx())?;println!("{}", pretty_format_batches(&collect(plan, ctx.task_ctx()).await?)?);Ok(())}
Before the round trip:
a | c | l | r
5 | 4 | | 4
4 | 4 | 50 | 3
3 | 4 | 50 | 2
2 | 4 | 30 | 1
1 | 4 | 30 |
After the round trip, c ignores the filter, l stops skipping nulls, and r returns the following row:
a | c | l | r
5 | 5 | |
4 | 5 | 50 | 5
3 | 5 | | 4
2 | 5 | 30 | 3
1 | 5 | | 2
Expected behavior
The decoded plan returns the same result as the original plan.
Additional context
Aggregates in AggregateExec (ordering, DISTINCT, IGNORE NULLS, FILTER, reversal, display names) already round-trip correctly. Only window expressions are affected.
Willingness to contribute
I plan to contribute a fix for this bug independently
Activity