Skip to content

Window expressions lose FILTER, IGNORE NULLS and reversal state in physical plan protobuf serialization #27206

Description

@shehabgamin

Describe the bug

Window queries can return different results after a physical plan protobuf round trip. PhysicalWindowExprNodehas no field for some of the state a window expression carries, sodatafusion-proto` drops it:

  • the FILTER (WHERE ...) of a window aggregate (only the logical plan side was fixed, in Support null_treatment, distinct, and filter for window functions in proto #17417)
  • IGNORE NULLS on window functions such as lag, lead, first_value and nth_value (the encoder always writes false)
  • the reversed flag that the optimizer sets when it evaluates lag / lead / nth_value over reversed input
  • the ordering and reversed flag of an aggregate used as a window function (array_agg(x ORDER BY y) OVER () built through the API)

To Reproduce

use datafusion::arrow::util::pretty::pretty_format_batches;
use datafusion::physical_plan::collect;
use datafusion::prelude::SessionContext;
use datafusion_proto::bytes::{physical_plan_from_bytes, physical_plan_to_bytes};

#[tokio::main]
async fn main() -> datafusion::error::Result<()> {
    let ctx = SessionContext::new();
    let sql = "SELECT a, \
                 count(*) FILTER (WHERE a > 1) OVER () AS c, \
                 lag(b) IGNORE NULLS OVER (ORDER BY a DESC) AS l, \
                 lag(a, 1) OVER (ORDER BY a) AS r \
               FROM (VALUES (1, 10), (2, NULL), (3, 30), (4, NULL), (5, 50)) AS t(a, b) \
               ORDER BY a DESC";
    let plan = ctx.sql(sql).await?.create_physical_plan().await?;
    let bytes = physical_plan_to_bytes(plan)?;
    let plan = physical_plan_from_bytes(&bytes, &ctx.task_ctx())?;
    println!("{}", pretty_format_batches(&collect(plan, ctx.task_ctx()).await?)?);
    Ok(())
}

Before the round trip:

a | c | l  | r
5 | 4 |    | 4
4 | 4 | 50 | 3
3 | 4 | 50 | 2
2 | 4 | 30 | 1
1 | 4 | 30 |

After the round trip, c ignores the filter, l stops skipping nulls, and r returns the following row:

a | c | l  | r
5 | 5 |    |
4 | 5 | 50 | 5
3 | 5 |    | 4
2 | 5 | 30 | 3
1 | 5 |    | 2

Expected behavior

The decoded plan returns the same result as the original plan.

Additional context

Aggregates in AggregateExec (ordering, DISTINCT, IGNORE NULLS, FILTER, reversal, display names) already round-trip correctly. Only window expressions are affected.

Willingness to contribute

I plan to contribute a fix for this bug independently

Activity

  1. added theissue type on Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions