Skip to content

Bug in lambda variable resolution #23068

Closed

Description

@fbx31

Describe the bug

The first parameter of array_filter and array_transform seems to not be correctly resolved if it is a lamba variable, instead, receive Null.

To Reproduce

    use datafusion::logical_expr::{col, lit, lambda, lambda_var};
    use  datafusion::functions::expr_fn::*;
    use  datafusion::functions_nested::expr_fn::*;
    use datafusion::functions_aggregate::expr_fn::*;
    use futures::TryStreamExt;

    let mut df = dataframe!(
        "id" => vec![1, 2],
        "name" => vec!["Alice", "Bob"],
    )?;

    let expr1 = array_transform(
        col("grouped"),
        lambda(["x"],
        concat_ws(
            lit("+"), 
            vec![
                    get_field(lambda_var("x"), "name"),
                    get_field(lambda_var("x"), "l"),
                ]
            )
        )
    );
    let expr2 = array_transform(
        col("grouped"),
        lambda(["x"],
        concat_ws(
            lit("+"), 
            vec![
                    get_field(lambda_var("x"), "name"),
                    array_filter(
                        get_field(lambda_var("x"), "l"),
                        lambda(["y"], lambda_var("y").gt_eq(lit(1)))
                    ), 
                ]
            )
        )
    );

    df = df
        .with_column("l", make_array(vec![lit("0"), lit("1"), lit("2")]))?
        .aggregate(
                vec![lit("true")],
                vec![array_agg(named_struct(vec![lit("id"), col("id"), lit("name"), col("name"), lit("l"), col("l")])).alias("grouped")],
            )?
        .with_column("test", expr1)?
        // .with_column("test2", expr2)?
        ;

    let plan = df.into_unoptimized_plan().resolve_lambda_variables()?.data;
    let session =  datafusion::execution::context::SessionContext::new();
    let exec = session.state().create_physical_plan(&plan).await?;
    let context = session.task_ctx();
    let data: Vec<RecordBatch> = datafusion::physical_plan::execute_stream(exec, context)?.try_collect::<Vec<_>>().await?;
    let format = datafusion::config::FormatOptions { types_info:true, ..Default::default() };
    println!("{}", datafusion::arrow::util::pretty::pretty_format_batches_with_options(&data, &(&format).try_into()?)?);

With this code, if you uncomment the line .with_column("test2", expr2)? you will get following plan error:
Error: Plan("array_filter expected a list as first argument, got Null")

Perhaps I missed something...?

Thanks

Expected behavior

The expr2 should run normally and produce a column test2 similar but not same as comun 'test' in addition to the following schema

+--------------+------------------------------------------------------------------------+----------------------------------+
| Utf8("true") | grouped                                                                | test                             |
| Utf8         | List(Struct("id": Int32, "name": Utf8, "l": List(Utf8)))               | List(Utf8)                       |
+--------------+------------------------------------------------------------------------+----------------------------------+
| true         | [{id: 1, name: Alice, l: [0, 1, 2]}, {id: 2, name: Bob, l: [0, 1, 2]}] | [Alice+[0, 1, 2], Bob+[0, 1, 2]] |
+--------------+------------------------------------------------------------------------+----------------------------------+

Additional context

No response

Activity

  1. added theissue type on Jun 21, 2026
  2. fbx31 commented on Jun 21, 2026

    @fbx31
    Author

    @gstvg Many thanks for the job done and all other contributors. As promised when the PR was issued, I am using and testing extensively the current functions and in the future will try to implement array_fold and array_sort_by (if not already done by someone else).
    But for now, facing this (tricky) issue.

  3. gstvg commented on Jun 22, 2026

    @gstvg
    Contributor

    Thanks @fbx31, this was indeed tricky to identify (I was expecting it to happen in more intricate parts of lambda support) but happily the fix was very simple, thanks for the report #23071

  4. gstvg commented on Jun 22, 2026

    @gstvg
    Contributor

    In the meantime, there are two ugly, temporarily fixes:

    • Call resolve_lambda_variables directly in the expressions that need it :
        df = df
            .with_column("l", make_array(vec![lit("0"), lit("1"), lit("2")]))?
            .aggregate(
                    vec![lit("true")],
                    vec![array_agg(named_struct(vec![lit("id"), col("id"), lit("name"), col("name"), lit("l"), col("l")])).alias("grouped")],
                )?;
    
        let schema = df.schema().clone();
        df = df.with_column("test", expr1.resolve_lambda_variables(&schema)?)?;
    
        let schema = df.schema().clone();
        df = df..with_column("test2", expr2.resolve_lambda_variables(&schema)?)?;
    
        let plan = df.into_unoptimized_plan(); // no need to call resolve_lambda_variables here
  5. added a commit that references this issue on Jul 17, 2026
    fb8fe7a
  6. added a commit that references this issue on Jul 18, 2026
    fcbc9bd
  7. gstvg commented on Jul 22, 2026

    @gstvg
    Contributor

    @fbx31 this should be fixed in the latest patch release 54.1.0

  8. fbx31 commented on Jul 22, 2026

    @fbx31
    Author

    Thanks @gstvg .
    Btw, I am finalizing lambda sort_by function but still lot of work (tests and docs and sql queries, dataframe api ok) to issue a PR. I am still struggling also with signature for optional arguments with default values (sort direction and null first), didn't succeed to find solution with current lambda signature api. Perhaps an idea or an example?
    Thanks

  9. added a commit that references this issue on Aug 12, 2026
    b1fb639
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions