Repository navigation
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
The first parameter of array_filter and array_transform seems to not be correctly resolved if it is a lamba variable, instead, receive Null.
To Reproduce
use datafusion::logical_expr::{col, lit, lambda, lambda_var};
use datafusion::functions::expr_fn::*;
use datafusion::functions_nested::expr_fn::*;
use datafusion::functions_aggregate::expr_fn::*;
use futures::TryStreamExt;
let mut df = dataframe!(
"id" => vec![1, 2],
"name" => vec!["Alice", "Bob"],
)?;
let expr1 = array_transform(
col("grouped"),
lambda(["x"],
concat_ws(
lit("+"),
vec![
get_field(lambda_var("x"), "name"),
get_field(lambda_var("x"), "l"),
]
)
)
);
let expr2 = array_transform(
col("grouped"),
lambda(["x"],
concat_ws(
lit("+"),
vec![
get_field(lambda_var("x"), "name"),
array_filter(
get_field(lambda_var("x"), "l"),
lambda(["y"], lambda_var("y").gt_eq(lit(1)))
),
]
)
)
);
df = df
.with_column("l", make_array(vec![lit("0"), lit("1"), lit("2")]))?
.aggregate(
vec![lit("true")],
vec![array_agg(named_struct(vec![lit("id"), col("id"), lit("name"), col("name"), lit("l"), col("l")])).alias("grouped")],
)?
.with_column("test", expr1)?
// .with_column("test2", expr2)?
;
let plan = df.into_unoptimized_plan().resolve_lambda_variables()?.data;
let session = datafusion::execution::context::SessionContext::new();
let exec = session.state().create_physical_plan(&plan).await?;
let context = session.task_ctx();
let data: Vec<RecordBatch> = datafusion::physical_plan::execute_stream(exec, context)?.try_collect::<Vec<_>>().await?;
let format = datafusion::config::FormatOptions { types_info:true, ..Default::default() };
println!("{}", datafusion::arrow::util::pretty::pretty_format_batches_with_options(&data, &(&format).try_into()?)?);
With this code, if you uncomment the line .with_column("test2", expr2)? you will get following plan error:
Error: Plan("array_filter expected a list as first argument, got Null")
Perhaps I missed something...?
Thanks
Expected behavior
The expr2 should run normally and produce a column test2 similar but not same as comun 'test' in addition to the following schema
+--------------+------------------------------------------------------------------------+----------------------------------+
| Utf8("true") | grouped | test |
| Utf8 | List(Struct("id": Int32, "name": Utf8, "l": List(Utf8))) | List(Utf8) |
+--------------+------------------------------------------------------------------------+----------------------------------+
| true | [{id: 1, name: Alice, l: [0, 1, 2]}, {id: 2, name: Bob, l: [0, 1, 2]}] | [Alice+[0, 1, 2], Bob+[0, 1, 2]] |
+--------------+------------------------------------------------------------------------+----------------------------------+
Additional context
No response
Reacted by Kumar Ujjawal
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working
Activity
fbx31 commentedon Jun 21, 2026
@gstvg Many thanks for the job done and all other contributors. As promised when the PR was issued, I am using and testing extensively the current functions and in the future will try to implement array_fold and array_sort_by (if not already done by someone else).
But for now, facing this (tricky) issue.
gstvg commentedon Jun 22, 2026
Thanks @fbx31, this was indeed tricky to identify (I was expecting it to happen in more intricate parts of lambda support) but happily the fix was very simple, thanks for the report #23071
gstvg commentedon Jun 22, 2026
In the meantime, there are two ugly, temporarily fixes:
resolve_lambda_variablesdirectly in the expressions that need it :df = df .with_column("l", make_array(vec![lit("0"), lit("1"), lit("2")]))? .aggregate( vec![lit("true")], vec![array_agg(named_struct(vec![lit("id"), col("id"), lit("name"), col("name"), lit("l"), col("l")])).alias("grouped")], )?; let schema = df.schema().clone(); df = df.with_column("test", expr1.resolve_lambda_variables(&schema)?)?; let schema = df.schema().clone(); df = df..with_column("test2", expr2.resolve_lambda_variables(&schema)?)?; let plan = df.into_unoptimized_plan(); // no need to call resolve_lambda_variables herecoerce_types, other cases and all other methods can just invoke the existing wrapped functionHandle nulls in type coercion of higher-order UDFs, map_extract, spar…
[branch-54] Handle nulls in type coercion of higher-order UDFs, map_e…
gstvg commentedon Jul 22, 2026
@fbx31 this should be fixed in the latest patch release
54.1.0fbx31 commentedon Jul 22, 2026
Thanks @gstvg .
Btw, I am finalizing lambda sort_by function but still lot of work (tests and docs and sql queries, dataframe api ok) to issue a PR. I am still struggling also with signature for optional arguments with default values (sort direction and null first), didn't succeed to find solution with current lambda signature api. Perhaps an idea or an example?
Thanks
Handle nulls in type coercion of higher-order UDFs, map_extract, spar…