Skip to content

Unnested fields are not filterable when using subqueries. #16695

Closed

Description

@hmadison

Describe the bug

When attempting to evaluate a logical plan which mixes un-nesting and sub-queries, an unfiltered call to show will return the full set of columns, but attempting to filter on any column returned yields the following error:

Error: SchemaError(FieldNotFound { field: Column { relation: None, name: "__unnest_placeholder(one.__unnest_placeholder(mock.metadata).product).name" }, valid_fields: [Column { relation: Some(Bare { table: "mock" }), name: "metadata" }] }, Some(""))

To Reproduce

With nested.ndjson as:

{"metadata": {"product": {"name": "Product Name"}}}

The following test case reproduces the error:

use datafusion::common::DataFusionError;
use datafusion::prelude::{NdJsonReadOptions, SessionContext};

#[tokio::test]
async fn test_bug() -> Result<(), DataFusionError> {
    let ctx = SessionContext::new();
    ctx.register_json(
        "mock",
        "tests/fixtures/nested.ndjson",
        NdJsonReadOptions::default().file_extension("ndjson")
    ).await?;

    let query = r#"
    WITH one AS (SELECT unnest(metadata) FROM mock),
         two AS (SELECT unnest("__unnest_placeholder(mock.metadata).product") FROM one)
    SELECT * FROM two WHERE "__unnest_placeholder(one.__unnest_placeholder(mock.metadata).product).name" == 'Product Name'
    "#;

    let frame = ctx.sql(query).await?;
    frame.show().await?;

    Ok(())
}

Expected behavior

I would expect that the query completes and applies the filter.

Additional context

No response

Activity

  1. akoshchiy commented on Jul 14, 2025

    @akoshchiy
    Contributor

    take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions