Skip to content

Overaggressive pruning of nested fields(?) breaks some valid queries unless parquet pushdown is disabled #27253

Description

@ianthetechie

Describe the bug

Valid queries against a parquet table with certain structural characteristics now fail with an execution error: Field common not found in struct.

As noted in the issue title, my guess is that this is due to overaggressive pruning of nested fields during pushdown, but that needs some further verification.

I have worked up a minimal reproducer (below) which you can replicate with the datafusion-cli.

To Reproduce

-- Set up: an external parquet table with a few rows.
-- Apologies for the somewhat oblique construction, but certain DataFusion versions
-- rejected a simpler version with map(['en'], [english]).
COPY (
    SELECT named_struct(
        'primary', local_name,
        'common', map([language], [english])
    ) AS names
    FROM (VALUES ('Local', 'en', 'English'), (NULL, 'en', 'Fallback'), ('', 'en', ''))
        AS input(local_name, language, english)
) TO '/tmp/datafusion-reproducer.parquet' STORED AS PARQUET;

CREATE EXTERNAL TABLE nested_names
STORED AS PARQUET
LOCATION '/tmp/datafusion-reproducer.parquet';

-- These are defaults for a while AFAIK but including for reproducibility.
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.optimizer.enable_leaf_expression_pushdown = true;

-- Expected:
-- +-----------------------------+
-- | nested_names.names[primary] |
-- +-----------------------------+
-- | Local                       |
-- | NULL                        |
-- +-----------------------------+
--
-- Actual:
-- Parquet error: External: External error: Error evaluating filter predicate
-- caused by
-- Execution error: Field common not found in struct
SELECT names.primary
FROM nested_names
WHERE coalesce(names.primary, names.common['en']) <> '';

SET datafusion.execution.parquet.pushdown_filters = false;
SET datafusion.optimizer.enable_leaf_expression_pushdown = false;

-- With BOTH of these settings disabled, it succeeds and returns Local and NULL.
-- It still fails if either one is true.
SELECT names.primary
FROM nested_names
WHERE coalesce(names.primary, names.common['en']) <> '';

-- Hacky SQL workaround: succeeds even with pushdown enabled.
SET datafusion.execution.parquet.pushdown_filters = true;
SET datafusion.optimizer.enable_leaf_expression_pushdown = true;

SELECT names.primary
FROM nested_names
WHERE coalesce(names.primary, map_extract(names.common, 'en')[1]) <> '';

Expected behavior

The original query should succeed as written with syntax like names.common['en'], without needing to disable predicate pushdown or SQL-level hacks.

Additional context

I haven't had a chance to do a full git bisect, but installing a few releases, I've narrowed it quite a bit. Release-wise, the bug first appeared in 54.0.0; 53.1.0 was fine, and it is still broken on main at the time of this writing.

Willingness to contribute

I would like to contribute a fix for this bug, but request guidance from the DataFusion community

Activity

  1. added theissue type on Nov 9, 2026
  2. ianthetechie commented on Nov 9, 2026

    @ianthetechie
    ContributorAuthor

    Ran a git bisect and identified #20854 as the breaking commit. I'll see if I can work up a minimal patch from here...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions