You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Join dynamic filter pushdown through AggregateExec can confuse same-named grouping columns #25264
Enabling join dynamic filter pushdown can silently discard a matching row when the filter passes through an AggregateExec that groups on two columns with the same name (here a.id and b.id from a nested join).
With the same data and SQL, disabling datafusion.optimizer.enable_join_dynamic_filter_pushdown returns one row; enabling it returns zero rows, without an error.
To Reproduce
Run the following SQL in a fresh datafusion-cli session. The two /tmp/ Parquet paths must not already exist.
SETdatafusion.optimizer.join_reordering = false;
COPY (SELECT'a1'AS id, 'x1'AS ty)
TO '/tmp/df_dynamic_filter_aggregate_a.parquet'
STORED AS PARQUET;
COPY (SELECT'x1'AS id)
TO '/tmp/df_dynamic_filter_aggregate_b.parquet'
STORED AS PARQUET;
CREATE EXTERNAL TABLE ta
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_a.parquet';
CREATE EXTERNAL TABLE tb
STORED AS PARQUET
LOCATION '/tmp/df_dynamic_filter_aggregate_b.parquet';
-- Returns one row: (x1, a1, x1).SETdatafusion.optimizer.enable_join_dynamic_filter_pushdown = false;
SELECTs.id, a.id, b.idFROM tb s
JOIN (
SELECTa.id, b.idFROM ta a LEFT JOIN tb b ONa.ty=b.idGROUP BYa.id, b.id
)
ONs.id=b.id;
-- Incorrectly returns zero rows.SETdatafusion.optimizer.enable_join_dynamic_filter_pushdown = true;
SELECTs.id, a.id, b.idFROM tb s
JOIN (
SELECTa.id, b.idFROM ta a LEFT JOIN tb b ONa.ty=b.idGROUP BYa.id, b.id
)
ONs.id=b.id;
Expected behavior
No response
Additional context
AggregateExec resolves pushed-down filter columns by name instead of by grouping position, so the filter on the grouping column b.id is applied to a.id. Found while working on #25244; the fix is included in #25259.
Activity
fix: map pushed-down filter columns by position instead of by name (a…
fix: map pushed-down filter columns by position instead of by name (a…
fix: map pushed-down filter columns by position instead of by name (a…
fix: map pushed-down filter columns by position instead of by name (a…