Repository navigation
Commit b376290
authored
fix: map pushed-down filter columns by position instead of by name (apache#25259)
## Which issue does this PR close?
- Closes apache#25244.
- Closes apache#25262.
- Closes apache#25263.
- Closes apache#25264.
- Closes apache#21246.
- Closes apache#25296.
## Rationale for this change
Enabling `datafusion.optimizer.enable_join_dynamic_filter_pushdown` can
silently drop matching rows when the probe side of a join contains
several columns with the same name, for example `a.id` and `b.id` from a
nested join. Physical filter pushdown resolved a pushed filter's columns
in the child schema by name, so a predicate on the second `id` column
was rewritten to the first `id` column, which holds a different value.
The same name-based mapping also affects TopK dynamic filters under the
default configuration (apache#25296). For a nested join of orders and payments
that both expose `amount`, `ORDER BY p.amount LIMIT 1` can push the
payment threshold onto `orders.amount`. Later matching orders are
incorrectly pruned, returning `(100, 30)` instead of `(300, 10)`.
Every operator that forwards parent filters had the same weakness, in
slightly different forms:
- `HashJoinExec` computed which output columns belong to each side by
position, but then resolved the child column by name.
- `FilterExec`, `SortExec`, `RepartitionExec`, `CoalesceBatchesExec`,
`UnionExec` and similar nodes used the generic name lookup even though
their output positions equal their input positions.
- `FilterExec` with an embedded projection resolved unconsumed parent
filters back into input coordinates by name.
- `ProjectionExec` looked up output aliases by name to find the
expression to substitute, so two outputs aliased `id` both mapped to the
first.
- `AggregateExec` restricted pushdown to grouping positions but resolved
the input column by name.
The join-filter regression queries return one row with dynamic filtering
disabled and zero rows with it enabled across these shapes. The
`FilterExec`, `ProjectionExec` and `AggregateExec` cases were found
while auditing the remaining name-based paths for apache#25244; they share the
root cause, so this PR fixes them together. The TopK case in apache#25296 is
fixed by the same positional mapping.
## What changes are included in this PR?
Built-in operators now remap filter columns by position. The deprecated
public API retains its historical name-based resolution for
compatibility.
- `FilterRemapper` uses a `ColumnMapping` enum: `Identity` preserves
positions and checks that names match; `Explicit` uses a caller-supplied
parent-output to child-input mapping and allows names to differ.
- `ChildFilterDescription::from_child` now maps by position. New
`ChildFilterDescription::from_child_with_column_mapping` takes an
explicit `HashMap<usize, usize>`.
- `HashJoinExec` builds the explicit mapping from its `column_indices`
and output projection. For semi joins, output join keys on the emitted
side are mapped to the paired key on the other side, which also supports
differently named keys.
- `FilterExec` maps through its embedded projection in both pushdown
phases and when folding unconsumed parent filters back into its
predicate.
- `ProjectionExec` substitutes the expression at each output position
instead of looking the alias up in the output schema.
- `AggregateExec` maps each grouping output position to the input column
that grouping expression reads; non-column grouping expressions are not
forwarded, as before.
- `ChildFilterDescription::from_child_with_allowed_indices` is kept as a
deprecated wrapper that preserves name-based resolution to the first
matching child field. It translates allowed parent column references
into an explicit positional mapping; new callers should supply positions
directly to avoid ambiguous duplicate names.
## What is the testing strategy for this PR?
- `dynamic_filter_pushdown_config.slt` gains four regression queries
over two small Parquet tables, run once with join dynamic filtering
disabled and once enabled, asserting identical rows: nested joins with
`RepartitionExec` between them (the query from apache#25244), a `FilterExec`
with an embedded projection, a `ProjectionExec` with duplicate aliases,
and an `AggregateExec` grouping on same-named columns. All four return
zero rows on `main` with dynamic filtering enabled.
- `dynamic_filter_pushdown_config.slt` also covers apache#25296 using nested
joins of orders, payments and customers, with one row per batch and one
row per orders Parquet row group. The query returns `(300, 10)` with
TopK dynamic filtering disabled, enabled, and enabled together with
Parquet filter pushdown. On `main` at `85d4cbb0a9`, both enabled cases
reproduce the incorrect `(100, 30)` result; all three pass on this
branch.
- `datafusion/core/tests/physical_optimizer/filter_pushdown.rs` gains
focused tests that call `gather_filters_for_pushdown` directly on
`RepartitionExec`, `HashJoinExec` (duplicate child columns with and
without projection, both semi join directions with differently named
keys, and a semi join whose key is not a plain column), `FilterExec`
with a projection in both phases, `ProjectionExec` with duplicate
aliases, and `AggregateExec` with reordered same-named grouping columns.
Further tests cover the deprecated `from_child_with_allowed_indices`
wrapper preserving the first name match, accepting allowed parent
indices outside the child schema, and rejecting unresolvable names, the
identity mapping rejecting a column whose name differs from the child
field at that position, and `FilterExec` rejecting an unconsumed parent
filter outside its projection.
Passed locally on the updated implementation:
- `cargo fmt --all`
- `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`
- `cargo test --profile ci -p datafusion --test core_integration
physical_optimizer::filter_pushdown` — 70 tests passed.
- `cargo test --profile ci --test sqllogictests --
dynamic_filter_pushdown_config.slt` — passed.
- The isolated apache#25296 regression fails on `main` in both TopK-enabled
configurations with the expected wrong-result mismatch.
## Are there any user-facing changes?
Queries that push join or TopK dynamic filters through operators with
duplicate column names now return the correct rows, including the
default-configuration TopK query in apache#25296.
API change in `datafusion-physical-plan`:
`ChildFilterDescription::from_child_with_allowed_indices` is deprecated
in favour of `from_child` and the new `from_child_with_column_mapping`;
the deprecated function preserves its previous name-based behavior.
Callers should migrate to explicit positions because duplicate names
make name resolution ambiguous. `ChildFilterDescription::from_child`
also resolves by position, which only affects callers that used it on a
node whose output positions differ from its child's.1 parent ec97d1c commit b376290
8 files changed
Lines changed: 942 additions & 168 deletions
File tree
- datafusion
- core/tests/physical_optimizer
- physical-plan/src
- aggregates
- joins/hash_join
- sqllogictest/test_files
- docs/source/library-user-guide/upgrading
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
178 | 178 | | |
179 | 179 | | |
180 | 180 | | |
181 | | - | |
| 181 | + | |
182 | 182 | | |
183 | 183 | | |
184 | 184 | | |
| |||
2205 | 2205 | | |
2206 | 2206 | | |
2207 | 2207 | | |
2208 | | - | |
2209 | | - | |
2210 | | - | |
2211 | | - | |
2212 | | - | |
2213 | | - | |
2214 | | - | |
| 2208 | + | |
| 2209 | + | |
| 2210 | + | |
| 2211 | + | |
| 2212 | + | |
| 2213 | + | |
| 2214 | + | |
| 2215 | + | |
| 2216 | + | |
| 2217 | + | |
| 2218 | + | |
| 2219 | + | |
| 2220 | + | |
| 2221 | + | |
| 2222 | + | |
| 2223 | + | |
| 2224 | + | |
| 2225 | + | |
| 2226 | + | |
| 2227 | + | |
| 2228 | + | |
2215 | 2229 | | |
2216 | 2230 | | |
2217 | 2231 | | |
| |||
2227 | 2241 | | |
2228 | 2242 | | |
2229 | 2243 | | |
2230 | | - | |
| 2244 | + | |
2231 | 2245 | | |
2232 | | - | |
| 2246 | + | |
2233 | 2247 | | |
2234 | 2248 | | |
2235 | 2249 | | |
| |||
3175 | 3189 | | |
3176 | 3190 | | |
3177 | 3191 | | |
| 3192 | + | |
3178 | 3193 | | |
3179 | 3194 | | |
3180 | 3195 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
| 36 | + | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| |||
57 | 57 | | |
58 | 58 | | |
59 | 59 | | |
60 | | - | |
| 60 | + | |
| 61 | + | |
61 | 62 | | |
62 | 63 | | |
63 | 64 | | |
| |||
66 | 67 | | |
67 | 68 | | |
68 | 69 | | |
69 | | - | |
| 70 | + | |
70 | 71 | | |
71 | 72 | | |
72 | 73 | | |
| |||
316 | 317 | | |
317 | 318 | | |
318 | 319 | | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
319 | 343 | | |
320 | 344 | | |
321 | 345 | | |
| |||
697 | 721 | | |
698 | 722 | | |
699 | 723 | | |
700 | | - | |
701 | | - | |
702 | | - | |
703 | | - | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
704 | 734 | | |
705 | | - | |
706 | | - | |
707 | | - | |
708 | | - | |
709 | | - | |
710 | | - | |
711 | | - | |
712 | | - | |
713 | | - | |
714 | | - | |
| 735 | + | |
715 | 736 | | |
716 | 737 | | |
717 | 738 | | |
| |||
735 | 756 | | |
736 | 757 | | |
737 | 758 | | |
738 | | - | |
739 | | - | |
740 | | - | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
741 | 769 | | |
742 | 770 | | |
743 | | - | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
744 | 778 | | |
745 | 779 | | |
746 | 780 | | |
| |||
0 commit comments