Skip to content

Commit b7becd9

Browse files
adriangbclaude
authored andcommitted
fix: derive scan equivalences only from filters the source applies exactly (apache#25780)
## Which issue does this PR close? - Closes apache#25779. ## Rationale for this change With `datafusion.execution.parquet.pushdown_filters = false` (the default), a Parquet scan claimed equivalence properties from a predicate that it uses only for pruning. An order-preserving repartition directly on the scan then merged on the wrong keys, and `ORDER BY` and `ORDER BY ... LIMIT` returned wrong results. The issue has the reproducer. ## What changes are included in this PR? | Change | File | | --- | --- | | New `FileSource::exact_filter()`: the part of `filter()` that every output row satisfies. Default `None`. | `datasource/src/file.rs` | | `FileScanConfig::eq_properties` uses `exact_filter()` instead of `filter()`. `filter()` is unchanged, because `statistics()` still needs it to mark row counts inexact after pruning. | `datasource/src/file_scan_config/mod.rs` | | `ParquetSource::exact_filter()` returns `None` when `pushdown_filters` is off. When it is on, it returns only the conjuncts that pass `can_expr_be_pushed_down_with_schemas`, the same test `try_pushdown_filters` uses before it replies `PushedDown::Yes`. | `datasource-parquet/src/source.rs` | | Upgrade guide entry for custom `FileSource` implementations. | `upgrading/56.0.0.md` | The behavior from apache#16686 / apache#16563 is kept: with `pushdown_filters = true` the scan still reports the equivalences, and the existing plans do not change. ## What is the testing strategy for this PR? - New `sqllogictest` cases in `push_down_filter_parquet.slt`: one `EXPLAIN` (the repartition now merges on `a, b`) and four result checks (`ORDER BY`, `ORDER BY ... LIMIT`, `GROUP BY ... ORDER BY`, and the `a = b` form). Without the fix, all 5 fail. - `cte.slt`: one expected plan changes. The scan no longer claims that `k` is constant from its pruning-only `k = 2` predicate, so it reports its real `[k]` ordering, and the single-input `RepartitionExec` now shows `maintains_sort_order=true`. - Differential check against DuckDB: 14 query shapes (order by, top-k, group by, window, hash join with dynamic filter, sort-merge join, `DISTINCT ON`, `first_value`, `a = b`) × 18 configurations (`pushdown_filters`, `prefer_existing_sort`, `repartition_file_scans`, `target_partitions`, `prefer_hash_join`). 15 failures before this change, 0 after. ## Are there any user-facing changes? Queries return correct results. A custom `FileSource` that applies its filter to every row must implement `exact_filter` to keep the equivalence properties of its scan (see the upgrade guide). This is a new trait method with a default, so it is not a breaking change at compile time. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
1 parent eb9854d commit b7becd9

6 files changed

Lines changed: 173 additions & 2 deletions

File tree

‎datafusion/datasource-parquet/src/source.rs‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,7 @@ use datafusion_datasource::file_scan_config::FileScanConfig;
5050
use datafusion_functions::core::file_row_index::FileRowIndexFunc;
5151
use datafusion_physical_expr::expressions::{Column, DynamicFilterTracking};
5252
use datafusion_physical_expr::projection::ProjectionExprs;
53+
use datafusion_physical_expr::utils::split_conjunction;
5354
use datafusion_physical_expr::{EquivalenceProperties, conjunction};
5455
use datafusion_physical_expr_adapter::DefaultPhysicalExprAdapterFactory;
5556
use datafusion_physical_expr_adapter::rewrite::{
@@ -702,6 +703,23 @@ impl FileSource for ParquetSource {
702703
self.predicate.clone()
703704
}
704705

706+
/// The predicate is applied to every row only when filter pushdown is
707+
/// enabled, and then only the conjuncts that can become a `RowFilter`.
708+
/// Otherwise the predicate is used only for pruning.
709+
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
710+
if !self.pushdown_filters() {
711+
return None;
712+
}
713+
let predicate = self.predicate.as_ref()?;
714+
let pushable_schema = self.table_schema.schema_without_virtual_columns();
715+
let exact = split_conjunction(predicate)
716+
.into_iter()
717+
.filter(|expr| can_expr_be_pushed_down_with_schemas(expr, pushable_schema))
718+
.cloned()
719+
.collect::<Vec<_>>();
720+
(!exact.is_empty()).then(|| conjunction(exact))
721+
}
722+
705723
fn with_batch_size(&self, batch_size: usize) -> Arc<dyn FileSource> {
706724
let mut conf = self.clone();
707725
conf.batch_size = Some(batch_size);

‎datafusion/datasource/src/file.rs‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,21 @@ pub trait FileSource: Any + Send + Sync {
114114
None
115115
}
116116

117+
/// Returns the part of [`Self::filter`] that every output row is
118+
/// guaranteed to satisfy.
119+
///
120+
/// [`FileScanConfig`] derives equivalence properties (constants and equal
121+
/// columns) from this filter. A filter that the source uses only to skip
122+
/// data (for example, row group or page pruning with statistics) does not
123+
/// remove every non-matching row, so it must not be returned here.
124+
///
125+
/// Returns `None` by default.
126+
///
127+
/// [`FileScanConfig`]: crate::file_scan_config::FileScanConfig
128+
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
129+
None
130+
}
131+
117132
/// Return the projection that will be applied to the output stream on top
118133
/// of [`Self::table_schema`].
119134
///

‎datafusion/datasource/src/file_scan_config/mod.rs‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -910,7 +910,9 @@ impl DataSource for FileScanConfig {
910910
)
911911
.with_constraints(self.constraints.clone());
912912

913-
if let Some(filter) = self.file_source.filter() {
913+
// Only a filter that the source applies exactly gives valid equivalences.
914+
// A filter that is used only for pruning lets non-matching rows through.
915+
if let Some(filter) = self.file_source.exact_filter() {
914916
// We need to remap column indexes to match the projected schema since that's what the equivalence properties deal with.
915917
// Note that this will *ignore* any non-projected columns: these don't factor into ordering / equivalence.
916918
match Self::add_filter_equivalence_info(&filter, &mut eq_properties, schema) {

‎datafusion/sqllogictest/test_files/cte.slt‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1700,7 +1700,7 @@ physical_plan
17001700
08)------------RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
17011701
09)--------------WorkTableExec: name=r
17021702
10)--FilterExec: k@0 = 2, projection=[v2@1]
1703-
11)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1
1703+
11)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=1, maintains_sort_order=true
17041704
12)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/cte/test.parquet]]}, projection=[k, v2], output_ordering=[k@0 ASC NULLS LAST], file_type=parquet, predicate=k@0 = 2, pruning_predicate=k_null_count@2 != row_count@3 AND k_min@0 <= 2 AND 2 <= k_max@1, required_guarantees=[k in (2)]
17051705

17061706
query II

‎datafusion/sqllogictest/test_files/push_down_filter_parquet.slt‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1336,3 +1336,111 @@ RESET datafusion.execution.parquet.pushdown_filters;
13361336

13371337
statement ok
13381338
drop table t;
1339+
1340+
# A predicate that the Parquet scan keeps only for pruning
1341+
# (pushdown_filters = false) must not make the scan claim equivalences.
1342+
# Each file is sorted by (a, b). If the scan claims that `a` is constant,
1343+
# the order-preserving repartition below the FilterExec merges on `b` only
1344+
# and the rows with a = 5 come out in the wrong order.
1345+
1346+
statement ok
1347+
COPY (SELECT * FROM (VALUES (3, 9), (5, 1)) v(a, b))
1348+
TO 'test_files/scratch/push_down_filter_parquet/pruning_only_eq/f1.parquet'
1349+
STORED AS PARQUET;
1350+
1351+
statement ok
1352+
COPY (SELECT * FROM (VALUES (5, 2)) v(a, b))
1353+
TO 'test_files/scratch/push_down_filter_parquet/pruning_only_eq/f2.parquet'
1354+
STORED AS PARQUET;
1355+
1356+
statement ok
1357+
CREATE EXTERNAL TABLE pruning_only_eq (a INT, b INT)
1358+
STORED AS PARQUET
1359+
LOCATION 'test_files/scratch/push_down_filter_parquet/pruning_only_eq/'
1360+
WITH ORDER (a ASC, b ASC);
1361+
1362+
statement ok
1363+
set datafusion.optimizer.repartition_file_scans = false;
1364+
1365+
statement ok
1366+
set datafusion.optimizer.prefer_existing_sort = true;
1367+
1368+
statement ok
1369+
set datafusion.execution.parquet.pushdown_filters = false;
1370+
1371+
statement ok
1372+
set datafusion.explain.physical_plan_only = true;
1373+
1374+
# The repartition merges on (a, b), the real file ordering
1375+
query TT
1376+
EXPLAIN SELECT b FROM pruning_only_eq WHERE a = 5 ORDER BY b LIMIT 1;
1377+
----
1378+
physical_plan
1379+
01)SortPreservingMergeExec: [b@0 ASC NULLS LAST], fetch=1
1380+
02)--FilterExec: a@0 = 5, projection=[b@1], fetch=1
1381+
03)----RepartitionExec: partitioning=RoundRobinBatch(4), input_partitions=2, preserve_order=true, sort_exprs=a@0 ASC NULLS LAST, b@1 ASC NULLS LAST
1382+
04)------DataSourceExec: file_groups={2 groups: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq/f1.parquet], [WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/push_down_filter_parquet/pruning_only_eq/f2.parquet]]}, projection=[a, b], output_ordering=[a@0 ASC NULLS LAST, b@1 ASC NULLS LAST], file_type=parquet, predicate=a@0 = 5, pruning_predicate=a_null_count@2 != row_count@3 AND a_min@0 <= 5 AND 5 <= a_max@1, required_guarantees=[a in (5)]
1383+
1384+
query I
1385+
SELECT b FROM pruning_only_eq WHERE a = 5 ORDER BY b LIMIT 1;
1386+
----
1387+
1
1388+
1389+
query II
1390+
SELECT a, b FROM pruning_only_eq WHERE a = 5 ORDER BY b;
1391+
----
1392+
5 1
1393+
5 2
1394+
1395+
statement ok
1396+
set datafusion.optimizer.prefer_existing_sort = false;
1397+
1398+
query III
1399+
SELECT a, b, count(*) FROM pruning_only_eq WHERE a = 5 GROUP BY a, b ORDER BY b;
1400+
----
1401+
5 1 1
1402+
5 2 1
1403+
1404+
# The same filter, with columns `a` and `b` in the opposite roles
1405+
statement ok
1406+
set datafusion.optimizer.prefer_existing_sort = true;
1407+
1408+
statement ok
1409+
COPY (SELECT * FROM (VALUES (1, 9), (3, 3)) v(a, b))
1410+
TO 'test_files/scratch/push_down_filter_parquet/pruning_only_col_eq/f1.parquet'
1411+
STORED AS PARQUET;
1412+
1413+
statement ok
1414+
COPY (SELECT * FROM (VALUES (5, 5)) v(a, b))
1415+
TO 'test_files/scratch/push_down_filter_parquet/pruning_only_col_eq/f2.parquet'
1416+
STORED AS PARQUET;
1417+
1418+
statement ok
1419+
CREATE EXTERNAL TABLE pruning_only_col_eq (a INT, b INT)
1420+
STORED AS PARQUET
1421+
LOCATION 'test_files/scratch/push_down_filter_parquet/pruning_only_col_eq/'
1422+
WITH ORDER (a ASC);
1423+
1424+
query II
1425+
SELECT a, b FROM pruning_only_col_eq WHERE a = b ORDER BY a;
1426+
----
1427+
3 3
1428+
5 5
1429+
1430+
statement ok
1431+
drop table pruning_only_eq;
1432+
1433+
statement ok
1434+
drop table pruning_only_col_eq;
1435+
1436+
statement ok
1437+
RESET datafusion.optimizer.repartition_file_scans;
1438+
1439+
statement ok
1440+
RESET datafusion.optimizer.prefer_existing_sort;
1441+
1442+
statement ok
1443+
RESET datafusion.execution.parquet.pushdown_filters;
1444+
1445+
statement ok
1446+
RESET datafusion.explain.physical_plan_only;

‎docs/source/library-user-guide/upgrading/56.0.0.md‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -632,3 +632,31 @@ Wire compatibility is directional:
632632
- A 55.0 reader must not consume an alias-preserving 56.0 MERGE payload. It
633633
ignores the unknown field but cannot preserve the qualifier required by the
634634
expressions, which can cause resolution failure or incorrect rebinding.
635+
636+
### `FileSource::exact_filter` controls scan equivalence properties
637+
638+
`FileScanConfig` no longer derives equivalence properties (constant columns
639+
and equal columns) from `FileSource::filter`. It uses the new
640+
`FileSource::exact_filter` method instead. This method returns the part of the
641+
filter that every output row satisfies. A filter that the source uses only for
642+
pruning (for example, Parquet row group statistics when
643+
`datafusion.execution.parquet.pushdown_filters` is `false`) lets
644+
non-matching rows through, and equivalences derived from it gave wrong query
645+
results.
646+
647+
`exact_filter` returns `None` by default. `ParquetSource` returns the pushed-down
648+
conjuncts when `pushdown_filters` is enabled.
649+
650+
**Migration guide:**
651+
652+
If your custom `FileSource` applies its filter to every row, implement
653+
`exact_filter` to keep the equivalence properties of the scan:
654+
655+
```rust,ignore
656+
impl FileSource for MySource {
657+
fn exact_filter(&self) -> Option<Arc<dyn PhysicalExpr>> {
658+
self.filter()
659+
}
660+
// ...
661+
}
662+
```

0 commit comments

Comments
 (0)