Skip to content

Commit 25beec8

Browse files
fix: give each SortExec reset_state copy its own TopK dynamic filter (revert apache#26670) (apache#27142)
## Which issue does this PR close? - Closes apache#27130. ## Rationale for this change As discussed in apache#27130, this reverts apache#26670 so copies made by `SortExec::reset_state()` no longer share one TopK dynamic filter. When a cached `LIMIT 1` plan was reset twice and both copies ran, the first one to finish tightened the shared filter and the second returned no rows. A plain revert brings back a different wrong result, though: a plan that has run once and is then reset still has its old bound in the scan's predicate (the scan keeps the filter it got at planning time). For `SELECT a FROM t ORDER BY a LIMIT 1` over a parquet file with pushdown enabled, the rerun returned no rows with the plain revert. So this keeps the in-place `reset()` from apache#26056 on the old filter and then gives the new plan its own filter, as @Kristofferjoha suggested keeping. | | main | plain revert | this PR | |---|---|---|---| | reset twice, run both copies (apache#27130) | 2nd: no rows | ok | ok | | run, reset, run again (parquet, pushdown) | ok | 2nd: no rows | ok | ## What changes are included in this PR? - `SortExec::reset_state` resets the old dynamic filter in place, then creates a new one for the reset plan. The `sort_concurrency` reset from apache#26242 is kept. (The plain revert did not apply cleanly because apache#26242 changed the same lines.) - Removes the recursive CTE pruning test apache#26670 added to `cte.slt`, since the scan now keeps the old filter (reset to `true`) and the pruning no longer happens, as in 55. - Adds two unit tests in `sorts/sort.rs`: the reproducer from apache#27130, and a test that the filter a scan would hold is back to `true` after `reset_state`. The second one fails with the plain revert. ## Are these changes tested? Yes. Both new tests fail on main or on the plain revert respectively, and pass here. All `datafusion-physical-plan` unit tests and sqllogictests pass. ## Are there any user-facing changes? The TopK dynamic filter inside a recursive CTE no longer prunes after the first iteration, as in 55 (the apache#26670 test's scan reads 300 rows instead of 30). Query results are unchanged. The proper fix (a fresh filter per copy that the scan also picks up, which would also cover `AggregateExec`) can follow after 56. This reopens apache#26642.
1 parent 407bef6 commit 25beec8

2 files changed

Lines changed: 84 additions & 84 deletions

File tree

  • datafusion

‎datafusion/physical-plan/src/sorts/sort.rs‎

Lines changed: 84 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1755,8 +1755,8 @@ impl ExecutionPlan for SortExec {
17551755
// we need to reset.
17561756
if let Some(filter) = &new_sort.filter {
17571757
filter.read().expr().reset();
1758+
new_sort.filter = Some(new_sort.create_filter());
17581759
}
1759-
new_sort.rebuild_filter_for_current_partitioning();
17601760
new_sort.metrics_set = ExecutionPlanMetricsSet::new();
17611761
new_sort.sort_concurrency = Arc::new(OnceLock::new());
17621762

@@ -4764,4 +4764,87 @@ mod tests {
47644764
assert_eq!(num_rows(&topk(false), None)?, Precision::Exact(10));
47654765
Ok(())
47664766
}
4767+
4768+
/// Regression test for <https://github.com/apache/datafusion/issues/27130>
4769+
///
4770+
/// Plans derived from the same TopK plan via `reset_state` must not share
4771+
/// dynamic filter state. This models a cached `LIMIT 1` plan that is reset
4772+
/// and executed by two queries at the same time: both copies are created
4773+
/// before either executes, and the first one to finish must not cause the
4774+
/// other to return an empty result.
4775+
#[tokio::test]
4776+
async fn test_topk_reset_state_copies_are_independent() -> Result<()> {
4777+
let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
4778+
let batch = RecordBatch::try_new(
4779+
Arc::clone(&schema),
4780+
vec![Arc::new(Int32Array::from(vec![3, 1, 2]))],
4781+
)?;
4782+
let input =
4783+
TestMemoryExec::try_new_exec(&[vec![batch]], Arc::clone(&schema), None)?;
4784+
let sort: Arc<dyn ExecutionPlan> = Arc::new(
4785+
SortExec::new(
4786+
[PhysicalSortExpr::new_default(Arc::new(Column::new("a", 0)))].into(),
4787+
input,
4788+
)
4789+
.with_fetch(Some(1)),
4790+
);
4791+
4792+
// Both executions are prepared before either runs
4793+
let first = Arc::clone(&sort).reset_state()?;
4794+
let second = Arc::clone(&sort).reset_state()?;
4795+
4796+
let task_ctx = Arc::new(TaskContext::default());
4797+
let first_result = collect(first, Arc::clone(&task_ctx)).await?;
4798+
let second_result = collect(second, task_ctx).await?;
4799+
4800+
assert_snapshot!(batches_to_string(&first_result), @r"
4801+
+---+
4802+
| a |
4803+
+---+
4804+
| 1 |
4805+
+---+
4806+
");
4807+
assert_snapshot!(batches_to_string(&second_result), @r"
4808+
+---+
4809+
| a |
4810+
+---+
4811+
| 1 |
4812+
+---+
4813+
");
4814+
4815+
Ok(())
4816+
}
4817+
4818+
/// A scan below a TopK keeps the dynamic filter it was given at planning
4819+
/// time. After the plan has run once, `reset_state` must clear that
4820+
/// filter's bounds, or running the reset plan again can return too few rows.
4821+
#[tokio::test]
4822+
async fn test_topk_reset_state_clears_pushed_down_filter() -> Result<()> {
4823+
let schema = Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]));
4824+
let batch = RecordBatch::try_new(
4825+
Arc::clone(&schema),
4826+
vec![Arc::new(Int32Array::from(vec![3, 1, 2]))],
4827+
)?;
4828+
let input =
4829+
TestMemoryExec::try_new_exec(&[vec![batch]], Arc::clone(&schema), None)?;
4830+
let sort: Arc<dyn ExecutionPlan> = Arc::new(
4831+
SortExec::new(
4832+
[PhysicalSortExpr::new_default(Arc::new(Column::new("a", 0)))].into(),
4833+
input,
4834+
)
4835+
.with_fetch(Some(1)),
4836+
);
4837+
// The filter a scan would have received during filter pushdown
4838+
let pushed_down = sort.dynamic_expressions_produced();
4839+
assert_eq!(pushed_down.len(), 1);
4840+
let pushed_down = Arc::clone(&pushed_down[0]);
4841+
4842+
collect(Arc::clone(&sort), Arc::new(TaskContext::default())).await?;
4843+
assert_snapshot!(pushed_down.to_string(), @"DynamicFilter [ a@0 IS NULL OR a@0 < 1 ]");
4844+
4845+
let _reset = Arc::clone(&sort).reset_state()?;
4846+
assert_snapshot!(pushed_down.to_string(), @"DynamicFilter [ true ]");
4847+
4848+
Ok(())
4849+
}
47674850
}

‎datafusion/sqllogictest/test_files/cte.slt‎

Lines changed: 0 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -1993,86 +1993,3 @@ SET datafusion.execution.target_partitions = 4;
19931993

19941994
statement ok
19951995
RESET datafusion.execution.parquet.pushdown_filters;
1996-
1997-
# TopK must reset its dynamic filter in place so the Parquet scan keeps receiving
1998-
# updated bounds after each recursive iteration (#26642). The minimum is in the
1999-
# first row group, so each iteration should scan only 10 rows instead of 100.
2000-
statement ok
2001-
SET datafusion.execution.target_partitions = 1;
2002-
2003-
statement ok
2004-
SET datafusion.execution.batch_size = 10;
2005-
2006-
statement ok
2007-
SET datafusion.execution.parquet.pushdown_filters = true;
2008-
2009-
statement ok
2010-
COPY (
2011-
SELECT CASE WHEN value = 1 THEN 1 ELSE 102 - value END AS k
2012-
FROM generate_series(1, 100)
2013-
) TO 'test_files/scratch/cte/topk_dynamic_filter.parquet'
2014-
OPTIONS ('format.max_row_group_size' '10');
2015-
2016-
statement ok
2017-
CREATE EXTERNAL TABLE topk_dynamic_filter STORED AS PARQUET
2018-
LOCATION 'test_files/scratch/cte/topk_dynamic_filter.parquet';
2019-
2020-
query I
2021-
WITH RECURSIVE w(n) AS (
2022-
SELECT 1
2023-
UNION ALL
2024-
SELECT w.n + s.k
2025-
FROM w CROSS JOIN (
2026-
SELECT k FROM topk_dynamic_filter ORDER BY k LIMIT 1
2027-
) s
2028-
WHERE w.n < 4
2029-
)
2030-
SELECT * FROM w ORDER BY n;
2031-
----
2032-
1
2033-
2
2034-
3
2035-
4
2036-
2037-
statement ok
2038-
SET datafusion.explain.analyze_categories = 'rows';
2039-
2040-
query TT
2041-
EXPLAIN ANALYZE
2042-
WITH RECURSIVE w(n) AS (
2043-
SELECT 1
2044-
UNION ALL
2045-
SELECT w.n + s.k
2046-
FROM w CROSS JOIN (
2047-
SELECT k FROM topk_dynamic_filter ORDER BY k LIMIT 1
2048-
) s
2049-
WHERE w.n < 4
2050-
)
2051-
SELECT * FROM w ORDER BY n;
2052-
----
2053-
Plan with Metrics
2054-
01)SortExec: expr=[n@0 ASC NULLS LAST], preserve_partitioning=[false], metrics=[output_rows=4, output_batches=1, spill_count=0, spilled_rows=0]
2055-
02)--RecursiveQueryExec: name=w, is_distinct=false, metrics=[output_rows=4, output_batches=4]
2056-
03)----ProjectionExec: expr=[CAST(1 AS Int64;N) as n], metrics=[output_rows=1, output_batches=1]
2057-
04)------PlaceholderRowExec, metrics=[]
2058-
05)----ProjectionExec: expr=[n@0 + k@1 as n], metrics=[]
2059-
06)------CrossJoinExec, metrics=[]
2060-
07)--------FilterExec: n@0 < 4, metrics=[]
2061-
08)----------WorkTableExec: name=w, metrics=[]
2062-
09)--------SortExec: TopK(fetch=1), expr=[k@0 ASC NULLS LAST], preserve_partitioning=[false], metrics=[]
2063-
10)----------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/cte/topk_dynamic_filter.parquet]]}, projection=[k], file_type=parquet, predicate=DynamicFilter [ true ], sort_order_for_reorder=[k@0 ASC NULLS LAST], dynamic_rg_pruning=eligible, metrics=[output_rows=30, output_batches=3, files_ranges_pruned_statistics=3 total → 3 matched, row_groups_pruned_statistics=30 total → 30 matched, page_index_pages_pruned=0 total → 0 matched, page_index_rows_pruned=0 total → 0 matched, limit_pruned_row_groups=0 total → 0 matched, batches_split=0, file_open_errors=0, file_scan_errors=0, files_opened=3, files_processed=3, num_predicate_creation_errors=0, predicate_evaluation_errors=0, pushdown_rows_matched=30, pushdown_rows_pruned=0, predicate_cache_inner_records=30, predicate_cache_records=30, scan_efficiency_ratio=9.96% (279/2.80 K)]
2064-
2065-
statement ok
2066-
DROP TABLE topk_dynamic_filter;
2067-
2068-
statement ok
2069-
RESET datafusion.explain.analyze_categories;
2070-
2071-
statement ok
2072-
SET datafusion.execution.target_partitions = 4;
2073-
2074-
statement ok
2075-
RESET datafusion.execution.batch_size;
2076-
2077-
statement ok
2078-
RESET datafusion.execution.parquet.pushdown_filters;

0 commit comments

Comments
 (0)