Repository navigation
Commit 25beec8
authored
fix: give each SortExec reset_state copy its own TopK dynamic filter (revert apache#26670) (apache#27142)
## Which issue does this PR close?
- Closes apache#27130.
## Rationale for this change
As discussed in apache#27130, this reverts apache#26670 so copies made by
`SortExec::reset_state()` no longer share one TopK dynamic filter. When
a cached `LIMIT 1` plan was reset twice and both copies ran, the first
one to finish tightened the shared filter and the second returned no
rows.
A plain revert brings back a different wrong result, though: a plan that
has run once and is then reset still has its old bound in the scan's
predicate (the scan keeps the filter it got at planning time). For
`SELECT a FROM t ORDER BY a LIMIT 1` over a parquet file with pushdown
enabled, the rerun returned no rows with the plain revert. So this keeps
the in-place `reset()` from apache#26056 on the old filter and then gives the
new plan its own filter, as @Kristofferjoha suggested keeping.
| | main | plain revert | this PR |
|---|---|---|---|
| reset twice, run both copies (apache#27130) | 2nd: no rows | ok | ok |
| run, reset, run again (parquet, pushdown) | ok | 2nd: no rows | ok |
## What changes are included in this PR?
- `SortExec::reset_state` resets the old dynamic filter in place, then
creates a new one for the reset plan. The `sort_concurrency` reset from
apache#26242 is kept. (The plain revert did not apply cleanly because apache#26242
changed the same lines.)
- Removes the recursive CTE pruning test apache#26670 added to `cte.slt`,
since the scan now keeps the old filter (reset to `true`) and the
pruning no longer happens, as in 55.
- Adds two unit tests in `sorts/sort.rs`: the reproducer from apache#27130,
and a test that the filter a scan would hold is back to `true` after
`reset_state`. The second one fails with the plain revert.
## Are these changes tested?
Yes. Both new tests fail on main or on the plain revert respectively,
and pass here. All `datafusion-physical-plan` unit tests and
sqllogictests pass.
## Are there any user-facing changes?
The TopK dynamic filter inside a recursive CTE no longer prunes after
the first iteration, as in 55 (the apache#26670 test's scan reads 300 rows
instead of 30). Query results are unchanged. The proper fix (a fresh
filter per copy that the scan also picks up, which would also cover
`AggregateExec`) can follow after 56.
This reopens apache#26642.1 parent 407bef6 commit 25beec8
2 files changed
Lines changed: 84 additions & 84 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1755 | 1755 | | |
1756 | 1756 | | |
1757 | 1757 | | |
| 1758 | + | |
1758 | 1759 | | |
1759 | | - | |
1760 | 1760 | | |
1761 | 1761 | | |
1762 | 1762 | | |
| |||
4764 | 4764 | | |
4765 | 4765 | | |
4766 | 4766 | | |
| 4767 | + | |
| 4768 | + | |
| 4769 | + | |
| 4770 | + | |
| 4771 | + | |
| 4772 | + | |
| 4773 | + | |
| 4774 | + | |
| 4775 | + | |
| 4776 | + | |
| 4777 | + | |
| 4778 | + | |
| 4779 | + | |
| 4780 | + | |
| 4781 | + | |
| 4782 | + | |
| 4783 | + | |
| 4784 | + | |
| 4785 | + | |
| 4786 | + | |
| 4787 | + | |
| 4788 | + | |
| 4789 | + | |
| 4790 | + | |
| 4791 | + | |
| 4792 | + | |
| 4793 | + | |
| 4794 | + | |
| 4795 | + | |
| 4796 | + | |
| 4797 | + | |
| 4798 | + | |
| 4799 | + | |
| 4800 | + | |
| 4801 | + | |
| 4802 | + | |
| 4803 | + | |
| 4804 | + | |
| 4805 | + | |
| 4806 | + | |
| 4807 | + | |
| 4808 | + | |
| 4809 | + | |
| 4810 | + | |
| 4811 | + | |
| 4812 | + | |
| 4813 | + | |
| 4814 | + | |
| 4815 | + | |
| 4816 | + | |
| 4817 | + | |
| 4818 | + | |
| 4819 | + | |
| 4820 | + | |
| 4821 | + | |
| 4822 | + | |
| 4823 | + | |
| 4824 | + | |
| 4825 | + | |
| 4826 | + | |
| 4827 | + | |
| 4828 | + | |
| 4829 | + | |
| 4830 | + | |
| 4831 | + | |
| 4832 | + | |
| 4833 | + | |
| 4834 | + | |
| 4835 | + | |
| 4836 | + | |
| 4837 | + | |
| 4838 | + | |
| 4839 | + | |
| 4840 | + | |
| 4841 | + | |
| 4842 | + | |
| 4843 | + | |
| 4844 | + | |
| 4845 | + | |
| 4846 | + | |
| 4847 | + | |
| 4848 | + | |
| 4849 | + | |
4767 | 4850 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1993 | 1993 | | |
1994 | 1994 | | |
1995 | 1995 | | |
1996 | | - | |
1997 | | - | |
1998 | | - | |
1999 | | - | |
2000 | | - | |
2001 | | - | |
2002 | | - | |
2003 | | - | |
2004 | | - | |
2005 | | - | |
2006 | | - | |
2007 | | - | |
2008 | | - | |
2009 | | - | |
2010 | | - | |
2011 | | - | |
2012 | | - | |
2013 | | - | |
2014 | | - | |
2015 | | - | |
2016 | | - | |
2017 | | - | |
2018 | | - | |
2019 | | - | |
2020 | | - | |
2021 | | - | |
2022 | | - | |
2023 | | - | |
2024 | | - | |
2025 | | - | |
2026 | | - | |
2027 | | - | |
2028 | | - | |
2029 | | - | |
2030 | | - | |
2031 | | - | |
2032 | | - | |
2033 | | - | |
2034 | | - | |
2035 | | - | |
2036 | | - | |
2037 | | - | |
2038 | | - | |
2039 | | - | |
2040 | | - | |
2041 | | - | |
2042 | | - | |
2043 | | - | |
2044 | | - | |
2045 | | - | |
2046 | | - | |
2047 | | - | |
2048 | | - | |
2049 | | - | |
2050 | | - | |
2051 | | - | |
2052 | | - | |
2053 | | - | |
2054 | | - | |
2055 | | - | |
2056 | | - | |
2057 | | - | |
2058 | | - | |
2059 | | - | |
2060 | | - | |
2061 | | - | |
2062 | | - | |
2063 | | - | |
2064 | | - | |
2065 | | - | |
2066 | | - | |
2067 | | - | |
2068 | | - | |
2069 | | - | |
2070 | | - | |
2071 | | - | |
2072 | | - | |
2073 | | - | |
2074 | | - | |
2075 | | - | |
2076 | | - | |
2077 | | - | |
2078 | | - | |
0 commit comments