Repository navigation
Commit 9ceca30
fix: build the IN list static filter only from literal-only items (apache#26083)
## Which issue does this PR close?
- Closes apache#26082.
## Rationale for this change
`n IN (2, 100, 101, CASE WHEN d > 5 THEN 1 ELSE d END * 5)` returns
wrong results on every row (repro in apache#26082).
`try_evaluate_constant_list` decides that a list is constant by
evaluating it on an empty batch, and `CaseExpr` returns its THEN branch
as a scalar for a zero-length WHEN mask, so the CASE item is frozen to
`5` in the static filter and never evaluated per row.
## What changes are included in this PR?
`try_evaluate_constant_list` returns `None` unless every leaf of every
list item is a `Literal`, so any other list takes the per-batch
evaluation `InListExpr` already uses for items that return arrays. Lists
of literals, casts of literals and other expressions over literals still
build the static filter from their empty-batch evaluation, as before.
Checking leaves rather than `collect_columns` keeps the rule independent
of which leaf types read the input (`Column`, `LambdaVariable`, or a
leaf defined outside DataFusion). The cost is that an item with a
non-literal leaf that is nonetheless constant, such as a stable function
the optimizer did not fold, is now evaluated per batch instead of once.
## What is the testing strategy for this PR?
A new section in `in_list.slt` runs the repro as a projection and as a
filter, and `test_in_list_case_item_reading_a_column` builds the same
expression through `in_list()`. Both fail on main and pass with this
change. The other `in_list` unit tests, the `physical-expr` lib tests,
the full sqllogictest suite and the `filter_pushdown` integration tests
pass unchanged.
## Are there any user-facing changes?
No API changes. IN lists whose items have a non-literal leaf return
correct results, evaluated per batch rather than through the static
filter.
---------
Co-authored-by: Jay Zhan <[email protected]>1 parent c4e72de commit 9ceca30
2 files changed
Lines changed: 72 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
22 | 22 | | |
23 | 23 | | |
24 | 24 | | |
| 25 | + | |
25 | 26 | | |
26 | 27 | | |
27 | 28 | | |
| |||
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| 35 | + | |
34 | 36 | | |
35 | 37 | | |
36 | 38 | | |
| |||
107 | 109 | | |
108 | 110 | | |
109 | 111 | | |
110 | | - | |
| 112 | + | |
111 | 113 | | |
112 | 114 | | |
113 | 115 | | |
| |||
117 | 119 | | |
118 | 120 | | |
119 | 121 | | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
120 | 131 | | |
121 | 132 | | |
122 | 133 | | |
| |||
539 | 550 | | |
540 | 551 | | |
541 | 552 | | |
542 | | - | |
| 553 | + | |
543 | 554 | | |
544 | 555 | | |
| 556 | + | |
545 | 557 | | |
546 | 558 | | |
547 | 559 | | |
| |||
3176 | 3188 | | |
3177 | 3189 | | |
3178 | 3190 | | |
| 3191 | + | |
| 3192 | + | |
| 3193 | + | |
| 3194 | + | |
| 3195 | + | |
| 3196 | + | |
| 3197 | + | |
| 3198 | + | |
| 3199 | + | |
| 3200 | + | |
| 3201 | + | |
| 3202 | + | |
| 3203 | + | |
| 3204 | + | |
| 3205 | + | |
| 3206 | + | |
| 3207 | + | |
| 3208 | + | |
| 3209 | + | |
| 3210 | + | |
| 3211 | + | |
| 3212 | + | |
| 3213 | + | |
| 3214 | + | |
| 3215 | + | |
| 3216 | + | |
| 3217 | + | |
3179 | 3218 | | |
3180 | 3219 | | |
3181 | 3220 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
827 | 827 | | |
828 | 828 | | |
829 | 829 | | |
| 830 | + | |
| 831 | + | |
| 832 | + | |
| 833 | + | |
| 834 | + | |
| 835 | + | |
| 836 | + | |
| 837 | + | |
| 838 | + | |
| 839 | + | |
| 840 | + | |
| 841 | + | |
| 842 | + | |
| 843 | + | |
| 844 | + | |
| 845 | + | |
| 846 | + | |
| 847 | + | |
| 848 | + | |
| 849 | + | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
| 857 | + | |
| 858 | + | |
| 859 | + | |
| 860 | + | |
0 commit comments