Repository navigation
Commit c1b29da
committed
[fix](fe) Do not fail on absent statistics of a slot a rewrite generated
### What problem does this PR solve?
Related PR: apache#66898
Problem Summary:
The P0 regression suite correctness_p0/test_column_prune_in_hash_join fails:
select 9 from tbl_a where not (
col_int_undef_signed not in (select 8 from tbl_b) and '2023-12-12' in ('2023-12-19'));
java.sql.SQLException: errCode = 2, detailMessage = Statistics of $c$1#23#false is not
derived, rowCount=6.0, derived expressions: [expr_cast(8 as INT)apache#24, col_int_undef_signed#2]
Two things make a legitimate lookup fail:
1. FilterEstimation#visitNot reads the statistics of every input slot of the child of NOT
(FilterEstimation.java:885) although only the key slots are estimated afterwards, inside
the `context.isKeySlot(slot)` branch. The input slots of the child include slots a rewrite
generated and the statistics never recorded, e.g. the alias slot of the unnested subquery
above, so the unused lookup is the one that fails.
2. Statistics#findColumnStatistics failed the query under fe_debug for every expression
without an entry. Statistics are derived for the columns of a relation, which is also what
OperativeColumnDerive decides to fetch, so only for such a column is a missing entry a bug
of the derivation. A slot a rewrite generated is not a column of a relation: besides the
alias slot of an unnested subquery (ColumnAliasGenerator writes "$c$<n>"), a mark join slot,
a generate slot, and so on all have no original column, and their statistics have always
been estimated as unknown.
Fix:
1. visitNot reads the statistics of a slot only when it is a key slot.
2. findColumnStatistics only fails under fe_debug when the expression is a slot of a column of
a relation (a SlotReference with an original column); any other expression without an entry
keeps degrading to unknown statistics, as in the production environment.
### Release note
None
### Check List (For Author)
- Test: Regression test, Unit Test
- Regression test, locally with fe_debug = true (fuzzy session variables, the condition of
the pipeline): correctness_p0/test_column_prune_in_hash_join passes; the whole
correctness_p0 directory passes except test_sort, which needs the test_query_db that only
the query_p0 data setup creates; shape_check (1215 suites of tpcds_sf100, tpcds_sf1000
(+ constraints, nopkfk, 10t_orc), tpch_sf1000 (+ constraints), ssb_sf100, clickbench,
others) has 0 failures.
- Unit test: StatisticsTest covers that a column of a relation without statistics fails
under fe_debug while a generated slot degrades to unknown statistics, and the statistics
suites (StatsCalculatorTest, OperativeColumnDeriveTest, FilterEstimationTest,
JoinEstimateTest, ExpressionEstimationTest, MemoStatsAndCostRecomputerTest,
ChildrenPropertiesRegulatorTest, ShuffleKeyPruneUtilsTest, AnalyzeTPCHTest,
RewriteTPCHTest) pass, 203 tests.
- Behavior changed: Yes. A missing entry for a generated slot no longer fails a query under
fe_debug; it degrades to unknown statistics. The check is kept for the columns of a relation,
which is what the operative slots derivation governs.
- Does this need documentation: No1 parent 564a779 commit c1b29da
3 files changed
Lines changed: 49 additions & 13 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris
- nereids/stats
- statistics/model
- test/java/org/apache/doris/statistics/model
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
882 | 882 | | |
883 | 883 | | |
884 | 884 | | |
885 | | - | |
886 | | - | |
887 | 885 | | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
888 | 891 | | |
889 | 892 | | |
890 | 893 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
90 | | - | |
91 | | - | |
92 | | - | |
93 | | - | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
94 | 95 | | |
95 | 96 | | |
96 | 97 | | |
| |||
100 | 101 | | |
101 | 102 | | |
102 | 103 | | |
103 | | - | |
| 104 | + | |
104 | 105 | | |
105 | 106 | | |
106 | 107 | | |
107 | 108 | | |
108 | 109 | | |
109 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
110 | 127 | | |
111 | 128 | | |
112 | 129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
20 | 23 | | |
21 | 24 | | |
22 | 25 | | |
| |||
40 | 43 | | |
41 | 44 | | |
42 | 45 | | |
43 | | - | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
44 | 50 | | |
45 | 51 | | |
46 | 52 | | |
| |||
50 | 56 | | |
51 | 57 | | |
52 | 58 | | |
53 | | - | |
54 | | - | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
55 | 62 | | |
56 | 63 | | |
57 | | - | |
| 64 | + | |
| 65 | + | |
58 | 66 | | |
59 | | - | |
60 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
61 | 71 | | |
62 | 72 | | |
63 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
64 | 80 | | |
65 | 81 | | |
66 | 82 | | |
| |||
0 commit comments