Skip to content

Commit c1b29da

Browse files
committed
[fix](fe) Do not fail on absent statistics of a slot a rewrite generated
### What problem does this PR solve? Related PR: apache#66898 Problem Summary: The P0 regression suite correctness_p0/test_column_prune_in_hash_join fails: select 9 from tbl_a where not ( col_int_undef_signed not in (select 8 from tbl_b) and '2023-12-12' in ('2023-12-19')); java.sql.SQLException: errCode = 2, detailMessage = Statistics of $c$1#23#false is not derived, rowCount=6.0, derived expressions: [expr_cast(8 as INT)apache#24, col_int_undef_signed#2] Two things make a legitimate lookup fail: 1. FilterEstimation#visitNot reads the statistics of every input slot of the child of NOT (FilterEstimation.java:885) although only the key slots are estimated afterwards, inside the `context.isKeySlot(slot)` branch. The input slots of the child include slots a rewrite generated and the statistics never recorded, e.g. the alias slot of the unnested subquery above, so the unused lookup is the one that fails. 2. Statistics#findColumnStatistics failed the query under fe_debug for every expression without an entry. Statistics are derived for the columns of a relation, which is also what OperativeColumnDerive decides to fetch, so only for such a column is a missing entry a bug of the derivation. A slot a rewrite generated is not a column of a relation: besides the alias slot of an unnested subquery (ColumnAliasGenerator writes "$c$<n>"), a mark join slot, a generate slot, and so on all have no original column, and their statistics have always been estimated as unknown. Fix: 1. visitNot reads the statistics of a slot only when it is a key slot. 2. findColumnStatistics only fails under fe_debug when the expression is a slot of a column of a relation (a SlotReference with an original column); any other expression without an entry keeps degrading to unknown statistics, as in the production environment. ### Release note None ### Check List (For Author) - Test: Regression test, Unit Test - Regression test, locally with fe_debug = true (fuzzy session variables, the condition of the pipeline): correctness_p0/test_column_prune_in_hash_join passes; the whole correctness_p0 directory passes except test_sort, which needs the test_query_db that only the query_p0 data setup creates; shape_check (1215 suites of tpcds_sf100, tpcds_sf1000 (+ constraints, nopkfk, 10t_orc), tpch_sf1000 (+ constraints), ssb_sf100, clickbench, others) has 0 failures. - Unit test: StatisticsTest covers that a column of a relation without statistics fails under fe_debug while a generated slot degrades to unknown statistics, and the statistics suites (StatsCalculatorTest, OperativeColumnDeriveTest, FilterEstimationTest, JoinEstimateTest, ExpressionEstimationTest, MemoStatsAndCostRecomputerTest, ChildrenPropertiesRegulatorTest, ShuffleKeyPruneUtilsTest, AnalyzeTPCHTest, RewriteTPCHTest) pass, 203 tests. - Behavior changed: Yes. A missing entry for a generated slot no longer fails a query under fe_debug; it degrades to unknown statistics. The check is kept for the columns of a relation, which is what the operative slots derivation governs. - Does this need documentation: No
1 parent 564a779 commit c1b29da

3 files changed

Lines changed: 49 additions & 13 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/nereids/stats/FilterEstimation.java‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -882,9 +882,12 @@ public Statistics visitNot(Not not, EstimationContext context) {
882882
StatisticsBuilder statisticsBuilder = new StatisticsBuilder(context.statistics).setRowCount(rowCount);
883883
// update key col stats
884884
for (Slot slot : not.child().getInputSlots()) {
885-
ColumnStatistic originColStats = context.statistics.findColumnStatistics(slot);
886-
ColumnStatistic childColStats = childStats.findColumnStatistics(slot);
887885
if (context.isKeySlot(slot)) {
886+
// read the statistics of this slot here, and not for every input slot of the child:
887+
// the child of NOT also contains slots this estimation does not care about, e.g. the
888+
// alias slot of an unnested subquery, and their statistics need not be recorded
889+
ColumnStatistic originColStats = context.statistics.findColumnStatistics(slot);
890+
ColumnStatistic childColStats = childStats.findColumnStatistics(slot);
888891
ColumnStatisticBuilder colBuilder = new ColumnStatisticBuilder(childColStats);
889892
// update column stats for
890893
// 1. not (A=B)

‎fe/fe-core/src/main/java/org/apache/doris/statistics/model/Statistics.java‎

Lines changed: 22 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -87,10 +87,11 @@ public Statistics(double rowCount, int widthInJoinCluster,
8787
* expression are simply unknown. Missing statistics therefore degrade to
8888
* {@link ColumnStatistic#UNKNOWN} instead of breaking the query.
8989
*
90-
* <p>Absence of statistics is however always a potential bug of the statistics derivation: when
91-
* {@code fe_debug} is on (as in the pipeline test environment) it is turned into a
92-
* NullPointerException so that the bug fails the test instead of silently degrading the plan,
93-
* while the production environment keeps returning unknown statistics.
90+
* <p>Absence of the statistics of a column of a relation is however a potential bug of the
91+
* statistics derivation: when {@code fe_debug} is on (as in the pipeline test environment) it is
92+
* turned into a NullPointerException so that the bug fails the test instead of silently
93+
* degrading the plan, while the production environment keeps returning unknown statistics. See
94+
* {@link #isStatisticsDerivationExpected} for the expressions this covers.
9495
*
9596
* <p>Use {@link #findColumnStatisticsOrNull} when the absence of statistics is an expected case
9697
* that the caller explicitly handles.
@@ -100,13 +101,29 @@ public ColumnStatistic findColumnStatistics(Expression expression) {
100101
if (columnStatistic != null) {
101102
return columnStatistic;
102103
}
103-
if (SessionVariable.isFeDebug()) {
104+
if (SessionVariable.isFeDebug() && isStatisticsDerivationExpected(expression)) {
104105
throw new NullPointerException("Statistics of " + expression + " is not derived, rowCount="
105106
+ rowCount + ", derived expressions: " + expressionToColumnStats.keySet());
106107
}
107108
return ColumnStatistic.UNKNOWN;
108109
}
109110

111+
/**
112+
* Whether a missing entry for the given expression means that the statistics derivation missed
113+
* something, i.e. whether it is a bug worth failing the query for while {@code fe_debug} is on.
114+
*
115+
* <p>The statistics of a plan are derived for the columns of a relation, which is also what
116+
* {@code OperativeColumnDerive} decides to fetch, so a column of a relation is expected to have
117+
* its statistics recorded. Slots that a rewrite generated are not columns of a relation and are
118+
* not expected to be recorded: for example the alias slot of an unnested subquery
119+
* ({@code $c$1}, see {@code ColumnAliasGenerator}), the slot of a mark join, or the slot of an
120+
* unnest. Their statistics have always been estimated as unknown.
121+
*/
122+
private static boolean isStatisticsDerivationExpected(Expression expression) {
123+
return expression instanceof SlotReference
124+
&& ((SlotReference) expression).getOriginalColumn().isPresent();
125+
}
126+
110127
/**
111128
* Returns the statistics of the given expression, or null when no statistics are recorded for
112129
* it. Use this method instead of {@link #findColumnStatistics} only when the caller really

‎fe/fe-core/src/test/java/org/apache/doris/statistics/model/StatisticsTest.java‎

Lines changed: 22 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,9 @@
1717

1818
package org.apache.doris.statistics.model;
1919

20+
import org.apache.doris.catalog.Column;
21+
import org.apache.doris.catalog.PrimitiveType;
22+
import org.apache.doris.nereids.trees.expressions.ExprId;
2023
import org.apache.doris.nereids.trees.expressions.SlotReference;
2124
import org.apache.doris.nereids.types.IntegerType;
2225
import org.apache.doris.nereids.util.MoreFieldsThread;
@@ -40,7 +43,10 @@ public void tearDown() {
4043
@Test
4144
public void testFindColumnStatisticsOfNotDerivedExpression() {
4245
SlotReference derivedSlot = SlotReference.of("derived", IntegerType.INSTANCE);
43-
SlotReference notDerivedSlot = SlotReference.of("not_derived", IntegerType.INSTANCE);
46+
SlotReference notDerivedColumn = relationColumn("not_derived");
47+
// a slot generated by a rewrite (an alias of an expression, e.g. a $c$ slot of an unnested
48+
// subquery) is not a column of a relation and is not expected to be recorded
49+
SlotReference generatedSlot = SlotReference.of("$c$1", IntegerType.INSTANCE);
4450
Statistics stats = new Statistics(100, 1,
4551
ImmutableMap.of(derivedSlot, new ColumnStatisticBuilder().setNdv(10).build()));
4652

@@ -50,17 +56,27 @@ public void testFindColumnStatisticsOfNotDerivedExpression() {
5056

5157
// Production: statistics that are not derived degrade to unknown statistics, so that a bug of
5258
// the statistics derivation cannot break a query.
53-
Assertions.assertTrue(stats.findColumnStatistics(notDerivedSlot).isUnKnown());
54-
Assertions.assertNull(stats.findColumnStatisticsOrNull(notDerivedSlot));
59+
Assertions.assertTrue(stats.findColumnStatistics(notDerivedColumn).isUnKnown());
60+
Assertions.assertTrue(stats.findColumnStatistics(generatedSlot).isUnKnown());
61+
Assertions.assertNull(stats.findColumnStatisticsOrNull(notDerivedColumn));
5562
Assertions.assertEquals(10.0, stats.findColumnStatistics(derivedSlot).ndv);
5663

57-
// The pipeline test environment (fe_debug = true) fails instead of silently degrading.
64+
// The pipeline test environment (fe_debug = true) fails instead of silently degrading when the
65+
// statistics of a column of a relation are missing.
5866
connectContext.getSessionVariable().feDebug = true;
59-
Assertions.assertThrows(NullPointerException.class, () -> stats.findColumnStatistics(notDerivedSlot));
60-
Assertions.assertNull(stats.findColumnStatisticsOrNull(notDerivedSlot));
67+
Assertions.assertThrows(NullPointerException.class, () -> stats.findColumnStatistics(notDerivedColumn));
68+
// a generated slot has no derivation to miss, so it still degrades to unknown statistics
69+
Assertions.assertTrue(stats.findColumnStatistics(generatedSlot).isUnKnown());
70+
Assertions.assertNull(stats.findColumnStatisticsOrNull(notDerivedColumn));
6171
Assertions.assertEquals(10.0, stats.findColumnStatistics(derivedSlot).ndv);
6272
}
6373

74+
private static SlotReference relationColumn(String name) {
75+
Column column = new Column(name, PrimitiveType.INT);
76+
return new SlotReference(new ExprId(1), name, IntegerType.INSTANCE, true,
77+
ImmutableList.of("test", "tbl"), null, column, null, column);
78+
}
79+
6480
@Test
6581
public void testAvgSizeAbnormal() {
6682
SlotReference slot = SlotReference.of("a", IntegerType.INSTANCE);

0 commit comments

Comments
 (0)