Skip to content

Commit 232ef31

Browse files
committed
[fix](fe) Keep the literals of a grouping definition in the learned key
### What problem does this solve? Problem Summary: a grouping key which names a definition of the sub tree is rendered in the key of the aggregation, and the literal mode folded the literals of that definition. The learned (constant agnostic) key of Aggregate(group by x) <- Project(a % n AS x) <- Scan(t) was therefore one for `n = 2` and for `n = 1000`, although with unchanged a = 0..999 the first produces 2 groups and the second 1000. The learned input matching only compares the scans (unchanged here), so a count collected for the first query was applied to the second. The grouping keys of an aggregation now keep their literals in both modes, exactly like the conditions of a join: a definition inside a key decides the number of groups, so it cannot be folded. The aggregation header is protected as well, so the fold of a pasted struct info copies it verbatim. A grouping key without a literal (a plain column) renders as before. ### Release note A learned hbo statistics entry of an aggregation is no longer applied to a query with another grouping definition. ### Check List (For Author) - Test: Unit Test - The two corpus cases `aggregate_grouping_by_a_computed_project` and `aggregate_grouping_by_another_computed_project` now differ in their constant agnostic keys, and a test pins that the definition is part of that key. The regenerated file moves only those two agnostic keys (the intended change); 197 cases of the stats package plus the HBO command suite pass. - Behavior changed: Yes - the constant agnostic key of an aggregation keeps the literals of its grouping keys. - Does this need documentation: No
1 parent 8288527 commit 232ef31

3 files changed

Lines changed: 42 additions & 11 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/nereids/stats/GroupStructInfo.java‎

Lines changed: 20 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1018,7 +1018,10 @@ private enum Kind {
10181018
FILE_SCAN("", false),
10191019
STORAGE_AGGREGATE("A", false),
10201020
FILTER("F", false),
1021-
AGGREGATE("A", false),
1021+
// a grouping key can name a definition of the sub tree (the output of a computed project), and the
1022+
// literals of that definition decide the number of groups: folding them would let the learned count
1023+
// of `GROUP BY a % 2` (two groups) be reused for `GROUP BY a % 1000` (a thousand groups)
1024+
AGGREGATE("A", true),
10221025
// a join's residual conditions decide which rows the join keeps (a range bound, a function of both
10231026
// sides), so their literals are identity: the published and learned keys of a join must not fold
10241027
// `t.b > u.b + 1000` and `t.b > u.b + 1` into one entry
@@ -1630,7 +1633,7 @@ private static void appendNode(StringBuilder out, Node node, LiteralMode mode, b
16301633
out.append(')');
16311634
return;
16321635
case AGGREGATE:
1633-
out.append("A{gb:").append(sortedConditions(node.conditions, mode)).append("}(");
1636+
out.append("A{gb:").append(sortedIdentityConditions(node.conditions)).append("}(");
16341637
appendChildren(out, node, mode, shape);
16351638
out.append(')');
16361639
return;
@@ -1715,6 +1718,21 @@ private static String distinctConditions(List<Expression> conditions, LiteralMod
17151718
return String.join(SEP, rendered);
17161719
}
17171720

1721+
/**
1722+
* The grouping keys of an aggregation, rendered with their literals in both modes (see
1723+
* {@link Kind#AGGREGATE}): a key which names a definition of the sub tree carries the literals of that
1724+
* definition, and they decide the number of groups. Sorted like every other condition collection,
1725+
* duplicates kept (a key repeated in the group by is a different plan).
1726+
*/
1727+
private static String sortedIdentityConditions(List<Expression> conditions) {
1728+
List<String> rendered = new ArrayList<>(conditions.size());
1729+
for (Expression condition : conditions) {
1730+
rendered.add(identityText(condition));
1731+
}
1732+
rendered.sort(String::compareTo);
1733+
return String.join(SEP, rendered);
1734+
}
1735+
17181736
/**
17191737
* The conditions of a join, rendered with their literals in both modes (see {@link Kind#JOIN}), sorted
17201738
* and de-duplicated like every other join condition collection.

‎fe/fe-core/src/test/java/org/apache/doris/nereids/stats/GroupStructInfoGoldenTest.java‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -494,6 +494,19 @@ void testAResidualConstantOfAJoinIsIdentity() {
494494
+ thousand.getCanonical(NO_LITERAL));
495495
}
496496

497+
@Test
498+
void testAGroupingDefinitionKeepsItsLiteralsInTheConstantAgnosticKey() {
499+
// the learned (constant agnostic) key of `GROUP BY a % 2` and of `GROUP BY a % 3` describe a
500+
// different number of groups: folding the literal of the definition would publish one count for
501+
// both queries
502+
GroupStructInfo byTwo = built("aggregate_grouping_by_a_computed_project");
503+
GroupStructInfo byThree = built("aggregate_grouping_by_another_computed_project");
504+
Assertions.assertNotEquals(byTwo.getFingerprint(NO_LITERAL), byThree.getFingerprint(NO_LITERAL),
505+
"two grouping definitions share a learned key");
506+
Assertions.assertTrue(byTwo.getCanonical(NO_LITERAL).contains("A{gb:Mod("),
507+
"the grouping definition is not in the key: " + byTwo.getCanonical(NO_LITERAL));
508+
}
509+
497510
@Test
498511
void testAComputedBranchOfASetOperationIsResolvedInItsMapping() {
499512
// the mapping of the second branch names its output slot, and the project which computes it is

‎fe/fe-core/src/test/resources/hbo-struct-info-golden.txt‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -95,9 +95,9 @@ skipReason =
9595
WITH_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0,v1,r0})
9696
WITH_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0})
9797
WITH_LITERAL.fingerprint = 8111ff4ab69f45665ef070e4aecd948b52e50fab0addd1d4f9ed93cf1baa0ca6
98-
NO_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(*))}(S{null.null.t1#ti:0,v1,r0})
99-
NO_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(*))}(S{null.null.t1#ti:0})
100-
NO_LITERAL.fingerprint = 610c60be55d01eb020344227f5ec6adfeb8ef1f141781b009ad57b32748f031c
98+
NO_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0,v1,r0})
99+
NO_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0})
100+
NO_LITERAL.fingerprint = 8111ff4ab69f45665ef070e4aecd948b52e50fab0addd1d4f9ed93cf1baa0ca6
101101
scans = 1
102102
scan[0] = table=null.null.t1 shapeIdentity=null.null.t1#ti:0 render=null.null.t1#ti:0,v1,r0 visibleVersion=1 scanRows=0 rowsKind=MEASURED selectedPartitions=0 totalPartitions=0 partitionIdentity= tabletIdentity= indexIdentity=
103103
[PLAN aggregate_grouping_by_another_computed_project]
@@ -107,9 +107,9 @@ skipReason =
107107
WITH_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(3:INT))}(S{null.null.t1#ti:0,v1,r0})
108108
WITH_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(3:INT))}(S{null.null.t1#ti:0})
109109
WITH_LITERAL.fingerprint = 72dd50a2dadcb8210dfee4aaa0bb6f68b85ccef48726c026a748c03074bc9b56
110-
NO_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(*))}(S{null.null.t1#ti:0,v1,r0})
111-
NO_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(*))}(S{null.null.t1#ti:0})
112-
NO_LITERAL.fingerprint = 610c60be55d01eb020344227f5ec6adfeb8ef1f141781b009ad57b32748f031c
110+
NO_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(3:INT))}(S{null.null.t1#ti:0,v1,r0})
111+
NO_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(3:INT))}(S{null.null.t1#ti:0})
112+
NO_LITERAL.fingerprint = 72dd50a2dadcb8210dfee4aaa0bb6f68b85ccef48726c026a748c03074bc9b56
113113
scans = 1
114114
scan[0] = table=null.null.t1 shapeIdentity=null.null.t1#ti:0 render=null.null.t1#ti:0,v1,r0 visibleVersion=1 scanRows=0 rowsKind=MEASURED selectedPartitions=0 totalPartitions=0 partitionIdentity= tabletIdentity= indexIdentity=
115115
[PLAN aggregate_grouping_by_the_same_expression_directly]
@@ -119,9 +119,9 @@ skipReason =
119119
WITH_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0,v1,r0})
120120
WITH_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0})
121121
WITH_LITERAL.fingerprint = 8111ff4ab69f45665ef070e4aecd948b52e50fab0addd1d4f9ed93cf1baa0ca6
122-
NO_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(*))}(S{null.null.t1#ti:0,v1,r0})
123-
NO_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(*))}(S{null.null.t1#ti:0})
124-
NO_LITERAL.fingerprint = 610c60be55d01eb020344227f5ec6adfeb8ef1f141781b009ad57b32748f031c
122+
NO_LITERAL.canonical = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0,v1,r0})
123+
NO_LITERAL.shape = A{gb:Mod(col(db.t1.id),lit(2:INT))}(S{null.null.t1#ti:0})
124+
NO_LITERAL.fingerprint = 8111ff4ab69f45665ef070e4aecd948b52e50fab0addd1d4f9ed93cf1baa0ca6
125125
scans = 1
126126
scan[0] = table=null.null.t1 shapeIdentity=null.null.t1#ti:0 render=null.null.t1#ti:0,v1,r0 visibleVersion=1 scanRows=0 rowsKind=MEASURED selectedPartitions=0 totalPartitions=0 partitionIdentity= tabletIdentity= indexIdentity=
127127
[PLAN aggregate_no_keys]

0 commit comments

Comments
 (0)