Skip to content

Commit 97f605e

Browse files
committed
test: refresh ANSI decimal AVG plan expectations
Regenerate the six Spark 4.x plan overrides for the intentional grouped ANSI decimal AVG fallback and document its scope. Preserve the shared and Spark 3.x expectations.
1 parent 728c55e commit 97f605e

7 files changed

Lines changed: 515 additions & 1 deletion

File tree

‎docs/source/user-guide/latest/compatibility/operators.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,13 @@ intermediate sum and check precision after dividing by the count. Comet instead
4343
overflow while accumulating that sum, which can discard a valid average.
4444

4545
Both partial and final aggregates stay in Spark, including when a shuffle separates them.
46-
Grouped averages and narrower decimal averages remain eligible for native execution.
46+
Narrower ungrouped decimal averages remain eligible for native execution.
47+
48+
Grouped `AVG` whose input remains `DECIMAL` after Spark optimization falls back in ANSI mode
49+
at every precision to preserve Spark's lazy overflow evaluation. Every stage evaluating that AVG,
50+
including other functions in the same aggregate operator, runs in Spark even without `LIMIT`.
51+
Grouped `TRY_AVG`, legacy-mode decimal `AVG`, and non-decimal `AVG` remain eligible for native
52+
execution, subject to their other restrictions.
4753

4854
## Window Functions
4955

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
TakeOrderedAndProject
2+
+- Project
3+
+- BroadcastHashJoin
4+
:- Project
5+
: +- BroadcastHashJoin
6+
: :- Project
7+
: : +- BroadcastHashJoin
8+
: : :- CometColumnarToRow
9+
: : : +- CometFilter
10+
: : : +- CometHashAggregate
11+
: : : +- CometExchange
12+
: : : +- CometHashAggregate
13+
: : : +- CometProject
14+
: : : +- CometBroadcastHashJoin
15+
: : : :- CometFilter
16+
: : : : +- CometNativeScan parquet spark_catalog.default.store_returns
17+
: : : : +- CometSubqueryBroadcast
18+
: : : : +- CometBroadcastExchange
19+
: : : : +- CometProject
20+
: : : : +- CometFilter
21+
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
22+
: : : +- CometBroadcastExchange
23+
: : : +- CometProject
24+
: : : +- CometFilter
25+
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
26+
: : +- BroadcastExchange
27+
: : +- Filter
28+
: : +- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
29+
: : +- Exchange
30+
: : +- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
31+
: : +- CometColumnarToRow
32+
: : +- CometHashAggregate
33+
: : +- CometExchange
34+
: : +- CometHashAggregate
35+
: : +- CometProject
36+
: : +- CometBroadcastHashJoin
37+
: : :- CometFilter
38+
: : : +- CometNativeScan parquet spark_catalog.default.store_returns
39+
: : : +- ReusedSubquery
40+
: : +- CometBroadcastExchange
41+
: : +- CometProject
42+
: : +- CometFilter
43+
: : +- CometNativeScan parquet spark_catalog.default.date_dim
44+
: +- BroadcastExchange
45+
: +- CometColumnarToRow
46+
: +- CometProject
47+
: +- CometFilter
48+
: +- CometNativeScan parquet spark_catalog.default.store
49+
+- BroadcastExchange
50+
+- CometColumnarToRow
51+
+- CometProject
52+
+- CometFilter
53+
+- CometNativeScan parquet spark_catalog.default.customer
54+
55+
Comet accelerated 34 out of 48 eligible operators (70%). Final plan contains 4 transitions between Spark and Comet. Accelerated expressions: 7 native, 0 codegen dispatch.
Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
TakeOrderedAndProject
2+
+- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
3+
+- Exchange
4+
+- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
5+
+- CometColumnarToRow
6+
+- CometExpand
7+
+- CometProject
8+
+- CometBroadcastHashJoin
9+
:- CometProject
10+
: +- CometBroadcastHashJoin
11+
: :- CometProject
12+
: : +- CometBroadcastHashJoin
13+
: : :- CometProject
14+
: : : +- CometBroadcastHashJoin
15+
: : : :- CometProject
16+
: : : : +- CometBroadcastHashJoin
17+
: : : : :- CometProject
18+
: : : : : +- CometBroadcastHashJoin
19+
: : : : : :- CometFilter
20+
: : : : : : +- CometNativeScan parquet spark_catalog.default.catalog_sales
21+
: : : : : : +- CometSubqueryBroadcast
22+
: : : : : : +- CometBroadcastExchange
23+
: : : : : : +- CometProject
24+
: : : : : : +- CometFilter
25+
: : : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
26+
: : : : : +- CometBroadcastExchange
27+
: : : : : +- CometProject
28+
: : : : : +- CometFilter
29+
: : : : : +- CometNativeScan parquet spark_catalog.default.customer_demographics
30+
: : : : +- CometBroadcastExchange
31+
: : : : +- CometProject
32+
: : : : +- CometFilter
33+
: : : : +- CometNativeScan parquet spark_catalog.default.customer
34+
: : : +- CometBroadcastExchange
35+
: : : +- CometFilter
36+
: : : +- CometNativeScan parquet spark_catalog.default.customer_demographics
37+
: : +- CometBroadcastExchange
38+
: : +- CometProject
39+
: : +- CometFilter
40+
: : +- CometNativeScan parquet spark_catalog.default.customer_address
41+
: +- CometBroadcastExchange
42+
: +- CometProject
43+
: +- CometFilter
44+
: +- CometNativeScan parquet spark_catalog.default.date_dim
45+
+- CometBroadcastExchange
46+
+- CometProject
47+
+- CometFilter
48+
+- CometNativeScan parquet spark_catalog.default.item
49+
50+
Comet accelerated 43 out of 47 eligible operators (91%). Final plan contains 1 transitions between Spark and Comet. Accelerated expressions: 5 native, 0 codegen dispatch.
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
TakeOrderedAndProject
2+
+- Project
3+
+- BroadcastHashJoin
4+
:- Project
5+
: +- BroadcastHashJoin
6+
: :- Project
7+
: : +- BroadcastHashJoin
8+
: : :- CometColumnarToRow
9+
: : : +- CometFilter
10+
: : : +- CometHashAggregate
11+
: : : +- CometExchange
12+
: : : +- CometHashAggregate
13+
: : : +- CometProject
14+
: : : +- CometBroadcastHashJoin
15+
: : : :- CometProject
16+
: : : : +- CometBroadcastHashJoin
17+
: : : : :- CometFilter
18+
: : : : : +- CometNativeScan parquet spark_catalog.default.web_returns
19+
: : : : : +- CometSubqueryBroadcast
20+
: : : : : +- CometBroadcastExchange
21+
: : : : : +- CometProject
22+
: : : : : +- CometFilter
23+
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
24+
: : : : +- CometBroadcastExchange
25+
: : : : +- CometProject
26+
: : : : +- CometFilter
27+
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
28+
: : : +- CometBroadcastExchange
29+
: : : +- CometProject
30+
: : : +- CometFilter
31+
: : : +- CometNativeScan parquet spark_catalog.default.customer_address
32+
: : +- BroadcastExchange
33+
: : +- Filter
34+
: : +- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
35+
: : +- Exchange
36+
: : +- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
37+
: : +- CometColumnarToRow
38+
: : +- CometHashAggregate
39+
: : +- CometExchange
40+
: : +- CometHashAggregate
41+
: : +- CometProject
42+
: : +- CometBroadcastHashJoin
43+
: : :- CometProject
44+
: : : +- CometBroadcastHashJoin
45+
: : : :- CometFilter
46+
: : : : +- CometNativeScan parquet spark_catalog.default.web_returns
47+
: : : : +- ReusedSubquery
48+
: : : +- CometBroadcastExchange
49+
: : : +- CometProject
50+
: : : +- CometFilter
51+
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
52+
: : +- CometBroadcastExchange
53+
: : +- CometProject
54+
: : +- CometFilter
55+
: : +- CometNativeScan parquet spark_catalog.default.customer_address
56+
: +- BroadcastExchange
57+
: +- CometColumnarToRow
58+
: +- CometProject
59+
: +- CometFilter
60+
: +- CometNativeScan parquet spark_catalog.default.customer
61+
+- BroadcastExchange
62+
+- CometColumnarToRow
63+
+- CometProject
64+
+- CometFilter
65+
+- CometNativeScan parquet spark_catalog.default.customer_address
66+
67+
Comet accelerated 46 out of 60 eligible operators (76%). Final plan contains 4 transitions between Spark and Comet. Accelerated expressions: 7 native, 0 codegen dispatch.
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
TakeOrderedAndProject
2+
+- Project
3+
+- BroadcastHashJoin
4+
:- CometColumnarToRow
5+
: +- CometProject
6+
: +- CometBroadcastHashJoin
7+
: :- CometProject
8+
: : +- CometBroadcastHashJoin
9+
: : :- CometFilter
10+
: : : +- CometNativeScan parquet spark_catalog.default.store
11+
: : +- CometBroadcastExchange
12+
: : +- CometFilter
13+
: : +- CometHashAggregate
14+
: : +- CometExchange
15+
: : +- CometHashAggregate
16+
: : +- CometProject
17+
: : +- CometBroadcastHashJoin
18+
: : :- CometFilter
19+
: : : +- CometNativeScan parquet spark_catalog.default.store_sales
20+
: : : +- CometSubqueryBroadcast
21+
: : : +- CometBroadcastExchange
22+
: : : +- CometProject
23+
: : : +- CometFilter
24+
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
25+
: : +- CometBroadcastExchange
26+
: : +- CometProject
27+
: : +- CometFilter
28+
: : +- CometNativeScan parquet spark_catalog.default.date_dim
29+
: +- CometBroadcastExchange
30+
: +- CometProject
31+
: +- CometFilter
32+
: +- CometNativeScan parquet spark_catalog.default.item
33+
+- BroadcastExchange
34+
+- Filter
35+
+- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
36+
+- Exchange
37+
+- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
38+
+- CometColumnarToRow
39+
+- CometHashAggregate
40+
+- CometExchange
41+
+- CometHashAggregate
42+
+- CometProject
43+
+- CometBroadcastHashJoin
44+
:- CometFilter
45+
: +- CometNativeScan parquet spark_catalog.default.store_sales
46+
: +- ReusedSubquery
47+
+- CometBroadcastExchange
48+
+- CometProject
49+
+- CometFilter
50+
+- CometNativeScan parquet spark_catalog.default.date_dim
51+
52+
Comet accelerated 39 out of 47 eligible operators (82%). Final plan contains 2 transitions between Spark and Comet. Accelerated expressions: 8 native, 0 codegen dispatch.
Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
TakeOrderedAndProject
2+
+- Project
3+
+- BroadcastHashJoin
4+
:- Project
5+
: +- BroadcastHashJoin
6+
: :- Project
7+
: : +- BroadcastHashJoin
8+
: : :- CometColumnarToRow
9+
: : : +- CometFilter
10+
: : : +- CometHashAggregate
11+
: : : +- CometExchange
12+
: : : +- CometHashAggregate
13+
: : : +- CometProject
14+
: : : +- CometBroadcastHashJoin
15+
: : : :- CometProject
16+
: : : : +- CometBroadcastHashJoin
17+
: : : : :- CometFilter
18+
: : : : : +- CometNativeScan parquet spark_catalog.default.catalog_returns
19+
: : : : : +- CometSubqueryBroadcast
20+
: : : : : +- CometBroadcastExchange
21+
: : : : : +- CometProject
22+
: : : : : +- CometFilter
23+
: : : : : +- CometNativeScan parquet spark_catalog.default.date_dim
24+
: : : : +- CometBroadcastExchange
25+
: : : : +- CometProject
26+
: : : : +- CometFilter
27+
: : : : +- CometNativeScan parquet spark_catalog.default.date_dim
28+
: : : +- CometBroadcastExchange
29+
: : : +- CometProject
30+
: : : +- CometFilter
31+
: : : +- CometNativeScan parquet spark_catalog.default.customer_address
32+
: : +- BroadcastExchange
33+
: : +- Filter
34+
: : +- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
35+
: : +- Exchange
36+
: : +- HashAggregate [COMET: Grouped decimal AVG in ANSI mode requires Spark's lazy group evaluation]
37+
: : +- CometColumnarToRow
38+
: : +- CometHashAggregate
39+
: : +- CometExchange
40+
: : +- CometHashAggregate
41+
: : +- CometProject
42+
: : +- CometBroadcastHashJoin
43+
: : :- CometProject
44+
: : : +- CometBroadcastHashJoin
45+
: : : :- CometFilter
46+
: : : : +- CometNativeScan parquet spark_catalog.default.catalog_returns
47+
: : : : +- ReusedSubquery
48+
: : : +- CometBroadcastExchange
49+
: : : +- CometProject
50+
: : : +- CometFilter
51+
: : : +- CometNativeScan parquet spark_catalog.default.date_dim
52+
: : +- CometBroadcastExchange
53+
: : +- CometProject
54+
: : +- CometFilter
55+
: : +- CometNativeScan parquet spark_catalog.default.customer_address
56+
: +- BroadcastExchange
57+
: +- CometColumnarToRow
58+
: +- CometProject
59+
: +- CometFilter
60+
: +- CometNativeScan parquet spark_catalog.default.customer
61+
+- BroadcastExchange
62+
+- CometColumnarToRow
63+
+- CometProject
64+
+- CometFilter
65+
+- CometNativeScan parquet spark_catalog.default.customer_address
66+
67+
Comet accelerated 46 out of 60 eligible operators (76%). Final plan contains 4 transitions between Spark and Comet. Accelerated expressions: 7 native, 0 codegen dispatch.

0 commit comments

Comments
 (0)