Repository navigation
Commit 4c3f048
authored
feat: disable native columnar-to-row conversion by default (#5114)
* feat: disable native columnar-to-row conversion by default
Change the default of spark.comet.exec.columnarToRow.native.enabled
from true to false. Isolated benchmarking in #5112 shows the native
converter is 3.7x slower per row than the JVM implementation at the
default batch size and up to 15.7x slower for small batches due to a
fixed JNI and FFI cost of roughly 10us per batch, while end-to-end
benchmarks and TPC-DS runs show no benefit over the JVM operator.
Also update the operator scaladoc, which claimed zero-copy strings and
reduced GC pressure: rows have been deep-copied to the JVM heap since
the fix for #3308, and #4440 found no measurable GC benefit.
Part of #5112
* fix: strip CometColumnarToRowExec in DPP subquery broadcast rewrite
The non-AQE DPP rewrite in CometExecRule only recognized
CometNativeColumnarToRowExec under the SubqueryBroadcastExec's
BroadcastExchangeExec. With the native conversion disabled by default,
the JVM CometColumnarToRowExec is used instead, and because it is
CodegenSupport the prepared subquery plan wraps it as
WholeStageCodegenExec(CometColumnarToRowExec(InputAdapter(cometPlan))).
The rewrite did not match this shape, so SubqueryBroadcastExec was
never converted to CometSubqueryBroadcastExec and DPP broadcast
exchange reuse stopped working. Unwrap the codegen and InputAdapter
layers so both columnar-to-row variants are handled.
* test: regenerate plan stability golden files for JVM columnar-to-row default
Regenerated with dev/regenerate-golden-files.sh for all Spark versions.
The only change is the operator rename from CometNativeColumnarToRow to
CometColumnarToRow at plan stage roots; CometSubqueryBroadcast and
exchange reuse are unchanged thanks to the DPP rewrite fix.
* test: accept CometColumnarToRowExec in SubquerySuite partition pruning test
* test: refresh golden files and plan docs for JVM columnar-to-row default
* test: accept codegen-wrapped CometColumnarToRow in CachedBatchSerializerNoUnwrapSuite
The Comet patch for `CachedBatchSerializerNoUnwrapSuite` accepted either the
Spark shape (`WholeStageCodegenExec(ColumnarToRowExec)`) or a cached plan whose
top node is in `org.apache.spark.sql.comet`. The latter only held for
`CometNativeColumnarToRowExec`, which does not support codegen. Now that the
JVM `CometColumnarToRowExec` is the default, the cached plan is
`WholeStageCodegenExec(CometColumnarToRow(...))`, so both checks failed:
isSparkShape was false, and isCometShape was false unexpected cached plan:
*(1) CometColumnarToRow
+- CometNativeScan parquet [count#822213] ...
Look through a `WholeStageCodegenExec` before classifying the transition node so
both Comet operators are accepted.
* fix: do not report decimal avg overflow when there is nothing to average
Closes #5148.
In ANSI mode, `avg` on a decimal column raised
`[ARITHMETIC_OVERFLOW] Overflow in sum of decimals` for an ungrouped aggregate
whose input was empty or entirely null, where Spark returns NULL.
`AvgDecimalAccumulator::merge_batch` derives `is_empty` from the nullness of the
merged counts, but a global partial aggregate over zero rows still emits one row
with `sum = NULL, count = 0` and that count is not null, so `is_empty` becomes
false. `evaluate` then treated the missing sum as an overflow. Guard that check
with `count > 0`, matching the neighbouring `is_not_null` check and the
`GroupsAccumulator` path (which is why only aggregates without GROUP BY were
affected).
This is what makes `LogicalPlanTagInSparkPlanSuite` q24a/q24b/q24-v2.7 fail in
the Spark 4.1 SQL lane once the JVM columnar-to-row operator is the default:
`CometColumnarToRowExec` supports codegen, so `checkGeneratedCode` calls
`WholeStageCodegenExec.doCodeGen`, which prepares the plan and eagerly executes
q24's `SELECT 0.05 * avg(netpaid) FROM ssales` scalar subquery against the
row-less TPC-DS tables that suite creates.1 parent 4b091ac commit 4c3f048
169 files changed
Lines changed: 411 additions & 320 deletions
File tree
- dev/diffs
- docs/source/user-guide/latest
- native/spark-expr/src/agg_funcs
- spark/src
- main/scala/org/apache
- comet
- rules
- spark/sql/comet
- test
- resources/tpcds-plan-stability
- approved-plans-v1_4-spark3_5
- q44
- q58
- q67
- q70
- q83
- approved-plans-v1_4-spark4_0
- q14b
- q44
- q54
- q58
- q6
- q83.ansi
- approved-plans-v1_4-spark4_1
- q33
- q49
- q56
- q60
- q66
- approved-plans-v1_4
- q10
- q11
- q12
- q13
- q14a
- q14b
- q15
- q16
- q17
- q18
- q19
- q1
- q20
- q21
- q22
- q23a
- q23b
- q24a
- q24b
- q25
- q26
- q27
- q28
- q29
- q2
- q30
- q31
- q32
- q33
- q34
- q35
- q36
- q37
- q38
- q39a
- q39b
- q3
- q40
- q41
- q42
- q43
- q44
- q45
- q46
- q47
- q48
- q49
- q4
- q50
- q51
- q52
- q53
- q54
- q55
- q56
- q57
- q58
- q59
- q5
- q60
- q61
- q62
- q63
- q64
- q65
- q66
- q67
- q68
- q69
- q6
- q70
- q71
- q72
- q73
- q74
- q75
- q76
- q77
- q78
- q79
- q7
- q80
- q81
- q82
- q83
- q84
- q85
- q86
- q87
- q88
- q89
- q8
- q90
- q91
- q92
- q93
- q94
- q95
- q96
- q97
- q98
- q99
- q9
- approved-plans-v2_7-spark3_5
- q67a
- q70a
- approved-plans-v2_7-spark4_0
- q14
- q6
- approved-plans-v2_7-spark4_1
- q14a
- q49
- approved-plans-v2_7-spark4_2/q77a
- approved-plans-v2_7
- q10a
- q11
- q12
- q14a
- q14
- q18a
- q20
- q22a
- q22
- q24
- q27a
- q34
- q35a
- q35
- q36a
- q47
- q49
- q51a
- q57
- q5a
- q64
- q67a
- q6
- q70a
- q72
- q74
- q75
- q77a
- q78
- q80a
- q86a
- q98
- scala/org/apache/comet/exec
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1124 | 1124 | | |
1125 | 1125 | | |
1126 | 1126 | | |
1127 | | - | |
| 1127 | + | |
1128 | 1128 | | |
1129 | 1129 | | |
1130 | 1130 | | |
1131 | 1131 | | |
1132 | 1132 | | |
1133 | 1133 | | |
1134 | | - | |
| 1134 | + | |
1135 | 1135 | | |
1136 | 1136 | | |
1137 | 1137 | | |
| |||
1140 | 1140 | | |
1141 | 1141 | | |
1142 | 1142 | | |
1143 | | - | |
| 1143 | + | |
1144 | 1144 | | |
1145 | 1145 | | |
1146 | 1146 | | |
| |||
1150 | 1150 | | |
1151 | 1151 | | |
1152 | 1152 | | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
1153 | 1165 | | |
1154 | 1166 | | |
1155 | 1167 | | |
| |||
1158 | 1170 | | |
1159 | 1171 | | |
1160 | 1172 | | |
1161 | | - | |
| 1173 | + | |
1162 | 1174 | | |
1163 | 1175 | | |
1164 | 1176 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1065 | 1065 | | |
1066 | 1066 | | |
1067 | 1067 | | |
1068 | | - | |
| 1068 | + | |
1069 | 1069 | | |
1070 | 1070 | | |
1071 | 1071 | | |
1072 | 1072 | | |
1073 | 1073 | | |
1074 | 1074 | | |
1075 | | - | |
| 1075 | + | |
1076 | 1076 | | |
1077 | 1077 | | |
1078 | 1078 | | |
| |||
1081 | 1081 | | |
1082 | 1082 | | |
1083 | 1083 | | |
1084 | | - | |
| 1084 | + | |
1085 | 1085 | | |
1086 | 1086 | | |
1087 | 1087 | | |
| |||
1091 | 1091 | | |
1092 | 1092 | | |
1093 | 1093 | | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
1094 | 1106 | | |
1095 | 1107 | | |
1096 | 1108 | | |
| |||
1099 | 1111 | | |
1100 | 1112 | | |
1101 | 1113 | | |
1102 | | - | |
| 1114 | + | |
1103 | 1115 | | |
1104 | 1116 | | |
1105 | 1117 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1298 | 1298 | | |
1299 | 1299 | | |
1300 | 1300 | | |
1301 | | - | |
| 1301 | + | |
1302 | 1302 | | |
1303 | 1303 | | |
1304 | 1304 | | |
1305 | 1305 | | |
1306 | 1306 | | |
1307 | 1307 | | |
1308 | | - | |
| 1308 | + | |
1309 | 1309 | | |
1310 | 1310 | | |
1311 | 1311 | | |
| |||
1314 | 1314 | | |
1315 | 1315 | | |
1316 | 1316 | | |
1317 | | - | |
| 1317 | + | |
1318 | 1318 | | |
1319 | 1319 | | |
1320 | 1320 | | |
| |||
1324 | 1324 | | |
1325 | 1325 | | |
1326 | 1326 | | |
| 1327 | + | |
| 1328 | + | |
| 1329 | + | |
| 1330 | + | |
| 1331 | + | |
| 1332 | + | |
| 1333 | + | |
| 1334 | + | |
| 1335 | + | |
| 1336 | + | |
| 1337 | + | |
| 1338 | + | |
1327 | 1339 | | |
1328 | 1340 | | |
1329 | 1341 | | |
| |||
1333 | 1345 | | |
1334 | 1346 | | |
1335 | 1347 | | |
1336 | | - | |
| 1348 | + | |
1337 | 1349 | | |
1338 | 1350 | | |
1339 | 1351 | | |
| |||
1342 | 1354 | | |
1343 | 1355 | | |
1344 | 1356 | | |
1345 | | - | |
| 1357 | + | |
1346 | 1358 | | |
1347 | 1359 | | |
1348 | 1360 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1397 | 1397 | | |
1398 | 1398 | | |
1399 | 1399 | | |
1400 | | - | |
| 1400 | + | |
1401 | 1401 | | |
1402 | 1402 | | |
1403 | 1403 | | |
1404 | 1404 | | |
1405 | 1405 | | |
1406 | 1406 | | |
1407 | | - | |
| 1407 | + | |
1408 | 1408 | | |
1409 | 1409 | | |
1410 | 1410 | | |
| |||
1413 | 1413 | | |
1414 | 1414 | | |
1415 | 1415 | | |
1416 | | - | |
| 1416 | + | |
1417 | 1417 | | |
1418 | 1418 | | |
1419 | 1419 | | |
| |||
1423 | 1423 | | |
1424 | 1424 | | |
1425 | 1425 | | |
| 1426 | + | |
| 1427 | + | |
| 1428 | + | |
| 1429 | + | |
| 1430 | + | |
| 1431 | + | |
| 1432 | + | |
| 1433 | + | |
| 1434 | + | |
| 1435 | + | |
| 1436 | + | |
| 1437 | + | |
1426 | 1438 | | |
1427 | 1439 | | |
1428 | 1440 | | |
| |||
1432 | 1444 | | |
1433 | 1445 | | |
1434 | 1446 | | |
1435 | | - | |
| 1447 | + | |
1436 | 1448 | | |
1437 | 1449 | | |
1438 | 1450 | | |
| |||
1441 | 1453 | | |
1442 | 1454 | | |
1443 | 1455 | | |
1444 | | - | |
| 1456 | + | |
1445 | 1457 | | |
1446 | 1458 | | |
1447 | 1459 | | |
| |||
2724 | 2736 | | |
2725 | 2737 | | |
2726 | 2738 | | |
2727 | | - | |
| 2739 | + | |
2728 | 2740 | | |
2729 | 2741 | | |
2730 | | - | |
| 2742 | + | |
2731 | 2743 | | |
2732 | 2744 | | |
2733 | 2745 | | |
| |||
2738 | 2750 | | |
2739 | 2751 | | |
2740 | 2752 | | |
2741 | | - | |
2742 | | - | |
2743 | | - | |
2744 | | - | |
2745 | | - | |
| 2753 | + | |
| 2754 | + | |
| 2755 | + | |
| 2756 | + | |
| 2757 | + | |
| 2758 | + | |
| 2759 | + | |
| 2760 | + | |
| 2761 | + | |
| 2762 | + | |
2746 | 2763 | | |
2747 | 2764 | | |
2748 | 2765 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
161 | 161 | | |
162 | 162 | | |
163 | 163 | | |
164 | | - | |
| 164 | + | |
165 | 165 | | |
166 | 166 | | |
167 | 167 | | |
| |||
0 commit comments