Repository navigation
Commit 1c6d38f
committed
fix: fall back for PivotFirst pivot columns and lists Comet cannot match
Spark finds a pivot value's slot in a Scala HashMap only for atomic pivot
columns, and even there a binary value matches by reference, so no row ever
matches before Spark 4.2. Arrays and structs (including the struct that a
pivot on several columns builds) are matched through an interpreted ordering,
and from 4.2 so are binary values and non-default collations. The native
lookup reproduces none of that, so those pivot columns now fall back.
Spark also sizes the aggregate buffer from the distinct pivot values, so a
list with duplicates (including 0.0 next to -0.0) made the native partial
aggregate emit more columns than the shuffle schema declares. Such lists fall
back too.
The SQL fixture now asserts native execution for every query that should run
natively, covers each pivot column type that does, and checks each fallback.1 parent a7ba06d commit 1c6d38f
4 files changed
Lines changed: 209 additions & 33 deletions
File tree
- docs/source
- contributor-guide/expression-audits
- user-guide/latest
- spark/src
- main/scala/org/apache/comet/serde
- test/resources/sql-tests/expressions/aggregate
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
111 | 111 | | |
112 | 112 | | |
113 | 113 | | |
114 | | - | |
| 114 | + | |
115 | 115 | | |
116 | 116 | | |
117 | | - | |
118 | | - | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
119 | 120 | | |
120 | 121 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
123 | 123 | | |
124 | 124 | | |
125 | 125 | | |
126 | | - | |
| 126 | + | |
127 | 127 | | |
128 | 128 | | |
129 | 129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1434 | 1434 | | |
1435 | 1435 | | |
1436 | 1436 | | |
1437 | | - | |
| 1437 | + | |
1438 | 1438 | | |
1439 | 1439 | | |
1440 | 1440 | | |
1441 | 1441 | | |
1442 | 1442 | | |
1443 | 1443 | | |
| 1444 | + | |
| 1445 | + | |
| 1446 | + | |
1444 | 1447 | | |
1445 | 1448 | | |
| 1449 | + | |
| 1450 | + | |
1446 | 1451 | | |
1447 | 1452 | | |
1448 | 1453 | | |
1449 | | - | |
| 1454 | + | |
| 1455 | + | |
| 1456 | + | |
| 1457 | + | |
| 1458 | + | |
| 1459 | + | |
| 1460 | + | |
| 1461 | + | |
| 1462 | + | |
| 1463 | + | |
| 1464 | + | |
| 1465 | + | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
| 1469 | + | |
| 1470 | + | |
| 1471 | + | |
1450 | 1472 | | |
1451 | 1473 | | |
| 1474 | + | |
1452 | 1475 | | |
1453 | 1476 | | |
| 1477 | + | |
| 1478 | + | |
1454 | 1479 | | |
1455 | 1480 | | |
| 1481 | + | |
| 1482 | + | |
| 1483 | + | |
| 1484 | + | |
| 1485 | + | |
1456 | 1486 | | |
1457 | 1487 | | |
1458 | 1488 | | |
| |||
0 commit comments