Skip to content

Commit 1c6d38f

Browse files
committed
fix: fall back for PivotFirst pivot columns and lists Comet cannot match
Spark finds a pivot value's slot in a Scala HashMap only for atomic pivot columns, and even there a binary value matches by reference, so no row ever matches before Spark 4.2. Arrays and structs (including the struct that a pivot on several columns builds) are matched through an interpreted ordering, and from 4.2 so are binary values and non-default collations. The native lookup reproduces none of that, so those pivot columns now fall back. Spark also sizes the aggregate buffer from the distinct pivot values, so a list with duplicates (including 0.0 next to -0.0) made the native partial aggregate emit more columns than the shuffle schema declares. Such lists fall back too. The SQL fixture now asserts native execution for every query that should run natively, covers each pivot column type that does, and checks each fallback.
1 parent a7ba06d commit 1c6d38f

4 files changed

Lines changed: 209 additions & 33 deletions

File tree

‎docs/source/contributor-guide/expression-audits/agg_funcs.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -111,10 +111,11 @@
111111

112112
## pivot_first
113113

114-
- Spark 3.4.3 (audited 2026-07-02): `PivotFirst(pivotColumn, valueColumn, pivotColumnValues)` is an internal `ImperativeAggregate` emitted only by the optimized-pivot fast path in `Analyzer.ResolvePivot`. It buckets `valueColumn` into an array of length `pivotColumnValues.size` indexed by matching `pivotColumn`. Null value columns are ignored, unmatched pivot values are dropped. Value types are gated by `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal).
114+
- Spark 3.4.3 (audited 2026-07-02): `PivotFirst(pivotColumn, valueColumn, pivotColumnValues)` is an internal `ImperativeAggregate` emitted only by the optimized-pivot fast path in `Analyzer.ResolvePivot`. It buckets `valueColumn` into an array of length `indexSize` indexed by matching `pivotColumn`. Null value columns are ignored, unmatched pivot values are dropped. Value types are gated by `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal). The index is a Scala `HashMap[Any, Int]` for an `AtomicType` pivot column, so a value matches by Scala `==` on the boxed Catalyst value: `-0.0` matches `0.0`, NaN matches nothing, and a binary value (an `Array[Byte]`) matches only itself by reference, so no row ever does. Other pivot columns (arrays, and the struct that a pivot on several columns builds) use a `TreeMap` ordered by `TypeUtils.getInterpretedOrdering`. `indexSize` is the number of distinct keys, and a repeated pivot value keeps its last position, so with duplicates the buffer is narrower than the list and a row matching a repeated value is written past the end of it.
115115
- Spark 3.5.8 (audited 2026-07-02): swaps `StructType.fromAttributes` for `DataTypeUtils.fromAttributes`; no behavior change.
116116
- Spark 4.0.1 (audited 2026-07-02): identical to 3.5.8.
117-
- Spark 4.1.1 (audited 2026-07-02): identical to 4.0.1. (Spark `master` adds a defensive `findPivotIndex` wrapper that returns `-1` for null keys on the non-`AtomicType` `TreeMap` path; not present in any released version Comet builds against, and Comet's HashMap-backed lookup handles `ScalarValue::Null` safely on all types.)
118-
- `CometPivotFirst` (in `spark/src/main/scala/org/apache/comet/serde/aggregates.scala`) forwards the aggregate to the native `SparkPivotFirst` UDAF (`native/spark-expr/src/agg_funcs/pivot_first.rs`) when the value type is in the supported set. State layout matches Spark's `aggBufferAttributes` (one scalar column per pivot slot) so the shuffle schema between Partial and Final stays consistent. `evaluate()` reassembles the slots into a `ListArray` matching `PivotFirst.dataType = ArrayType(valueDataType)`.
117+
- Spark 4.1.1 (audited 2026-07-02): identical to 4.0.1.
118+
- Spark 4.2.0 (audited 2026-10-12): the `TreeMap` is chosen for every pivot column type without proper equals (`!TypeUtils.typeWithProperEquals`), which adds binary and strings with a non-binary-equality collation, so binary pivot values now match by content. A `findPivotIndex` wrapper returns `-1` for a null key on the `TreeMap` path instead of throwing.
119+
- `CometPivotFirst` (in `spark/src/main/scala/org/apache/comet/serde/aggregates.scala`) forwards the aggregate to the native `SparkPivotFirst` UDAF (`native/spark-expr/src/agg_funcs/pivot_first.rs`) when the value type is in the supported set and the pivot column is a Boolean, Byte, Short, Int, Long, Float, Double, Decimal, UTF8_BINARY String, Date, Timestamp or TimestampNTZ. Those are the types Spark keys by value equality on every version, which the native lookup reproduces, including the float rules. Other pivot columns fall back, as does a list where `indexSize` is smaller than the list (duplicates, including `0.0` next to `-0.0`). State layout matches Spark's `aggBufferAttributes` (one scalar column per pivot slot) so the shuffle schema between Partial and Final stays consistent. The result reassembles the slots into a `ListArray` matching `PivotFirst.dataType = ArrayType(valueDataType)`.
119120

120121
[Spark Expression Support]: ../../user-guide/latest/expressions.md

‎docs/source/user-guide/latest/expressions.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -123,7 +123,7 @@ The tables below list every Spark built-in expression with its current status.
123123
| `percentile_approx` | ✅ | Native | Alias of `approx_percentile`; same restrictions apply |
124124
| `percentile_cont` | ✅ | — | Spark 4.0+ `WITHIN GROUP (ORDER BY ...)`; ascending only runs natively, `DESC` falls back to Spark |
125125
| `percentile_disc` | 🔜 | — | Percentile aggregate |
126-
| `pivot_first` | ✅ | Native | Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark's standard filtered-aggregate path. |
126+
| `pivot_first` | ✅ | Native | Internal aggregate for the optimized `PIVOT` fast path. Value type must be in `PivotFirst.supportsDataType` (Boolean, Byte, Short, Int, Long, Float, Double, Decimal); other types keep Spark's standard filtered-aggregate path. Binary, array and struct pivot columns (including pivots on more than one column), non-default collations and pivot lists with duplicate values fall back to Spark |
127127
| `regr_avgx` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
128128
| `regr_avgy` | ✅ | — | Native: Spark rewrites to `Average` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |
129129
| `regr_count` | ✅ | — | Native: Spark rewrites to `Count` (tests in [#4551](https://github.com/apache/datafusion-comet/pull/4551)) |

‎spark/src/main/scala/org/apache/comet/serde/aggregates.scala‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1434,25 +1434,55 @@ object CometMode extends CometAggregateExpressionSerde[Mode] with CometTypeShim
14341434
}
14351435
}
14361436

1437-
object CometPivotFirst extends CometAggregateExpressionSerde[PivotFirst] {
1437+
object CometPivotFirst extends CometAggregateExpressionSerde[PivotFirst] with CometTypeShim {
14381438

14391439
// Delegate to Spark's own PivotFirst.supportsDataType so the two lists cannot drift if a
14401440
// future Spark version adds a value type to the fast-path gate.
14411441
private def unsupportedValueTypeReason(dt: DataType): String =
14421442
s"Unsupported value data type: $dt"
14431443

1444+
private def unsupportedPivotTypeReason(dt: DataType): String =
1445+
s"Unsupported pivot column data type: $dt"
1446+
14441447
private val emptyPivotValuesReason = "Pivot values list is empty"
14451448

1449+
private val duplicatePivotValuesReason = "Pivot values list contains duplicates"
1450+
14461451
override def getUnsupportedReasons(): Seq[String] = Seq(
14471452
"Value data type outside PivotFirst.supportsDataType " +
14481453
"(Boolean, Byte, Short, Int, Long, Float, Double, Decimal)",
1449-
emptyPivotValuesReason)
1454+
"Pivot column of a type other than Boolean, Byte, Short, Int, Long, Float, Double, " +
1455+
"Decimal, String (UTF8_BINARY collation), Date, Timestamp or TimestampNTZ, which " +
1456+
"includes the struct that a pivot on more than one column builds",
1457+
emptyPivotValuesReason,
1458+
s"$duplicatePivotValuesReason, including `0.0` next to `-0.0`")
1459+
1460+
// The pivot column types for which Spark finds a value's slot in a `HashMap` keyed by plain
1461+
// value equality, which the native lookup reproduces. Spark keys arrays and structs in a
1462+
// `TreeMap` by their interpreted ordering. It keys binary values by reference identity before
1463+
// 4.2 and by ordering from 4.2, which orders strings with a non-UTF8_BINARY collation too.
1464+
private def isSupportedPivotType(dt: DataType): Boolean = dt match {
1465+
case BooleanType | ByteType | ShortType | IntegerType | LongType => true
1466+
case FloatType | DoubleType => true
1467+
case _: DecimalType => true
1468+
case DateType | TimestampType | TimestampNTZType => true
1469+
case st: StringType => !isStringCollationType(st)
1470+
case _ => false
1471+
}
14501472

14511473
override def getSupportLevel(expr: PivotFirst): SupportLevel = {
1474+
val pivotType = expr.pivotColumn.dataType
14521475
if (!PivotFirst.supportsDataType(expr.valueDataType)) {
14531476
Unsupported(Some(unsupportedValueTypeReason(expr.valueDataType)))
1477+
} else if (!isSupportedPivotType(pivotType)) {
1478+
Unsupported(Some(unsupportedPivotTypeReason(pivotType)))
14541479
} else if (expr.pivotColumnValues.isEmpty) {
14551480
Unsupported(Some(emptyPivotValuesReason))
1481+
} else if (expr.indexSize != expr.pivotColumnValues.length) {
1482+
// Spark sizes the aggregate buffer from its index of the pivot values, which holds a
1483+
// repeated value once, while the native aggregate keeps one slot per list entry. The
1484+
// native partial aggregate would emit more columns than the shuffle schema declares.
1485+
Unsupported(Some(duplicatePivotValuesReason))
14561486
} else {
14571487
Compatible()
14581488
}

0 commit comments

Comments
 (0)