Skip to content

Commit 0c4279a

Browse files
committed
perf: add a GroupsAccumulator for PivotFirst
PivotFirst only had a row Accumulator, so grouped aggregation went through DataFusion's GroupsAccumulatorAdapter, which visits every group's state on each batch, and the accumulator built a ScalarValue for every input row. The grouped accumulator keeps every group's slots in one flat, group-major buffer with a validity bitmap, and returns them as the list without copying. Pivot values are looked up in a typed index built once per aggregate, with Spark's float rules in a new float_semantics::ieee_equality_key helper. The ungrouped accumulator shares the index. A pivot list with a repeated value is rejected at construction, since the serde falls back for it.
1 parent 1c6d38f commit 0c4279a

3 files changed

Lines changed: 927 additions & 212 deletions

File tree

‎native/core/src/execution/planner.rs‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3567,9 +3567,8 @@ impl PhysicalPlanner {
35673567
self.create_expr(expr.value_column.as_ref().unwrap(), Arc::clone(&schema))?;
35683568
let value_type = to_arrow_datatype(expr.value_datatype.as_ref().unwrap());
35693569
// Reconstruct pivot values as ScalarValues from the serialized Literal exprs.
3570-
// The Scala serde builds them as Literal(v, pivot_column.dataType), so extracting
3571-
// via Literal downcast gives us the exact ScalarValue the update path compares
3572-
// against per input row.
3570+
// The Scala serde builds them as Literal(v, pivot_column.dataType), so they carry
3571+
// the pivot column's type, which `SparkPivotFirst` builds its lookup index from.
35733572
let mut pivot_values = Vec::with_capacity(expr.pivot_values.len());
35743573
for lit_expr in &expr.pivot_values {
35753574
let physical = self.create_expr(lit_expr, Arc::clone(&schema))?;
@@ -3580,8 +3579,10 @@ impl PhysicalPlanner {
35803579
})?;
35813580
pivot_values.push(literal.value().clone());
35823581
}
3583-
let func =
3584-
AggregateUDF::new_from_impl(SparkPivotFirst::new(value_type, pivot_values));
3582+
let func = AggregateUDF::new_from_impl(SparkPivotFirst::try_new(
3583+
value_type,
3584+
pivot_values,
3585+
)?);
35853586
Self::create_aggr_func_expr("pivot_first", schema, vec![pivot_col, value_col], func)
35863587
}
35873588
}

0 commit comments

Comments
 (0)