Skip to content

Commit 329bba2

Browse files
authored
SQL style null-on-empty semantics for Sum and Mean (#9113)
This changes the existing `vortex.sum` to use SQL-style null-on-empty semantics. ## New Sum semantics | Input | Result | | --- | --- | | No rows | `null` | | All values null | `null` | | Valid values summing to zero | `0` | | Integer or decimal overflow | `null` | | All NaNs with the default `skip_nans` | `0` | | Any NaN with `include_nans` | `NaN` | NaNs count as valid values even when skipped, which is why an all-NaN Sum is zero rather than empty. ## Partial state Every new Sum partial is: ```text { sum: T, is_overflow: bool, is_empty: bool, } ``` The `sum` field is non-nullable. The outer struct is nullable so grouped or null rows can still be represented. `skip_nans` remains aggregate configuration and is not part of the partial. A fresh accumulator begins as: ```text { sum: 0, is_overflow: false, is_empty: true } ``` Observing a valid value clears `is_empty`. Empty states are merge identities, overflow is absorbing, and finalization returns null when either `is_empty` or `is_overflow` is true. Grouped Sum kernels produce this same state directly. As a result, `list_sum` no longer needs a separate empty-list mask: null, empty, and all-null lists naturally finalize to null. ## Mean Mean remains a combined `Sum + Count` aggregate and now uses the canonical Sum state. It still finalizes as `sum / count` and retains its `count == 0` guard. - Empty or all-null Mean returns `null`. - All-NaN Mean with `skip_nans` returns `null` because count is zero. - An overflowed Sum produces a null Mean. - Normal input produces `sum / count`. ## Backwards compatibility Legacy scalar partials are normalized into the canonical struct: | Incoming partial | Canonical interpretation | Final behavior | | --- | --- | --- | | Legacy scalar `x` | `{ sum: x, is_overflow: false, is_empty: false }` | `x` | | Legacy scalar `0` | `{ sum: 0, is_overflow: false, is_empty: false }` | `0` | | Legacy scalar `null` | `{ sum: 0, is_overflow: true, is_empty: false }` | `null` | | New struct partial | Used directly | New semantics | This preserves zero-on-empty when an old partial appears. The legacy scalar representation could not distinguish an empty Sum from a genuine zero, so every legacy zero remains a nonempty zero. Both scalar and array partials are normalized. Cached scalar `Stat::Sum` values also pass through this compatibility path. ## Sum options and zone-map deserialization Sum now uses `SumAggregateOpts { skip_nans, struct_partial }`. New/default Sum options set and serialize `struct_partial = true`, selecting the canonical struct partial. Historical Sum options serialized only `skip_nans`. The new options decoder is wire-compatible with those bytes and treats a missing `struct_partial` field as `false`. That selects the legacy nullable scalar partial dtype when an existing zone map is deserialized. This is needed because a zoned layout derives its stored stats-table schema from the deserialized aggregate options. The option therefore lets old zone maps validate and read their scalar Sum fields, while newly written zone maps derive the struct Sum field. Legacy scalar fields are normalized once after reading, and the rest of pruning uses the canonical representation. No separate Sum-specific zoned metadata version or schema branch is needed. ## Forward compatibility New files store struct Sum partials in zoned stats tables. Readers from before this change ignore the new protobuf option field, derive the historical scalar Sum dtype, and reject the struct stats field during dtype validation. This is intentional under Vortex's current compatibility contract: new readers remain able to read older files, but older readers are not yet guaranteed to read newer files. The zoned metadata version remains unchanged because `struct_partial` disambiguates the stored schema for new readers; bumping the version would only make older readers reject the layout earlier. Closes #9084. --------- Signed-off-by: Matt Katz <[email protected]>
1 parent 19f771f commit 329bba2

27 files changed

Lines changed: 1901 additions & 370 deletions

File tree

‎vortex-array/benches/aggregate_grouped.rs‎

Lines changed: 59 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -11,12 +11,12 @@ use rand::RngExt;
1111
use rand::SeedableRng;
1212
use rand::rngs::StdRng;
1313
use vortex_array::ArrayRef;
14+
use vortex_array::Canonical;
1415
use vortex_array::IntoArray;
1516
use vortex_array::VortexSessionExecute;
1617
use vortex_array::aggregate_fn::AggregateFnVTable;
1718
use vortex_array::aggregate_fn::DynGroupedAccumulator;
1819
use vortex_array::aggregate_fn::GroupedAccumulator;
19-
use vortex_array::aggregate_fn::NumericalAggregateOpts;
2020
use vortex_array::aggregate_fn::fns::count::Count;
2121
use vortex_array::aggregate_fn::fns::sum::Sum;
2222
use vortex_array::arrays::ListViewArray;
@@ -154,14 +154,12 @@ fn list_element_dtype(list_view: &ArrayRef) -> DType {
154154

155155
fn grouped_accumulator<V>(list_view: &ArrayRef, vtable: V) -> ArrayRef
156156
where
157-
V: AggregateFnVTable<Options = NumericalAggregateOpts> + Clone,
157+
V: AggregateFnVTable + Clone,
158+
V::Options: Default,
158159
{
159-
let mut acc = GroupedAccumulator::try_new(
160-
vtable,
161-
NumericalAggregateOpts::default(),
162-
list_element_dtype(list_view),
163-
)
164-
.unwrap();
160+
let mut acc =
161+
GroupedAccumulator::try_new(vtable, V::Options::default(), list_element_dtype(list_view))
162+
.unwrap();
165163
acc.accumulate_list(list_view, &mut SESSION.create_execution_ctx())
166164
.unwrap();
167165
divan::black_box(acc.finish().unwrap())
@@ -199,6 +197,59 @@ fn sum_f64_clustered_nulls(bencher: Bencher) {
199197
.bench_refs(|input| grouped_accumulator(input, Sum));
200198
}
201199

200+
/// Like [`grouped_accumulator`], but executes the lazy finalize result to canonical so the
201+
/// bench measures the full cost of producing usable sums.
202+
fn grouped_accumulator_canonical<V>(list_view: &ArrayRef, vtable: V) -> ArrayRef
203+
where
204+
V: AggregateFnVTable + Clone,
205+
V::Options: Default,
206+
{
207+
let mut acc =
208+
GroupedAccumulator::try_new(vtable, V::Options::default(), list_element_dtype(list_view))
209+
.unwrap();
210+
let mut ctx = SESSION.create_execution_ctx();
211+
acc.accumulate_list(list_view, &mut ctx).unwrap();
212+
let result = acc
213+
.finish()
214+
.unwrap()
215+
.execute::<Canonical>(&mut ctx)
216+
.unwrap()
217+
.into_array();
218+
divan::black_box(result)
219+
}
220+
221+
#[divan::bench]
222+
fn canonical_sum_i32_nullable_all_valid(bencher: Bencher) {
223+
let input = i32_nullable_all_valid_input();
224+
bencher
225+
.with_inputs(|| &input)
226+
.bench_refs(|input| grouped_accumulator_canonical(input, Sum));
227+
}
228+
229+
#[divan::bench]
230+
fn canonical_sum_i32_clustered_nulls(bencher: Bencher) {
231+
let input = i32_clustered_nulls_input();
232+
bencher
233+
.with_inputs(|| &input)
234+
.bench_refs(|input| grouped_accumulator_canonical(input, Sum));
235+
}
236+
237+
#[divan::bench]
238+
fn canonical_sum_f64_all_valid(bencher: Bencher) {
239+
let input = f64_all_valid_input();
240+
bencher
241+
.with_inputs(|| &input)
242+
.bench_refs(|input| grouped_accumulator_canonical(input, Sum));
243+
}
244+
245+
#[divan::bench]
246+
fn canonical_sum_f64_clustered_nulls(bencher: Bencher) {
247+
let input = f64_clustered_nulls_input();
248+
bencher
249+
.with_inputs(|| &input)
250+
.bench_refs(|input| grouped_accumulator_canonical(input, Sum));
251+
}
252+
202253
#[divan::bench]
203254
fn count_i32_clustered_nulls(bencher: Bencher) {
204255
let input = i32_clustered_nulls_input();

‎vortex-array/src/aggregate_fn/accumulator.rs‎

Lines changed: 76 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -119,9 +119,18 @@ impl<V: AggregateFnVTable> DynAccumulator for Accumulator<V> {
119119
batch.dtype()
120120
);
121121

122-
// 0. Legacy stats bridge: if this aggregate is still cached under a legacy Stat slot,
123-
// consume that exact stat before kernel dispatch or decode.
122+
// Stat::Sum stores a finalized scalar rather than a partial. Give Sum's vtable the first
123+
// chance to consume that cache before dispatching an encoding kernel.
124+
let checked_cached_sum = Stat::from_aggregate_fn(&self.aggregate_fn) == Some(Stat::Sum)
125+
&& batch.statistics().get(Stat::Sum).is_exact();
126+
if checked_cached_sum && self.vtable.try_accumulate(&mut self.partial, batch, ctx)? {
127+
return Ok(());
128+
}
129+
130+
// 0. Cached stats bridge: consume an exact partial from the aggregate's Stat slot before
131+
// kernel dispatch or decode. Sum is handled above because its cache stores a result.
124132
if let Some(stat) = Stat::from_aggregate_fn(&self.aggregate_fn)
133+
&& stat != Stat::Sum
125134
&& let Precision::Exact(partial) = batch.statistics().get(stat)
126135
{
127136
let partial = if partial.dtype() == &self.partial_dtype {
@@ -167,7 +176,7 @@ impl<V: AggregateFnVTable> DynAccumulator for Accumulator<V> {
167176
}
168177

169178
// 2. Allow the vtable to short-circuit on the raw array before decompression.
170-
if self.vtable.try_accumulate(&mut self.partial, batch, ctx)? {
179+
if !checked_cached_sum && self.vtable.try_accumulate(&mut self.partial, batch, ctx)? {
171180
return Ok(());
172181
}
173182

@@ -280,6 +289,7 @@ mod tests {
280289
use crate::aggregate_fn::combined::PairOptions;
281290
use crate::aggregate_fn::fns::mean::Mean;
282291
use crate::aggregate_fn::fns::sum::Sum;
292+
use crate::aggregate_fn::fns::sum::SumAggregateOpts;
283293
use crate::aggregate_fn::kernels::DynAggregateKernel;
284294
use crate::aggregate_fn::session::AggregateFnSession;
285295
use crate::array::VTable;
@@ -288,7 +298,10 @@ mod tests {
288298
use crate::dtype::DType;
289299
use crate::dtype::Nullability;
290300
use crate::dtype::PType;
301+
use crate::expr::stats::Precision;
302+
use crate::expr::stats::Stat;
291303
use crate::scalar::Scalar;
304+
use crate::scalar::ScalarValue;
292305

293306
/// Mean partial sentinel `{sum: 42.0, count: 1}` — distinguishable from the
294307
/// natural fan-out result `{sum: 7.0, count: 1}` that `Combined::try_accumulate`
@@ -320,7 +333,7 @@ mod tests {
320333
}
321334
}
322335

323-
/// Sum partial sentinel `42.0` — distinguishable from the natural Sum of
336+
/// Sum partial sentinel `{sum: 42.0, is_overflow: false, is_empty: false}` — distinguishable from the natural Sum of
324337
/// `dict_of_seven()` which is `7.0`.
325338
#[derive(Debug)]
326339
struct SentinelSumPartialKernel;
@@ -331,7 +344,7 @@ mod tests {
331344
_batch: &ArrayRef,
332345
_ctx: &mut ExecutionCtx,
333346
) -> VortexResult<Option<Scalar>> {
334-
Ok(Some(Scalar::primitive(42.0f64, Nullability::Nullable)))
347+
Ok(Some(sum_partial(42.0)))
335348
}
336349
}
337350

@@ -350,7 +363,7 @@ mod tests {
350363
Accumulator::try_new(
351364
Mean::combined(),
352365
PairOptions(
353-
NumericalAggregateOpts::default(),
366+
SumAggregateOpts::default(),
354367
NumericalAggregateOpts::default(),
355368
),
356369
dtype,
@@ -359,11 +372,24 @@ mod tests {
359372

360373
fn sentinel_partial() -> Scalar {
361374
let acc = mean_f64_accumulator().expect("build accumulator");
362-
let sum = Scalar::primitive(42.0f64, Nullability::Nullable);
375+
let sum = sum_partial(42.0);
363376
let count = Scalar::primitive(1u64, Nullability::NonNullable);
364377
Scalar::struct_(acc.partial_dtype, vec![sum, count])
365378
}
366379

380+
fn sum_partial(value: f64) -> Scalar {
381+
let dtype = DType::Primitive(PType::F64, Nullability::NonNullable);
382+
let acc = Accumulator::try_new(Sum, SumAggregateOpts::default(), dtype).expect("sum");
383+
Scalar::struct_(
384+
acc.partial_dtype,
385+
vec![
386+
Scalar::primitive(value, Nullability::NonNullable),
387+
Scalar::bool(false, Nullability::NonNullable),
388+
Scalar::bool(false, Nullability::NonNullable),
389+
],
390+
)
391+
}
392+
367393
/// Kernel registered for `(Dict, Combined<Mean>)` fires in preference to
368394
/// `Combined::try_accumulate`'s fan-out path — proves the dispatch reorder.
369395
#[test]
@@ -381,7 +407,13 @@ mod tests {
381407

382408
let s = partial.as_struct();
383409
assert_eq!(
384-
s.field("sum").unwrap().as_primitive().as_::<f64>(),
410+
s.field("sum")
411+
.unwrap()
412+
.as_struct()
413+
.field("sum")
414+
.unwrap()
415+
.as_primitive()
416+
.as_::<f64>(),
385417
Some(42.0)
386418
);
387419
assert_eq!(
@@ -408,7 +440,13 @@ mod tests {
408440

409441
let s = partial.as_struct();
410442
assert_eq!(
411-
s.field("sum").unwrap().as_primitive().as_::<f64>(),
443+
s.field("sum")
444+
.unwrap()
445+
.as_struct()
446+
.field("sum")
447+
.unwrap()
448+
.as_primitive()
449+
.as_::<f64>(),
412450
Some(7.0)
413451
);
414452
assert_eq!(
@@ -439,7 +477,13 @@ mod tests {
439477
// via `Combined<Mean>`'s fan-out. `Count`'s native `try_accumulate` reads the
440478
// batch's valid_count, so count is the real 1.
441479
assert_eq!(
442-
s.field("sum").unwrap().as_primitive().as_::<f64>(),
480+
s.field("sum")
481+
.unwrap()
482+
.as_struct()
483+
.field("sum")
484+
.unwrap()
485+
.as_primitive()
486+
.as_::<f64>(),
443487
Some(42.0)
444488
);
445489
assert_eq!(
@@ -448,4 +492,26 @@ mod tests {
448492
);
449493
Ok(())
450494
}
495+
496+
#[test]
497+
fn cached_sum_precedes_encoding_kernel() -> VortexResult<()> {
498+
static KERNEL: SentinelSumPartialKernel = SentinelSumPartialKernel;
499+
let session = fresh_session();
500+
session
501+
.get::<AggregateFnSession>()
502+
.register_aggregate_kernel(Dict.id(), Some(Sum.id()), &KERNEL);
503+
let mut ctx = session.create_execution_ctx();
504+
505+
let batch = dict_of_seven();
506+
batch
507+
.statistics()
508+
.set(Stat::Sum, Precision::Exact(ScalarValue::from(11.0f64)));
509+
510+
let dtype = DType::Primitive(PType::F64, Nullability::NonNullable);
511+
let mut acc = Accumulator::try_new(Sum, SumAggregateOpts::default(), dtype)?;
512+
acc.accumulate(&batch, &mut ctx)?;
513+
514+
assert_eq!(acc.finish()?.as_primitive().as_::<f64>(), Some(11.0));
515+
Ok(())
516+
}
451517
}

‎vortex-array/src/aggregate_fn/accumulator_grouped.rs‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -273,7 +273,10 @@ impl<V: AggregateFnVTable> DynGroupedAccumulator for GroupedAccumulator<V> {
273273
}
274274

275275
fn flush(&mut self) -> VortexResult<ArrayRef> {
276-
let states = std::mem::take(&mut self.partials);
276+
let mut states = std::mem::take(&mut self.partials);
277+
if states.len() == 1 {
278+
return Ok(states.pop().vortex_expect("checked one partial"));
279+
}
277280
Ok(ChunkedArray::try_new(states, self.partial_dtype.clone())?.into_array())
278281
}
279282

‎vortex-array/src/aggregate_fn/fns/mean/mod.rs‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ use crate::aggregate_fn::combined::CombinedOptions;
1919
use crate::aggregate_fn::combined::PairOptions;
2020
use crate::aggregate_fn::fns::count::Count;
2121
use crate::aggregate_fn::fns::sum::Sum;
22+
use crate::aggregate_fn::fns::sum::SumAggregateOpts;
2223
use crate::aggregate_fn::fns::sum::sum_decimal_dtype;
2324
use crate::arrays::ConstantArray;
2425
use crate::builtins::ArrayBuiltins;
@@ -40,7 +41,7 @@ pub fn mean(array: &ArrayRef, ctx: &mut ExecutionCtx) -> VortexResult<Scalar> {
4041
let mut acc = Accumulator::try_new(
4142
Mean::combined(),
4243
PairOptions(
43-
NumericalAggregateOpts::default(),
44+
SumAggregateOpts::default(),
4445
NumericalAggregateOpts::default(),
4546
),
4647
array.dtype().clone(),
@@ -319,7 +320,7 @@ mod tests {
319320
let keep_nans = NumericalAggregateOpts::include_nans();
320321
let mut acc = Accumulator::try_new(
321322
Mean::combined(),
322-
PairOptions(keep_nans, keep_nans),
323+
PairOptions(keep_nans.into(), keep_nans),
323324
array.dtype().clone(),
324325
)?;
325326
acc.accumulate(&array, &mut ctx)?;
@@ -411,7 +412,7 @@ mod tests {
411412
let mut acc = Accumulator::try_new(
412413
Mean::combined(),
413414
PairOptions(
414-
NumericalAggregateOpts::default(),
415+
SumAggregateOpts::default(),
415416
NumericalAggregateOpts::default(),
416417
),
417418
dtype,
@@ -446,7 +447,7 @@ mod tests {
446447
let mut acc = Accumulator::try_new(
447448
Mean::combined(),
448449
PairOptions(
449-
NumericalAggregateOpts::default(),
450+
SumAggregateOpts::default(),
450451
NumericalAggregateOpts::default(),
451452
),
452453
DType::Primitive(PType::F64, Nullability::Nullable),
@@ -474,7 +475,7 @@ mod tests {
474475
let mut acc = GroupedAccumulator::try_new(
475476
Mean::combined(),
476477
PairOptions(
477-
NumericalAggregateOpts::default(),
478+
SumAggregateOpts::default(),
478479
NumericalAggregateOpts::default(),
479480
),
480481
DType::Primitive(PType::F64, Nullability::Nullable),

0 commit comments

Comments
 (0)