Repository navigation
Commit 329bba2
authored
SQL style null-on-empty semantics for Sum and Mean (#9113)
This changes the existing `vortex.sum` to use SQL-style null-on-empty
semantics.
## New Sum semantics
| Input | Result |
| --- | --- |
| No rows | `null` |
| All values null | `null` |
| Valid values summing to zero | `0` |
| Integer or decimal overflow | `null` |
| All NaNs with the default `skip_nans` | `0` |
| Any NaN with `include_nans` | `NaN` |
NaNs count as valid values even when skipped, which is why an all-NaN
Sum is zero rather than empty.
## Partial state
Every new Sum partial is:
```text
{
sum: T,
is_overflow: bool,
is_empty: bool,
}
```
The `sum` field is non-nullable. The outer struct is nullable so grouped
or null rows can still be represented. `skip_nans` remains aggregate
configuration and is not part of the partial.
A fresh accumulator begins as:
```text
{ sum: 0, is_overflow: false, is_empty: true }
```
Observing a valid value clears `is_empty`. Empty states are merge
identities, overflow is absorbing, and finalization returns null when
either `is_empty` or `is_overflow` is true.
Grouped Sum kernels produce this same state directly. As a result,
`list_sum` no longer needs a separate empty-list mask: null, empty, and
all-null lists naturally finalize to null.
## Mean
Mean remains a combined `Sum + Count` aggregate and now uses the
canonical Sum state. It still finalizes as `sum / count` and retains its
`count == 0` guard.
- Empty or all-null Mean returns `null`.
- All-NaN Mean with `skip_nans` returns `null` because count is zero.
- An overflowed Sum produces a null Mean.
- Normal input produces `sum / count`.
## Backwards compatibility
Legacy scalar partials are normalized into the canonical struct:
| Incoming partial | Canonical interpretation | Final behavior |
| --- | --- | --- |
| Legacy scalar `x` | `{ sum: x, is_overflow: false, is_empty: false }`
| `x` |
| Legacy scalar `0` | `{ sum: 0, is_overflow: false, is_empty: false }`
| `0` |
| Legacy scalar `null` | `{ sum: 0, is_overflow: true, is_empty: false
}` | `null` |
| New struct partial | Used directly | New semantics |
This preserves zero-on-empty when an old partial appears. The legacy
scalar representation could not distinguish an empty Sum from a genuine
zero, so every legacy zero remains a nonempty zero.
Both scalar and array partials are normalized. Cached scalar `Stat::Sum`
values also pass through this compatibility path.
## Sum options and zone-map deserialization
Sum now uses `SumAggregateOpts { skip_nans, struct_partial }`.
New/default Sum options set and serialize `struct_partial = true`,
selecting the canonical struct partial.
Historical Sum options serialized only `skip_nans`. The new options
decoder is wire-compatible with those bytes and treats a missing
`struct_partial` field as `false`. That selects the legacy nullable
scalar partial dtype when an existing zone map is deserialized.
This is needed because a zoned layout derives its stored stats-table
schema from the deserialized aggregate options. The option therefore
lets old zone maps validate and read their scalar Sum fields, while
newly written zone maps derive the struct Sum field. Legacy scalar
fields are normalized once after reading, and the rest of pruning uses
the canonical representation.
No separate Sum-specific zoned metadata version or schema branch is
needed.
## Forward compatibility
New files store struct Sum partials in zoned stats tables. Readers from
before this change ignore the new protobuf option field, derive the
historical scalar Sum dtype, and reject the struct stats field during
dtype validation.
This is intentional under Vortex's current compatibility contract: new
readers remain able to read older files, but older readers are not yet
guaranteed to read newer files. The zoned metadata version remains
unchanged because `struct_partial` disambiguates the stored schema for
new readers; bumping the version would only make older readers reject
the layout earlier.
Closes #9084.
---------
Signed-off-by: Matt Katz <[email protected]>1 parent 19f771f commit 329bba2
27 files changed
Lines changed: 1901 additions & 370 deletions
File tree
- vortex-array
- benches
- src
- aggregate_fn
- fns
- mean
- sum
- arrays/chunked/compute
- expr/stats
- scalar_fn/fns
- stats
- vortex-duckdb/src/convert
- vortex-layout/src/layouts/zoned
- vortex-proto
- proto
- src/generated
- vortex-python/src
- vortex-sqllogictest/slt/duckdb
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | | - | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
154 | 154 | | |
155 | 155 | | |
156 | 156 | | |
157 | | - | |
| 157 | + | |
| 158 | + | |
158 | 159 | | |
159 | | - | |
160 | | - | |
161 | | - | |
162 | | - | |
163 | | - | |
164 | | - | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
165 | 163 | | |
166 | 164 | | |
167 | 165 | | |
| |||
199 | 197 | | |
200 | 198 | | |
201 | 199 | | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
202 | 253 | | |
203 | 254 | | |
204 | 255 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
122 | | - | |
123 | | - | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
124 | 132 | | |
| 133 | + | |
125 | 134 | | |
126 | 135 | | |
127 | 136 | | |
| |||
167 | 176 | | |
168 | 177 | | |
169 | 178 | | |
170 | | - | |
| 179 | + | |
171 | 180 | | |
172 | 181 | | |
173 | 182 | | |
| |||
280 | 289 | | |
281 | 290 | | |
282 | 291 | | |
| 292 | + | |
283 | 293 | | |
284 | 294 | | |
285 | 295 | | |
| |||
288 | 298 | | |
289 | 299 | | |
290 | 300 | | |
| 301 | + | |
| 302 | + | |
291 | 303 | | |
| 304 | + | |
292 | 305 | | |
293 | 306 | | |
294 | 307 | | |
| |||
320 | 333 | | |
321 | 334 | | |
322 | 335 | | |
323 | | - | |
| 336 | + | |
324 | 337 | | |
325 | 338 | | |
326 | 339 | | |
| |||
331 | 344 | | |
332 | 345 | | |
333 | 346 | | |
334 | | - | |
| 347 | + | |
335 | 348 | | |
336 | 349 | | |
337 | 350 | | |
| |||
350 | 363 | | |
351 | 364 | | |
352 | 365 | | |
353 | | - | |
| 366 | + | |
354 | 367 | | |
355 | 368 | | |
356 | 369 | | |
| |||
359 | 372 | | |
360 | 373 | | |
361 | 374 | | |
362 | | - | |
| 375 | + | |
363 | 376 | | |
364 | 377 | | |
365 | 378 | | |
366 | 379 | | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
367 | 393 | | |
368 | 394 | | |
369 | 395 | | |
| |||
381 | 407 | | |
382 | 408 | | |
383 | 409 | | |
384 | | - | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
385 | 417 | | |
386 | 418 | | |
387 | 419 | | |
| |||
408 | 440 | | |
409 | 441 | | |
410 | 442 | | |
411 | | - | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
412 | 450 | | |
413 | 451 | | |
414 | 452 | | |
| |||
439 | 477 | | |
440 | 478 | | |
441 | 479 | | |
442 | | - | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
443 | 487 | | |
444 | 488 | | |
445 | 489 | | |
| |||
448 | 492 | | |
449 | 493 | | |
450 | 494 | | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
451 | 517 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
273 | 273 | | |
274 | 274 | | |
275 | 275 | | |
276 | | - | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
277 | 280 | | |
278 | 281 | | |
279 | 282 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
40 | 41 | | |
41 | 42 | | |
42 | 43 | | |
43 | | - | |
| 44 | + | |
44 | 45 | | |
45 | 46 | | |
46 | 47 | | |
| |||
319 | 320 | | |
320 | 321 | | |
321 | 322 | | |
322 | | - | |
| 323 | + | |
323 | 324 | | |
324 | 325 | | |
325 | 326 | | |
| |||
411 | 412 | | |
412 | 413 | | |
413 | 414 | | |
414 | | - | |
| 415 | + | |
415 | 416 | | |
416 | 417 | | |
417 | 418 | | |
| |||
446 | 447 | | |
447 | 448 | | |
448 | 449 | | |
449 | | - | |
| 450 | + | |
450 | 451 | | |
451 | 452 | | |
452 | 453 | | |
| |||
474 | 475 | | |
475 | 476 | | |
476 | 477 | | |
477 | | - | |
| 478 | + | |
478 | 479 | | |
479 | 480 | | |
480 | 481 | | |
| |||
0 commit comments