Repository navigation
Commit df2667f
committed
feat(physical-plan): add GroupColumn support for FixedSizeList / List / LargeList / Struct in multi-column GROUP BY
Addresses apache#22682 by adding column-native GroupColumn implementations for
the nested types, so a multi-column GROUP BY containing any of these no
longer falls back from `GroupValuesColumn` to the byte-encoded
`GroupValuesRows` path.
## Summary
Today `multi_group_by::supported_schema` is all-or-nothing: if any single
GROUP BY column is not in the `supported_type` allow-list, the whole
grouping falls back to `GroupValuesRows`, even when every other column
would have qualified for the fast column-wise + cross-column short-circuit
path. The most common offender in real workloads is a single nested
column (e.g. `LargeList<Struct<Utf8, LargeUtf8>>`) that drags an
otherwise primitive-heavy 30+ column GROUP BY onto the slow path.
This PR adds:
- `FixedSizeListGroupValueBuilder<T>` for `FixedSizeList<primitive>`.
Uses an inner `PrimitiveGroupValueBuilder` for the flat child storage.
- `StructGroupValueBuilder` for `Struct<...>` whose fields are themselves
`GroupColumn`-supported. Recursive composition: each child field is
itself constructed via the factory.
- `ListGroupValueBuilder<O>` (`O: OffsetSizeTrait`) for `List<T>` and
`LargeList<T>`. Owns an offsets `Vec<O>` and a child `Box<dyn
GroupColumn>` built via the factory. Null outer rows occupy a
zero-length range so the offset buffer stays monotonic without
consuming child slots.
- A recursive `make_group_column(field: &Field) -> Result<Box<dyn
GroupColumn>>` factory that replaces the giant inline match in
`GroupValuesColumn::intern`. The factory walks nested fields and
recursively builds children for `Struct` / `List` / `LargeList` cases.
- `supported_type` recursively checks children for `List` / `LargeList`
/ `Struct`. `FixedSizeList` remains restricted to primitive children
in this PR; composing FSL with nested children can follow in a
separate PR.
## Tests (38 new, 68/68 group_values pass)
Per-builder:
- append / build round trip
- `equal_to` for outer nulls, inner nulls, identical, different (and
for List: different lengths, empty)
- **sliced input array** (offset != 0) — critical real-world case where
the input array carries a non-zero slice offset
- `take_n` boundary cases: `n=0`, `n=len`, prefix-contains-nulls
- `vectorized_equal_to` / `vectorized_append` match per-row reference
- `size()` grows with appends
- `build` on an empty builder returns an empty array
Composition:
- `Struct<Struct<...>>` (two-level nested struct)
- `List<List<Int32>>` (two-level nested list)
- `LargeList<Struct<Utf8, Int32>>` (mirrors the production-shape nested
type that motivated this PR)
Dispatcher and invariants:
- Each builder is exercised end-to-end via `new_group_values` to prove
the dispatcher routes the schema to `GroupValuesColumn` (not
`GroupValuesRows`) and dedup is correct across batches.
- `supported_type` accepts every primitive + nested combination listed
above and rejects `Float16`, `Decimal256`, `Time64(Second)`,
`List<Float16>`, `Struct{Float16}`, `FixedSizeList<Utf8>`, etc.
- A `supported_type` <-> `make_group_column` consistency fuzz over 21
supported + 9 unsupported types asserts the biconditional. Locks the
invariant against future drift between the allow-list and the
dispatcher.
## Known limitations
- `FixedSizeList<T>` is restricted to primitive `T`. Composing FSL with
`Struct` or `List` children can follow in a separate PR.
- `vectorized_equal_to` / `vectorized_append` for the new builders fall
back to per-row loops. They are correctness-equivalent to the
specialized vectorized paths used by the primitive builders, but a
follow-up can add type-specialized batched comparators where
beneficial.
- `Map` support is not in this PR; ordering semantics need a separate
discussion.
## No regression on existing types
The factory refactor preserves the existing specialization order:
primitive / byte / byte-view / boolean / decimal128 builders are still
matched first. Only types that were previously rejected fall through to
the new builders. All existing aggregate tests (115/115) pass unchanged.1 parent 48adae4 commit df2667f
4 files changed
Lines changed: 2468 additions & 185 deletions
File tree
- datafusion/physical-plan/src/aggregates/group_values/multi_group_by
0 commit comments