Skip to content

Commit df2667f

Browse files
committed
feat(physical-plan): add GroupColumn support for FixedSizeList / List / LargeList / Struct in multi-column GROUP BY
Addresses apache#22682 by adding column-native GroupColumn implementations for the nested types, so a multi-column GROUP BY containing any of these no longer falls back from `GroupValuesColumn` to the byte-encoded `GroupValuesRows` path. ## Summary Today `multi_group_by::supported_schema` is all-or-nothing: if any single GROUP BY column is not in the `supported_type` allow-list, the whole grouping falls back to `GroupValuesRows`, even when every other column would have qualified for the fast column-wise + cross-column short-circuit path. The most common offender in real workloads is a single nested column (e.g. `LargeList<Struct<Utf8, LargeUtf8>>`) that drags an otherwise primitive-heavy 30+ column GROUP BY onto the slow path. This PR adds: - `FixedSizeListGroupValueBuilder<T>` for `FixedSizeList<primitive>`. Uses an inner `PrimitiveGroupValueBuilder` for the flat child storage. - `StructGroupValueBuilder` for `Struct<...>` whose fields are themselves `GroupColumn`-supported. Recursive composition: each child field is itself constructed via the factory. - `ListGroupValueBuilder<O>` (`O: OffsetSizeTrait`) for `List<T>` and `LargeList<T>`. Owns an offsets `Vec<O>` and a child `Box<dyn GroupColumn>` built via the factory. Null outer rows occupy a zero-length range so the offset buffer stays monotonic without consuming child slots. - A recursive `make_group_column(field: &Field) -> Result<Box<dyn GroupColumn>>` factory that replaces the giant inline match in `GroupValuesColumn::intern`. The factory walks nested fields and recursively builds children for `Struct` / `List` / `LargeList` cases. - `supported_type` recursively checks children for `List` / `LargeList` / `Struct`. `FixedSizeList` remains restricted to primitive children in this PR; composing FSL with nested children can follow in a separate PR. ## Tests (38 new, 68/68 group_values pass) Per-builder: - append / build round trip - `equal_to` for outer nulls, inner nulls, identical, different (and for List: different lengths, empty) - **sliced input array** (offset != 0) — critical real-world case where the input array carries a non-zero slice offset - `take_n` boundary cases: `n=0`, `n=len`, prefix-contains-nulls - `vectorized_equal_to` / `vectorized_append` match per-row reference - `size()` grows with appends - `build` on an empty builder returns an empty array Composition: - `Struct<Struct<...>>` (two-level nested struct) - `List<List<Int32>>` (two-level nested list) - `LargeList<Struct<Utf8, Int32>>` (mirrors the production-shape nested type that motivated this PR) Dispatcher and invariants: - Each builder is exercised end-to-end via `new_group_values` to prove the dispatcher routes the schema to `GroupValuesColumn` (not `GroupValuesRows`) and dedup is correct across batches. - `supported_type` accepts every primitive + nested combination listed above and rejects `Float16`, `Decimal256`, `Time64(Second)`, `List<Float16>`, `Struct{Float16}`, `FixedSizeList<Utf8>`, etc. - A `supported_type` <-> `make_group_column` consistency fuzz over 21 supported + 9 unsupported types asserts the biconditional. Locks the invariant against future drift between the allow-list and the dispatcher. ## Known limitations - `FixedSizeList<T>` is restricted to primitive `T`. Composing FSL with `Struct` or `List` children can follow in a separate PR. - `vectorized_equal_to` / `vectorized_append` for the new builders fall back to per-row loops. They are correctness-equivalent to the specialized vectorized paths used by the primitive builders, but a follow-up can add type-specialized batched comparators where beneficial. - `Map` support is not in this PR; ordering semantics need a separate discussion. ## No regression on existing types The factory refactor preserves the existing specialization order: primitive / byte / byte-view / boolean / decimal128 builders are still matched first. Only types that were previously rejected fall through to the new builders. All existing aggregate tests (115/115) pass unchanged.
1 parent 48adae4 commit df2667f

4 files changed

Lines changed: 2468 additions & 185 deletions

File tree

0 commit comments

Comments
 (0)