You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit d54ce65
Browse filesBrowse the repository at this point in the historyBrowse files
authored
perf: optimize arrays_zip perfect list zips (apache#22285)
## Which issue does this PR close?
- Closesapache#22225.
## Rationale for this change
`arrays_zip` currently uses the general `MutableArrayData` path even
when all regular `ListArray` inputs are already perfectly aligned. In
that case, the output list offsets match the inputs and each struct
child column can reuse the corresponding input values array instead of
copying one row at a time.
## What changes are included in this PR?
- Add a fast path for perfect regular `ListArray` zips that reuses the
first input's offsets and clones the input values arrays into the output
struct children.
- Keep the existing general path for ragged inputs, `LargeList`,
`FixedSizeList`, `Null` inputs, and null rows that would require
padding.
- Add unit coverage for offset/value reuse, zero-length null rows, and
null rows with hidden values falling back to the general path.
- Rename the no-null benchmark case to `arrays_zip_perfect_zip_8192`.
## Are these changes tested?
Yes. Local checks run:
- `cargo fmt --all`
- `cargo test -p datafusion-functions-nested`
- `cargo clippy -p datafusion-functions-nested --all-targets
--all-features -- -D warnings`
- `CARGO_TARGET_DIR=C:\df-target cargo clippy --all-targets
--all-features -- -D warnings`
- `CARGO_TARGET_DIR=C:\df-target cargo bench -p
datafusion-functions-nested --bench arrays_zip -- --warm-up-time 1
--measurement-time 2 --sample-size 10`
Latest local benchmark sample:
- `arrays_zip_perfect_zip_8192`: `11.234 ?s 11.600 ?s 12.045 ?s`
- `arrays_zip_10pct_nulls_8192`: `4.3463 ms 4.5531 ms 4.7898 ms`
## Are there any user-facing changes?
No. This is an internal performance optimization with the same
`arrays_zip` output semantics.
## Review follow-up validation
2026-05-21:
- `cargo fmt --all --check` passed after formatting.
- `cargo test -p datafusion-functions-nested
perfect_zip_uses_supplied_field_names --lib` could not compile locally
because this Windows GNU toolchain is missing `dlltool.exe`.
- `cargo clippy -p datafusion-functions-nested --all-targets -- -D
warnings` hit the same local `dlltool.exe` toolchain blocker before
checking the crate.
---------
Co-authored-by: Puneet Dixit <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <[email protected]>
Co-authored-by: puneetdixit200 <[email protected]>
0 commit comments