Skip to content

Optimize arrays_zip for perfectly aligned arrays - #22245

Closed
BipashaBi wants to merge 2 commits into
apache:mainfrom
BipashaBi:arrays-zip-fast-path
Closed

Optimize arrays_zip for perfectly aligned arrays#22245
BipashaBi wants to merge 2 commits into
apache:mainfrom
BipashaBi:arrays-zip-fast-path

Conversation

@BipashaBi

@BipashaBi BipashaBi commented May 16, 2026 •
edited by Jefffrey
Loading

Copy link
Copy Markdown

Which issue does this PR close?

Rationale for this change

arrays_zip currently reconstructs output arrays row-by-row using MutableArrayData even when all input arrays are perfectly aligned and require no null padding.

In the perfectly aligned case, the output can reuse the existing child arrays and offsets directly, avoiding unnecessary copying and allocations.

What changes are included in this PR?

  • Added a fast-path optimization for perfectly aligned arrays
  • Added is_perfect_zip() validation helper
  • Added try_fast_path() zero-copy execution path
  • Reused child value arrays directly when possible
  • Reused list offsets instead of rebuilding them
  • Preserved the existing implementation as a fallback for ragged/null-padded inputs
  • Added a dedicated fast-path test

Are these changes tested?

  • Added a dedicated fast-path test for perfectly aligned arrays
  • Ran cargo check
  • Ran cargo test --package datafusion-functions-nested
  • Verified existing tests continue to pass

Are there any user-facing changes?

No user-facing changes. This is an internal performance optimization for arrays_zip.

@github-actions github-actions Bot added the functions Changes to functions implementation label May 16, 2026
@Jefffrey Jefffrey mentioned this pull request May 20, 2026
Merged
@Omega359

Omega359 commented May 23, 2026

Copy link
Copy Markdown
Contributor

There is a correctness issue: the fast path reuses the first concrete input’s list null bitmap as the output null bitmap at arrays_zip.rs. The slow path marks an output row null only when all concrete input lists are null. The fast path currently marks it according to the first input.

That changes semantics when inputs are perfectly aligned but have different list-level nulls.

select arrays_zip(a,b) as zipped
from (values
  (null::int[], []::int[]),
  ([]::int[], null::int[]),
  (null::int[], null::int[])
) as t(a,b);

upstream/main returns:

[]
[]
NULL

This PR returns:

NULL
[]
NULL

I would suggest adding this to the arrays_zip.slt file

# mixed NULL list and non-null empty list: result row is non-null empty
query ?
select arrays_zip(a, b) from (values
  (null::int[], []::int[]),
  ([]::int[], null::int[]),
  (null::int[], null::int[])
) as t(a, b);
----
[]
[]
NULL

Attached is a benchmark update to test the performance of this change.
arrays_zip_benchmark_main.patch

@adriangb

adriangb commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

#22285 is merged so I think we can close this. Thanks for contribution @BipashaBi

@adriangb adriangb closed this Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

functions Changes to functions implementation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Optimize arrays_zip to avoid row-by-row copying in the perfect-zip case

3 participants

@BipashaBi @Omega359 @adriangb