Repository navigation
Optimize arrays_zip to avoid row-by-row copying in the perfect-zip case #22225
Description
Is your feature request related to a problem or challenge?
arrays_zip (arrays_zip_inner_with_field in datafusion/functions-nested/src/arrays_zip.rs) always assembles its output by walking every row through per-column MutableArrayData builders, copying each input slice one row at a time (builder.extend(0, start, end)) and padding shorter rows with NULLs (builder.extend_nulls(...)).
When the inputs form a perfect zip — every input array has identical per-row element lengths, no null list rows with non-zero element slots, and therefore no null padding is needed — this row-by-row copy is wasted work. In that case the resulting struct child columns are bit-identical to the (concatenated) input value arrays, and the list offsets are identical to the inputs' offsets.
Describe the solution you'd like
Detect the perfect-zip case up front and skip the MutableArrayData path entirely:
- Build the output struct child columns directly from the original input value
ArrayRefs (clone / concat, no per-row copy). - Reuse an input array's offset buffer for the resulting
ListArrayinstead of rebuilding it.
This keeps the existing general path as a fallback for the ragged / null-padded cases.
Describe alternatives you've considered
Keep the current always-copy implementation. It is correct but does avoidable work for the common case where all zipped arrays line up.
Additional context
Raised by @paleolimbot while reviewing #21984:
Not here, but for the perfect zip (all value arrays the same length, no nulls with non-zero element slot lengths, no null padding needed) this should ideally be just clones of the original arrayrefs
Split out of #21984 (a metadata-propagation bugfix) since this is an orthogonal performance optimization that warrants its own benchmarks and edge-case tests.
Activity
comphead commentedon May 15, 2026
@adriangb are you planning to work on that? @hsiang-c cc
adriangb commentedon May 15, 2026
No, I am happy to have someone else handle it!
comphead commentedon May 15, 2026
Okie, we'll probably pick it up later next week as this function is heavily used in some of our workloads
BipashaBi commentedon May 16, 2026
greetings @adriangb !
i have a PR which solves this issue. Please kindly check it ASAP.
puneetdixit200 commentedon May 16, 2026
I’d like to work on this optimization. I can add a benchmark for the perfect-zip case, avoid row-by-row copying where input arrays align, and verify output compatibility with existing nested-array tests. Please assign this to me.
adriangb commentedon May 16, 2026
Hi @BipashaBi, a link to the PR would be much more helpful than a screenshot. For what it’s worth I also suggest to not ask for things “ASAP”: reviewers have limited time and nowadays especially with so much AI generated code are under a lot of pressure. Wording like “when you have time” will land a lot smoother.
@puneetdixit200 I agree that adding benchmarks first as an isolated PR is the way to go.
Since @comphead already said they’d work on this and they have a real high scale system to test against I suggest we wait to see what they come up with. Maybe @puneetdixit200 @BipashaBi can help with code review of that PR? Time spent on code review is much more valuable than code submissions, it’s a much more limited resource.
perf: optimize arrays_zip perfect list zips (apache#22285)