You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
list_extract: replace per-row MutableArrayData gather with take and zip #5100
list_extract in native/spark-expr/src/array_funcs/list_extract.rs gathers one element per row from the flat values buffer with a per-row MutableArrayData loop (extend one element, or extend from the one-row default array, or extend nulls). This is a hand-rolled take, and per-row extend calls are slower than a single gather.
Describe the potential solution
Two-pass structure:
Compute an index array per row: Some(start + adjusted_index), None for null list, null ordinal, or out-of-bounds, erroring early for fail_on_error exactly as today (the row-by-row index computation preserves error-before-result ordering).
One arrow::compute::take(values, &indices, None) (take handles null indices as null output and works on any element type, including nested).
Default substitution with arrow::compute::kernels::zip::zip(mask, default_scalar, taken), where the mask fires only on non-null-list, non-null-index out-of-bounds rows.
Additional context
Null propagation is identical; the mask construction for default substitution is the only subtle part. Likely a performance win (one gather versus per-row extends).
Found during an audit of native code that replicates existing arrow-rs kernels.
Activity
peterxcli commentedon Jul 31, 2026
take