Repository navigation
Conversation
Merging this PR will not alter performance1 warning
Comparing Footnotes
|
list_transform's lambda body into list layout element reads
list_transform's lambda body into list layout element readslist_transform's lambda body into ListLayout element projection
mhk197
marked this pull request as ready for review
October 29, 2026 11:50
mhk197
force-pushed
the
mk/list-transform-element-pushdown
branch
from
November 5, 2026 15:58
ef22a72 to
d9f0f02
Compare
mhk197
changed the base branch from
develop
to
mk/datafusion-array-transform-pr
November 5, 2026 15:58
mhk197
changed the base branch from
mk/datafusion-array-transform-pr
to
mk/list-masked-element-reads
November 5, 2026 16:52
mhk197
force-pushed
the
mk/list-transform-element-pushdown
branch
from
November 5, 2026 17:40
d9f0f02 to
c7b6347
Compare
Signed-off-by: Matt Katz <[email protected]>
robert3005
force-pushed
the
mk/list-transform-element-pushdown
branch
from
November 6, 2026 11:02
c7b6347 to
ec1d74b
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The list reader evaluates a
list_transformlambda body inside the elements read, so it reads only the element fields the body uses. A body that can fail runs only on the elements of selected valid lists: the read takes the bounded path from #11272 and masks out null lists' elements, which can hold anything.Changes
split_list_transform(list/expr.rs): onelist_transform($, λ), with the list read whole nowhere else, becomes λ's body over the elements and the rest of the expression over the lists of its values.project_all(list/reader.rs): a fallible body over a nullable list takes the bounded path, whereselect_listskeeps only selected valid lists.step_into_struct_fieldandItemarepub(crate)so the list reader can bind the body over the elements.list<struct<a, b, c, d, e>>,x -> x.a + x.blist<struct<a, b, c, d, e>>,x -> x.aandlist_length(l)list<struct<a, b>>, 30% null lists overMAX,x -> x.a + x.bTesting
list_transform_reads_only_what_its_body_uses: the segments each expression requests, nested lists and fallible bodies included.fallible_body_sees_only_selected_valid_lists:x + 1over lists hidingMAXunder nulls and in an unselected list.Follow-up
list_transformlambdas into oneListLayoutelement read #11273 pushes several lambdas, and a lambda beside the whole list, into one read.