Skip to content

Push several list_transform lambdas into one ListLayout element read - #11273

Draft
mhk197 wants to merge 1 commit into
mk/list-transform-element-pushdownfrom
mk/list-transform-multi-lambda
Draft

mhk197 wants to merge 1 commit into
mk/list-transform-element-pushdownfrom
mk/list-transform-multi-lambda

Conversation

@mhk197

@mhk197 mhk197 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Several distinct list_transform lambdas over one list, or a lambda beside the list read whole, go into one elements read. The elements reader packs one value per lambda, plus the raw element when needed, and the lists come back as a struct of lists sharing the offsets and validity. Over the whole stack, the microbench geomean of 112 cases is 0.65x of develop.

Changes

  • split_list_transform (list/expr.rs): packs the bodies as "0", "1", ... and rewrites each use to its list.
  • lists_over (list/reader.rs): one list per pack field, extracted lazily from the packed elements.

Testing

  • list_transform_reads_only_what_its_body_uses::two_lambdas now reads only a and f.
  • random_list_uses_match_memory covers two lambdas beside list_sum, list_length, is_null, and the whole list.

Follow-up

  • Cache the split per expression, and look through list_transform for is_null and list_length.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant