Skip to content

Push list_transform's lambda body into ListLayout element projection - #10948

Open
mhk197 wants to merge 1 commit into
mk/list-masked-element-readsfrom
mk/list-transform-element-pushdown
Open

mhk197 wants to merge 1 commit into
mk/list-masked-element-readsfrom
mk/list-transform-element-pushdown

Conversation

@mhk197

@mhk197 mhk197 commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

The list reader evaluates a list_transform lambda body inside the elements read, so it reads only the element fields the body uses. A body that can fail runs only on the elements of selected valid lists: the read takes the bounded path from #11272 and masks out null lists' elements, which can hold anything.

Changes

  • split_list_transform (list/expr.rs): one list_transform($, λ), with the list read whole nowhere else, becomes λ's body over the elements and the rest of the expression over the lists of its values.
  • project_all (list/reader.rs): a fallible body over a nullable list takes the bounded path, where select_lists keeps only selected valid lists.
  • step_into_struct_field and Item are pub(crate) so the list reader can bind the body over the elements.
microbench (list-layout file, full scan unless noted) develop this PR MB read
list<struct<a, b, c, d, e>>, x -> x.a + x.b 1.96 ms 1.77 ms 15.6 → 5.1
list<struct<a, b, c, d, e>>, x -> x.a and list_length(l) 1.42 ms 1.21 ms 15.6 → 1.7
list<struct<a, b>>, 30% null lists over MAX, x -> x.a + x.b 4.24 ms 2.19 ms 3.1 → 3.1

Testing

  • list_transform_reads_only_what_its_body_uses: the segments each expression requests, nested lists and fallible bodies included.
  • fallible_body_sees_only_selected_valid_lists: x + 1 over lists hiding MAX under nulls and in an unselected list.

Follow-up

@codspeed

codspeed Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Merging this PR will not alter performance

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

✅ 1917 untouched benchmarks
⏩ 47 skipped benchmarks1


Comparing mk/list-transform-element-pushdown (ec1d74b) with develop (345b6d9)2

Open in CodSpeed

Footnotes

  1. 47 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on mk/list-masked-element-reads (34f556e) during the generation of this report, so develop (345b6d9) was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

@mhk197 mhk197 changed the title Push list_transform's lambda body into list layout element reads Push list_transform's lambda body into list layout element reads Oct 29, 2026
@mhk197 mhk197 changed the title Push list_transform's lambda body into list layout element reads Push list_transform's lambda body into ListLayout element projection Oct 29, 2026
@mhk197
mhk197 marked this pull request as ready for review October 29, 2026 11:50
@mhk197
mhk197 force-pushed the mk/list-transform-element-pushdown branch from ef22a72 to d9f0f02 Compare November 5, 2026 15:58
@mhk197
mhk197 changed the base branch from develop to mk/datafusion-array-transform-pr November 5, 2026 15:58
@mhk197
mhk197 added this pull request to stack #11032 November 5, 2026 15:58
@mhk197 mhk197 added the changelog/performance A performance improvement label Nov 5, 2026
@mhk197
mhk197 removed this pull request from stack #11032 November 5, 2026 16:52
@mhk197
mhk197 changed the base branch from mk/datafusion-array-transform-pr to mk/list-masked-element-reads November 5, 2026 16:52
@mhk197
mhk197 added this pull request to stack #11274 November 5, 2026 16:52
@mhk197
mhk197 force-pushed the mk/list-transform-element-pushdown branch from d9f0f02 to c7b6347 Compare November 5, 2026 17:40
@robert3005
robert3005 force-pushed the mk/list-transform-element-pushdown branch from c7b6347 to ec1d74b Compare November 6, 2026 11:02

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/performance A performance improvement

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant