Repository navigation
spark: prune struct fields inside arrays (blocked on list layout) #16
Description
Goal
Extend nested schema pruning (#9, #10) to struct fields inside arrays. For example, SELECT a.f3 FROM t where a is array<struct<...>> should read only f3.
Why it's blocked
Pruning only saves I/O when the file stores a list column with the list layout (vortex.list), which keeps the elements as their own layout tree.
vortex.listis only in theunstableedition (vortex-edition/src/declarations/unstable/mod.rs).- The JNI session enables only
CORE_2026_08_3(vortex-jni/src/session.rs), so Spark writes every list column as a single flat leaf. - Reading any element field of a flat leaf reads the whole list segment, so there's nothing to prune.
Work once vortex.list is in an edition we enable
- List reader pushdown (
vortex-layout/src/layouts/list/reader.rs): readlist_transform($, λ)as offsets and validity plusλevaluated by the elements reader, for lambdas that can't fail. Today it reads every element and applies the lambda afterwards. maskmarked infallible, so Spark'smask(pack(...), is_not_null(x))lambdas qualify for (1).- Struct reader: accept
list_transformover a single field in the direct-projection path. - JNI:
Expression.variableandExpression.listTransform. - Spark:
NestedSchemaPruningbuildslist_transformforArrayType(StructType). Maps can follow.
Upstream risk
list_transform landed in vortex-data#10503. The open draft vortex-data#10696 proposes a different design for higher-order functions; if it's chosen, steps 1, 3 and 4 change shape.
Activity
gitRasheed commentedon Oct 28, 2026
Status check, 2026-10-28. Still blocked.
vortex.listis still only in theunstableedition on upstream develop (40e0b1c), and the JNI session still enables onlyCORE_2026_08_3.list_transform's lambda body intoListLayoutelement projection vortex-data/vortex#10948 pushes an infalliblelist_transformlambda's body into the list layout's element reads. If it merges, step 1 is done and step 3 likely shrinks, since it already reusesstep_into_struct_field.maskdoesn't overrideis_infallible, so Spark'smask(pack(...), is_not_null(x))lambdas won't take that path yet.Nothing to do here until
vortex.listlands in an edition we enable. I'll recheck when either upstream draft moves.ChristopherWirt commentedon Nov 2, 2026
Resolved on develop by #35, incorporating #42, with list-layout edition isolation subsequently merged in #57. Spark now generates native list-transform projections for struct fields inside arrays when files use vortex.listLayout=true. VortexNestedSchemaPruningIoTest covers array-element field I/O pruning and nested/null/empty-array correctness against unpruned reads and Parquet. Flat-layout files correctly remain whole-leaf reads, as described in this issue. Confirmed against develop abb4830 by source and existing regression inspection; no tests were rerun for this audit.