## Summary
Closes #9
Spark asks the scan for only the nested fields a query uses, but the
connector read every nested column whole. `SELECT s.b` decoded all of
`s`.
Struct columns are now pruned all the way down to the native scan, at
any depth. Fields inside arrays and maps are still read whole; that
needs `list_transform`
([vortex-data#10503](vortex-data#10503)).
## Changes
**Spark connector**
- `pruneColumns` keeps Spark's pruned type for struct columns.
- `NestedSchemaPruning` turns the pruned read schema into a native
projection:
- `select([b], $.s)` when only leaf fields are pruned. This keeps the
struct's own nulls.
- `mask(pack(..), is_not_null($.s))` when a child struct is pruned too.
`pack` has no nulls, so the mask carries them over. Non-nullable structs
skip the mask.
**JNI**
- Adds an `Expression.mask` binding.
**`StructReader`**
- No changes. Upstream
[vortex-data#10536](vortex-data#10536)
collapses single-field partitions, so the Spark projections are read
field by field without a special case. An earlier version of this PR
extended `StructReader`'s direct projection path, which vortex-data#10536 removed.
**Tests**
- Spark: null structs, null leaves, nulls at each level of a struct
inside a struct, and a pushed filter on a field that was pruned away.
- Layout: segment-count tests, now run against vortex-data#10536's partitioned
path. A pruned nullable struct column reads only its kept fields plus
its validity.
- `CountingSegmentSource` now lives in `segments/test.rs`, replacing
three copies.
## Benchmark
Measured before the merge of vortex-data#10536. JMH, 2 forks × 10 iterations,
release native library, Spark 4.1 local[4]. The table has 2M rows, with
a nullable 20-field struct `s` and the same struct nested at
`outer.inner`. `nestedSchemaPruning=false` is the old behaviour, where
the whole struct is read.
| Query | Pruning on | Pruning off |
|---|---|---|
| `max(s.f3)` | 26.2 ± 2.3 ms | 40.2 ± 3.4 ms |
| `max(outer.inner.f3)` | 22.0 ± 1.1 ms | 37.1 ± 2.3 ms |
## Checks
- `cargo test -p vortex-layout --lib`: 696 passed
- `cargo clippy -p vortex-layout --all-targets --all-features -- -D
warnings`: clean
- `cargo +nightly-2026-09-10 fmt --check -p vortex-layout -p
vortex-jni`: clean
- `./gradlew spotlessCheck :vortex-jni:test :vortex-spark_2.13:test
:vortex-spark_2.12:test`: all pass (JNI 97, Spark 2.13 225, Spark 2.12
225 with 2 Spark-4-only tests skipped).
- Not run since vortex-data#10536: the benchmark, the full workspace suite.
## Known gaps
- `Expression` factories don't keep their inputs reachable while the
native call runs. In theory GC can free a temporary first. This affects
every factory, not just the new `mask`, so it's left for a separate fix.
- `count(*)`, or a query that selects only partition columns, still
reads every column. This predates the PR.
- Pruning two levels inside a nullable struct reads the whole inner
struct. For `nullable_outer.inner.c`, `inner.a` and `inner.b` are still
read; sibling fields of `inner` are skipped. The cause is upstream:
`StructReader` masks `$.inner` with the parent's validity before the
`select`, so the `select` can't be pushed down into `inner`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Signed-off-by: Christopher Wirt <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Replace
StructReader'sPartitioned::Directspecial case with a generalfix in partitioned evaluation:
BoundPartitionedExpr::collapse_single_partitionsunwraps partitions thathold one sub-expression (
pack(a_0: e)becomese) and rewrites the root's$.a.a_0accesses to$.a. Field readers then evaluate$directlyinstead of wrapping every field in a one-field struct.
rather than hash order.
PartitionedEvalcaches the collapsed expression together withRootFields. When the root only packs partition results into anon-nullable struct, projection assembles the
StructArraydirectlyinstead of applying the root expression on every split.