feat(spark): push nested struct pruning down to the native scan by ChristopherWirt · Pull Request #10957 · vortex-data/vortex

ChristopherWirt · 2026-10-29T10:25:59Z

Summary

Tracking Issue: Spark connector reads whole nested columns when a query uses one field #10956

Spark asks the scan for only the nested fields a query uses, but the connector reads every nested column whole. SELECT s.b decodes all of s.

Struct columns are now pruned all the way down to the native scan, at any depth. Fields inside arrays and maps are still read whole.

#9658 rewrites the connector as a FileFormat and will conflict with this. Since it's not clear when it will land, this change targets the current connector. I'm happy to port it onto #9658 afterwards.

Changes

Spark connector

pruneColumns keeps Spark's pruned type for struct columns. Other columns keep their table type, as in fix(spark): return the requested field when Spark prunes nested fields #10724.
NestedSchemaPruning turns the pruned read schema into a native projection:
- select([b], $.s) when only leaf fields are pruned. This keeps the struct's own nulls.
- mask(pack(..), is_not_null($.s)) when a child struct is pruned too. pack has no nulls, so the mask carries them over. Non-nullable structs skip the mask.

JNI

Adds an Expression.mask binding.

vortex-layout

No reader changes. Since perf: collapse single-value partitions instead of a direct struct projection path #10536, StructReader reads these projections field by field.
New segment-count tests: a pruned nullable struct column reads only its kept fields plus its validity.
CountingSegmentSource now lives in segments/test.rs, replacing four copies.

Tests

Spark: null structs, null leaves, nulls at each level of a struct inside a struct, and a pushed filter on a field that was pruned away.
The null checks select IS NULL next to a pruned field, so they go through the pruned read, and assert the read schema Spark asks for.
Two files that differ only in nullability, read through a pruned nested struct. This covers the order where schema inference picks the nullable file. The other order fails on develop too, and fix(spark): read every column as nullable #11020 fixes it.

Tested with:

cargo test -p vortex-layout --lib: 696 passed
./gradlew :vortex-jni:test :vortex-spark_2.13:test :vortex-spark_2.12:test: all pass

Benchmark

JMH, 2 forks × 10 iterations, release native library, Spark 4.1 local[4]. The table has 2M rows, with a nullable 20-field struct s and the same struct nested at outer.inner. nestedSchemaPruning=false is the old behaviour, where the whole struct is read.

Query	Pruning on	Pruning off
`max(s.f3)`	23.6 ± 0.9 ms	36.9 ± 1.2 ms
`max(outer.inner.f3)`	21.4 ± 1.8 ms	35.0 ± 1.5 ms

Known gaps

Pruning two levels inside a nullable struct reads the whole inner struct. For nullable_outer.inner.c, inner.a and inner.b are still read, but sibling fields of inner are skipped. StructReader masks $.inner with the parent's validity before the select, so the select isn't pushed into inner.

AI assistance

Disclosed per the AI policy: the investigation, code and tests were drafted with Claude (Claude Code) and reviewed by me.

robert3005

Just the one thing about nullability

robert3005 · 2026-10-29T18:13:36Z

+        }
+        Expression packed = Expression.pack(names, children, false);
+        // `pack` builds a struct with no nulls, so carry over the source struct's nulls.
+        return nullable ? Expression.mask(packed, Expression.isNotNull(struct)) : packed;


There's a chance that this differs from the file nullability, can you add a test where two files differ in nullability only and make sure they get read correctly?

Good catch. I added a test with two files that differ only in nullability, reading a pruned nested struct through the masked path: a2a20e6. It passes, and fails if the mask is removed.

It only covers the order where the inferred file is the nullable one. In the other order, reads crash on develop too, with or without this PR, because schema inference reads one file and keeps its nullability (#11019). #11020 makes every column nullable, as Spark's file sources do. Once it lands I'll add the other order here.

…ullable Port the open Spark connector fixes to the FileFormat connector. From #10957, struct columns are now pruned to the nested fields that a query uses, at any depth. VortexScanBuilder keeps Spark's pruned type for struct columns. The partition reader projects them with select, or with pack masked by the struct's validity when a child struct is also pruned. Add the Expression.mask JNI binding that this needs. From #11020, add regression tests for files that differ only in nullability and for a user schema that declares a column non-nullable. FileTable and HadoopFsRelation already read every column as nullable, so no reader change is necessary. Signed-off-by: Robert Kruszewski <[email protected]>

robert3005 · 2026-10-30T14:38:07Z

You will need to resolve conflicts here as well

Spark asks the scan for only the nested fields a query uses, but the connector read every nested column whole, so SELECT s.b decoded all of s. pruneColumns now keeps Spark's pruned type for struct columns, and NestedSchemaPruning turns the pruned read schema into a native projection: select([b], $.s) when only leaf fields are pruned, or mask(pack(..), is_not_null($.s)) when a child struct is pruned too. Fields inside arrays and maps are still read whole. Adds an Expression.mask JNI binding, and moves the duplicated CountingSegmentSource test helper into segments/test.rs. Signed-off-by: Christopher Wirt <[email protected]>

…llability, in both orders Signed-off-by: Christopher Wirt <[email protected]>

ChristopherWirt · 2026-10-30T14:42:43Z

Rebased on #11020. filesDifferingInNullability now runs both file orders (nullable file inferred or not), and both pass.

codspeed · 2026-10-30T15:01:33Z

Merging this PR will improve performance by 21.12%

1 warning

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚡ 3 improved benchmarks
✅ 2315 untouched benchmarks

Performance Changes

	Mode	Benchmark	`BASE`	`HEAD`	Efficiency
⚡	Walltime	`filter_neon[64]`	3.1 µs	2.4 µs	+26.7%
⚡	Walltime	`threshold_neon[64]`	4 µs	3.3 µs	+20.89%
⚡	Walltime	`threshold_neon[80]`	4.7 µs	4.1 µs	+16.02%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.

_{Comparing quantcast:cwirt/spark-nested-schema-pruning (dd898c3) with develop (19dcd2f)}

joseph-isaacs · 2026-10-30T17:23:28Z

Looks like there is another merge conflict sorry

Signed-off-by: Christopher Wirt <[email protected]>

…ullable Port the Spark connector fixes from develop to the FileFormat connector. From #10957, struct columns are now pruned to the nested fields that a query uses, at any depth. VortexScanBuilder keeps Spark's pruned type for struct columns. The partition reader projects them with select, or with pack masked by the struct's validity when a child struct is also pruned. The projection uses the file's own field names, which can differ in case from the Spark names. From #11020, keep the regression tests for files that differ only in nullability and for a user schema that declares a column non-nullable. FileTable and HadoopFsRelation already read every column as nullable, so no reader change is necessary. Signed-off-by: Robert Kruszewski <[email protected]>

…11083) Follow-up to #10957. The reader matches top-level columns against each file, but it builds the nested projection from Spark's schema and never checks it against the file's struct. When a file's struct differs from the read schema, Vortex returns wrong rows or fails where Parquet returns the right ones: 1. Narrower schema. A file with `c<x, y>` read as `c STRUCT<y>` returns x's values for `c.y`. 2. Name case. A file with `s<a>` read as `s STRUCT<A>` with `caseSensitive=false` throws `A not found`. 3. Missing field. Files with `s<a>` and `s<a, b>` read as `s<a, b>` throw instead of returning a null `b` for the first file. 4. Field order. Files with `c<x, y>` and `c<y, x>` swap x and y for the second file. Cases 3 and 4 need a supplied schema, because schema merging still requires the same struct fields in every file. --------- Signed-off-by: gitRasheed <[email protected]> Signed-off-by: Robert Kruszewski <[email protected]>

robert3005 reviewed Oct 29, 2026

View reviewed changes

ChristopherWirt mentioned this pull request Oct 30, 2026

Spark reads fail with NullPointerException when files differ in nullability #11019

Closed

robert3005 approved these changes Oct 30, 2026

View reviewed changes

robert3005 enabled auto-merge (squash) October 30, 2026 14:39

test(spark): read a pruned nested struct across files differing in nu…

7c4353b

…llability, in both orders Signed-off-by: Christopher Wirt <[email protected]>

auto-merge was automatically disabled October 30, 2026 14:42
Head branch was pushed to by a user without write access

ChristopherWirt force-pushed the cwirt/spark-nested-schema-pruning branch from a2a20e6 to 7c4353b Compare October 30, 2026 14:42

Merge branch 'develop' into cwirt/spark-nested-schema-pruning

719f33a

robert3005 added the changelog/feature A new feature label Oct 30, 2026

robert3005 enabled auto-merge (squash) October 30, 2026 14:57

ChristopherWirt added 2 commits October 30, 2026 15:17

Merge branch 'develop' into cwirt/spark-nested-schema-pruning

0e878f7

Merge branch 'develop' into cwirt/spark-nested-schema-pruning

4540f3e

Merge upstream develop and resolve scan test helper conflict

8fcec6a

Signed-off-by: Christopher Wirt <[email protected]>

auto-merge was automatically disabled October 30, 2026 17:24
Head branch was pushed to by a user without write access

ChristopherWirt added 2 commits October 30, 2026 17:29

Merge branch 'develop' into cwirt/spark-nested-schema-pruning

8562bc4

Merge upstream develop into nested schema pruning

dd898c3

Signed-off-by: Christopher Wirt <[email protected]>

robert3005 merged commit 7baa376 into vortex-data:develop Oct 30, 2026
74 checks passed

gitRasheed mentioned this pull request Nov 2, 2026

fix(spark): resolve pruned nested fields against each file's struct #11083

Merged

robert3005 mentioned this pull request Nov 11, 2026

Spark connector reads whole nested columns when a query uses one field #10956

Closed

Navigation Menu

Repository navigation

feat(spark): push nested struct pruning down to the native scan - #10957

feat(spark): push nested struct pruning down to the native scan#10957
robert3005 merged 8 commits into
vortex-data:developfrom
quantcast:cwirt/spark-nested-schema-pruning

ChristopherWirt commented Oct 29, 2026 •

edited

Loading

Uh oh!

robert3005 left a comment

Uh oh!

robert3005 Oct 29, 2026

Uh oh!

ChristopherWirt Oct 30, 2026

Uh oh!

robert3005 commented Oct 30, 2026

Uh oh!

ChristopherWirt commented Oct 30, 2026

Uh oh!

codspeed Bot commented Oct 30, 2026 •

edited

Loading

Uh oh!

joseph-isaacs commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ChristopherWirt commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Benchmark

Known gaps

AI assistance

Uh oh!

robert3005 left a comment

Choose a reason for hiding this comment

Uh oh!

robert3005 Oct 29, 2026

Choose a reason for hiding this comment

Uh oh!

ChristopherWirt Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

robert3005 commented Oct 30, 2026

Uh oh!

ChristopherWirt commented Oct 30, 2026

Uh oh!

codspeed Bot commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merging this PR will improve performance by 21.12%

Performance Changes

Uh oh!

joseph-isaacs commented Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChristopherWirt commented Oct 29, 2026 •

edited

Loading

codspeed Bot commented Oct 30, 2026 •

edited

Loading