Repository navigation
fix(spark): resolve pruned nested fields against each file's struct - #11083
Merged
robert3005 merged 3 commits intoNov 2, 2026
Merged
Conversation
The reader matched top-level columns against each file, but built the nested projection from the table schema and trusted that a read struct had the file's fields, in the file's order and case. When a supplied schema did not, a read returned another field's values, failed on a field named in another case, indexed past the struct's children, or swapped fields stored in another order, where Parquet returns the right rows. The projection is now built per file against that file's Arrow schema. Columns and struct fields resolve by name under Spark's case sensitivity, through structs nested in structs, and are read under the file's names in Spark's order. A nullable column or field the file does not carry reads as null, and its struct keeps its own nulls. Pushed filters bind only paths the file carries under the same names, and Spark evaluates the rest. Signed-off-by: gitRasheed <[email protected]>
…other type Resolving a read schema scanned every file field for each read field, at every struct level, and filter restriction did the same. Both now build a name lookup once per struct, so wide schemas resolve in linear time. A read struct over a file field that is neither a struct nor an extension type now fails with the file and field path, instead of handing Spark a column of the wrong type. Test two files with different struct layouts in one query, and the new error. Use Arrow's extension-name constant, and document what the class resolves and what it still reads by position. Signed-off-by: Robert Kruszewski <[email protected]>
Merging this PR will degrade performance by 19.95%1 warning
Warning Please fix the performance issues or acknowledge them on CodSpeed. Performance Changes
Tip Investigate this regression by commenting Comparing 
|
Signed-off-by: Robert Kruszewski <[email protected]>
robert3005
approved these changes
Nov 2, 2026
robert3005
enabled auto-merge (squash)
November 2, 2026 14:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #10957. The reader matches top-level columns against each file, but it builds the nested projection from Spark's schema and never checks it against the file's struct. When a file's struct differs from the read schema, Vortex returns wrong rows or fails where Parquet returns the right ones:
c<x, y>read asc STRUCT<y>returns x's values forc.y.s<a>read ass STRUCT<A>withcaseSensitive=falsethrowsA not found.s<a>ands<a, b>read ass<a, b>throw instead of returning a nullbfor the first file.c<x, y>andc<y, x>swap x and y for the second file.Cases 3 and 4 need a supplied schema, because schema merging still requires the same struct fields in every file.
Changes
spark.sql.caseSensitiveat every struct level. A nullable field the file lacks reads as null, and its struct keeps its own nulls.Known gap: structs inside arrays and maps are still read whole and by position, and so are extension types that export as Arrow structs, such as Variant.
Tests
VortexNestedSchemaResolutionTestcovers the four cases and compares each struct query with Parquet, with nested schema pruning off and on. It also reads a file with an unsigned field the query doesn't select. These 8 tests fail on develop. A ninth test covers partition columns spelled in another case and a partition-only read. Spark handles those itself for a file format, so it passes on develop and guards the reader change.Checked locally:
:vortex-spark-3.5_2.12:test,:vortex-spark-3.5_2.13:test,:vortex-spark-4.0_2.13:test,spark41CompatTest,spark42CompatTestandspotlessCheckpass.I used AI tools to help write and review this change.