Repository navigation
perf(spark): project no columns for an empty required schema - #19
Closed
gitRasheed wants to merge 1 commit into
Closed
gitRasheed wants to merge 1 commit into
gitRasheed wants to merge 1 commit into
Conversation
VortexPartitionReader skipped the projection when Spark required no columns (count(*), count(1), df.count()), and the native scan then substituted root(), decoding every column of every file. Always send the select, so an empty schema reads only row counts; pushed filters still apply. Measured on a 1M-row, 18-column table: 2.9 s to 0.35 s. Signed-off-by: gitRasheed <[email protected]>
ChristopherWirt
approved these changes
Oct 30, 2026
Collaborator
|
Closing in favor of #35. When #35 merges, the FileFormat connector replaces this reader and always sends a projection, including an empty projection for queries that require no data columns; it also supports footer-backed COUNT(*) where applicable. The equivalent behavior and regression coverage live in #35, so this old-connector implementation is superseded. This is not being closed because the change has already landed on develop. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
count(*),count(1)anddf.count()decoded every column of every file.When Spark required no columns,
VortexPartitionReaderset no projection, and the native scan substitutesroot()for a missing one, so it read the whole table to count rows. On a 1M-row, 18-column table,count(*)took 2.9 s; with an empty projection it takes 0.35 s.Changes
The reader always sends a projection. For an empty required schema,
NestedSchemaPruning.projectiongivesselect([], root), which reads only row counts. Pushed filters still apply.Testing
VortexPartitionReaderTest.emptyProjectionReadsNoColumnsreads with an empty schema, with and without a pushed filter, and checks the batches have no columns and the right row counts. Fails on develop:expected: <0> but was: <3>columns.Checked locally:
./gradlew --parallel --continue javadoc check, and:vortex-spark_2.13:test -PsparkTestVersion=4.2.0.I used AI tools to help write and review this change.