You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Schema inference reads every file's footer on the driver by default #100
VortexFooterReader.inferSchema merges the schema of every file listed, because vortex.mergeSchema defaults to true (VortexFooterReader.java:92). It reads each footer from the driver, with vortex.footerParallelism threads (default 8, :49). Unlike vortex.stats.maxFiles, nothing caps how many files it reads.
Measured: loading 8,192 files of one directory spent about 6.5 minutes of driver time in mapFooters, about 21 footers a second, while the cluster sat idle. A dataset of about 200k files would take hours before any task starts. Setting vortex.mergeSchema=false brought it down to one footer read.
Proposal, any of:
Default mergeSchema to false, as Spark's Parquet source does. Callers whose files' schemas differ opt in.
Or above a threshold of files, merge footers in a Spark job, as Parquet's mergeSchemasInParallel does, rather than on the driver.
Or cap the files merged as vortex.stats.maxFiles does, and say so in the error if their schemas then disagree.
Activity