Skip to content

Schema inference reads every file's footer on the driver by default #100

Closed

Description

@ChristopherWirt

VortexFooterReader.inferSchema merges the schema of every file listed, because vortex.mergeSchema defaults to true (VortexFooterReader.java:92). It reads each footer from the driver, with vortex.footerParallelism threads (default 8, :49). Unlike vortex.stats.maxFiles, nothing caps how many files it reads.

Measured: loading 8,192 files of one directory spent about 6.5 minutes of driver time in mapFooters, about 21 footers a second, while the cluster sat idle. A dataset of about 200k files would take hours before any task starts. Setting vortex.mergeSchema=false brought it down to one footer read.

Proposal, any of:

  • Default mergeSchema to false, as Spark's Parquet source does. Callers whose files' schemas differ opt in.
  • Or above a threshold of files, merge footers in a Spark job, as Parquet's mergeSchemasInParallel does, rather than on the driver.
  • Or cap the files merged as vortex.stats.maxFiles does, and say so in the error if their schemas then disagree.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions