-
Notifications
You must be signed in to change notification settings - Fork 2k
Closed
Labels
enhancementNew feature or requestNew feature or requestperformanceMake DataFusion fasterMake DataFusion faster
Description
Follow on to #6337.
Currently when reading an arrow file from a stream, the entire stream is parsed as a file in order to determine the schema:
This will result in parsing the stream multiple times (once to determine the schema and again later to actually build RecordBatches from the stream).
Can we be more efficient here by only looking as far into the stream as necessary to read the schema?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestperformanceMake DataFusion fasterMake DataFusion faster