Skip to content

Allow specifying projection in ParquetRecordBatchReader::try_new_with_row_groups #8326

@yeya24

Description

@yeya24

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The current ParquetRecordBatchReader::try_new_with_row_groups function always uses ProjectionMask::all(), which forces reading all columns from the parquet file even when only specific columns are needed. This is inefficient for use cases where you only want to read a subset of columns, as it results in unnecessary I/O and memory usage.

Describe the solution you'd like

Modify the try_new_with_row_groups function to accept a projection: &ProjectionMask parameter instead of hardcoding ProjectionMask::all(). This will allow callers to specify exactly which columns they want to read, enabling more efficient data access patterns.

Describe alternatives you've considered

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementAny new improvement worthy of a entry in the changelog

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions