Skip to content

Push ChunkReader into SerializedPageReader #2463

@tustvold

Description

@tustvold

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Unlike SerializedFileReader, or SerializedRowGroupReader, SerializedPageReader is created with a T: Read instead of R: ChunkReader.

In order to preserve this when creating the async reader, I created a separate InMemoryColumnChunkReader.

Similarly when adding page skipping support in #2044 a VecDeque<T> was passed into the constructor, to avoid changing the signature.

Whilst working on #2460 we find ourselves in the same position where the lack of a ChunkReader complicates matters

Describe the solution you'd like

I think we have reached a point where introducing a breaking change to push the ChunkReader into SerializedPageReader is justified, as it will allow unifying InMemoryColumnChunkReader with SerializedPageReader and eliminating a load of additional complexity

Describe alternatives you've considered

Additional context

FYI @Ted-Jiang @thinkharderdev

Metadata

Metadata

Assignees

Labels

enhancementAny new improvement worthy of a entry in the changelogparquetChanges to the parquet crate

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions