-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Unlike SerializedFileReader, or SerializedRowGroupReader, SerializedPageReader is created with a T: Read instead of R: ChunkReader.
In order to preserve this when creating the async reader, I created a separate InMemoryColumnChunkReader.
Similarly when adding page skipping support in #2044 a VecDeque<T> was passed into the constructor, to avoid changing the signature.
Whilst working on #2460 we find ourselves in the same position where the lack of a ChunkReader complicates matters
Describe the solution you'd like
I think we have reached a point where introducing a breaking change to push the ChunkReader into SerializedPageReader is justified, as it will allow unifying InMemoryColumnChunkReader with SerializedPageReader and eliminating a load of additional complexity
Describe alternatives you've considered
Additional context