Repository navigation
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Unlike streams of JSON / CSV, the data that the parquet reader needs next i is not easy to predict as it depends on the filters, the row groups, which columns are requested, etc.
Now that we have the initial PushDecoder in this PR
We will be in the position to add an API for the decoder to communicate what data will be needed next
Describe the solution you'd like
I would like an API that allows users of the Parquet decoder to have more fine grained control over peeking
Describe alternatives you've considered
Here is an idea from @adriangb on https://github.com/apache/arrow-rs/pull/7997/files#r2444922393
a method along the lines of try_peek()? It'd be cool if it returned some structure that allowed fine grained control of the peeking:
let max_ranges = 32;
let max_bytes = 1024 * 1024 * 32;
let mut current_bytes = 0;
let mut ranges = Vec::new();
let mut peek = decoder.peek()
loop {
match peek.next() {
PeekResult::Range(range) => {
ranges.push(range);
current_bytes += range.end - range.start;
if ranges.len() > max_ranges { break }
if current_bytes > max_bytes { break }
PeekResult::End { break }
}
}Here is another potential API from the original ticket:
// Create a decoder for decoding parquet data as above
let mut decoder: ParquetDecoderBuilder = ...;
// As the decoder up from what data it will need, start prefetching data if desired
while let Some(pre_request) = decoder.peek_next_requests() {
// note that this is a peek and if we call peek again in the
// future, we may get a different set of pre_requests (for example
// if the decoder has applied a row filter and ruled out
// some row groups or data pages)
start_prefetch(pre_request);
}// push data to the decoder as before, but hopefully the reader
// will have already prefetched some of the data
Additional context
Activity
[-][Paruqet] PushDecoder: Add a peek API to support pre-fetching[/-][+][Parquet] PushDecoder: Add a peek API to support pre-fetching[/+]friendlymatthew commentedon Oct 20, 2025
Hi hi, I think the push based decoder looks super cool. I had a couple of questions:
I'm trying to understand when
peek()should return ranges. Should it only return ranges when the decoder is inWaitingOnFilterDataorWaitingOnDatastates? For other states (likeStart,Filters, orStartData), it would return empty because determining needed ranges would require running the state machineThis form of behavior would look something like:
Or should
peek()try to look ahead and predict future ranges (which would require simulating state transitions)PeekResulthas 2 variants (Range(Range<u64>)andEnd), which is basically reinventingOption<Range<u64>>. I wonder if we should just simplifypeekto:adriangb commentedon Oct 21, 2025
I think that's the idea / is necessary. E.g. it will "simulate" filters filtering nothing out, the worst case scenario. If you ask it for a little bit of data it's probably accurate. If you ask it for a lot it might be quite an over-estimate.
alamb commentedon Oct 21, 2025
Thank you @friendlymatthew and @adriangb
What I would suggest for this ticket is starting out by writing a test / usecase and then figuring out what API you need.
For example try to write a Parquet reader that
I am not sure exactly how this would work -- some ideas:
PushDecoder::take_next_row_group()that would return a new PushDecoder for evaluating only a single row group which could be processed in parallel?The more I write this the more I like the PushDecoder::take_next_row_group idea...
Something like
igorcalabria commentedon Feb 2, 2026
We did some tests with the push based API and found that pre-fetching would be really beneficial for the no filter, no projection cases. (
SELECT * ...). We did something like thisWith better pre-fetching it's pretty plausible that the network would be saturated (I'm only focusing on bandwidth here because this a "read all" case). I just wonder if "peeking" is the best API design for this. I was thinking more in the lines of a "scan plan" or something similar. Depending on the reader params, all ranges + decoders could be exposed in single call which is easily schedulable across tasks
alamb commentedon Feb 4, 2026
Thank you for the report @igorcalabria
Another thing I discovered while working on this code is the existing API https://docs.rs/parquet/latest/parquet/arrow/async_reader/struct.ParquetRecordBatchStream.html#method.next_row_group
(this is a pretty thin wrapper over the push decoder
try_next_readerAPI)That being said it will still fetch the ranges sequentially which is not ideal
Another thought I had for your usecase is to, as you say, create an individual PushDecoder for each row group (or some other RowSelection) and then run them all in parallel 🤔
feat(parquet): add ParquetPushDecoder::scan_plan for read-ahead plann…