Skip to content

Commit bcbbda1

Browse files
adriangbclaude
andcommitted
streaming: fetch small files in a single wave
The required-only first fetch keeps TTFB low on large files, but for a file whose whole plan fits the readahead window it turns one fetch wave into two serial waves — an extra round trip per file, which dominates many-small-file workloads (TPC-DS under simulated latency regressed 1.3x). Fetch the entire plan in one wave when it fits the window; required-only kicks in only for plans larger than the window, preserving the giant-row-group TTFB and memory wins. Co-Authored-By: Claude Fable 5 <[email protected]>
1 parent 5ff7af6 commit bcbbda1

1 file changed

Lines changed: 18 additions & 4 deletions

File tree

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 18 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -976,8 +976,10 @@ pub(crate) fn build_streaming_stream(
976976
}
977977
let sync_reader = builder.build()?;
978978

979+
let total_plan_bytes: u64 = plan.iter().map(|p| p.range.end - p.range.start).sum();
979980
let state = StreamingScanState {
980981
plan,
982+
total_plan_bytes,
981983
total_selected,
982984
batch_size: batch_size as u64,
983985
window,
@@ -1001,6 +1003,12 @@ pub(crate) fn build_streaming_stream(
10011003

10021004
pub(crate) struct StreamingScanState {
10031005
plan: Vec<PlanPage>,
1006+
/// Total bytes across all plan pages. Plans that fit the window are
1007+
/// fetched in one wave (see the inline-fetch site) — splitting a small
1008+
/// file's fetch into a required wave plus a readahead wave costs an
1009+
/// extra round trip per file, which dominates many-small-file workloads
1010+
/// (measured: TPC-DS under simulated latency).
1011+
total_plan_bytes: u64,
10041012
total_selected: u64,
10051013
batch_size: u64,
10061014
window: u64,
@@ -1125,10 +1133,16 @@ impl StreamingScanState {
11251133
continue;
11261134
}
11271135
ReaderSlot::Idle(mut reader) => {
1128-
// Fetch only the pages the next batch requires —
1129-
// readahead happens in the background (step 2), so
1130-
// the blocking fetch stays small and TTFB low.
1131-
let end = self.next_gulp_end(true);
1136+
// When the whole plan fits the readahead window
1137+
// (small files), fetch it in a single wave — the
1138+
// extra round trip of a required-only wave would
1139+
// dominate. For plans larger than the window, fetch
1140+
// only what the next batch requires so the blocking
1141+
// wave — and therefore time-to-first-batch — stays
1142+
// small; readahead happens in the background
1143+
// (step 2).
1144+
let required_only = self.total_plan_bytes > self.window;
1145+
let end = self.next_gulp_end(required_only);
11321146
let ranges: Vec<Range<u64>> = self.plan[self.fetched_idx..end]
11331147
.iter()
11341148
.map(|p| p.range.clone())

0 commit comments

Comments
 (0)