Skip to content

Commit d79165f

Browse files
adriangbclaude
andcommitted
feat(parquet): add bytes_processed scan-completion metric
Backport of apache#24524 (upstream a6e2d3f), carried here so platform can use it before the fork moves to a DataFusion release that contains it. Nothing previously reported how far along a scan was. `bytes_scanned` counts only the bytes fetched, so pruning and projection pushdown leave it understating progress by a factor that varies per query; `files_processed` only moves when a whole file completes, so a scan that opens every file at once reads zero for the entire first wave however much work is in flight. `bytes_processed` counts the bytes a scan is finished with, whether it read them or proved it did not need them. Over the life of a file, or of one byte range of a file split for parallelism, it advances by exactly that range's size, so `bytes_processed / total file bytes` is a completion fraction that needs no second metric to normalise against. Scope vs upstream: identical, except for the two hunks that adapt `sync_rg_plan_to_decoder_frontier` and `advance_rg_plan_to`. Those come from apache#24352, which this branch does not carry, so they are dropped. `ByteProgress`'s credit-on-drop still holds the metric's invariant; only the mid-scan crediting of a row group that arrow-rs finishes without handing back a reader is deferred to file close. Verified file by file against the upstream squash: every file whose base matched upstream's is byte-identical after the change, and `push_decoder.rs` differs only by those dropped hunks. Tested on this branch: the full `datafusion-datasource-parquet` suite (172 passed, including the seven `bytes_processed` invariant tests) and all four sqllogictest files the change touches. Co-Authored-By: Claude Opus 5 <[email protected]>
1 parent db02f34 commit d79165f

11 files changed

Lines changed: 674 additions & 27 deletions

File tree

‎datafusion/core/tests/sql/explain_analyze.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -889,6 +889,7 @@ async fn parquet_explain_analyze() {
889889
);
890890
assert_contains!(&formatted, "output_rows_skew=0%");
891891
assert_contains!(&formatted, "scan_efficiency_ratio=13.99%");
892+
assert_contains!(&formatted, "bytes_processed=");
892893

893894
// The order of metrics is expected to be the same as the actual pruning order
894895
// (file-> row-group -> page)

‎datafusion/datasource-parquet/src/metrics.rs‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,31 @@ use datafusion_physical_plan::metrics::{
2727
/// This component is a subject to **change** in near future and is exposed for low level integrations
2828
/// through [`ParquetFileReaderFactory`].
2929
///
30+
/// # The `bytes_processed` metric
31+
///
32+
/// Not every metric the parquet scan reports is a field on this struct. `bytes_processed` — the
33+
/// number of bytes the scan is finished with, whether it read them or proved by pruning that it
34+
/// did not need them — is registered straight onto the plan's metrics set, because only the
35+
/// internal progress guard ever touches it and a public field here would make every future
36+
/// metric a breaking change for anyone building this struct with a literal.
37+
///
38+
/// Read it the way `EXPLAIN ANALYZE` does, by name off the plan's metrics:
39+
///
40+
/// ```no_run
41+
/// # use datafusion_physical_plan::ExecutionPlan;
42+
/// # fn completion_fraction(plan: &dyn ExecutionPlan, total_file_bytes: u64) -> Option<f64> {
43+
/// let processed = plan.metrics()?.sum_by_name("bytes_processed")?.as_usize();
44+
/// Some(processed as f64 / total_file_bytes as f64)
45+
/// # }
46+
/// ```
47+
///
48+
/// Over the life of a file — or of one byte range of a file split for parallelism — it advances
49+
/// by exactly that file's size, which is what makes the ratio above a completion fraction rather
50+
/// than just another counter. Note that it measures work *resolved*, not time spent: bytes that
51+
/// pruning removes are credited the moment they are proved unnecessary, and proving that is
52+
/// nearly free. It is the right numerator for a progress bar, and only a rough one for
53+
/// predicting how much longer a query will take.
54+
///
3055
/// [`ParquetFileReaderFactory`]: super::ParquetFileReaderFactory
3156
#[derive(Debug, Clone)]
3257
pub struct ParquetFileMetrics {
@@ -103,6 +128,67 @@ pub struct ParquetFileMetrics {
103128
pub predicate_cache_records: Gauge,
104129
}
105130

131+
/// Tracks how much of one file — or one byte range of a file — a scan has
132+
/// finished with, crediting [`ParquetFileMetrics`]'s `bytes_processed` as it goes.
133+
///
134+
/// Every credit is clamped to the bytes left in the budget, and whatever is
135+
/// left over is credited on drop. The counter therefore advances by exactly the
136+
/// size of the range being scanned however the scan ends: normally, at a
137+
/// `LIMIT`, when a dynamic filter proves the rest of the file irrelevant, or on
138+
/// an error — including one that stops the file being opened at all, which is
139+
/// why the guard is created before the fallible stages of opening rather than
140+
/// alongside the decoder. That total is what makes the metric usable as a
141+
/// completion fraction rather than just another counter.
142+
///
143+
/// The clamp and the final top-up also absorb two small inexactnesses in
144+
/// crediting by row group: a file is slightly larger than the sum of its row
145+
/// groups (the footer, the page index and any padding belong to no row group),
146+
/// and a row group is assigned to a byte range by the offset of its first page,
147+
/// so a range's row groups do not add up to precisely its length.
148+
///
149+
/// The budget is held as a `usize` because [`Count`] is, so the two cannot
150+
/// disagree: on a 32-bit target a range longer than `usize::MAX` saturates once,
151+
/// here, rather than letting the remaining-byte arithmetic run ahead of what the
152+
/// counter can record. [`ParquetFileMetrics::bytes_scanned`] has the same
153+
/// ceiling.
154+
#[derive(Debug)]
155+
pub(crate) struct ByteProgress {
156+
/// Bytes of the scanned range not yet credited.
157+
remaining: usize,
158+
bytes_processed: Count,
159+
}
160+
161+
impl ByteProgress {
162+
/// Start tracking a range of `total` bytes.
163+
pub(crate) fn new(total: u64, bytes_processed: Count) -> Self {
164+
Self {
165+
remaining: saturating_usize(total),
166+
bytes_processed,
167+
}
168+
}
169+
170+
/// Record that the scan is finished with `bytes` more of the range.
171+
pub(crate) fn credit(&mut self, bytes: u64) {
172+
let bytes = saturating_usize(bytes).min(self.remaining);
173+
self.remaining -= bytes;
174+
self.bytes_processed.add(bytes);
175+
}
176+
}
177+
178+
/// Narrow a byte count to the width [`Count`] stores, saturating rather than
179+
/// wrapping. Lossless on 64-bit targets.
180+
fn saturating_usize(bytes: u64) -> usize {
181+
usize::try_from(bytes).unwrap_or(usize::MAX)
182+
}
183+
184+
impl Drop for ByteProgress {
185+
fn drop(&mut self) {
186+
let remaining = self.remaining;
187+
self.remaining = 0;
188+
self.bytes_processed.add(remaining);
189+
}
190+
}
191+
106192
impl ParquetFileMetrics {
107193
/// Create new metrics
108194
pub fn new(
@@ -234,6 +320,39 @@ impl ParquetFileMetrics {
234320
}
235321
}
236322

323+
/// The `bytes_processed` counter for one file: the total number of bytes the
324+
/// scan is finished with, whether they were read or skipped.
325+
///
326+
/// Where [`Self::bytes_scanned`] counts only the bytes fetched from the
327+
/// object store, this counts every byte the scan has resolved: the bytes it
328+
/// read, plus the bytes of the row groups (and whole files) that pruning
329+
/// proved cannot contribute. Over the lifetime of a file it therefore sums
330+
/// to that file's size — or, for a file split into byte ranges for
331+
/// parallelism, to the size of the range — so
332+
/// `bytes_processed / total file bytes` is a scan completion fraction,
333+
/// which `bytes_scanned` on its own is not: it understates progress by
334+
/// however much pruning and projection pushdown saved.
335+
///
336+
/// Credited at row-group granularity: a row group's bytes land when the
337+
/// scan is done with it. Crediting a row group's bytes progressively as its
338+
/// rows are decoded is left to a follow-up.
339+
///
340+
/// Built on demand rather than held on [`ParquetFileMetrics`] because only
341+
/// [`ByteProgress`] ever touches it, and a public field would make every
342+
/// future metric added here a breaking change for anyone constructing the
343+
/// struct with a literal.
344+
pub(crate) fn bytes_processed_counter(
345+
metrics: &ExecutionPlanMetricsSet,
346+
partition: usize,
347+
filename: &str,
348+
) -> Count {
349+
MetricBuilder::new(metrics)
350+
.with_new_label("filename", filename.to_string())
351+
.with_type(MetricType::Summary)
352+
.with_category(MetricCategory::Bytes)
353+
.counter("bytes_processed", partition)
354+
}
355+
237356
/// Record pages whose page-index pruning was skipped because the containing
238357
/// row group was fully matched by row-group statistics.
239358
///

‎datafusion/datasource-parquet/src/opener/early_stop.rs‎

Lines changed: 166 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,8 +38,15 @@ pub(super) struct EarlyStoppingStream<S> {
3838
done: bool,
3939
file_pruner: FilePruner,
4040
files_ranges_pruned_statistics: PruningMetrics,
41-
/// The inner stream
42-
inner: S,
41+
/// The inner stream, dropped as soon as this stream is done with it.
42+
///
43+
/// Held as an `Option` so finishing releases the decoder — and the buffers
44+
/// and per-file metric state it owns — at the moment we stop reading, not
45+
/// whenever the caller gets around to dropping this wrapper. Notably the
46+
/// scan's byte-progress accounting is completed by that drop, so deferring
47+
/// it would leave the file reading as partially scanned after the scan had
48+
/// demonstrably finished with it.
49+
inner: Option<S>,
4350
}
4451

4552
impl<S> EarlyStoppingStream<S> {
@@ -50,11 +57,17 @@ impl<S> EarlyStoppingStream<S> {
5057
) -> Self {
5158
Self {
5259
done: false,
53-
inner: stream,
60+
inner: Some(stream),
5461
file_pruner,
5562
files_ranges_pruned_statistics,
5663
}
5764
}
65+
66+
/// Mark the stream finished and release the inner stream.
67+
fn finish(&mut self) {
68+
self.done = true;
69+
self.inner = None;
70+
}
5871
}
5972

6073
impl<S> EarlyStoppingStream<S>
@@ -70,7 +83,7 @@ where
7083
self.files_ranges_pruned_statistics.add_pruned(1);
7184
// Previously this file range has been counted as matched
7285
self.files_ranges_pruned_statistics.subtract_matched(1);
73-
self.done = true;
86+
self.finish();
7487
Ok(None)
7588
} else {
7689
// Return the adapted batch
@@ -92,10 +105,13 @@ where
92105
if self.done {
93106
return Poll::Ready(None);
94107
}
95-
match ready!(self.inner.poll_next_unpin(cx)) {
108+
let Some(inner) = self.inner.as_mut() else {
109+
return Poll::Ready(None);
110+
};
111+
match ready!(inner.poll_next_unpin(cx)) {
96112
None => {
97113
// input done
98-
self.done = true;
114+
self.finish();
99115
Poll::Ready(None)
100116
}
101117
Some(input_batch) => {
@@ -105,3 +121,147 @@ where
105121
}
106122
}
107123
}
124+
125+
#[cfg(test)]
126+
mod tests {
127+
use super::*;
128+
129+
use std::sync::Arc;
130+
use std::sync::atomic::{AtomicBool, Ordering};
131+
132+
use arrow::array::{Int32Array, RecordBatch};
133+
use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
134+
use datafusion_common::{
135+
ColumnStatistics, ScalarValue, Statistics, stats::Precision,
136+
};
137+
use datafusion_datasource::PartitionedFile;
138+
use datafusion_physical_expr::PhysicalExpr;
139+
use datafusion_physical_expr::expressions::{
140+
BinaryExpr, Column, DynamicFilterPhysicalExpr, Literal,
141+
};
142+
use datafusion_physical_plan::metrics::Count;
143+
use futures::stream;
144+
145+
/// An inner stream that records when it is dropped, standing in for the
146+
/// decoder whose drop completes the scan's byte accounting.
147+
struct DropRecordingStream<S> {
148+
inner: S,
149+
dropped: Arc<AtomicBool>,
150+
}
151+
152+
impl<S: Stream + Unpin> Stream for DropRecordingStream<S> {
153+
type Item = S::Item;
154+
155+
fn poll_next(
156+
mut self: Pin<&mut Self>,
157+
cx: &mut Context<'_>,
158+
) -> Poll<Option<Self::Item>> {
159+
self.inner.poll_next_unpin(cx)
160+
}
161+
}
162+
163+
impl<S> Drop for DropRecordingStream<S> {
164+
fn drop(&mut self) {
165+
self.dropped.store(true, Ordering::Relaxed);
166+
}
167+
}
168+
169+
fn schema() -> SchemaRef {
170+
Arc::new(Schema::new(vec![Field::new("a", DataType::Int32, false)]))
171+
}
172+
173+
/// A file whose only column holds values 1..=9, so a predicate demanding
174+
/// larger values prunes it outright.
175+
fn file_with_stats() -> PartitionedFile {
176+
// Built field by field rather than from `Statistics::new_unknown`, which
177+
// already seeds one entry per column, so that column 0 carries these
178+
// bounds rather than an unknown placeholder.
179+
let statistics = Statistics {
180+
num_rows: Precision::Absent,
181+
total_byte_size: Precision::Absent,
182+
column_statistics: vec![
183+
ColumnStatistics::new_unknown()
184+
.with_min_value(Precision::Exact(ScalarValue::Int32(Some(1))))
185+
.with_max_value(Precision::Exact(ScalarValue::Int32(Some(9))))
186+
.with_null_count(Precision::Exact(0)),
187+
],
188+
};
189+
PartitionedFile::new("test.parquet".to_string(), 1_000)
190+
.with_statistics(Arc::new(statistics))
191+
}
192+
193+
fn pruning_filter(schema: &SchemaRef) -> FilePruner {
194+
let expr: Arc<dyn PhysicalExpr> = Arc::new(BinaryExpr::new(
195+
Arc::new(Column::new("a", 0)),
196+
datafusion_expr::Operator::Gt,
197+
Arc::new(Literal::new(ScalarValue::Int32(Some(100)))),
198+
));
199+
let dynamic: Arc<dyn PhysicalExpr> = Arc::new(DynamicFilterPhysicalExpr::new(
200+
expr.children().into_iter().map(Arc::clone).collect(),
201+
expr,
202+
));
203+
FilePruner::try_new(dynamic, schema, &file_with_stats(), Count::new())
204+
.expect("file has statistics, so a pruner can be built")
205+
}
206+
207+
fn batch(schema: &SchemaRef) -> RecordBatch {
208+
RecordBatch::try_new(
209+
Arc::clone(schema),
210+
vec![Arc::new(Int32Array::from(vec![1]))],
211+
)
212+
.unwrap()
213+
}
214+
215+
/// Stopping early must release the inner stream there and then. The decoder's
216+
/// drop is what completes this file's byte-progress accounting, so holding it
217+
/// until the caller drops the wrapper would leave the scan reporting a file it
218+
/// has finished with as still partly unread.
219+
#[tokio::test]
220+
async fn stopping_early_releases_the_inner_stream() {
221+
let schema = schema();
222+
let dropped = Arc::new(AtomicBool::new(false));
223+
let inner = DropRecordingStream {
224+
inner: stream::iter(vec![Ok(batch(&schema)), Ok(batch(&schema))]),
225+
dropped: Arc::clone(&dropped),
226+
};
227+
228+
let mut early_stopping = EarlyStoppingStream::new(
229+
inner,
230+
pruning_filter(&schema),
231+
PruningMetrics::new(),
232+
);
233+
234+
assert!(
235+
early_stopping.next().await.is_none(),
236+
"the filter prunes every row, so the first batch must end the stream",
237+
);
238+
assert!(
239+
dropped.load(Ordering::Relaxed),
240+
"the inner stream must be released when the scan stops, not when the \
241+
wrapper is eventually dropped",
242+
);
243+
}
244+
245+
/// The same must hold when the inner stream simply runs out.
246+
#[tokio::test]
247+
async fn exhausting_the_inner_stream_releases_it() {
248+
let schema = schema();
249+
let dropped = Arc::new(AtomicBool::new(false));
250+
let inner = DropRecordingStream {
251+
inner: stream::iter(Vec::<Result<RecordBatch>>::new()),
252+
dropped: Arc::clone(&dropped),
253+
};
254+
255+
let mut early_stopping = EarlyStoppingStream::new(
256+
inner,
257+
pruning_filter(&schema),
258+
PruningMetrics::new(),
259+
);
260+
261+
assert!(early_stopping.next().await.is_none());
262+
assert!(
263+
dropped.load(Ordering::Relaxed),
264+
"an exhausted inner stream must be released too",
265+
);
266+
}
267+
}

0 commit comments

Comments
 (0)