Skip to content

Commit 0e3c398

Browse files
authored
fix: count Parquet decoding in the scan's elapsed_compute (apache#25486)
## Which issue does this PR close? - Part of apache#18195. This is the last unchecked item, "Fix `elapsed_compute` baseline metrics not counting issue". ## Rationale for this change `EXPLAIN ANALYZE` reports an `elapsed_compute` for a Parquet scan that is far too small. For a single-partition `SELECT *` over a 1M row file, the scan takes about 155 ms and reports about 0.1 ms of compute, so the metric is useless for telling whether a query is bound by the scan. apache#20767 added an `elapsed_compute` timer in `PushDecoderStreamState::transition`, but it wraps `copy_arrow_reader_metrics` and `project_batch` only. The decoding happens before the timer starts (in `try_decode` at the time of that PR, in `reader.next()` today), so the metric went from a few nanoseconds to a fraction of a millisecond and is still missing where the time goes. The CSV fix in apache#18901 times `reader.next()`, and this PR does the equivalent for Parquet. ## What changes are included in this PR? `PushDecoderStreamState::transition` now runs one timer for the whole transition and pauses it only across `get_byte_ranges(...).await`. Everything else in the loop is CPU work: decoding in `reader.next()`, row group pruning and decoder rebuilds at row group boundaries, `try_next_reader`, `push_ranges` and the projection. That matches the guidance in the issue to count metadata handling and payload decoding. On the overhead concern raised on apache#20767: there is still one timer start per batch, as there is today. The only additional timer operations are a stop and a restart around each byte range fetch, which happens per row group rather than per batch. The timer is created from a clone of the `Time` metric, which shares the underlying counter, so the guard does not borrow `self` across the `&mut self` calls in the loop. It records when it is dropped, which covers every return. ## What is the testing strategy for this PR? New test `parquet_scan_elapsed_compute_includes_decoding` in `datafusion/core/tests/sql/explain_analyze.rs`. It writes a 1M row Parquet file, scans it with one target partition so the scan runs on one thread, and asserts that the scan's `elapsed_compute` is at least a quarter of the wall time of the scan. The existing tests only assert that `elapsed_compute` is greater than zero, which a timer that misses decoding still satisfies. This is also an answer to the question on apache#20767 of whether the metric can be tested: comparing against wall time measured in the same run avoids depending on machine speed. Measured locally in a debug build, three runs each, within 1% of each other: | | `elapsed_compute` | wall time | | --- | --- | --- | | main | 0.10 ms | 155 ms | | this PR | 151 ms | 155 ms | The test fails on main and passes with this change, and it passed five consecutive runs. The threshold of a quarter leaves a wide margin on both sides, and because it compares against wall time measured in the same run it does not depend on machine speed. **Is the new number right?** Two checks, done with a throwaway test that is not part of this PR, on the same 1M row file: - Against arrow-rs alone. Decoding the file with `ParquetRecordBatchReaderBuilder` at a batch size of 8192, with no DataFusion involved, took 145 to 148 ms over three runs. The scan's `elapsed_compute` with this change was 145 to 147 ms. - I/O is excluded. With the file served from an `InMemory` store wrapped in object_store's `ThrottledStore`, which sleeps on every get: | injected delay per get | wall time | `elapsed_compute` | | --- | --- | --- | | 0 ms | 149 ms | 149 ms | | 100 ms | 274 ms | 167 ms | | 300 ms | 471 ms | 167 ms | Tripling the delay adds 200 ms of wall time and leaves `elapsed_compute` unchanged. The step from 149 to 167 ms is CPU time and not a leak of I/O wait: wall time minus the injected delay is about 171 to 174 ms in those runs, so the work itself was slower once the thread had gone idle. The metric also scales with the work: 500k rows report 77 ms against 147 ms for 1M, a single `Int64` column reports 38 ms, and a single string column 76 ms. In every case `elapsed_compute` stayed below the wall time, by 0.5 to 5 ms. Also run locally: `cargo fmt --all -- --check`, `cargo clippy --all-targets --all-features -- -D warnings` for `datafusion-datasource-parquet` and for the `core_integration` test target, `cargo test -p datafusion-datasource-parquet` (267 passed), the `sql::explain_analyze` tests (30 passed) and the `parquet_integration` tests (242 passed). ## Are there any user-facing changes? `elapsed_compute` for Parquet scans in `EXPLAIN ANALYZE` becomes much larger, because it now includes decoding. No API changes.
1 parent d20936c commit 0e3c398

2 files changed

Lines changed: 69 additions & 3 deletions

File tree

‎datafusion/core/tests/sql/explain_analyze.rs‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -908,6 +908,64 @@ async fn parquet_explain_analyze() {
908908
);
909909
}
910910

911+
/// The Parquet scan's `elapsed_compute` must cover decoding, which dominates a
912+
/// full scan, and not only the projection of batches that are already decoded.
913+
/// See <https://github.com/apache/datafusion/issues/18195>.
914+
#[tokio::test]
915+
async fn parquet_scan_elapsed_compute_includes_decoding() -> Result<()> {
916+
use datafusion::datasource::source::DataSourceExec;
917+
use datafusion_common::instant::Instant;
918+
use std::time::Duration;
919+
920+
// A single partition keeps the scan on one thread, so its compute time
921+
// cannot exceed the wall time of the query.
922+
let ctx =
923+
SessionContext::new_with_config(SessionConfig::new().with_target_partitions(1));
924+
let tmp_dir = TempDir::new()?;
925+
let path = tmp_dir.path().join("data.parquet");
926+
let path = path.to_str().unwrap();
927+
ctx.sql(&format!(
928+
"COPY (SELECT value AS a, value % 1000 AS b, cast(value AS varchar) AS c \
929+
FROM generate_series(1, 1000000)) TO '{path}' STORED AS PARQUET"
930+
))
931+
.await?
932+
.collect()
933+
.await?;
934+
ctx.register_parquet("t", path, ParquetReadOptions::default())
935+
.await?;
936+
937+
let plan = ctx
938+
.sql("SELECT * FROM t")
939+
.await?
940+
.create_physical_plan()
941+
.await?;
942+
let start = Instant::now();
943+
collect(Arc::clone(&plan), ctx.task_ctx()).await?;
944+
let wall_time = start.elapsed();
945+
946+
assert!(
947+
plan.is::<DataSourceExec>(),
948+
"expected a bare scan, got:\n{}",
949+
DisplayableExecutionPlan::new(plan.as_ref()).indent(true)
950+
);
951+
let elapsed_compute = Duration::from_nanos(
952+
plan.metrics()
953+
.unwrap()
954+
.aggregate_by_name()
955+
.elapsed_compute()
956+
.unwrap() as u64,
957+
);
958+
959+
// Reading a local file is cheap next to decoding it, so compute should be
960+
// most of the wall time. A timer that misses decoding reports well under 1%.
961+
assert!(
962+
elapsed_compute * 4 > wall_time,
963+
"elapsed_compute {elapsed_compute:?} should be at least a quarter of the \
964+
scan's wall time {wall_time:?}"
965+
);
966+
Ok(())
967+
}
968+
911969
// This test reproduces the behavior described in
912970
// https://github.com/apache/datafusion/issues/16684 where projection
913971
// pushdown with recursive CTEs could fail to remove unused columns

‎datafusion/datasource-parquet/src/push_decoder.rs‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -430,16 +430,21 @@ impl PushDecoderStreamState {
430430
/// miri where `&mut self` creates a single opaque borrow that conflicts
431431
/// with `unfold`'s ownership across yield points.
432432
async fn transition(mut self) -> Option<(Result<RecordBatch>, Self)> {
433+
// Everything below is CPU work (decoding, row group pruning, building
434+
// readers, projection) except fetching byte ranges, so the timer runs
435+
// for the whole transition and is paused only across that await.
436+
// Cloning `Time` shares the underlying counter and keeps the guard
437+
// from borrowing `self`. The guard records on drop, which covers
438+
// every return.
439+
let elapsed_compute = self.baseline_metrics.elapsed_compute().clone();
440+
let mut timer = elapsed_compute.timer();
433441
loop {
434442
// Step 1: drain a batch from the active reader if any.
435443
if let Some(reader) = self.active_reader.as_mut() {
436444
match reader.next() {
437445
Some(Ok(batch)) => {
438-
let mut timer = self.baseline_metrics.elapsed_compute().timer();
439446
self.copy_arrow_reader_metrics();
440447
let result = self.project_batch(&batch);
441-
timer.stop();
442-
drop(timer);
443448
return Some((result, self));
444449
}
445450
Some(Err(e)) => {
@@ -501,11 +506,14 @@ impl PushDecoderStreamState {
501506
let decoder = self.decoder.as_mut().expect("decoder present");
502507
match decoder.try_next_reader() {
503508
Ok(DecodeResult::NeedsData(ranges)) => {
509+
// I/O, not compute.
510+
timer.stop();
504511
let data = self
505512
.reader
506513
.get_byte_ranges(ranges.clone())
507514
.await
508515
.map_err(DataFusionError::from);
516+
timer.restart();
509517
match data {
510518
Ok(data) => {
511519
if let Err(e) = self

0 commit comments

Comments
 (0)