Skip to content

Commit caaf0fa

Browse files
authored
bench: measure projected Parquet scans across file boundaries (apache#27016)
## Which issue does this PR close? Part of apache#26682. ## Rationale for this change When each Parquet file has one row group, a reader can hide the next file's data request by starting it while decoding the current file. A benchmark that scans several partitions concurrently can hide the same delay even when each reader waits for its current file to finish. This gives apache#23492 and the cross-file experiment on apache#24086 a shared, single-partition workload. ## What changes are included in this PR? - Extend `sql_query_with_io` with a projected scan over 20 single-row-group files in one scan partition. The same SQL and bytes run against an in-memory store and a store with 100 ms of simulated read latency. Planning and data generation happen outside the timed loop. Each execution checks the sum of the selected column over 20 million rows. - Add a two-file object-store test. The full projection must read the selected column chunk once from each file and return all four values. With `LIMIT 1`, only one file's payload is needed. File statistics are disabled so the scan gets both files, and the assertions allow either file order. ## What is the testing strategy for this PR? `cargo test -p datafusion --test core_integration object_store_access` passed all 24 tests. The extended workspace suite passed 13,396 Rust tests and all 545 SQL logic files. Workspace formatting and Clippy with all targets and features passed. To reproduce the benchmark, run ```sh cargo bench -p datafusion --bench sql_query_with_io -- parquet_cross_file_projection ``` Local Criterion baseline on the PR branch, using a release build on a shared host, four pinned CPUs and ten samples per store. | Store | Estimated scan time | 95% confidence interval | | --- | ---: | ---: | | In memory | 24.6 ms | 24.3–24.9 ms | | 100 ms per read request | 2.059 s | 2.057–2.060 s | To check that the benchmark detects cross-file read-ahead, I also ran the same workload against the [public cross-file prototype from apache#24086](adriangb@64e67f5). Both arms used the same release binary on four pinned CPUs, with file-open depth 2, a 16 MiB read-ahead window and a shared 32 MiB budget. Only the eager first-read cap changed, from 0 to 16 MiB. I ran off/on/on/off, with ten samples per process. Each cell is the median of its two process medians. | Store on the prototype | Eager reads off | Eager reads on | Elapsed time change | | --- | ---: | ---: | ---: | | In memory | 29.22 ms | 30.73 ms | 5.2% longer | | 100 ms per read request | 2.065 s | 1.036 s | 49.8% shorter | The prototype roughly halved elapsed time on the simulated high-latency store and added about 1.5 ms on the in-memory control. Both configurations produced the same expected sum. A real object-store workload could have a different balance of I/O and decoding. ## Are there any user-facing changes? No. This adds benchmark and integration-test coverage.
1 parent ea33cce commit caaf0fa

2 files changed

Lines changed: 132 additions & 6 deletions

File tree

‎datafusion/core/benches/sql_query_with_io.rs‎

Lines changed: 46 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ use datafusion::{
1616
prelude::{SessionConfig, SessionContext},
1717
};
1818
use datafusion_execution::runtime_env::RuntimeEnv;
19+
use datafusion_physical_plan::{collect, execution_plan::reset_plan_states};
1920
use itertools::Itertools;
2021
use object_store::{
2122
ObjectStore, ObjectStoreExt,
@@ -67,7 +68,7 @@ fn create_parquet_file(rng: &mut StdRng, id_offset: usize) -> Bytes {
6768
let mut data = Vec::<u8>::new();
6869
let mut writer = ArrowWriter::try_new(&mut data, schema, Default::default()).unwrap();
6970
writer.write(&batch).unwrap();
70-
writer.close().unwrap();
71+
assert_eq!(writer.close().unwrap().num_row_groups(), 1);
7172

7273
data.into()
7374
}
@@ -101,15 +102,19 @@ where
101102
Arc::new(ThrottledStore::new(object_store, config))
102103
}
103104

104-
async fn setup_context(object_store: Arc<dyn ObjectStore>) -> SessionContext {
105-
let config = SessionConfig::new().with_target_partitions(THREADS);
105+
async fn setup_context(
106+
object_store: Arc<dyn ObjectStore>,
107+
config: SessionConfig,
108+
) -> SessionContext {
106109
let rt = Arc::new(RuntimeEnv::default());
107110
rt.register_object_store(&Url::parse("data://my_store").unwrap(), object_store);
108111
let context = SessionContext::new_with_config_rt(config, rt);
109112

110113
for table_id in 0..TABLES {
111114
let table_name = table_name(table_id);
112-
let file_format = ParquetFormat::default().with_enable_pruning(true);
115+
let file_format = ParquetFormat::default()
116+
.with_options(context.copied_table_options().parquet)
117+
.with_enable_pruning(true);
113118
let options = ListingOptions::new(Arc::new(file_format))
114119
.with_table_partition_cols(vec![(
115120
String::from("partition"),
@@ -167,15 +172,19 @@ fn bench_query(
167172

168173
fn criterion_benchmark(c: &mut Criterion) {
169174
env_logger::try_init().ok();
170-
let store = make_store_slow(InMemory::new());
175+
let store: Arc<dyn ObjectStore> = Arc::new(InMemory::new());
176+
let slow_store = make_store_slow(Arc::clone(&store));
171177
let rt = tokio::runtime::Builder::new_multi_thread()
172178
.worker_threads(THREADS)
173179
.enable_time()
174180
.build()
175181
.unwrap();
176182

177183
rt.block_on(setup_files(Arc::clone(&store)));
178-
let ctx = rt.block_on(setup_context(store));
184+
let ctx = rt.block_on(setup_context(
185+
Arc::clone(&slow_store),
186+
SessionConfig::new().with_target_partitions(THREADS),
187+
));
179188

180189
let table0_name = table_name(0);
181190
let select_query = format!("SELECT * FROM {table0_name}");
@@ -244,6 +253,37 @@ fn criterion_benchmark(c: &mut Criterion) {
244253
&format!("{join_query} WHERE {table0_name}.partition = 0"),
245254
PARTITION_FILES * FILE_ROWS,
246255
);
256+
257+
// Each file contains one row group. Using one scan partition makes crossing
258+
// a file boundary the only opportunity to read ahead into another group.
259+
let mut group = c.benchmark_group("parquet_cross_file_projection");
260+
group.sample_size(10);
261+
group.sampling_mode(SamplingMode::Flat);
262+
group.measurement_time(Duration::from_secs(25));
263+
for (name, store) in [("in_memory", store), ("100ms_get", slow_store)] {
264+
let config = SessionConfig::from_env().unwrap().with_target_partitions(1);
265+
let ctx = rt.block_on(setup_context(store, config));
266+
let plan = rt.block_on(async {
267+
ctx.sql("SELECT sum(id) FROM table_0")
268+
.await
269+
.unwrap()
270+
.create_physical_plan()
271+
.await
272+
.unwrap()
273+
});
274+
let rows_per_partition = (PARTITION_FILES * FILE_ROWS) as u64;
275+
let expected =
276+
TABLE_PARTITIONS as u64 * rows_per_partition * (rows_per_partition - 1) / 2;
277+
let scan = || async {
278+
let plan = reset_plan_states(Arc::clone(&plan)).unwrap();
279+
let batches = collect(plan, ctx.task_ctx()).await.unwrap();
280+
let sum =
281+
datafusion_common::cast::as_uint64_array(batches[0].column(0)).unwrap();
282+
assert_eq!(sum.value(0), expected);
283+
};
284+
group.bench_function(name, |b| b.to_async(&rt).iter(scan));
285+
}
286+
group.finish();
247287
}
248288

249289
criterion_group!(benches, criterion_benchmark);

‎datafusion/core/tests/datasource/object_store_access.rs‎

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -961,6 +961,92 @@ async fn query_single_parquet_file() {
961961
);
962962
}
963963

964+
#[tokio::test]
965+
async fn projected_parquet_ranges_across_files() -> datafusion_common::Result<()> {
966+
// Keep both files in one scan partition, and read just the footer when
967+
// looking up metadata so those requests are distinct from column chunks.
968+
let mut test = Test::new().with_parquet_metadata_size_hint(Some(8));
969+
test.query("SET datafusion.execution.target_partitions = 1")
970+
.await;
971+
// Let the scan enforce LIMIT. Exact row counts would let the planner
972+
// discard the second file before the scan starts.
973+
test.query("SET datafusion.execution.collect_statistics = false")
974+
.await;
975+
let mut selected_chunks = vec![];
976+
for (path, value) in [("data/a.parquet", 1), ("data/b.parquet", 3)] {
977+
let unused: ArrayRef = Arc::new(Int32Array::from(vec![0, 0]));
978+
let selected: ArrayRef = Arc::new(Int32Array::from(vec![value, value + 1]));
979+
let batch =
980+
RecordBatch::try_from_iter([("unused", unused), ("selected", selected)])?;
981+
let mut bytes = vec![];
982+
// A single data page gives the projected column a single payload range.
983+
let props = parquet::file::properties::WriterProperties::builder()
984+
.set_dictionary_enabled(false)
985+
.build();
986+
let mut writer = parquet::arrow::ArrowWriter::try_new(
987+
&mut bytes,
988+
batch.schema(),
989+
Some(props),
990+
)?;
991+
writer.write(&batch)?;
992+
let metadata = writer.close()?;
993+
let (start, len) = metadata.row_group(0).column(1).byte_range();
994+
selected_chunks.push((Path::from(path), bytes.len() as u64, start..start + len));
995+
test = test.with_bytes(path, bytes).await;
996+
}
997+
let test = test.register_parquet("t", "/data/").await;
998+
999+
for (query, files_to_read) in [
1000+
("SELECT selected FROM t", 2),
1001+
("SELECT selected FROM t LIMIT 1", 1),
1002+
] {
1003+
test.object_store.clear_requests();
1004+
let batches = test.session_context.sql(query).await?.collect().await?;
1005+
let mut values = vec![];
1006+
for batch in &batches {
1007+
values.extend_from_slice(
1008+
datafusion_common::cast::as_int32_array(batch.column(0))?.values(),
1009+
);
1010+
}
1011+
values.sort_unstable();
1012+
if files_to_read == 2 {
1013+
assert_eq!(values, [1, 2, 3, 4]);
1014+
} else {
1015+
assert!(matches!(values.as_slice(), [1] | [3]));
1016+
}
1017+
1018+
let requests = test.object_store.recorded_requests();
1019+
let mut files_read = 0;
1020+
for (file, size, selected) in &selected_chunks {
1021+
let mut data_ranges = vec![];
1022+
for request in &requests {
1023+
match request {
1024+
RequestDetails::GetRanges { path, ranges } if path == file => {
1025+
data_ranges.extend_from_slice(ranges);
1026+
}
1027+
RequestDetails::GetOpts { path, get_options }
1028+
if path == file && !get_options.head =>
1029+
{
1030+
data_ranges.push(match &get_options.range {
1031+
Some(range) => range.as_range(*size).unwrap(),
1032+
None => 0..*size,
1033+
});
1034+
}
1035+
_ => {}
1036+
}
1037+
}
1038+
// Ignore footer reads. The selected column is the last data chunk.
1039+
data_ranges.retain(|range| range.start < selected.end);
1040+
if !data_ranges.is_empty() {
1041+
assert_eq!(data_ranges, std::slice::from_ref(selected), "{file}");
1042+
files_read += 1;
1043+
}
1044+
}
1045+
assert_eq!(files_read, files_to_read);
1046+
}
1047+
Ok(())
1048+
}
1049+
9641050
#[tokio::test]
9651051
async fn query_single_parquet_file_with_single_predicate() {
9661052
let test = Test::new().with_single_file_parquet().await;

0 commit comments

Comments
 (0)