Skip to content

Commit 6be7a15

Browse files
committed
address review: record prefetched opens as cache misses and document the prefetch
A prefetched open now records the storage reads of the load it waited for, so scan_io_metadata_cache_hits keeps meaning 'served from the cache' and scan_io_metadata_prefetches counts the loads that overlapped a previous file. An encrypted scan no longer builds the prefetch at all. The ranges test now opens through the factory, and the tuning guide documents the config.
1 parent 33d7f93 commit 6be7a15

6 files changed

Lines changed: 109 additions & 80 deletions

File tree

‎docs/source/user-guide/latest/metrics.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -230,7 +230,7 @@ execution metrics. Counters accumulate per scan operator; they do not instrument
230230
| `scan_io_object_store_response_bytes_read` | Response bytes actually consumed at that API, including bytes fetched between coalesced ranges. Not HTTP wire bytes. |
231231
| `scan_io_metadata_cache_hits` | Successful, cache-eligible metadata opens requiring no storage reads. |
232232
| `scan_io_metadata_cache_misses` | Successful, cache-eligible metadata opens requiring storage reads. Failed opens and encrypted opens, which bypass this shared cache, increment neither cache counter. |
233-
| `scan_io_metadata_prefetches` | Metadata loads that read storage before their file's open, started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). The open that uses one counts as a cache hit. |
233+
| `scan_io_metadata_prefetches` | Metadata opens that used a load started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). Each also counts as a cache miss. |
234234
| `row_groups_pruned_dynamic_filter` | Row groups skipped by a runtime predicate whose value changes during execution, such as a local TopK threshold. |
235235

236236
Reader-level and object-store bytes are two views of the same reads; do not add them together.

‎docs/source/user-guide/latest/tuning/scans.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,18 @@ more splits overall, so some idle tasks may remain — tune the value against yo
6464
See [#3817](https://github.com/apache/datafusion-comet/issues/3817#issuecomment-4193279630) for a
6565
worked example and further discussion.
6666

67+
### Prefetching the Next File's Metadata
68+
69+
A native Parquet scan opens a task's files one after another, and each open waits for a round trip to read
70+
the file's footer and page index before its data fetch can start. For tasks that read many small files from
71+
high-latency storage such as S3, that wait is paid once per file. Set
72+
`spark.comet.parquet.prefetchNextFileMetadata.enabled` to `true` to load the next file's footer and page
73+
index in the background while the current file is read. The setting is off by default.
74+
75+
The cost is one extra footer and page index read for a file that is never opened, such as when a scan stops
76+
early or the file is pruned before it opens. Scans of encrypted Parquet files never prefetch. The
77+
`scan_io_metadata_prefetches` metric counts the opens that used a prefetched load.
78+
6779
## Iceberg Scan Tuning
6880

6981
Comet's native Iceberg scan (`spark.comet.scan.icebergNative.enabled`, enabled by default) reads each

‎native/core/src/parquet/eager_page_index_reader_factory.rs‎

Lines changed: 66 additions & 45 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,11 @@
8686
//! task instead of reading the footer itself. This assumes DataFusion opens the files in the
8787
//! group's order, which holds unless the source reorders them (`ParquetSource::reorder_files`);
8888
//! a reordered scan would only waste its prefetches.
89+
//!
90+
//! This is a stopgap for the overlap DataFusion dropped. It can go once `FileStream` opens the next
91+
//! file while the current reader runs again; apache/datafusion#5209 (buffer more than one file
92+
//! open) and apache/datafusion#18470 (prefetch the next row group) are the open issues that would
93+
//! bring it back.
8994
9095
use crate::parquet::datetime_rebase::stamp_int96_leaves;
9196
use crate::parquet::parquet_support::SparkParquetOptions;
@@ -674,7 +679,14 @@ impl AsyncFileReader for EagerPageIndexReader {
674679
None => None,
675680
};
676681
let (metadata, storage_reads) = match prefetched {
677-
Some(metadata) => (metadata, 0),
682+
Some((metadata, storage_reads)) => {
683+
// The load read from storage ahead of this open, so the open records a cache
684+
// miss, as it would have without the prefetch, and the load counts as overlapped.
685+
if storage_reads > 0 {
686+
scan_io_metrics.metadata_prefetches.add(1);
687+
}
688+
(metadata, storage_reads)
689+
}
678690
None => {
679691
load_metadata(
680692
store,
@@ -832,7 +844,8 @@ async fn load_metadata(
832844

833845
/// A load of a file's metadata started before DataFusion opens the file. Dropping it aborts the
834846
/// load.
835-
type MetadataLoad = SpawnedTask<Option<Arc<ParquetMetaData>>>;
847+
/// Its result carries the number of storage reads it took, for the open that uses it to record.
848+
type MetadataLoad = SpawnedTask<Option<(Arc<ParquetMetaData>, usize)>>;
836849

837850
/// The scan-wide state of `with_next_file_metadata_prefetch`.
838851
#[derive(Debug)]
@@ -917,21 +930,17 @@ impl NextFileMetadataPrefetch {
917930
let object_meta = object_meta.clone();
918931
let location = object_meta.location.clone();
919932
let load = SpawnedTask::spawn(async move {
920-
let (metadata, storage_reads) = load_metadata(
933+
load_metadata(
921934
store,
922-
Arc::clone(&scan_io_metrics),
935+
scan_io_metrics,
923936
metadata_cache,
924937
&object_meta,
925938
metadata_size_hint,
926939
None,
927940
Some(PageIndexPolicy::Optional),
928941
)
929942
.await
930-
.ok()?;
931-
if storage_reads > 0 {
932-
scan_io_metrics.metadata_prefetches.add(1);
933-
}
934-
Some(metadata)
943+
.ok()
935944
});
936945
Some((location, load))
937946
}
@@ -1845,8 +1854,9 @@ mod tests {
18451854
in_flight.is_none()
18461855
}
18471856

1848-
/// Opening a file loads the next file's metadata in the background, so only the first open
1849-
/// reads storage. Opening the last file starts nothing.
1857+
/// Opening a file loads the next file's metadata in the background. Every file is read from
1858+
/// storage, so each open is a cache miss, and the two loads that ran ahead of their open are
1859+
/// counted as prefetches. Opening the last file starts nothing.
18501860
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
18511861
async fn opening_a_file_loads_the_next_files_metadata_ahead() {
18521862
let store = Arc::new(InMemory::new());
@@ -1860,8 +1870,8 @@ mod tests {
18601870
assert!(metadata.column_index().is_some() && metadata.offset_index().is_some());
18611871
}
18621872

1863-
assert_eq!(metric(&metrics, "scan_io_metadata_cache_misses"), 1);
1864-
assert_eq!(metric(&metrics, "scan_io_metadata_cache_hits"), 2);
1873+
assert_eq!(metric(&metrics, "scan_io_metadata_cache_misses"), 3);
1874+
assert_eq!(metric(&metrics, "scan_io_metadata_cache_hits"), 0);
18651875
assert_eq!(metric(&metrics, "scan_io_metadata_prefetches"), 2);
18661876
assert_eq!(metric(&metrics, "scan_io_footer_reads"), 3);
18671877
assert!(in_flight_is_empty(&factory));
@@ -1886,8 +1896,9 @@ mod tests {
18861896
open(&factory, file, &metrics).await.unwrap();
18871897
}
18881898

1889-
assert_eq!(metric(&metrics, "scan_io_metadata_cache_misses"), 2);
1890-
assert_eq!(metric(&metrics, "scan_io_metadata_cache_hits"), 1);
1899+
assert_eq!(metric(&metrics, "scan_io_metadata_cache_misses"), 3);
1900+
assert_eq!(metric(&metrics, "scan_io_metadata_cache_hits"), 0);
1901+
assert_eq!(metric(&metrics, "scan_io_metadata_prefetches"), 1);
18911902
assert!(in_flight_is_empty(&factory));
18921903
}
18931904

@@ -1908,46 +1919,55 @@ mod tests {
19081919
store.open();
19091920
assert_eq!(metadata.await.unwrap().file_metadata().num_rows(), 2);
19101921

1911-
assert_eq!(metric(&metrics, "scan_io_metadata_cache_misses"), 1);
1912-
assert_eq!(metric(&metrics, "scan_io_metadata_cache_hits"), 1);
1922+
assert_eq!(metric(&metrics, "scan_io_metadata_cache_misses"), 2);
1923+
assert_eq!(metric(&metrics, "scan_io_metadata_cache_hits"), 0);
19131924
assert_eq!(metric(&metrics, "scan_io_metadata_prefetches"), 1);
19141925
assert_eq!(metric(&metrics, "scan_io_footer_reads"), 2);
19151926
}
19161927

19171928
/// The next file is the next one with a different path. Further ranges of a file share its
1918-
/// metadata, so they share its next file too.
1919-
#[test]
1920-
fn the_next_file_skips_further_ranges_of_the_same_path() {
1921-
let file = |path: &str, range: Option<(i64, i64)>| {
1922-
let file = PartitionedFile::new(path.to_string(), 100);
1923-
match range {
1924-
Some((start, end)) => file.with_range(start, end),
1925-
None => file,
1926-
}
1927-
};
1928-
let files = vec![
1929-
file("a", Some((0, 50))),
1930-
file("a", Some((50, 100))),
1931-
file("b", None),
1929+
/// metadata, so they share its next file too, and opening them doesn't start a second load.
1930+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
1931+
async fn the_next_file_skips_further_ranges_of_the_same_path() {
1932+
let store = Arc::new(GatedStore::new("unused.parquet"));
1933+
let a = put_int_file(&store.inner, "a.parquet", None).await;
1934+
let b = put_int_file(&store.inner, "b.parquet", None).await;
1935+
let files = [
1936+
a.clone().with_range(0, 50),
1937+
a.with_range(50, 100),
1938+
b.clone(),
19321939
];
1933-
let runtime = datafusion::execution::runtime_env::RuntimeEnv::default();
1934-
let prefetch = NextFileMetadataPrefetch::new(
1935-
&files,
1936-
Arc::new(InMemory::new()),
1937-
Arc::new(ScanIoMetrics::new(&ExecutionPlanMetricsSet::new())),
1938-
runtime.cache_manager.get_file_metadata_cache(),
1939-
);
1940+
let metrics = ExecutionPlanMetricsSet::new();
1941+
let factory = prefetching_factory(Arc::clone(&store) as _, &files, &metrics);
1942+
1943+
for file in &files {
1944+
open(&factory, file, &metrics).await.unwrap();
1945+
}
1946+
1947+
// The ranges of a share one load. The second range finds it already in the cache.
19401948
let next = |file: &PartitionedFile| {
1949+
let prefetch = factory.next_file_prefetch.as_ref().unwrap();
19411950
prefetch
19421951
.next_files
19431952
.get(&(file.object_meta.location.clone(), file.range.clone()))
19441953
.map(|(next, _)| next.location.to_string())
19451954
};
1946-
1947-
assert_eq!(next(&files[0]).as_deref(), Some("b"));
1948-
assert_eq!(next(&files[1]).as_deref(), Some("b"));
1955+
assert_eq!(next(&files[0]).as_deref(), Some("b.parquet"));
1956+
assert_eq!(next(&files[1]).as_deref(), Some("b.parquet"));
19491957
assert_eq!(next(&files[2]), None);
1950-
assert_eq!(next(&file("a", Some((0, 10)))), None);
1958+
// b is read once, by the load that the first range of a started.
1959+
let single_load = Arc::new(GatedStore::new("unused.parquet"));
1960+
let alone = put_int_file(&single_load.inner, "b.parquet", None).await;
1961+
let alone_metrics = ExecutionPlanMetricsSet::new();
1962+
let alone_factory = prefetching_factory(
1963+
Arc::clone(&single_load) as _,
1964+
std::slice::from_ref(&alone),
1965+
&alone_metrics,
1966+
);
1967+
open(&alone_factory, &alone, &alone_metrics).await.unwrap();
1968+
assert_eq!(store.reads("b.parquet"), single_load.reads("b.parquet"));
1969+
assert_eq!(metric(&metrics, "scan_io_metadata_prefetches"), 1);
1970+
assert!(in_flight_is_empty(&factory));
19511971
}
19521972

19531973
/// A prefetch that fails yields nothing, so the file's open loads the metadata itself and
@@ -1975,9 +1995,10 @@ mod tests {
19751995
.contains("Failed to fetch metadata for file bad.parquet"),
19761996
"unexpected error: {error}"
19771997
);
1978-
let reads_per_load = store.reads("bad.parquet") / 2;
1979-
assert!(reads_per_load > 0);
1980-
assert_eq!(store.reads("bad.parquet"), 2 * reads_per_load);
1998+
assert!(
1999+
store.reads("bad.parquet") >= 2,
2000+
"both the prefetch and the open should read bad.parquet",
2001+
);
19812002
assert_eq!(metric(&metrics, "scan_io_metadata_prefetches"), 0);
19822003
}
19832004

‎native/core/src/parquet/parquet_exec.rs‎

Lines changed: 18 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -244,12 +244,14 @@ pub(crate) fn init_datasource_exec(
244244
// its footer metadata (see `datetime_rebase.rs`), which is how the expression adapter
245245
// attributes timestamp columns to Spark's INT64 vs INT96 rebase specs.
246246
let scan_io_source = scan_io_source(object_store_backend);
247-
// Comet's scans read one file group, the only shape the prefetch handles.
247+
// Comet's scans read one file group, the only shape the prefetch handles. An encrypted scan
248+
// never prefetches, since a prefetch carries no decryption properties.
248249
let prefetch_files = match file_groups.as_slice() {
249250
[files]
250-
if session_config
251-
.get_extension::<CometParquetScanOptions>()
252-
.is_some_and(|options| options.prefetch_next_file_metadata) =>
251+
if !encryption_enabled
252+
&& session_config
253+
.get_extension::<CometParquetScanOptions>()
254+
.is_some_and(|options| options.prefetch_next_file_metadata) =>
253255
{
254256
Some(files.as_slice())
255257
}
@@ -1350,9 +1352,8 @@ mod tests {
13501352
}
13511353

13521354
/// The prefetch is opt-in: a scan of several files loads metadata ahead only when the session
1353-
/// sets `prefetch_next_file_metadata`. With it, only the first file's open reads metadata from
1354-
/// storage, since each later file's footer and page index were loaded while the file before
1355-
/// it was read, and the batches match a scan without it.
1355+
/// sets `prefetch_next_file_metadata`. With it, each later file's footer and page index are
1356+
/// loaded while the file before it is read, and the batches match a scan without it.
13561357
#[tokio::test]
13571358
async fn prefetching_the_next_files_metadata_keeps_the_scan_result() {
13581359
let fixtures: Vec<_> = (0..3).map(|_| write_scan_io_fixture()).collect();
@@ -1363,14 +1364,12 @@ mod tests {
13631364
.collect();
13641365

13651366
let mut results = vec![];
1366-
// `None` is a session without the options, which every Comet plan attaches.
1367-
for prefetch in [None, Some(false), Some(true)] {
1368-
let mut config = datafusion::prelude::SessionConfig::new();
1369-
if let Some(prefetch) = prefetch {
1370-
config = config.with_extension(Arc::new(CometParquetScanOptions {
1367+
for prefetch in [false, true] {
1368+
let config = datafusion::prelude::SessionConfig::new().with_extension(Arc::new(
1369+
CometParquetScanOptions {
13711370
prefetch_next_file_metadata: prefetch,
1372-
}));
1373-
}
1371+
},
1372+
));
13741373
let session_ctx = Arc::new(SessionContext::new_with_config(config));
13751374
let scan = init_test_scan(
13761375
Arc::clone(&schema),
@@ -1390,16 +1389,13 @@ mod tests {
13901389
3_000
13911390
);
13921391

1393-
let (misses, hits, prefetches) = if prefetch == Some(true) {
1394-
(1, 2, 2)
1395-
} else {
1396-
(3, 0, 0)
1397-
};
1398-
assert_eq!(scan_metric(&scan, "scan_io_metadata_cache_misses"), misses);
1399-
assert_eq!(scan_metric(&scan, "scan_io_metadata_cache_hits"), hits);
1392+
// Every file is read from storage either way, so the prefetch only moves two of the
1393+
// loads ahead of their open.
1394+
assert_eq!(scan_metric(&scan, "scan_io_metadata_cache_misses"), 3);
1395+
assert_eq!(scan_metric(&scan, "scan_io_metadata_cache_hits"), 0);
14001396
assert_eq!(
14011397
scan_metric(&scan, "scan_io_metadata_prefetches"),
1402-
prefetches
1398+
if prefetch { 2 } else { 0 }
14031399
);
14041400
assert_eq!(scan_metric(&scan, "scan_io_footer_reads"), 3);
14051401
results.push(batches);

‎spark/src/main/scala/org/apache/spark/sql/comet/CometMetricNode.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -452,7 +452,9 @@ object CometMetricNode {
452452
"scan_io_metadata_cache_misses" ->
453453
SQLMetrics.createMetric(sc, "Metadata loads requiring storage I/O"),
454454
"scan_io_metadata_prefetches" ->
455-
SQLMetrics.createMetric(sc, "Metadata loads read from storage ahead of the file's open"),
455+
SQLMetrics.createMetric(
456+
sc,
457+
"Metadata loads read from storage ahead of the file's open and used by it"),
456458
"pushdown_rows_pruned" ->
457459
SQLMetrics.createMetric(sc, "Rows filtered out by predicates pushed into parquet scan"),
458460
"pushdown_rows_matched" ->

‎spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala‎

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -3488,26 +3488,24 @@ class CometExecSuite extends CometTestBase {
34883488
withTempPath { dir =>
34893489
val path = dir.getCanonicalPath
34903490
spark.range(0, 4000, 1, 4).toDF("id").write.parquet(path)
3491-
// `None` leaves the config unset: the prefetch is opt-in.
3492-
Seq(Some(true), Some(false), None).foreach { prefetch =>
3493-
val prefetchConf = prefetch.map { enabled =>
3494-
CometConf.COMET_PARQUET_PREFETCH_NEXT_FILE_METADATA_ENABLED.key -> enabled.toString
3495-
}
3491+
Seq(true, false).foreach { prefetch =>
3492+
val prefetchConf =
3493+
CometConf.COMET_PARQUET_PREFETCH_NEXT_FILE_METADATA_ENABLED.key -> prefetch.toString
34963494
// Pack the four files into one task.
34973495
val packConfs = Seq(
34983496
SQLConf.FILES_MIN_PARTITION_NUM.key -> "1",
34993497
SQLConf.FILES_OPEN_COST_IN_BYTES.key -> "1")
3500-
withSQLConf(packConfs ++ prefetchConf: _*) {
3498+
withSQLConf((packConfs :+ prefetchConf): _*) {
35013499
val (_, cometPlan) = checkSparkAnswerAndOperator(spark.read.parquet(path))
35023500
val scan = find(cometPlan)(_.isInstanceOf[CometNativeScanExec])
35033501
assert(scan.isDefined, "Expected to find a CometNativeScanExec")
35043502
assert(scan.get.executeColumnar().getNumPartitions == 1)
35053503
val metrics = scan.get.metrics
3506-
// With the prefetch, each open after the first finds its metadata already loaded.
3507-
val (misses, hits, prefetches) = if (prefetch.contains(true)) (1, 3, 3) else (4, 0, 0)
3508-
assert(metrics("scan_io_metadata_cache_misses").value == misses)
3509-
assert(metrics("scan_io_metadata_cache_hits").value == hits)
3510-
assert(metrics("scan_io_metadata_prefetches").value == prefetches)
3504+
// Every file is read from storage either way. With the prefetch, each open after the
3505+
// first finds its metadata already loaded.
3506+
assert(metrics("scan_io_metadata_cache_misses").value == 4)
3507+
assert(metrics("scan_io_metadata_cache_hits").value == 0)
3508+
assert(metrics("scan_io_metadata_prefetches").value == (if (prefetch) 3 else 0))
35113509
assert(metrics("scan_io_footer_reads").value == 4)
35123510
}
35133511
}

0 commit comments

Comments
 (0)