You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6be7a15
Browse filesBrowse the repository at this point in the historyBrowse files
address review: record prefetched opens as cache misses and document the prefetch
A prefetched open now records the storage reads of the load it waited for, so
scan_io_metadata_cache_hits keeps meaning 'served from the cache' and
scan_io_metadata_prefetches counts the loads that overlapped a previous file.
An encrypted scan no longer builds the prefetch at all. The ranges test now
opens through the factory, and the tuning guide documents the config.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/metrics.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -230,7 +230,7 @@ execution metrics. Counters accumulate per scan operator; they do not instrument
230
230
|`scan_io_object_store_response_bytes_read`| Response bytes actually consumed at that API, including bytes fetched between coalesced ranges. Not HTTP wire bytes. |
231
231
|`scan_io_metadata_cache_hits`| Successful, cache-eligible metadata opens requiring no storage reads. |
232
232
|`scan_io_metadata_cache_misses`| Successful, cache-eligible metadata opens requiring storage reads. Failed opens and encrypted opens, which bypass this shared cache, increment neither cache counter. |
233
-
|`scan_io_metadata_prefetches`| Metadata loads that read storage before their file's open, started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). The open that uses one counts as a cache hit.|
233
+
|`scan_io_metadata_prefetches`| Metadata opens that used a load started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). Each also counts as a cache miss. |
234
234
|`row_groups_pruned_dynamic_filter`| Row groups skipped by a runtime predicate whose value changes during execution, such as a local TopK threshold. |
235
235
236
236
Reader-level and object-store bytes are two views of the same reads; do not add them together.
Copy file name to clipboardExpand all lines: docs/source/user-guide/latest/tuning/scans.md
+12Lines changed: 12 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -64,6 +64,18 @@ more splits overall, so some idle tasks may remain — tune the value against yo
64
64
See [#3817](https://github.com/apache/datafusion-comet/issues/3817#issuecomment-4193279630) for a
65
65
worked example and further discussion.
66
66
67
+
### Prefetching the Next File's Metadata
68
+
69
+
A native Parquet scan opens a task's files one after another, and each open waits for a round trip to read
70
+
the file's footer and page index before its data fetch can start. For tasks that read many small files from
71
+
high-latency storage such as S3, that wait is paid once per file. Set
72
+
`spark.comet.parquet.prefetchNextFileMetadata.enabled` to `true` to load the next file's footer and page
73
+
index in the background while the current file is read. The setting is off by default.
74
+
75
+
The cost is one extra footer and page index read for a file that is never opened, such as when a scan stops
76
+
early or the file is pruned before it opens. Scans of encrypted Parquet files never prefetch. The
77
+
`scan_io_metadata_prefetches` metric counts the opens that used a prefetched load.
78
+
67
79
## Iceberg Scan Tuning
68
80
69
81
Comet's native Iceberg scan (`spark.comet.scan.icebergNative.enabled`, enabled by default) reads each
0 commit comments