Skip to content

Commit 37ba40d

Browse files
committed
address review: say that only loads which read storage count as prefetches
1 parent 9a248d8 commit 37ba40d

2 files changed

Lines changed: 4 additions & 3 deletions

File tree

‎docs/source/user-guide/latest/metrics.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -230,14 +230,15 @@ execution metrics. Counters accumulate per scan operator; they do not instrument
230230
| `scan_io_object_store_response_bytes_read` | Response bytes actually consumed at that API, including bytes fetched between coalesced ranges. Not HTTP wire bytes. |
231231
| `scan_io_metadata_cache_hits` | Successful, cache-eligible metadata opens requiring no storage reads. |
232232
| `scan_io_metadata_cache_misses` | Successful, cache-eligible metadata opens requiring storage reads. Failed opens and encrypted opens, which bypass this shared cache, increment neither cache counter. |
233-
| `scan_io_metadata_prefetches` | Metadata opens that used a load started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). Each also counts as a cache miss. |
233+
| `scan_io_metadata_prefetches` | Opens that used a load which read storage ahead of them, started when the previous file opened (`spark.comet.parquet.prefetchNextFileMetadata.enabled`). Each also counts as a cache miss. |
234234
| `row_groups_pruned_dynamic_filter` | Row groups skipped by a runtime predicate whose value changes during execution, such as a local TopK threshold. |
235235

236236
Reader-level and object-store bytes are two views of the same reads; do not add them together.
237237
Likewise, footer bytes are a subset of metadata bytes, not a third reader-level category. A warm
238238
metadata-cache hit contributes no new metadata or footer I/O. A prefetched load's footer and
239239
metadata bytes count when the prefetch reads them, so a prefetch that no open uses, because the
240-
scan stopped early or pruned the file before opening it, adds bytes without a matching cache hit.
240+
scan stopped early or pruned the file before opening it, adds bytes with no matching cache miss and
241+
no `scan_io_metadata_prefetches`.
241242
A valid plaintext footer followed by a page-index failure still contributes footer bytes; an
242243
invalid plaintext footer does not.
243244

‎docs/source/user-guide/latest/parquet.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ index in the background while the current file is read. The setting is off by de
225225

226226
The cost is one extra footer and page index read for a file that is never opened, such as when a scan stops
227227
early or the file is pruned before it opens. Scans of encrypted Parquet files never prefetch. The
228-
`scan_io_metadata_prefetches` metric counts the opens that used a prefetched load.
228+
`scan_io_metadata_prefetches` metric counts the opens that used a load which read storage ahead of them.
229229

230230
## Known Issues
231231

0 commit comments

Comments
 (0)