Skip to content

Commit 9947811

Browse files
committed
docs: note that a running task's preloaded footers can be evicted by the next task
1 parent f674a3e commit 9947811

1 file changed

Lines changed: 6 additions & 2 deletions

File tree

‎native/core/src/parquet/shared_metadata_cache.rs‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,8 +49,12 @@
4949
//! otherwise evict each other's entries, among them the footers the Delta deletion vector planning
5050
//! loads before the scan opens them. The limit stays up while a task finishes and others run on,
5151
//! because the cache evicts the least recently used entry whoever loaded it, so lowering it would
52-
//! take those footers from a running task as readily as the finished task's. Each task takes its
53-
//! share when it asks for the cache and gives it back when its `RuntimeEnv` drops it, so the limit
52+
//! take those footers from a running task as readily as the finished task's. That does not make
53+
//! the footers a running task preloaded safe: they are older than the finished task's entries, so
54+
//! when the next task loads its own, they are evicted first, and the scan that needed them fetches
55+
//! them again. This is an accepted cost, a refetch of a footer a task of its own would have kept,
56+
//! and only when the entries the stage's tasks load exceed what the limit holds. Each task takes
57+
//! its share when it asks for the cache and gives it back when its `RuntimeEnv` drops it, so the limit
5458
//! follows the tasks that actually run, whatever the executor's configured cores say. Shared
5559
//! entries are stored once, so the tasks of a stage hold no more than they did, and a retained
5660
//! cache holds no more than one task's.

0 commit comments

Comments
 (0)