Skip to content

Commit bb81cb8

Browse files
kosiewOmega359
authored andcommitted
Make StatisticsContext cache entries node-lifetime safe (apache#25929)
## Which issue does this PR close? - Part of apache#25141 ## Rationale for this change `StatisticsContext` can be shared across statistics walks and plan rewrites, but its cache is keyed by plan-node pointer addresses. Previously, callers had to reset the cache after a rewritten node was dropped to prevent a later allocation from reusing the same address and incorrectly hitting stale statistics or provider extensions. This change makes cached entries retain the plan node whose pointer supplied the cache key. As long as an entry remains cached, that node remains alive and its address cannot be reused by a different plan node. This makes `reset_cache` optional for correctness and leaves it as lifecycle/resource control for bounding retained memory. ## What changes are included in this PR? - Adds `CacheEntry<T>`, which stores both the cached value and an `Arc<dyn ExecutionPlan>` retaining the node associated with the cache key. - Applies retained-node cache entries consistently to core statistics and provider extensions. - Adds `compute_arc` and `compute_extended_arc` for callers that have an `Arc<dyn ExecutionPlan>` and want the root node itself to be memoized. - Keeps the existing borrowed `compute` and `compute_extended` APIs, while only memoizing descendants whose owning `Arc`s are available. - Updates `EnsureRequirements` to use `compute_arc` and removes cache resets after plan rewrites. - Updates `StatisticsContext` and `reset_cache` documentation to describe the new lifetime guarantees and the remaining resource-management role of cache resets. ## Are these changes tested? Yes. The patch adds and updates unit tests covering: - `context_cache_retains_plan_lifetime` - `owned_parent_and_leaf_are_released_by_reset` - `owned_parent_and_leaf_are_released_when_context_drops` - `borrowed_parent_is_not_cached_but_retains_children` - `borrowed_compute_hits_root_cached_by_compute_arc` - `extension_cache_retains_plan_lifetime` - Existing cache/reset and provider-extension tests updated to use the retained-`Arc` paths where appropriate. ## Are there any user-facing changes? No user-facing query behavior changes are intended. This changes the lifetime and memoization behavior of `StatisticsContext` cache entries so that shared contexts remain safe across plan rewrites. The patch also adds the public `StatisticsContext::compute_arc` and `StatisticsContext::compute_extended_arc` methods for callers that want root-node memoization. ## LLM-generated code disclosure This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.
1 parent 8b7d81d commit bb81cb8

3 files changed

Lines changed: 262 additions & 56 deletions

File tree

‎datafusion/physical-optimizer/src/ensure_requirements/enforce_distribution.rs‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1010,7 +1010,7 @@ fn get_repartition_requirement_status(
10101010
// Decide whether adding a round robin is beneficial depending on
10111011
// the statistical information we have on the number of rows:
10121012
let roundrobin_beneficial_stats = match stats_ctx
1013-
.compute(child.as_ref(), &StatisticsArgs::new())?
1013+
.compute_arc(child, &StatisticsArgs::new())?
10141014
.num_rows
10151015
{
10161016
Precision::Exact(n_rows) => n_rows > batch_size,
@@ -1353,9 +1353,7 @@ pub fn ensure_distribution(
13531353
/// `stats_ctx` carries the memoization cache used to answer the child
13541354
/// statistics queries behind the repartition decisions. Share one context
13551355
/// across the whole traversal so each subtree is computed once rather than
1356-
/// once per ancestor. The cache is keyed by raw plan-node pointers, so reset
1357-
/// it (via [`StatisticsContext::reset_cache`]) after any node whose plan
1358-
/// pointer actually changed.
1356+
/// once per ancestor.
13591357
#[expect(
13601358
deprecated,
13611359
reason = "HashPartitioned is accepted during the KeyPartitioned migration"

‎datafusion/physical-optimizer/src/ensure_requirements/mod.rs‎

Lines changed: 1 addition & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -225,24 +225,12 @@ impl PhysicalOptimizerRule for EnsureRequirements {
225225
// subtree's statistics are computed once instead of once per ancestor.
226226
// Build it from the session's statistics registry so registered providers
227227
// are consulted (an empty registry, the default, is unchanged behavior).
228-
// `StatsCache` is keyed by raw node pointer, so reset it after any node
229-
// whose plan pointer actually changed: a rewrite can free a cached node
230-
// and a later allocation could reuse its address. A node that makes no
231-
// change cannot free anything, so the cache safely persists across the
232-
// no-op nodes that dominate a deep plan.
233228
let stats_ctx = match context.statistics_registry() {
234229
Some(registry) => StatisticsContext::new_with_registry(registry.clone()),
235230
None => StatisticsContext::new(),
236231
};
237232
let dist_ctx = dist_ctx
238-
.transform_up(|ctx| {
239-
let before = Arc::clone(&ctx.plan);
240-
let result = ensure_distribution_with_stats(ctx, config, &stats_ctx)?;
241-
if !Arc::ptr_eq(&before, &result.data.plan) {
242-
stats_ctx.reset_cache();
243-
}
244-
Ok(result)
245-
})
233+
.transform_up(|ctx| ensure_distribution_with_stats(ctx, config, &stats_ctx))
246234
.data()?;
247235

248236
// Step 2b: Sorting enforcement (bottom-up) — runs on distribution-fixed plan

0 commit comments

Comments
 (0)