Repository navigation
Commit bb81cb8
Make StatisticsContext cache entries node-lifetime safe (apache#25929)
## Which issue does this PR close?
- Part of apache#25141
## Rationale for this change
`StatisticsContext` can be shared across statistics walks and plan
rewrites, but its cache is keyed by plan-node pointer addresses.
Previously, callers had to reset the cache after a rewritten node was
dropped to prevent a later allocation from reusing the same address and
incorrectly hitting stale statistics or provider extensions.
This change makes cached entries retain the plan node whose pointer
supplied the cache key. As long as an entry remains cached, that node
remains alive and its address cannot be reused by a different plan node.
This makes `reset_cache` optional for correctness and leaves it as
lifecycle/resource control for bounding retained memory.
## What changes are included in this PR?
- Adds `CacheEntry<T>`, which stores both the cached value and an
`Arc<dyn ExecutionPlan>` retaining the node associated with the cache
key.
- Applies retained-node cache entries consistently to core statistics
and provider extensions.
- Adds `compute_arc` and `compute_extended_arc` for callers that have an
`Arc<dyn ExecutionPlan>` and want the root node itself to be memoized.
- Keeps the existing borrowed `compute` and `compute_extended` APIs,
while only memoizing descendants whose owning `Arc`s are available.
- Updates `EnsureRequirements` to use `compute_arc` and removes cache
resets after plan rewrites.
- Updates `StatisticsContext` and `reset_cache` documentation to
describe the new lifetime guarantees and the remaining
resource-management role of cache resets.
## Are these changes tested?
Yes. The patch adds and updates unit tests covering:
- `context_cache_retains_plan_lifetime`
- `owned_parent_and_leaf_are_released_by_reset`
- `owned_parent_and_leaf_are_released_when_context_drops`
- `borrowed_parent_is_not_cached_but_retains_children`
- `borrowed_compute_hits_root_cached_by_compute_arc`
- `extension_cache_retains_plan_lifetime`
- Existing cache/reset and provider-extension tests updated to use the
retained-`Arc` paths where appropriate.
## Are there any user-facing changes?
No user-facing query behavior changes are intended. This changes the
lifetime and memoization behavior of `StatisticsContext` cache entries
so that shared contexts remain safe across plan rewrites.
The patch also adds the public `StatisticsContext::compute_arc` and
`StatisticsContext::compute_extended_arc` methods for callers that want
root-node memoization.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated
content has been manually reviewed.1 parent 8b7d81d commit bb81cb8
3 files changed
Lines changed: 262 additions & 56 deletions
File tree
- datafusion
- physical-optimizer/src/ensure_requirements
- physical-plan/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1010 | 1010 | | |
1011 | 1011 | | |
1012 | 1012 | | |
1013 | | - | |
| 1013 | + | |
1014 | 1014 | | |
1015 | 1015 | | |
1016 | 1016 | | |
| |||
1353 | 1353 | | |
1354 | 1354 | | |
1355 | 1355 | | |
1356 | | - | |
1357 | | - | |
1358 | | - | |
| 1356 | + | |
1359 | 1357 | | |
1360 | 1358 | | |
1361 | 1359 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
225 | 225 | | |
226 | 226 | | |
227 | 227 | | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
232 | | - | |
233 | 228 | | |
234 | 229 | | |
235 | 230 | | |
236 | 231 | | |
237 | 232 | | |
238 | | - | |
239 | | - | |
240 | | - | |
241 | | - | |
242 | | - | |
243 | | - | |
244 | | - | |
245 | | - | |
| 233 | + | |
246 | 234 | | |
247 | 235 | | |
248 | 236 | | |
| |||
0 commit comments