You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2c568b4
Browse filesBrowse the repository at this point in the historyBrowse files
perf: share one StatisticsContext across all physical optimizer rules (apache#26094)
## Which issue does this PR close?
- Part of apache#8227.
- Related to apache#25141.
## Rationale for this change
Each physical optimizer rule that reads statistics creates its own
`StatisticsContext`, so the statistics of the same plan nodes are
computed again in every rule. `JoinSelection` creates a new context for
every `get_stats` call, so it has no caching at all, even within its own
pass.
apache#25098 showed the gain of sharing one context inside
`EnsureDistribution`. apache#25929 made cache entries keep the plan node they
were computed for, so a context can now be shared safely across plan
rewrites. This PR uses that to share one `StatisticsContext` across all
the rules of one `optimize_physical_plan` call.
On TPC-DS (98 queries) and TPC-H (21 queries), sf1 Parquet, planned with
a local harness (not part of this PR):
| | `main` | this PR |
|---|---|---|
| Statistics cache misses (statistics computed from scratch) | 46,022 |
25,428 (-45%) |
| Physical planning time, sum over all queries | 294.8 / 293.5 ms |
274.3 / 279.7 ms (about -6%) |
| Physical plans | | identical for all 119 queries |
The largest gain is TPC-DS q64: cache misses go from 3,785 to 531 and
planning time drops by about 21%.
## What changes are included in this PR?
- `StatisticsContext` stores its cache in a `parking_lot::Mutex` instead
of `Rc<RefCell>`, so it is `Send + Sync`. This is needed because
`PhysicalOptimizerContext: Send + Sync`. The lock is held only for
single map lookups and inserts, never across the recursive walk. The
`compute_statistics` benchmark shows no difference from `main` (all
cases within ±4%, in both directions).
- New `PhysicalOptimizerContext::statistics_context()`, which returns
`None` by default. `DefaultPhysicalPlanner` creates one context per
`optimize_physical_plan` call, built from the session's statistics
registry, and returns it to every rule.
- `ConfigOnlyContext` also owns a `StatisticsContext`, so a rule called
through `optimize()` shares one cache for its whole pass.
- `JoinSelection`, `EnsureRequirements` (including
`PlanSize::from_plan`), `AggregateStatistics` and `LimitPushdown` use
the shared context. When a context does not share one (for example, a
context received through FFI), the rules create a new context from the
statistics registry, as they did before.
- New default method `PhysicalOptimizerContext::compute_statistics`,
which custom rules can call on the context they already receive. Without
it, a custom rule would have to repeat the fallback itself (use the
shared context, otherwise build one from `statistics_registry()`), and
the obvious shortcut, `StatisticsContext::new()`, does not consult the
registered statistics providers. With it, custom rules behave exactly
like the built-in ones.
- The fallback itself is the public function `with_statistics_context`.
Rules that walk the plan (`EnsureRequirements`, `LimitPushdown`) use it
to get one `StatisticsContext` for the whole traversal, so a fallback
context still caches within the pass.
- `pushdown_limit_helper` is deprecated in favour of the new
`pushdown_limit_helper_with_stats`, which takes a `&StatisticsContext`.
This is the same pattern as `ensure_distribution_with_stats`.
- The shared cache keeps every plan node whose statistics were computed
alive until `optimize_physical_plan` returns, including nodes that a
later rule replaces. Only the rules that read statistics add entries,
and the memory is released at the end of physical planning.
## What is the testing strategy for this PR?
- New `optimizer_rules_share_statistics_context` test in
`physical_planner.rs`: two rules compute the root statistics through the
shared context, and the second one gets the `Arc` cached by the first.
- Existing tests cover the rewired rules. All sqllogictests pass with no
plan changes.
## Are there any user-facing changes?
No breaking changes.
- New default methods `PhysicalOptimizerContext::statistics_context()`
and `PhysicalOptimizerContext::compute_statistics()`.
- New public functions `datafusion_session::with_statistics_context` and
`pushdown_limit_helper_with_stats`; `pushdown_limit_helper` is
deprecated. All are described in the 56.0.0 upgrade guide.
- `StatisticsContext` is now `Send + Sync`.
- `AggregateStatistics`, `LimitPushdown` and `PlanSize::from_plan` now
consult the session's statistics providers, as `JoinSelection` and
`EnsureRequirements` already do. Without registered providers (the
default), their results are unchanged.
----
Disclaimer: I used AI to assist in the code generation, I have manually
reviewed the output and it matches my intention and understanding.
---------
Co-authored-by: Qi Zhu <[email protected]>
#[expect(clippy::allow_attributes)]// See https://github.com/apache/datafusion/issues/18881#issuecomment-3621545670
51
-
#[allow(clippy::only_used_in_recursion)]// See https://github.com/rust-lang/rust-clippy/issues/14566
0 commit comments