Skip to content

Align statistics cache sharing test comments with reset-free contract - #25965

Merged
alamb merged 2 commits into
apache:mainfrom
kosiew:cacherobust-03-25141
Oct 29, 2026
Merged

alamb merged 2 commits into
apache:mainfrom
kosiew:cacherobust-03-25141

Conversation

@kosiew

@kosiew kosiew commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

This is the last of a series of PR which

Rationale for this change

The ensure_distribution_shares_statistics_cache regression test still describes cache reset behavior that is no longer part of the intended StatisticsContext contract.

The issue moves cache correctness away from an EnsureRequirements rewrite-specific reset protocol. The existing regression should therefore describe only what it actually verifies: that sharing one StatisticsContext across the distribution pass reduces repeated statistics computation compared with creating a fresh context for each node.

What changes are included in this PR?

This PR updates comments and Rustdoc around ensure_distribution_shares_statistics_cache to remove references to reset_cache() behavior.

In particular, it:

  • describes the regression in terms of sharing a StatisticsContext across nodes rather than resetting the cache
  • clarifies that the baseline allocates a fresh StatisticsContext per node
  • removes the implication that the fixpoint plan is important because it prevents cache resets
  • narrows the final assertion comment to the invariant the test actually checks: a shared context performs fewer statistics computations than a fresh context per node

There are no executable code changes in this patch.

Are these changes tested?

The existing ensure_distribution_shares_statistics_cache regression test is unchanged.

It continues to compare the real EnsureRequirements path using a shared StatisticsContext against a baseline that creates a fresh context per node at depths 4 and 12, and asserts that the shared path performs fewer leaf-statistics computations.

No new test cases are added by this patch.

Are there any user-facing changes?

No. This patch only updates test comments and Rustdoc to match the intended statistics-cache contract. There are no public API or runtime behavior changes in this patch.

LLM-generated code disclosure

This PR includes LLM-generated code and comments. All LLM-generated content has been manually reviewed.

@github-actions github-actions Bot added the core Core DataFusion crate label Oct 2, 2026
@kosiew
kosiew requested a review from zhuqi-lucas October 2, 2026 10:43
@kosiew
kosiew marked this pull request as ready for review October 2, 2026 10:43
@codecov-commenter

codecov-commenter commented Oct 2, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.64%. Comparing base (416002a) to head (630034f).
⚠️ Report is 16 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #25965      +/-   ##
==========================================
+ Coverage   82.59%   82.64%   +0.04%     
==========================================
  Files        1145     1147       +2     
  Lines      444096   445440    +1344     
  Branches   444096   445440    +1344     
==========================================
+ Hits       366806   368113    +1307     
+ Misses      55021    54975      -46     
- Partials    22269    22352      +83     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@asolimando asolimando left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kosiew LGTM, one minor nit, thanks for the improvement over StatisticsContext cache!

Comment on lines +5954 to +5958
// The measured arm drives the real `EnsureRequirements` rule, so the shared
// context under test is the one the rule actually uses — reimplementing it
// here would keep passing even if the rule stopped sharing. The baseline arm
// allocates a fresh `StatisticsContext` per node, reproducing the behavior
// before this change.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I wonder if this comment is needed at all, the test reads well without, and it seems to cover why we did the change rather than the current state of the test itself. wdyt?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @kosiew and @asolimando

@alamb
alamb added this pull request to the merge queue Oct 29, 2026
@alamb alamb added documentation Improvements or additions to documentation development-process Related to development process of DataFusion and removed documentation Improvements or additions to documentation labels Oct 29, 2026
Merged via the queue into apache:main with commit c426589 Oct 29, 2026
43 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate development-process Related to development process of DataFusion v56.0.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

StatisticsContext: make the pointer-keyed cache robust against node reuse when shared across a plan-rewriting pass

4 participants