Repository navigation
chore: Upgrade to DataFusion ~53. - #4351
Merged
Merged
Conversation
stuhood
added
the
Do Not Cherry Pick
* Removed the deprecated `statistics` method from `PgSearchScanPlan`. * Changed the `properties` field to be wrapped in an `Arc<PlanProperties>` and updated the method signature to return `&Arc<PlanProperties>`. * Derived `std::fmt::Debug` on `PhysicalOptimizerRule` implementations.
stuhood
force-pushed
the
stuhood.datafusion-53
branch
from
March 12, 2026 23:58
6bc2b26 to
42db159
Compare
philippemnoel
approved these changes
Mar 12, 2026
stuhood
marked this pull request as ready for review
March 13, 2026 00:20
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4351 +/- ##
==========================================
- Coverage 81.42% 81.18% -0.25%
==========================================
Files 183 185 +2
Lines 43220 43937 +717
==========================================
+ Hits 35194 35671 +477
- Misses 8026 8266 +240
🚀 New features to boost your workflow:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
pg_search 'docs' (25M rows)
Details
| Benchmark suite | Current: 42db159 | Previous: b75f1da | Ratio |
|---|---|---|---|
aggregate_sort |
13450.506000000001 median ms |
13432.6875 median ms |
1.00 |
aggregate_sort - alternative 1 |
13406.9125 median ms |
13513.29 median ms |
0.99 |
disjunctive_search |
824.0550000000001 median ms |
873.5765 median ms |
0.94 |
disjunctive_search - alternative 1 |
146562.48200000002 median ms |
146315.6745 median ms |
1.00 |
distinct_parent_sort |
5957.7735 median ms |
6031.9265 median ms |
0.99 |
distinct_parent_sort - alternative 1 |
7714.0785 median ms |
7699.352000000001 median ms |
1.00 |
foreign_filter_local_sort |
553.7085 median ms |
575.4185 median ms |
0.96 |
foreign_filter_local_sort - alternative 1 |
489.1565 median ms |
494.526 median ms |
0.99 |
hierarchical_content-no-scores-large |
1987.775 median ms |
2061.0475 median ms |
0.96 |
hierarchical_content-no-scores-large - alternative 1 |
111.60300000000001 median ms |
113.297 median ms |
0.99 |
hierarchical_content-no-scores-small |
519.383 median ms |
520.0025 median ms |
1.00 |
hierarchical_content-no-scores-small - alternative 1 |
107.63300000000001 median ms |
107.819 median ms |
1.00 |
hierarchical_content-scores-large |
987.941 median ms |
1088.142 median ms |
0.91 |
hierarchical_content-scores-large - alternative 1 |
712.9675 median ms |
721.3305 median ms |
0.99 |
hierarchical_content-scores-large - alternative 2 |
1045.964 median ms |
1122.7669999999998 median ms |
0.93 |
hierarchical_content-scores-small |
699.2165 median ms |
703.947 median ms |
0.99 |
hierarchical_content-scores-small - alternative 1 |
704.156 median ms |
711.0495 median ms |
0.99 |
paging-string-max |
15.135 median ms |
15.2105 median ms |
1.00 |
paging-string-median |
45.453 median ms |
46.65 median ms |
0.97 |
paging-string-min |
59.939499999999995 median ms |
61.078500000000005 median ms |
0.98 |
permissioned_search |
1408.641 median ms |
1486.5375 median ms |
0.95 |
permissioned_search - alternative 1 |
1895.147 median ms |
1975.8400000000001 median ms |
0.96 |
semi_join_filter |
459.57050000000004 median ms |
451.7555 median ms |
1.02 |
semi_join_filter - alternative 1 |
345.1735 median ms |
345.726 median ms |
1.00 |
semi_join_filter - alternative 2 |
460.206 median ms |
459.887 median ms |
1.00 |
semi_join_filter - alternative 3 |
2652.833 median ms |
2670.651 median ms |
0.99 |
This comment was automatically generated by workflow using github-action-benchmark.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Upgrade to what will become DataFusion
53by building atopbranch-53.Why
In order to make it easier to backport changes to
0.22.x, and hopefully give that branch a longer life than it might have otherwise.How
statisticsmethod fromPgSearchScanPlan.propertiesfield to be wrapped in anArc<PlanProperties>and updated the method signature to return&Arc<PlanProperties>.DebugforPhysicalOptimizerRuleimplementations.Tests
There are many plan changes. But having triaged them, they seem to be primarily that:
dynamic_filterscounts dropped.GlobalLimitExecin favor of feat: Push limit into hash join apache/datafusion#20228, which pushes the limit into a hash join.