feat: upgrade datafusion to 51 #9202
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Failed to generate code suggestions for PR |
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request upgrades the DataFusion query engine from version 50.3.0 to 51.0.0, along with related dependencies including Arrow (56→57), Parquet (56→57), and various OpenTelemetry/Tonic libraries. The upgrade adapts to breaking API changes in the DataFusion ecosystem, particularly around physical plan construction, codec interfaces, and configuration.
Key Changes:
- Updated DataFusion APIs for physical plan codecs (SessionContext → TaskContext)
- Adapted to fallible
UnionExec::try_new()constructor - Updated protobuf/tonic stack from 0.13 to 0.14
- Fixed return value handling for
span.set_parent()
Reviewed changes
Copilot reviewed 34 out of 37 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Cargo.toml, Cargo.lock | Upgraded DataFusion, Arrow, Parquet, Tonic, and OpenTelemetry dependencies |
| src/service/search/datafusion/exec.rs | Updated SQL dialect configuration to use enum instead of string |
| src/service/search/datafusion/table_provider/*.rs | Updated physical plan APIs and ParquetFileReaderFactory signature |
| src/service/search/datafusion/distributed_plan/codec/*.rs | Migrated from FunctionRegistry to TaskContext in codec interfaces |
| src/service/search/grpc/flight.rs | Updated physical plan deserialization to use TaskContext |
| src/flight/src/*.rs | Updated IPC encoding to use CompressionContext |
| src/proto/build.rs | Switched from tonic::codec to tonic_prost for protobuf encoding |
| src/handler/grpc/**/*.rs | Fixed span.set_parent() return value handling |
| src/infra/src/storage/mod.rs | Updated parquet metadata API to use FooterTail |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
hengfeiyang
approved these changes
Nov 25, 2025
Shrinath-O2
added
Testing-Completed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.