perf: use datafusion native listingtable #9259
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Failed to generate code suggestions for PR |
Auto-generated translation updates from English source file. 🤖 Generated with automated translation workflow
…openobserve into datafusion-schema-evolution
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request refactors the search infrastructure to use DataFusion's native ListingTable instead of a custom implementation. The PR removes the manual schema version grouping logic and delegates schema evolution to DataFusion's DefaultSchemaAdapterFactory. Key architectural changes include:
- Replacing custom
NewListingTablewith a newListingTableAdapterwrapper - Using DataFusion's native schema evolution capabilities
- Simplifying file handling by removing schema version-based file grouping
- Optimizing memory usage by using
Arc<BitVec>for segment IDs instead of cloning
Reviewed changes
Copilot reviewed 28 out of 28 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
src/service/search/grpc/wal.rs |
Removed schema version grouping logic; simplified to single table creation |
src/service/search/grpc/storage.rs |
Removed schema version handling; streamlined file processing |
src/service/search/grpc/utils.rs |
Deleted file containing UTF8 view schema conversion utility |
src/service/search/grpc/mod.rs |
Removed utils module reference |
src/service/search/datafusion/table_provider/parquet_reader.rs |
Deleted custom parquet reader implementation |
src/service/search/datafusion/table_provider/mod.rs |
Removed custom NewListingTable implementation and related helper functions |
src/service/search/datafusion/table_provider/listing_adapter.rs |
Added new adapter wrapping DataFusion's ListingTable with index condition handling |
src/service/search/datafusion/table_provider/helpers.rs |
Cleaned up by removing unused helper functions |
src/service/search/datafusion/table_provider/memtable.rs |
Updated imports to use helpers from new location |
src/service/search/datafusion/storage/file_list.rs |
Changed to use Arc<BitVec> to avoid cloning segment IDs |
src/service/search/datafusion/mod.rs |
Removed file_type module reference |
src/service/search/datafusion/file_type.rs |
Deleted custom file type enum |
src/service/search/datafusion/exec.rs |
Simplified TableBuilder by removing schema diff rules and optimization flags |
src/service/promql/search/grpc/storage.rs |
Updated to use new simplified register_table signature |
src/service/file_list_dump.rs |
Changed signature to pass files by value |
src/service/compact/merge.rs |
Removed schema diff generation logic; simplified to single table creation |
src/job/files/parquet.rs |
Updated to use simplified TableBuilder API |
src/config/src/meta/stream.rs |
Changed segment_ids field to use Arc<BitVec> |
web/src/locales/languages/*.json |
Added translations for new UI features (insights, alerts, pipeline types) |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
hengfeiyang
approved these changes
Nov 27, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
DefaultSchemaAdapterFactoryto do schema evolutionListingAdapterto do index fallback logicalListingAdapterCachedParquetFileReaderFactory