Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 0.13.0#19728
Open
alamb wants to merge 60 commits intoapache:mainfrom
Open
Upgrade DataFusion to arrow-rs/parquet 58.0.0 / object_store 0.13.0#19728alamb wants to merge 60 commits intoapache:mainfrom
object_store 0.13.0#19728alamb wants to merge 60 commits intoapache:mainfrom
Conversation
github-actions
bot
added
sqllogictest
alamb
commented
Jan 10, 2026
alamb
commented
Jan 10, 2026
| | alltypes_plain.parquet | 1851 | 8882 | 2 | page_index=false | | ||
| | alltypes_tiny_pages.parquet | 454233 | 269266 | 2 | page_index=true | | ||
| | lz4_raw_compressed_larger.parquet | 380836 | 1347 | 2 | page_index=false | | ||
| | alltypes_tiny_pages.parquet | 454233 | 269074 | 2 | page_index=true | |
Contributor
Author
There was a problem hiding this comment.
I think this reduction in metadata size is a direct consequence of @WaterWhisperer's PR to improve PageEncoding representation
Contributor
|
Run benchmarks |
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
|
run benchmarks |
Contributor
|
run benchmark tpch |
|
🤖 |
|
🤖: Benchmark completed Details
|
|
🤖: Benchmark completed Details
|
alamb
changed the title
object_store 13.0.0object_store 13.0.0
alamb
commented
Feb 23, 2026
| let timestamp = Utc::now(); | ||
| let range = options.range.clone(); | ||
|
|
||
| let head = options.head; |
Contributor
Author
There was a problem hiding this comment.
A substantial amount of the changes in this PR are due to the upgrade to object_store 0.13 where several of the trait methods are consolidated (e.g. get, get_opts, head, etc) have been consolidated.
You can see the upgrade guide here: https://docs.rs/object_store/latest/object_store/trait.ObjectStore.html#upgrade-guide-for-0130
alamb
commented
Feb 23, 2026
|
|
||
| let props = WriterProperties::builder() | ||
| .set_max_row_group_size(2) | ||
| .set_max_row_group_row_count(Some(2)) |
Contributor
Author
There was a problem hiding this comment.
This configuration was renamed in
alamb
commented
Feb 23, 2026
| .location | ||
| .parts() | ||
| .last() | ||
| .next_back() |
Contributor
Author
There was a problem hiding this comment.
clippy told me to do this -- I am not sure why it doesn't do so on main
alamb
commented
Feb 23, 2026
| _opts: PutOptions, | ||
| ) -> object_store::Result<PutResult> { | ||
| Err(object_store::Error::NotImplemented) | ||
| unimplemented!() |
Contributor
Author
There was a problem hiding this comment.
the NotImplemented error now contains new fields about operations that are not implemented. I just kept the code simple and used unimplemented instead
alamb
commented
Feb 23, 2026
| Total Requests: 2 | ||
| - HEAD path=csv_table.csv | ||
| - GET path=csv_table.csv | ||
| - GET (opts) path=csv_table.csv head=true |
Contributor
Author
There was a problem hiding this comment.
There are many fewer methods in the ObjectStore trait now, and the tests have been updated to reflect that. The actual calls are all still the same
alamb
commented
Feb 23, 2026
| // no content for head requests | ||
| GetResultPayload::Stream(stream::empty().boxed()) | ||
| } else if let Some(range) = options.range { | ||
| let GetRange::Bounded(range) = range else { |
Contributor
Author
There was a problem hiding this comment.
I had to inline the implementation of head() and get_range() here
alamb
commented
Feb 23, 2026
|
|
||
| let stop = if !self.include_upper_bound { | ||
| Date32Type::subtract_month_day_nano(stop, step) | ||
| Date32Type::subtract_month_day_nano_opt(stop, step).ok_or_else(|| { |
Contributor
Author
There was a problem hiding this comment.
due to apache/arrow-rs#9144 from @cht42 (the underlying library now checks for overflow and returns None rather than panic'ing)
| 07)│ DataSourceExec │ | ||
| 08)│ -------------------- │ | ||
| 09)│ bytes: 1040 │ | ||
| 09)│ bytes: 1024 │ |
Contributor
Author
There was a problem hiding this comment.
the size of parquet files changes slightly from version to version (as the embedded version changes, etc)
alamb
changed the title
object_store 13.0.0object_store 0.13.0
Contributor
Author
|
Ok, I think this PR is ready to go |
Contributor
Author
|
run benchmarks |
github-actions
bot
added
the
documentation
|
🤖 |
|
🤖: Benchmark completed Details
|
Contributor
Author
|
run benchmark clickbench_partitioned |
|
🤖 |
|
🤖: Benchmark completed Details
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
58.0.0(January 2026) arrow-rs#8466Oustanding issues
Rationale for this change
Keep datafusion up to date (and test Arrow using DataFusion tests)
What changes are included in this PR?
Are these changes tested?
Are there any user-facing changes?