Repository navigation
deps: upgrade to DataFusion 55.0 and Arrow/Parquet 59.2 - #2958
Conversation
mbutrovich
changed the title
|
I will get this cleaned up and ready for review today now that crates are available. |
mbutrovich
changed the title
gabotechs
left a comment
gabotechs
left a comment
There was a problem hiding this comment.
👍 Everything looks good here, just the expected mechanical changes from the datfusion@55 upgrade.
|
@mbutrovich Care to rebase this so this can be merged? :) |
|
@CTTY would like to see the 0.11 release branch be cut ahead of this version bump, with the reasoning being that DataFusion 54.1 can be considered stable at this point and consumers may benefit from that. Please can you confirm you're aligned, and we can go ahead with cutting that branch before merging this? |
|
That doesn't stop the rebase of course :) |
|
Yes, I'm leaning toward releasing 0.11 before bumping DataFusion to v55 |
No concerns from me. Seems like 0.11 will happen pretty quickly. Note that I'm about to head on vacation until after Labor Day (US) so would need a committer to keep resolving the merge conflicts as they arise. |
|
Given we've now cut the 0.11.x branch, I think we're good to move forward here towards merging into |
|
@dannycjones should we wait until RC passes before merging? |
I think we're OK to go ahead merging. I am planning to backport anything to branch |
|
thanks @mbutrovich and everyone for the review! |
|
hello team, can we create a release for this? Probably as the last version before the migration. |
Hi, is there anything that is preventing you from using the migrated integration? I'd expect the migrated crates to be published very soon. We want to release 0.11.0 so the released datafusion support can stay on the last stable datafusion version(54.1) |
|
@CTTY thank you for the update! in my case, I only use the core module of iceberg-rust, so I would expect a release with arrow 59. |
| "maturin>=1.9.4,<2.0", | ||
| "pytest>=8.3.2", | ||
| "datafusion>=52,<55", | ||
| "datafusion==54.*", |
There was a problem hiding this comment.
@mbutrovich why do we target 54 here? I'm not super familiar with the Python bindings and I see these are test dependencies.
* bump df to 55.0 (head on main) and arrow to 59.1 * upmerge main after opendal bump * fix some CI failures * python version bumps * bump to branch-55 and rc2 commit. bump to arrow and parquet 59.2 * bump to branch-55 and rc2 commit. bump to arrow and parquet 59.2 * bump to final crates, fix API changes * update public API * gate test
Which issue does this PR close?
What changes are included in this PR?
Are these changes tested?
Existing tests.
AI Disclosure
Claude helped with some mechanical stuff as this was in development (bumping WIP branches and commits, API changes) but I understand and support the changes.