Move forward to datafusion 52#291
Conversation
|
🤔 The ballista dependency is in fact a problem. Let me create a separate PR removing ballista. |
|
Done, hopefully this will make your life easier |
|
Awesome, let me try to get this PR working then! |
# Conflicts: # Cargo.lock # benchmarks/Cargo.toml
`RoundRobinBatch`
|
I think the fact that the Previously, we were marking the If you are down for doing it, I'm more than happy to guide you, otherwise I can make a PR to this one with the necessary changes. |
|
I took a look and the necessary changes for fixing the tests are pretty nasty. I don't think it's fair to throw you into the depths of the distributed planner, so I made a new PR to this one fixing the remaining issues: |
…df52-fix Adapt distributed planner for v52
|
Thank you, merged in. I'll give whatever's left broken a shot tomorrow - or should #294 be merged first? |
|
If CI passes, let’s pull this one in directly and just do follow ups. Thanks for handling this! |
gabotechs
left a comment
gabotechs
left a comment
There was a problem hiding this comment.
Thanks @marc-pydantic !
2 participants
This PR moves everything to Datafusion 52, which has been released yesterday/today (depending where you are!). All dependencies that are shared with
datafusionare upgraded to the same version the Datafusion 52 release is using to minimize friction. Other that that, it was just a minor patch so far.The big issue is with
ballista: It depends transitively on a different version of the nativelzma, due to a differentapache-avroversion being amongst its dependencies. There's no way to reconcile this due to C's lack of namespaces, so I commented out ballista for now. That's why this PR is a draft for now, please let me know how to proceed!Update: I reviewed all
cargo instasnapshots, but a second pair of eyes can't hurt. From what I understand these are exclusively changes in the optimizer causing lessCoalesceBatchesExecandRoundRobinBatchnodes to be emitted.