Expose datafusion table provider as python binding#1324
Conversation
|
cc @timsaucer for helping me with the integration, finally got around to this |
| import datafusion | ||
|
|
||
| assert ( | ||
| datafusion.__version__ >= "45" |
There was a problem hiding this comment.
Do we need to make sure that rust and python are using the same version of datafusion?
cc @alamb for ideas.
There was a problem hiding this comment.
They do not need to strictly be the same version, but there have been a couple of small ffi api changes that were breaking. I need to track down exactly which versions were compatible. I will try to get to this soon.
Once the ffi api is stable you can use different versions of datafusion rust and python. I have demonstrated this in other projects.
There was a problem hiding this comment.
That's really great! Thank you @timsaucer for the help.
kevinjqliu
force-pushed
the
kevinjqliu/datafusion-iceberg-table-provider
branch
from
eb0fe40 to
cf5bd54
Compare
Xuanwo
left a comment
Xuanwo
left a comment
There was a problem hiding this comment.
Mostly LGTM, cc @liurenjie1024 for another look.
sdd
left a comment
sdd
left a comment
There was a problem hiding this comment.
LGTM now, thanks @kevinjqliu 🙌🏼
|
Thanks everyone! |
4 participants
Which issue does this PR close?
IcebergTableProviderto python #865What changes are included in this PR?
This PR creates a new
IcebergDataFusionTablepython class and exposes it through the newpyiceberg_core.datafusionmodule.The goal of exposing
IcebergDataFusionTableis to be able to register the Iceberg table provider to datafusion-python, using theregister_table_providerAPI.See the usage example in
bindings/python/tests/test_datafusion_table_provider.pyThe integration relies on the
FFI_TableProviderAPI as described in https://datafusion.apache.org/python/user-guide/io/table_provider.htmlNote that this integration only works for
datafusion >= 45due to this issue apache/datafusion#13851Are these changes tested?
Yes, unit tests.
To build and test locally: