@jorgecarleitao what is the rational behind wrapping (FFI_ArrowArray, FFI_ArrowSchema) with ArrowArray and the children with ArrowArrayChild ?

Codecov Report

Merging #439 (7acd3a5) into master (ef88876) will decrease coverage by 0.05%.
The diff coverage is 63.19%.

@@            Coverage Diff             @@
##           master     #439      +/-   ##
==========================================
- Coverage   82.76%   82.71%   -0.06%     
==========================================
  Files         165      166       +1     
  Lines       45724    45853     +129     
==========================================
+ Hits        37844    37927      +83     
- Misses       7880     7926      +46     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ef88876...7acd3a5. Read the comment docs.

At the time I had limited knowledge of what I was doing. I think that we could refactor this so that the arrays are shared and the datatypes are copied, since we do not really need to share metadata.

Thanks for the review, I'm going to add more tests.

While the conversions do copy between the C schema and rust DataType/Field/Schema structs cc @pitrou to confirm that the lifetimes are properly handled. This PR shouldn't affect how the arrays get exchanged.

@jorgecarleitao what do you think about implementing arrow <-> pyarrow interoperability directly in the arrow-rs repository and use that from the datafusion bindings?

The clippy errors should be unrelated.

@jorgecarleitao what do you think about implementing arrow <-> pyarrow interoperability directly in the arrow-rs repository and use that from the datafusion bindings?

@ritchie46 you might be interested in this, as I saw https://github.com/pola-rs/polars/tree/master/py-polars/src/arrow_interop

@jorgecarleitao what do you think about implementing arrow <-> pyarrow interoperability directly in the arrow-rs repository and use that from the datafusion bindings?

I think that it is a good idea.

I am not sure how to execute on it though, as I never built a Rust library out of pyo3. E.g. DataFusion requires some conversions that are currently called from Rust (not Python). Is it enough for it to depend on "arrow-rs-python"?

So, I would say let's try it and see how this goes in terms of packaging and dependency management.

@jorgecarleitao created the relevant follow-up tickets and cleaned up the PR.
@pitrou didn't notice any outstanding problems, so I think this should be ready to go now.

@jorgecarleitao @alamb resolved conflicts with 8672274

@jorgecarleitao may I go ahead and merge it?

@kszucs I took the liberty of rebasing this PR and fixing clippy -- it should be good to go now once the CI passes

Thanks everyone! Merging it then.