Repository navigation
Use arrow-avro for performance and improved type support #14097
Description
Is your feature request related to a problem or challenge?
See the new blog post Announcing arrow-avro in Arrow Rust for deatils
Currently DataFusion has its own avro --> arrow implementation in: https://github.com/apache/datafusion/blob/54a5d3fd3f98997f048c2e3e7ba67d7bfe2f124d/datafusion/core/src/datasource/avro_to_arrow/mod.rs
This was added way back in the day by @Igosuki and @jorgecarleitao in
Recently there is work upstream in arrow-rs to add additional avro support:
For example, the recent PR from @jecsand838
Describe the solution you'd like
One arrow-avro is more mature, it would be great to consider removing the DataFusion specific implementation and using the upstream arrow-avro one
This would allow DataFusion to take advantage of the upstream work and keep our codebase a bit simpler
Describe alternatives you've considered
No response
Additional context
No response
Activity
getChan commentedon Jan 19, 2025
take
getChan commentedon Jan 25, 2025
I'm waiting for the arrow-avro PR below as it might include changes to the public API.
apache/arrow-rs#6965
I will resume related work once this PR is merged and arrow-avro is released.
jecsand838 commentedon Sep 15, 2025
@getChan @alamb
I had a quick update I wanted to give.
Once apache/arrow-rs#8293, apache/arrow-rs#8348, and apache/arrow-rs#8349 are reviewed and merged in, The
arrow-avroReaderwill be functionally ready for use in DataFusion. We are getting extremely close.Also on the
arrow-avroWriterside, we have about 4-5 PRs left as well.getChan commentedon Sep 15, 2025
@jecsand838
Thank you for the update. I'll get started on the work soon. If you'd like to work on this, you can take this issue.
getChan commentedon Sep 17, 2025
waiting arrow-avro 56.0.0 release.
for use brand-new reader APIs. like ReaderBuilder
alamb commentedon Sep 17, 2025
FYI @getChan I don't think we are going to release the arrow-avro crate for a while
You can start testing immediately using crates.io patch in Cargo.toml
An example of how to do so is here
arrow/parquet54.2.0 #14628Something like
[-]Consider using upstream arrow-avro reader[/-][+]Use upstream arrow-avro reader[/+][-]Use upstream arrow-avro reader[/-][+]Use new arrow-avro reader for performance and type safety[/+][-]Use new arrow-avro reader for performance and type safety[/-][+]Use new arrow-avro reader for performance and improved support[/+][-]Use new arrow-avro reader for performance and improved support[/-][+]Use `arrow-avro` for performance and improved type support[/+]alamb commentedon Nov 20, 2025
@getChan and @jecsand838 , @EmilyMatt has been looking into the current avro reader, see
Maybe you can work together to clean up avro support in DataFusion by consolidating on the new crate
EmilyMatt commentedon Nov 23, 2025
Sure, would love to help, what is actually missing right now?
alamb commentedon Nov 25, 2025
I think @getChan and @jecsand838 are working out some schema issues:
Maybe you can help push that along?
EmilyMatt commentedon Nov 26, 2025
@alamb I've also encountered some schema issues in my attempt to create a reader using arrow-avro, documented here:
apache/arrow-rs#8928
Regardless, I will open a PR soon in arrow-rs for an AsyncReader that uses object_store and should be very easily integrated into DataFusion in lieu of the current reader.
EmilyMatt commentedon Nov 26, 2025
Note the following arrow-rs PR:
apache/arrow-rs#8930
jecsand838 commentedon Nov 26, 2025
We're making progress. The biggest issue so far is getting projection working as far I can tell atleast. @getChan
@EmilyMatt Thank you for creating these issues. I do like the idea of an OCF AsyncReader, this makes sense to me.
Migrate Avro reader to arrow-avro and remove internal conversion code (…