You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Avro is a widely used binary, row-oriented data encoding. It is very similar to protobuf, and has seen very wide adoption in the data ecosystem, especially for streaming workloads.
Describe the solution you'd like
A new arrow_avro crate will provide vectorised support for reading and writing avro data. The APIs should be designed in such a way as to work for the various different container formats for avro encoded data, including single object encoding, object container files and message even if first-class support is not provided for all these framing mechanisms.
Describe alternatives you've considered
Additional context
DataFusion has some avro support, however, it is based on the row-based apache_avro crate and is therefore likely extremely sub-optimal.
Reacted by Samrose, Kousuke Saruta, Ben Chambers, Devin D'Angelo, Xuanwo, Jorge, Kalan Snyder, coded4alex, Santiago Fraire Willemoes, Yaroslav Tkachenko and 4 more
I think this may have gotten bumped by other priorities and I think @tustvold plans to wrap up whatever his current state is while he works on other things
Hello,
I have recently rewritten a Rust avro crate from ~scratch due to performance and ergonomics issues with the apache-avro crate. (It achieves x10 perf compared to the apache-avro crate).
This crate is soon hitting 1.0.
I suspect part of that work could be reused to solve this issue, and I could probably help with this.
Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code?
Thanks 🙂
Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code?
I think this means "reading avro records into Array directly
There are more sophisticated ways of implementing this feature, for example the tape based methods of the JSON and CSV readers in this crate
What I would personally recommend doing is:
Make a PR with a relatively basic (can be missing features) Avro Reader / writer and use that to work out the interface that is desired (@tustvold may already have this sitting on a branch somewhere)
Implement a basic reader/writer, perhaps using apache-avro or perhaps another implementation, including broad test coverage
Work on optimizing the implementation (using the existing coverage)
(@tustvold may already have this sitting on a branch somewhere)
I have most of the bits and pieces and hope to push something up in the coming week or two. The major missing thing at the moment is tests.
Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code
it appears to rely on knowing the data schema at compile time
It doesn't 😊
It goes through serde, but that also works with unknown structure: for example you can convert any avro with dynamic schema to json (and vice versa) just by plugging them via serde_transcode.
It seems that the main idea would be to have a custom implementation of DeserializeSeed for arrow records 🤔
(Most likely that would also enable plugging any serde format pretty easily afterwards as well since you could likely plug any deserializer (except maybe you'd want to read the schema in advance, but maybe it's not even necessary if you determine the structure as you receive the first records).)
For anyone following along, thanks to the near mythic efforts of @jecsand838 and @nathaniel-d-ef we will be releasing arrow-avro publically as part of the 57 release:
Activity
alamb commentedon Oct 2, 2023
I think having a native Avro --> Arrow code will be good as well as continue to encourage additional use of Arrow in the ecosystem.
bjchambers commentedon Oct 2, 2023
This would be awesome!
Samrose-Ahmed commentedon Oct 2, 2023
Awesome happy to review any pr and help with tests.
alamb commentedon Oct 30, 2023
I think this may have gotten bumped by other priorities and I think @tustvold plans to wrap up whatever his current state is while he works on other things
Map AvroSchema to Arrow (apache#4886)
tustvold commentedon Oct 30, 2023
I intend to keep working away at this in the background, but any help on reviews would be most appreciated
Jefffrey commentedon Nov 13, 2023
FYI a duplicate issue also exists: #727
alamb commentedon Nov 18, 2023
Today I learned that there is a version of the avro reader/writer in arrow2:
https://github.com/jorgecarleitao/arrow2/tree/3ddc6a10c6fbc2d0f85a9f66eeb46112abd07029/src/io/avro
Map AvroSchema to Arrow (#4886) (#5009)
Ten0 commentedon Jan 15, 2024
Hello,
I have recently rewritten a Rust avro crate from ~scratch due to performance and ergonomics issues with the apache-avro crate. (It achieves x10 perf compared to the apache-avro crate).
This crate is soon hitting 1.0.
I suspect part of that work could be reused to solve this issue, and I could probably help with this.
Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code?
Thanks 🙂
alamb commentedon Jan 16, 2024
I think this means "reading avro records into
ArraydirectlyHere is the way it is implemented in datafusion:
https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/avro_to_arrow/reader.rs
There are more sophisticated ways of implementing this feature, for example the tape based methods of the JSON and CSV readers in this crate
What I would personally recommend doing is:
tustvold commentedon Jan 16, 2024
I have most of the bits and pieces and hope to push something up in the coming week or two. The major missing thing at the moment is tests.
I had a brief look at https://github.com/Ten0/serde_avro_fast and I am not sure it necessarily would be a good fit for arrow-rs as it appears to rely on knowing the data schema at compile time, but I could be completely off base here
Ten0 commentedon Jan 16, 2024
Heyy,
Thanks for the quick answers.
It doesn't 😊
It goes through serde, but that also works with unknown structure: for example you can convert any avro with dynamic schema to json (and vice versa) just by plugging them via
serde_transcode.It seems that the main idea would be to have a custom implementation of
DeserializeSeedfor arrow records 🤔(Most likely that would also enable plugging any serde format pretty easily afterwards as well since you could likely plug any deserializer (except maybe you'd want to read the schema in advance, but maybe it's not even necessary if you determine the structure as you receive the first records).)
68 remaining items
Arrow-Avro: Resolve named field discrepancies (#8546)
Add arrow-avro README.md file (#8556)
Arrow-avro Writer Dense Union support (#8550)
alippai commentedon Oct 10, 2025
https://arrow.apache.org/docs/dev/status.html#third-party-data-formats this can be updated when arrow-avro is released
Add support for 64-bit Schema Registry IDs (Id64) in arrow-avro (#8575)
Add support for run-end encoded (REE) arrays in arrow-avro (#8584)
Add `ArrowError::AvroError`, remaining types and roundtrip tests to `…
Update publishing instructions to publish arrow-avro as part of 57
Update publishing instructions to publish arrow-avro as part of 57 (#…
alamb commentedon Oct 17, 2025
For anyone following along, thanks to the near mythic efforts of @jecsand838 and @nathaniel-d-ef we will be releasing arrow-avro publically as part of the 57 release:
57.0.0(October 2025) #7835[Website]: Blog post about arrow-avro Rust crate (#712)
GH-47930: Update Rust support status for Avro format in documentation (…
apacheGH-47930: Update Rust support status for Avro format in documen…