Skip to content

Add Avro Support #4886

Closed
Closed

Description

@tustvold

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Avro is a widely used binary, row-oriented data encoding. It is very similar to protobuf, and has seen very wide adoption in the data ecosystem, especially for streaming workloads.

Describe the solution you'd like

A new arrow_avro crate will provide vectorised support for reading and writing avro data. The APIs should be designed in such a way as to work for the various different container formats for avro encoded data, including single object encoding, object container files and message even if first-class support is not provided for all these framing mechanisms.

Describe alternatives you've considered

Additional context

DataFusion has some avro support, however, it is based on the row-based apache_avro crate and is therefore likely extremely sub-optimal.

FYI @Samrose-Ahmed @sarutak @devinjdangelo I intend to work on this, but any help with reviews / testing would be most welcome

Subtasks

Activity

  1. self-assigned this
    on Oct 2, 2023
  2. alamb commented on Oct 2, 2023

    @alamb
    Contributor

    I think having a native Avro --> Arrow code will be good as well as continue to encourage additional use of Arrow in the ecosystem.

  3. bjchambers commented on Oct 2, 2023

    @bjchambers
    Contributor

    This would be awesome!

  4. Samrose-Ahmed commented on Oct 2, 2023

    @Samrose-Ahmed
    Contributor

    Awesome happy to review any pr and help with tests.

  5. alamb commented on Oct 30, 2023

    @alamb
    Contributor

    I think this may have gotten bumped by other priorities and I think @tustvold plans to wrap up whatever his current state is while he works on other things

  6. tustvold commented on Oct 30, 2023

    @tustvold
    ContributorAuthor

    I intend to keep working away at this in the background, but any help on reviews would be most appreciated

  7. Jefffrey commented on Nov 13, 2023

    @Jefffrey
    Contributor

    FYI a duplicate issue also exists: #727

  8. alamb commented on Nov 18, 2023

    @alamb
    Contributor
  9. added a commit that references this issue on Jan 11, 2024
  10. Ten0 commented on Jan 15, 2024

    @Ten0

    Hello,
    I have recently rewritten a Rust avro crate from ~scratch due to performance and ergonomics issues with the apache-avro crate. (It achieves x10 perf compared to the apache-avro crate).
    This crate is soon hitting 1.0.
    I suspect part of that work could be reused to solve this issue, and I could probably help with this.
    Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code?
    Thanks 🙂

  11. alamb commented on Jan 16, 2024

    @alamb
    Contributor

    Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code?

    I think this means "reading avro records into Array directly

    Here is the way it is implemented in datafusion:
    https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/avro_to_arrow/reader.rs

    There are more sophisticated ways of implementing this feature, for example the tape based methods of the JSON and CSV readers in this crate

    What I would personally recommend doing is:

    1. Make a PR with a relatively basic (can be missing features) Avro Reader / writer and use that to work out the interface that is desired (@tustvold may already have this sitting on a branch somewhere)
    2. Implement a basic reader/writer, perhaps using apache-avro or perhaps another implementation, including broad test coverage
    3. Work on optimizing the implementation (using the existing coverage)
  12. tustvold commented on Jan 16, 2024

    @tustvold
    ContributorAuthor

    (@tustvold may already have this sitting on a branch somewhere)

    I have most of the bits and pieces and hope to push something up in the coming week or two. The major missing thing at the moment is tests.

    Would you mind explaining in a bit more detail what you mean by "vectorized support for reading and writing avro data", and point me to where that would plug in the code

    I had a brief look at https://github.com/Ten0/serde_avro_fast and I am not sure it necessarily would be a good fit for arrow-rs as it appears to rely on knowing the data schema at compile time, but I could be completely off base here

  13. Ten0 commented on Jan 16, 2024

    @Ten0

    Heyy,

    Thanks for the quick answers.

    it appears to rely on knowing the data schema at compile time

    It doesn't 😊
    It goes through serde, but that also works with unknown structure: for example you can convert any avro with dynamic schema to json (and vice versa) just by plugging them via serde_transcode.
    It seems that the main idea would be to have a custom implementation of DeserializeSeed for arrow records 🤔
    (Most likely that would also enable plugging any serde format pretty easily afterwards as well since you could likely plug any deserializer (except maybe you'd want to read the schema in advance, but maybe it's not even necessary if you determine the structure as you receive the first records).)

  14. 68 remaining items

  15. alippai commented on Oct 10, 2025

    @alippai
    Contributor
  16. alamb commented on Oct 17, 2025

    @alamb
    Contributor

    For anyone following along, thanks to the near mythic efforts of @jecsand838 and @nathaniel-d-ef we will be releasing arrow-avro publically as part of the 57 release:

  17. added 2 commits that reference this issue on Oct 23, 2025
  18. added a commit that references this issue on May 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions