Skip to content

[EPIC] [Parquet] Implement Variant type support in Parquet #6736

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Parquet recently adopted the Variant type from Spark: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md

Details on

Describe the solution you'd like
I would like to implement variant support in parquet-rs

Additional context
I am not sure if any other parquet implementations have implemented this yet / if there are example parquet files. I will attempt to find out

Shredding Support

Full variant_get Support

Parquet File Integration

Arrow --> Variant conversions

Variant infrastructure

Related PRs

** Related Community Resources**

Activity

  1. CurtHagenlocher commented on Nov 15, 2024

    @CurtHagenlocher
    Contributor

    There's an implementation in Spark (try here for starters) but when I last looked ~two months ago there was no binary test data; only some round trips via JSON.

  2. tustvold commented on Dec 4, 2024

    @tustvold
    Contributor

    I do wonder if a precursor to supporting this would be some way to translate / represent the variant data in arrow, whilst there are non-arrow APIs, they'd likely struggle to accommodate this addition, and aren't how the vast majority of people consume parquet data using this crate.

  3. findepi commented on Dec 4, 2024

    @findepi
    Member

    From arrow perspective, would that be a new DataType, or rather a convention of using DataType::Struct with two Binary fields?

    A fully performance variant implementation should be able to leverage file-level column disaggregation (shredding), but I do think this could come as a follow-up to a "normal" Variant type implementation.

  4. tustvold commented on Dec 4, 2024

    @tustvold
    Contributor

    From arrow perspective, would that be a new DataType, or rather a convention of using DataType::Struct with two Binary fields?

    I don't know, I've not really been following the variant proposal close enough to weigh in here. However, my understanding is that shredding is one of the major motivators for this getting added to parquet, as without it you might as well just embed any record format, e.g. Avro. I therefore suspect most use-cases will be at least partially shredded, and the reader will need to handle this case. This is especially true given the variant_value is NULL when the data is shredded, as opposed to say duplicating the content (which would have its own issues TBC), and so we can't just ignore the shredded data.

    Unfortunately I can't see an obvious way to be able to represent this sort of semi-structured data within the arrow format without introducing a new DataType that is able to accommodate arrays having the same type, but different child layouts...

    TLDR I suspect actioning this will require arrow defining a way to represent semi-structured data...

  5. findepi commented on Dec 4, 2024

    @findepi
    Member

    There needs to be a way to represent a series of variant values having "no type in common" (variant integer, variant boolean, variant varchar, etc all mixed up). For that some blob-like representation with internal structure seems natural.
    Then there should be a way to carry-on the shredded columns without having to put them back into that blob, so yes, one type, different child layouts.
    It feels to me that the runtime representation will end up being similar to what is defined in Parquet (https://github.com/apache/parquet-format/blob/master/VariantShredding.md)... so maybe it should be the same representation to provide for an efficient read path.

  6. findepi commented on Dec 4, 2024

    @findepi
    Member

    When considering what to do in Arrow, we should also keep an eye on the ongoing effort in Iceberg apache/iceberg#10392 (comment)
    This could inform some design decisions.
    cc @Xuanwo

  7. alamb commented on Jan 24, 2025

    @alamb
    ContributorAuthor

    There is now a related proposal being added to paruqet (shredding the variant type:

    FWIW I would love to have some rust representation (aka someone who wanted to implement the variant type in the rust parquet decoder). If you are interested there are relevant conversations going on on the parquet mailing list

  8. alamb commented on Jan 24, 2025

    @alamb
    ContributorAuthor

    to be clear, we may not be able to merge / really use variants in rust until it gets into Arrow, but we can work out how it would work in parquet (maybe with the non arrow interfaces) first

  9. CurtHagenlocher commented on Jan 25, 2025

    @CurtHagenlocher
    Contributor

    There's a pure Python implementation in the Spark repo. It's almost standalone, having a dependency only on PySparkValueError.

  10. alamb commented on Mar 6, 2025

    @alamb
    ContributorAuthor

    I spent some time listening and thinking about this on the parquet call yesterday: https://lists.apache.org/thread/cnn6264g56jktrwmplz89x8cgkcvr4ql

    Note there is a thread on the arrow mailing list about adding variant support in arrow-rs:

    (and it looks like @wjones127 made some sort of demo using an extension type in datafusion):

    Unfortunately I can't see an obvious way to be able to represent this sort of semi-structured data within the arrow format

    What I suggest is that the parquet reader reads variant columns as Binary / LargeBinary with an arrow extension type annotation, which would let downstream projects interpret / read the extension type correctly

    I think one challenge will be "how to tell the parquet writer to write / annotate the columns as variant"

    Before we can do anything useful with the variant type, we'll need a library to parse / interpret a variant value (aka the equivalent of a JSON parser / set of objects)

  11. alamb commented on Mar 6, 2025

    @alamb
    ContributorAuthor

    So the first step as I see it is that someone has to code up / find a Rust implementation for working with variant values. This could be a port or inspired by the ones @CurtHagenlocher 's points at here:

    There's a pure Python implementation in the Spark repo.

    There's an implementation in Spark (try here for starters)

    Once we have such a library, we can then figure out if/how it should be used in the parquet reader/writer directly

  12. wjones127 commented on Mar 6, 2025

    @wjones127
    Member

    I haven't had time to work on this recently, but for a rust implementation for working with variant values, anyone should feel free to work off what I had started in the https://github.com/datafusion-contrib/datafusion-functions-variant/tree/main/open-variant repo. There is a core open-variant crate there that's just meant to be reading and writing variant values.

  13. 52 remaining items

  14. alamb commented on Aug 2, 2025

    @alamb
    ContributorAuthor

    In case anyone is interested, I have a proposal for how to implement reading of shredded variants here:

    Any feedback would be most welcome

  15. alamb commented on Aug 6, 2025

    @alamb
    ContributorAuthor

    I filed quite a few items under the Arrow --> Variant conversions heading above that it would be great to get help with

  16. alamb commented on Aug 13, 2025

    @alamb
    ContributorAuthor

    I filed a ticket to discuss integrating variant support for real into the parquet crate:

  17. alamb commented on Aug 14, 2025

    @alamb
    ContributorAuthor

    Here is a proposal to add a new feature flag to the parquet crate to support Variant:

  18. alamb commented on Sep 21, 2025

    @alamb
    ContributorAuthor

    Since this ticket is getting out of control with the number of linked issues (it is hard to understand what the current status is and what needs help), what I am thinking we'll do is that once I merge the support for reading/writing the parquet logical type I'll close this ticket as "done" (as we'll have completed the initial implementation) and file a follow on EPIC with remaining work

    The major additional areas I think we are missing are support for shredded Lists/Arrays and nested lists/structs, and some additional type support

  19. alamb commented on Sep 26, 2025

    @alamb
    ContributorAuthor

    Now that we merged support for reading/writing Variant LogicalTypes to parquet (via the following PR) let'c claim that we have completed the "initial support" for Variant

    There is still more work to do, so I filed an additional follow on EPIC to try and organize this work

    A huge shoutout to everyone who helped make this happen (and I am sorry if I missed you)

    @PinkCrow007 @mkarbo @scovich @superserious-dev @friendlymatthew @micoo227 @Weijun-H @harshmotw-db @odysa @carpecodeum @viirya @klion26 @adriangb @kosiew @liamzwbao @codephage2020 @ding-young @mbrobbel @petern48 @sdf-jkl @abacef and @mprammer

  20. HuaHuaY commented on Jun 29, 2026

    @HuaHuaY

    Can we remove WIP state in parquet-variant/src/lib.rs:27 now?

  21. kevinjqliu commented on Jun 29, 2026

    @kevinjqliu

    great catch @HuaHuaY

    //! ## 🚧 Work In Progress
    //!
    //! This crate is under active development and is not yet ready for production use.
    //! If you are interested in helping, you can find more information on the GitHub [Variant issue]
    //!
    //! [Variant issue]: https://github.com/apache/arrow-rs/issues/6736

    the epic is closed (#6736), so i'd say yes!

    would you like to open a PR to remove those lines?

  22. HuaHuaY commented on Jun 29, 2026

    @HuaHuaY

    would you like to open a PR to remove those lines?

    Hi @kevinjqliu, I have opened a PR to remove WIP comments in three files #10238. Please take a look.

  23. sdf-jkl commented on Jun 29, 2026

    @sdf-jkl
    Member

    @kevinjqliu @HuaHuaY we're still working on finalizing Variant support in #8480.

    The majority of work is done, but there are still some breaking issues that need handling before finalizing Variant:

    The rest I'd say are additive and can be done after we make Variant production ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature/additionparquetChanges to the parquet crate

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions