Repository navigation
Closed
Labels
enhancementNew feature/additionNew feature/additionparquetChanges to the parquet crateChanges to the parquet crate
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Parquet recently adopted the Variant type from Spark: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
Details on
Describe the solution you'd like
I would like to implement variant support in parquet-rs
Additional context
I am not sure if any other parquet implementations have implemented this yet / if there are example parquet files. I will attempt to find out
Shredding Support
- [Variant] Add low level support for shredding and unshredding #7715
- [Variant] [Shredding] Support typed_access for
Boolean#8329 - [Variant] [Shredding] Support typed_access for
Date32#8330 - [Variant] [Shredding] Support typed_access for
Timestamp(Microsecond, _)andTimestamp(Nanosecond, _)#8331 - [Variant] [Shredding] Support typed_access for
Utf8andBinaryView#8333 - [Variant] [Shredding] Support typed_access for
FixedSizeBinary#8335 - [Variant] API to construct Shredded Variant Arrays #7895
- [Variant] Implement
VariantArray::valuefor shredded variants #8091 - [Variant] Support Shredded Objects in
variant_get: typed path access (STEP 1) #8150 - [Variant] Implement
ShreddingState::AllNullvariant #8088 - [Variant] Support
variant_getkernel for shredded variants #7941 - [Variant] Support Shredded Objects in
variant_get#8083 - [Variant] Implement a
shred_variantfunction #8361
Full variant_get Support
- [Variant] Allow appending raw object/list bytes to variant builders #8141
- [Variant] Support creating Variants with pre-existing Metadata #8152
Parquet File Integration
- [Variant]
VariantArray::data_typereturnsStructType, causingArray::as_structto panic #8319 - Support mapping canonical extension types to Parquet LogicalTypes #7063
- [Variant] Integration tests for reading parquet w/ Variants #8084
- [Variant] Support reading/writing Parquet Variant LogicalType #8370
- [Variant] writing a VariantArray to parquet panics #8296
- [Variant] Add
variantfeature toparquetcrate #8132 - Add example Variant data and parquet files parquet-testing#75
Arrow --> Variant conversions
- [Variant] cast_to_variant will panic on certain
Date64or Timestamp Values values #8155 - [Variant] Rename
batch_json_string_to_variantandbatch_variant_to_json_stringjson_to_variant #8144 - [Variant] Support
StringViewandLargeStringin ´batch_json_string_to_variant` #8145 - [Variant]: Implement
DataType::List/LargeListsupport forcast_to_variantkernel #8060 - [Variant]: Implement
DataType::Dictionarysupport forcast_to_variantkernel #8062 - [Variant]: Implement
DataType::Mapsupport forcast_to_variantkernel #8063 - [Variant]: Implement
DataType::RunEndEncodedsupport forcast_to_variantkernel #8064 - [Variant] Implement
cast_to_variantkernel #8043 - [Variant]: Implement
DataType::Utf8/LargeUtf8/Utf8Viewsupport forcast_to_variantkernel #8049 - [Variant]: Implement
DataType::Binary/LargeBinary/BinaryViewsupport forcast_to_variantkernel #8050 - [Variant]: Implement
DataType::FixedSizeBinarysupport forcast_to_variantkernel #8051 - [Variant]: Implement
DataType::Booleansupport forcast_to_variantkernel #8052 - [Variant]: Implement
DataType::Nullsupport forcast_to_variantkernel #8053 - [Variant]: Implement
DataType::Date32 / DataType::Date64support forcast_to_variantkernel #8054 - [Variant]: Implement
DataType::Time32/Time64support forcast_to_variantkernel #8055 - [Variant]: Implement
DataType::Intervalsupport forcast_to_variantkernel #8056 - [Variant]: Implement
DataType::Float16support forcast_to_variantkernel #8057 - [Variant]: Implement
DataType::Timestamp(..)support forcast_to_variantkernel #8058 - [Variant]: Implement
DataType::Decimal32/Decimal64/Decimal128/Decimal256support forcast_to_variantkernel #8059 - [Variant]: Implement
DataType::Structsupport forcast_to_variantkernel #8061
Variant infrastructure
- [Variant] Add
Variant::as_f16#8228 -
validatedandis_fully_validatedflags doesn't need to be part of PartialEq #7952 - [Variant] Avoid extra allocation in
ObjectBuilder#7899 - [Variant] Avoid extra allocation in list builder #7977
- [Variant] Convert JSON to Variant with fewer copies #7964
- [Variant]
impl FromIteratorfprVariantPath#7955 - [Variant] remove VariantMetadata::dictionary_size #7947
- [Variant] Improve
VariantArrayperformance by storing the index of the metadata and value arrays #7920 - [Variant] Test and implement efficient building for "large" Arrays #7699
- [Variant] Add
ListBuilder::with_valuefor convenience #7951 - [Variant] Add
ObjectBuilder::with_fieldfor convenience #7949 - [Variant] Rust API to Read Variant Values #7423
- [Variant] Implement read support for remaining primitive types #7630
- Initial Builder API for Creating Variant Values #7653
- [Variant]: Rust API to Create Variant Values #7424
- [Variant] Improve API for iterating over values of a VariantList #7685
- [Variant] Add Variant::as_object and Variant::as_list #7755
- [Variant] Consider validating variants on creation (rather than read) #7684
- [Variant] Implement
VariantObject::fieldandVariantObject::fields#7665 - [Variant] More efficient determination of String vs ShortString #7700
- [Variant] Panic when appending Object or List to VariantBuilder #7701
- [Variant] Introduce structs for Variant::Decimal types #7660
- [Variant] Support Nested Data in
VariantBuilder#7696 - [Variant] Improved API for accessing Variant Objects and lists #7756
- [Variant] Add negative tests for reading invalid primitive variant values #7645
- Variant: Write Variant Values as JSON #7426
- [Variant] Add input validation in
VariantBuilder#7697 - [Variant] Minor: make fields in
VariantDecimal*private, add examples #7770 - [Variant] Add flag in
ObjectBuilderto control validation behavior on duplicate field write #7777 - [Variant] Avoid second copy of field name in MetadataBuilder #7814
- [Variant] Add testing for invalid variants (fuzz testing??) #7842
- [Variant] Remove explicit ObjectBuilder::finish() and ListBuilder::finish and move to
Dropimpl #7780 - [Variant] Field lookup with out of bounds index causes unwanted behavior #7784
- [Variant] Make it harder to forget to finish a pending parent i n ObjectBuilder #7798
- [Variant] Move JSON related functionality to different crate. #7800
- [Variant] make
serde_jsonan optional dependency ofparquet-variant#7775 - [Variant] Add tests for invalid variant values (aka verify invalid inputs) #7681
- Variant: Read/Parse JSON value as Variant #7425
- [Variant] If
ObjectBuilder::finalizeis not called, the resulting Variant object is malformed. #7863 - [Variant] Improve VariantBuilder when creating field name dictionaries / sorted dictionaries #7698
- [Variant] Tests for creating "large"
VariantLists #7820 - [Variant] Impl PartialEq for VariantObject #7943 #7948
- [Variant] Panic when appending nested objects to VariantBuilder #7907
- [Variant] Tests for creating "large"
VariantObjectss #7821 - [Variant][Compute] Add batch processing for Variant-JSON String conversion #7883
- [Variant] Offer
simdutf8as an optional dependency when validating metadata #7902 - [Variant] VariantMetadata, VariantList and VariantObject are too big for Copy #7831
- [Variant] Define basic convenience methods for variant pathing #7894
- [Variant] Converting variant to JSON string seems slow #7869
- [Variant]
test_json_to_variant_object_very_largetakes over 20s #7872 - [Variant] Support VariantBuilder to write to buffers owned by the caller #7805
Related PRs
** Related Community Resources**
- Parquet C/C++ variant implementation from @neilechao: GH-45937: [C++][Parquet] Variant logical type definition arrow#45375
- Rust Variant impelementation from @jonhoo @wjones127 and others: https://github.com/datafusion-contrib/datafusion-functions-variant
Reacted by mwish, Kostas Papageorgiou, Yan Tingwang, Malthe Karbo and Austin Liu
Reacted by Ed Seidl, Zhang Li, emkornfield, Soumil Nitin Shah, debugmiller and Malthe Karbo
Metadata
Metadata
Assignees
Labels
enhancementNew feature/additionNew feature/additionparquetChanges to the parquet crateChanges to the parquet crate
Activity
CurtHagenlocher commentedon Nov 15, 2024
There's an implementation in Spark (try here for starters) but when I last looked ~two months ago there was no binary test data; only some round trips via JSON.
tustvold commentedon Dec 4, 2024
I do wonder if a precursor to supporting this would be some way to translate / represent the variant data in arrow, whilst there are non-arrow APIs, they'd likely struggle to accommodate this addition, and aren't how the vast majority of people consume parquet data using this crate.
findepi commentedon Dec 4, 2024
From arrow perspective, would that be a new DataType, or rather a convention of using DataType::Struct with two Binary fields?
A fully performance variant implementation should be able to leverage file-level column disaggregation (shredding), but I do think this could come as a follow-up to a "normal" Variant type implementation.
tustvold commentedon Dec 4, 2024
I don't know, I've not really been following the variant proposal close enough to weigh in here. However, my understanding is that shredding is one of the major motivators for this getting added to parquet, as without it you might as well just embed any record format, e.g. Avro. I therefore suspect most use-cases will be at least partially shredded, and the reader will need to handle this case. This is especially true given the variant_value is NULL when the data is shredded, as opposed to say duplicating the content (which would have its own issues TBC), and so we can't just ignore the shredded data.
Unfortunately I can't see an obvious way to be able to represent this sort of semi-structured data within the arrow format without introducing a new DataType that is able to accommodate arrays having the same type, but different child layouts...
TLDR I suspect actioning this will require arrow defining a way to represent semi-structured data...
findepi commentedon Dec 4, 2024
There needs to be a way to represent a series of variant values having "no type in common" (variant integer, variant boolean, variant varchar, etc all mixed up). For that some blob-like representation with internal structure seems natural.
Then there should be a way to carry-on the shredded columns without having to put them back into that blob, so yes, one type, different child layouts.
It feels to me that the runtime representation will end up being similar to what is defined in Parquet (https://github.com/apache/parquet-format/blob/master/VariantShredding.md)... so maybe it should be the same representation to provide for an efficient read path.
findepi commentedon Dec 4, 2024
When considering what to do in Arrow, we should also keep an eye on the ongoing effort in Iceberg apache/iceberg#10392 (comment)
This could inform some design decisions.
cc @Xuanwo
alamb commentedon Jan 24, 2025
There is now a related proposal being added to paruqet (shredding the variant type:
FWIW I would love to have some rust representation (aka someone who wanted to implement the variant type in the rust parquet decoder). If you are interested there are relevant conversations going on on the parquet mailing list
alamb commentedon Jan 24, 2025
to be clear, we may not be able to merge / really use variants in rust until it gets into Arrow, but we can work out how it would work in parquet (maybe with the non arrow interfaces) first
CurtHagenlocher commentedon Jan 25, 2025
There's a pure Python implementation in the Spark repo. It's almost standalone, having a dependency only on
PySparkValueError.alamb commentedon Mar 6, 2025
I spent some time listening and thinking about this on the parquet call yesterday: https://lists.apache.org/thread/cnn6264g56jktrwmplz89x8cgkcvr4ql
Note there is a thread on the arrow mailing list about adding variant support in arrow-rs:
(and it looks like @wjones127 made some sort of demo using an extension type in datafusion):
What I suggest is that the parquet reader reads variant columns as
Binary/LargeBinarywith an arrow extension type annotation, which would let downstream projects interpret / read the extension type correctlyI think one challenge will be "how to tell the parquet writer to write / annotate the columns as variant"
Before we can do anything useful with the variant type, we'll need a library to parse / interpret a variant value (aka the equivalent of a JSON parser / set of objects)
alamb commentedon Mar 6, 2025
So the first step as I see it is that someone has to code up / find a Rust implementation for working with variant values. This could be a port or inspired by the ones @CurtHagenlocher 's points at here:
Once we have such a library, we can then figure out if/how it should be used in the parquet reader/writer directly
wjones127 commentedon Mar 6, 2025
I haven't had time to work on this recently, but for a rust implementation for working with variant values, anyone should feel free to work off what I had started in the https://github.com/datafusion-contrib/datafusion-functions-variant/tree/main/open-variant repo. There is a core
open-variantcrate there that's just meant to be reading and writing variant values.alamb commentedon Mar 6, 2025
The current variant spec appears to be
52 remaining items
alamb commentedon Aug 2, 2025
In case anyone is interested, I have a proposal for how to implement reading of shredded variants here:
variant_getand ShreddedVariantArray#8021Any feedback would be most welcome
alamb commentedon Aug 6, 2025
I filed quite a few items under the Arrow --> Variant conversions heading above that it would be great to get help with
DataType::Utf8/LargeUtf8/Utf8Viewsupport forcast_to_variantkernel #8049DataType::Binary/LargeBinary/BinaryViewsupport forcast_to_variantkernel #8050DataType::FixedSizeBinarysupport forcast_to_variantkernel #8051DataType::Booleansupport forcast_to_variantkernel #8052DataType::Nullsupport forcast_to_variantkernel #8053DataType::Date32 / DataType::Date64support forcast_to_variantkernel #8054DataType::Time32/Time64support forcast_to_variantkernel #8055DataType::Intervalsupport forcast_to_variantkernel #8056DataType::Float16support forcast_to_variantkernel #8057DataType::Timestamp(..)support forcast_to_variantkernel #8058DataType::Decimal32/Decimal64/Decimal128/Decimal256support forcast_to_variantkernel #8059DataType::List/LargeListsupport forcast_to_variantkernel #8060DataType::Structsupport forcast_to_variantkernel #8061DataType::Dictionarysupport forcast_to_variantkernel #8062DataType::Mapsupport forcast_to_variantkernel #8063DataType::RunEndEncodedsupport forcast_to_variantkernel #8064alamb commentedon Aug 7, 2025
I filed a bunch more tickets for those of you who might be Variant curious and want to help out
variant_get#8082variant_get#8083variant_get#8086variant_get#8087ShreddingState::AllNullvariant #8088[Variant] Add `variant_get` and Shredded `VariantArray` (#8021)
alamb commentedon Aug 13, 2025
I filed a ticket to discuss integrating variant support for real into the
parquetcrate:variantfeature toparquetcrate #8132alamb commentedon Aug 14, 2025
Here is a proposal to add a new feature flag to the parquet crate to support
Variant:variant_experimentalfeature toparquetcrate #8133alamb commentedon Sep 12, 2025
I updated the
shredded_variantintegration test and that PR is now ready for reviewvariant_integrationtest to use final approvedparquet-testingdata #8325I also files a bunch of tickets to add support for the additional data types in the tests:
Boolean#8329Date32#8330Timestamp(Microsecond, _)andTimestamp(Nanosecond, _)#8331Decimal128#8332Utf8andBinaryView#8333Time64(Microsecond)#8334FixedSizeBinary#8335Struct#8336List#8337alamb commentedon Sep 21, 2025
Since this ticket is getting out of control with the number of linked issues (it is hard to understand what the current status is and what needs help), what I am thinking we'll do is that once I merge the support for reading/writing the parquet logical type I'll close this ticket as "done" (as we'll have completed the initial implementation) and file a follow on EPIC with remaining work
The major additional areas I think we are missing are support for shredded Lists/Arrays and nested lists/structs, and some additional type support
alamb commentedon Sep 26, 2025
Now that we merged support for reading/writing Variant LogicalTypes to parquet (via the following PR) let'c claim that we have completed the "initial support" for Variant
There is still more work to do, so I filed an additional follow on EPIC to try and organize this work
A huge shoutout to everyone who helped make this happen (and I am sorry if I missed you)
@PinkCrow007 @mkarbo @scovich @superserious-dev @friendlymatthew @micoo227 @Weijun-H @harshmotw-db @odysa @carpecodeum @viirya @klion26 @adriangb @kosiew @liamzwbao @codephage2020 @ding-young @mbrobbel @petern48 @sdf-jkl @abacef and @mprammer
feat(parquet-variant): add Dictionary and REE variant_to_arrow support (
HuaHuaY commentedon Jun 29, 2026
Can we remove WIP state in
parquet-variant/src/lib.rs:27now?kevinjqliu commentedon Jun 29, 2026
great catch @HuaHuaY
arrow-rs/parquet-variant/src/lib.rs
Lines 27 to 32 in 7616e10
the epic is closed (#6736), so i'd say yes!
would you like to open a PR to remove those lines?
HuaHuaY commentedon Jun 29, 2026
Hi @kevinjqliu, I have opened a PR to remove WIP comments in three files #10238. Please take a look.
sdf-jkl commentedon Jun 29, 2026
@kevinjqliu @HuaHuaY we're still working on finalizing
Variantsupport in #8480.The majority of work is done, but there are still some
breakingissues that need handling before finalizingVariant:variant_getto cast kernel #8982 and it's sub issues.The rest I'd say are additive and can be done after we make
Variantproduction ready.