Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Note this is likely one of the most complex parts of implementing Shredded Variants , so it is not a good first task
We are trying to support the general case of the variant_get function, which allows runtime dynamic access to Variants (either shredded or unshredded).
- We found in [Variant] Support Shredded Objects in
variant_get#8083 that supporting variant_get is quite complicated (see here), so we are proposing to brake it down into multiple piece.
This ticket tracks
Support variant_get for any input (shredded or otherwise), any depth of object field path steps, and casting to one primitive data type, eg. Some(DataType::Int32). This should close the loop and is potentially a good candidate for first PR.
Implementing this functionality will likely require the basic representation for shredded Variant arrays along with path traversal in variant_get. However, it does NOT cover the following (which are / will be broken into separate tickets)
- Support for retrieving as other data types (e.g.
Some(DataType::Utf8)) - Retrieving any arbitrary path and returning what is there (no type specified)
- Retrieving an arbitrary path as a "Struct" (aka implementing shredding)
- Retrieving any arbitrary path as a Variant (aka "unshredding")
Describe the solution you'd like
@scovich sketched out a high level design for Shredded Objects (see Representing Variant In Arrow Proposal: "Shredding an Object" and Variant Shredding::Objects) in this PR
So roughly that means supporting
// get the named field of variant object as a typed field
variant_get(array, "$.field_name", DataType::Int)Where $.field_name represents some arbitrary VariantPath such as a for field "a", or a.b for field "b" of field "a"
This should work for:
- Variants where the field_name is in a typed_value
- Variants where the field_name is not in the typed value
Describe alternatives you've considered
- Add a test that manually constructs a shredded variant array (follow the example in the arrow proposal)
- Add a test that calls variant_get appropriately
- Implement the code
I suggest getting this working for non-nested obejcts first, and then working on nesting / pathing as a second pR
Additional context
Reference
Activity
[-][Variant] Support Shredded Objects in `variant_get`:[/-][+][Variant] Support Shredded Objects in `variant_get`: typed path access[/+][-][Variant] Support Shredded Objects in `variant_get`: typed path access[/-][+][Variant] Support Shredded Objects in `variant_get`: typed path access (STEP 1)[/+]scovich commentedon Aug 15, 2025
One primitive data type, right? Objects and Arrays being a separate effort?
alamb commentedon Aug 15, 2025
yes, updated
carpecodeum commentedon Aug 15, 2025
Hey, @alamb @scovich I would like to take this
alamb commentedon Aug 15, 2025
Thanks @carpecodeum !
I strongly suspect this will be one of the more complicated ones, but I think it would be a great exercise
Perhaps you can start by writing some tests and then adapt some of the code in
?
alamb commentedon Aug 15, 2025
Specifically, the tests will ensure we all agree what we are trying to accomplish / implement. We don't need to implement all possible combinations now, but getting the basics would be helpful
Perhaps we can start with something simple like an input object like
Row 1
{ "a" : { "x": 55 }, "b": 42 }Row 2
{ "a" : { "x": "foo" }, "b": 42 }Where x is stored in a
typed_valuecolumnHowever, given that row2 has "x" as a string, it will need to be stored as a variant I think
So then
variant_get(array, "a.x", Int32)should return this I believe:Maybe @scovich can checi me on that
scovich commentedon Aug 15, 2025
That sounds about right. For completeness, we should a couple more rows:
{ "a": { "y": 10 }, "b": 42 }and
... which covers the other two ways shredding can shake out (path not present, and parent not a struct).
Note:
Whichever PR goes first has to implement the basic framework (both the pathing code that
variant_getcalls to navigate shredded columns, and the builder-side code that drills into a binary variant column). The pathfinding PR has all the pieces we need, I think, so it shouldn't be too hard?[Variant] Support Shredded Objects in variant_get (take 2) (apache#8280)