Skip to content

[Variant] Support Shredded Objects in variant_get: typed path access (STEP 1) #8150

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Note this is likely one of the most complex parts of implementing Shredded Variants , so it is not a good first task

We are trying to support the general case of the variant_get function, which allows runtime dynamic access to Variants (either shredded or unshredded).

This ticket tracks
Support variant_get for any input (shredded or otherwise), any depth of object field path steps, and casting to one primitive data type, eg. Some(DataType::Int32). This should close the loop and is potentially a good candidate for first PR.

Implementing this functionality will likely require the basic representation for shredded Variant arrays along with path traversal in variant_get. However, it does NOT cover the following (which are / will be broken into separate tickets)

  • Support for retrieving as other data types (e.g. Some(DataType::Utf8))
  • Retrieving any arbitrary path and returning what is there (no type specified)
  • Retrieving an arbitrary path as a "Struct" (aka implementing shredding)
  • Retrieving any arbitrary path as a Variant (aka "unshredding")

Describe the solution you'd like
@scovich sketched out a high level design for Shredded Objects (see Representing Variant In Arrow Proposal: "Shredding an Object" and Variant Shredding::Objects) in this PR

So roughly that means supporting

// get the named field of variant object as a typed field 
variant_get(array, "$.field_name", DataType::Int)

Where $.field_name represents some arbitrary VariantPath such as a for field "a", or a.b for field "b" of field "a"

This should work for:

  1. Variants where the field_name is in a typed_value
  2. Variants where the field_name is not in the typed value

Describe alternatives you've considered

  1. Add a test that manually constructs a shredded variant array (follow the example in the arrow proposal)
  2. Add a test that calls variant_get appropriately
  3. Implement the code

I suggest getting this working for non-nested obejcts first, and then working on nesting / pathing as a second pR

Additional context

Reference

Activity

  1. changed the title [-][Variant] Support Shredded Objects in `variant_get`:[/-] [+][Variant] Support Shredded Objects in `variant_get`: typed path access[/+] on Aug 15, 2025
  2. changed the title [-][Variant] Support Shredded Objects in `variant_get`: typed path access[/-] [+][Variant] Support Shredded Objects in `variant_get`: typed path access (STEP 1)[/+] on Aug 15, 2025
  3. scovich commented on Aug 15, 2025

    @scovich
    Contributor

    Support variant_get for any input (shredded or otherwise), any depth of object field path steps, and casting to one data type, eg. Some(DataType::Int32).

    One primitive data type, right? Objects and Arrays being a separate effort?

  4. alamb commented on Aug 15, 2025

    @alamb
    ContributorAuthor

    Support variant_get for any input (shredded or otherwise), any depth of object field path steps, and casting to one data type, eg. Some(DataType::Int32).

    One primitive data type, right? Objects and Arrays being a separate effort?

    yes, updated

  5. carpecodeum commented on Aug 15, 2025

    @carpecodeum
    Contributor

    Hey, @alamb @scovich I would like to take this

  6. alamb commented on Aug 15, 2025

    @alamb
    ContributorAuthor

    Thanks @carpecodeum !

    I strongly suspect this will be one of the more complicated ones, but I think it would be a great exercise

    Perhaps you can start by writing some tests and then adapt some of the code in

    ?

  7. alamb commented on Aug 15, 2025

    @alamb
    ContributorAuthor

    Specifically, the tests will ensure we all agree what we are trying to accomplish / implement. We don't need to implement all possible combinations now, but getting the basics would be helpful

    Perhaps we can start with something simple like an input object like

    Row 1

    {
      "a" : {
        "x": 55
      },
      "b": 42
    }

    Row 2

    {
      "a" : {
        "x": "foo"
      },
      "b": 42
    }

    Where x is stored in a typed_value column

    However, given that row2 has "x" as a string, it will need to be stored as a variant I think

    So then variant_get(array, "a.x", Int32) should return this I believe:

    55
    NULL
    

    Maybe @scovich can checi me on that

  8. scovich commented on Aug 15, 2025

    @scovich
    Contributor

    That sounds about right. For completeness, we should a couple more rows:

    {
      "a": {
          "y": 10
      },
      "b": 42
    }

    and

    {
      "a": 20,
      "b": 42
    }

    ... which covers the other two ways shredding can shake out (path not present, and parent not a struct).

    Note:

    Whichever PR goes first has to implement the basic framework (both the pathing code that variant_get calls to navigate shredded columns, and the builder-side code that drills into a binary variant column). The pathfinding PR has all the pieces we need, I think, so it shouldn't be too hard?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions