Skip to content

[Variant] Integration tests for reading parquet w/ Variants #8084

Closed

Description

@alamb

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

We are implementing Variant, and part of doing so is ensuring that the Rust implementation is compatible with other implementations

Describe the solution you'd like
I would like a harness that

  1. For each test listed in cases.json
  2. Reads the expected variant.bin file and creates a Variant from it
  3. Reads the data from the Parquet file (should be a StructArray)
  4. Construct a VariantArray
  5. Read the value from the VariantArray (using both VariantArray::value and the variant_get kernel)
  6. Verify that the value from the VariantArray matches the Variant read from the .bin file

Many of the cases will not to pass initially -- that is fine, we should just mark them as #[ignore] or something and we will enable them as we work through the Variant implementation

Describe alternatives you've considered

I recommend using this PR from @rdblue

Additional context

Activity

  1. alamb commented on Aug 13, 2025

    @alamb
    ContributorAuthor

    Update here @carpecodeum has added the initial basic tests in

    There are a few more follow on items:

    1. Test reading the values as VariantArray (this requires some finagling with feature flags / dependencies that I am working on)
    2. Add #test style tests for all the cases (@carpecodeum / @mprammer is this something you able to help with?)
    3. Potential cleanups from [VARIANT] Initial integration tests for variant reads #8104
  2. carpecodeum commented on Aug 14, 2025

    @carpecodeum
    Contributor

    Update here @carpecodeum has added the initial basic tests in

    There are a few more follow on items:

    1. Test reading the values as VariantArray (this requires some finagling with feature flags / dependencies that I am working on)
    2. Add #test style tests for all the cases (@carpecodeum / @mprammer is this something you able to help with?)
    3. Potential cleanups from [VARIANT] Initial integration tests for variant reads #8104

    Yep we can help with that, sorry missed this comment

  3. alamb commented on Aug 14, 2025

    @alamb
    ContributorAuthor

    Thanks @carpecodeum -- I filed this ticket to track adding a feature flag / making it easier to use Variant with

    I think it will be tricky (b/c I don't really know what to suggest / don't have a clear goal yet)

    Perhaps you can work on 2 and 3 while I try and sort out #8132 ?

  4. alamb commented on Aug 14, 2025

    @alamb
    ContributorAuthor

    Here is a PR to make the variant code available in the parquet crate

  5. alamb commented on Sep 9, 2025

    @alamb
    ContributorAuthor

    Now that we have merged #8133 and the variant files have been merged into parquet-testing

    We can update the tests to run with the new features

  6. alamb commented on Sep 11, 2025

    @alamb
    ContributorAuthor

    I am working on updating the test harness with the finalized testing files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions