Skip to content

epic: Initial implementation of this crate #2

Closed

Description

@friendlymatthew

A checklist of items that I plan to tackle. Please feel free to edit and open issues related to a certain topic

udfs to support

Databricks supports the following variant-related udfs: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-functions-builtin#variant-functions. I think it would be very cool to achieve 1:1 parity with their functionality

Apache spark's variant functions: https://spark.apache.org/docs/latest/api/sql/search.html?q=variant

misc

questions

It would also be good to compile a list of questions/ideas/limitations that arise from interfacing with arrow's parquet-variant libraries:

Activity

  1. added a commit that references this issue on Oct 10, 2025
  2. alamb commented on Oct 14, 2025

    @alamb
    Collaborator

    Databricks supports the following variant-related udfs: https://docs.databricks.com/gcp/en/sql/language-manual/sql-ref-functions-builtin#variant-functions. I think it would be very cool to achieve 1:1 parity with their functionality

    I agree this would be a great target to shoot for (maybe even put a note in the readme that the goal of this crate is to provide databricks / spark compatible variant functions)

    The databricks list seems pretty similar to what is in spark: https://spark.apache.org/docs/latest/api/sql/index.html

  3. alamb commented on Oct 14, 2025

    @alamb
    Collaborator

    misc

    I recommend putting the examples in the crate readme / library home page as rustdocs to make sure they keep running and make them easier to discover

    • Pick a license

    I recommend Apache 2.0 , given it is well known and is straightforward to apply / understand, and consistent with most other DataFusion libraries.

    questions

    It would also be good to compile a list of questions/ideas/limitations that arise from interfacing with arrow's parquet-variant libraries:

    This is a great list! I think you are probably the first person to really try and integrate Variant so this feedback is super valuable.

    These would be great tickets to file in arrow-rs and attach to the epic. Would you like me to do so? Or would you rather do so yourself?

    • Why does VariantArray not implement IntoIter?
    • variant_to_json should accept an optional format configurations. Currently the library dictates how to map specialized Variant types to JSON (e.g. timestamps are always formatted as a string)
    • There is a lot of ceremony to go from a Variant to a VariantArray. Maybe VariantArray should impl From<IntoIterator<Item = Variant>>?
    • When checking if 2 VariantArrays are equal, it is a bit odd that it will panic when calling .value(i) when the ith position has empty metadata and value?
    • Why doesn't VariantArray impl PartialEq?

    I don't think there are great reasons for these rough edges, and I suspect we can make them all much easier

  4. alamb commented on Oct 14, 2025

    @alamb
    Collaborator

    There is also a list of functions in the original iceberg proposal here that might be good to reference:

  5. friendlymatthew commented on Nov 13, 2025

    @friendlymatthew
    MemberAuthor

    cc @XiangpengHao if you have any thoughts/ideas

  6. XiangpengHao commented on Nov 13, 2025

    @XiangpengHao
    Collaborator

    Thanks @friendlymatthew for looping me in! I'm working on this through the lens of json_bench: https://github.com/ClickHouse/JSONBench

    Basically convert all json data to variant, and run the benchmark queries through variant_get. I bet we can improve performance in several places.

  7. alamb commented on Nov 14, 2025

    @alamb
    Collaborator

    Basically convert all json data to variant, and run the benchmark queries through variant_get. I bet we can improve performance in several places.

    I will be disappointed if you can't find several 2-3x improvements without looking hard.

    As some added incentive, you want to turn your student army loose on this, I would be willing to offer "I'll take you to dinner in Madison" type prizes for any student who produces a merged PR that gives a 2x or better performance improvement in some important variant operation (like json --> variant)

  8. friendlymatthew commented on Nov 14, 2025

    @friendlymatthew
    MemberAuthor

    As some added incentive, you want to turn your student army loose on this, I would be willing to offer "I'll take you to dinner in Madison" type prizes for any student who produces a merged PR that gives a 2x or better performance improvement in some important variant operation (like json --> variant)

    I wonder if this incentive is only for UW–Madison students and not for starving engineers in NYC

  9. alamb commented on Nov 17, 2025

    @alamb
    Collaborator

    As some added incentive, you want to turn your student army loose on this, I would be willing to offer "I'll take you to dinner in Madison" type prizes for any student who produces a merged PR that gives a 2x or better performance improvement in some important variant operation (like json --> variant)

    I wonder if this incentive is only for UW–Madison students and not for starving engineers in NYC

    I will extend the offer to any such engineers in NYC, on a case by case basis (I do not have unlimited budget 😆 )

  10. alamb commented on Dec 2, 2025

    @alamb
    Collaborator

    I found an interesting blog about variant here: https://www.e6data.com/blog/faster-json-sql-variant-data-type

    Specifically, I think the description of variant_get and the : and :: operators are valuable and worth considering for this crate

  11. sunng87 commented on Dec 3, 2025

    @sunng87
    Member

    I have been looking into these part too. Postgres has some operators defined for json data access https://www.postgresql.org/docs/18/functions-json.html#FUNCTIONS-JSON and these operators are already defined but not implemented in datafusion and sqlparser.

    In postgres, the json path query is implemented as a UDF jsonb_path_query. There is no operator for it. @? as a related one but only runs testing. : can be a good shortcut. :: type casting is important here because we need to pass type coercion for those operators that can generate multiple data types of return value.

    By the way, just want to confirm is it possible to extend operators in an external library? I remember I saw custom planner implementation that rewrites operator into udf call.

  12. alamb commented on Dec 3, 2025

    @alamb
    Collaborator

    By the way, just want to confirm is it possible to extend operators in an external library? I remember I saw custom planner implementation that rewrites operator into udf call.

    I am not sure -- if it isn't possible we should add it.

  13. alamb commented on Dec 3, 2025

    @alamb
    Collaborator
  14. Samyak2 commented on Dec 3, 2025

    @Samyak2
    Contributor

    Thanks for the mention @alamb! I would love to help with the effort here.

    Regarding : operator: databricks supports the operator for both JSON string and variant (ref). This makes it a lot easier to switch a column from string to variant - the changes needed in queries would be minimal. Snowflake also uses a similar syntax in their variant implementation (ref - note that this is not the same as the "open variant" type we're talking about here). Not that we need to use a proprietary engine as a reference, but they operate in the same analytics space and users would expect some familiarity in syntax.

    In practice too, I have noticed that users prefer colon over variant_get (I don't have a source for this). Personally, column:field::int feels much more natural over variant_get(column, '$.field', 'int').

  15. alamb commented on Dec 4, 2025

    @alamb
    Collaborator

    By the way, just want to confirm is it possible to extend operators in an external library? I remember I saw custom planner implementation that rewrites operator into udf call.

    I am not sure -- if it isn't possible we should add it.

    BTW @sunng87 I found that yes we can plan custom operators using an ExprPlanner -- see here:

    https://github.com/apache/datafusion/blob/0f83c1d233499e80fd9b0baf89dd624099c1d1ba/datafusion/core/tests/user_defined/expr_planner.rs#L42-L41

  16. Samyak2 commented on Dec 4, 2025

    @Samyak2
    Contributor

    Can we create a separate issue for colon : operator? I want to take a stab at it

  17. Samyak2 commented on Dec 4, 2025

    @Samyak2
    Contributor

    By the way, just want to confirm is it possible to extend operators in an external library? I remember I saw custom planner implementation that rewrites operator into udf call.

    I am not sure -- if it isn't possible we should add it.

    BTW @sunng87 I found that yes we can plan custom operators using an ExprPlanner -- see here:

    https://github.com/apache/datafusion/blob/0f83c1d233499e80fd9b0baf89dd624099c1d1ba/datafusion/core/tests/user_defined/expr_planner.rs#L42-L41

    One potential blocker I'm running into: I don't see a colon operator in https://docs.rs/sqlparser/latest/sqlparser/ast/enum.BinaryOperator.html

    BinaryOperator::Custom(String) could work, but I don't see any usage of that in datafusion currently, so I'm guessing it needs a custom dialect implementation. The closest PR I could find is this - apache/datafusion-sqlparser-rs#533 (which was closed by the author for being an incorrect/incomplete implementation).

    Any ideas here? Should we start with adding colon operator to sqlparser's GenericDialect? or should this be done in a custom dialect?

  18. alamb commented on Dec 4, 2025

    @alamb
    Collaborator

    I concur -- the parser doesn't seem to support :

    (venv) andrewlamb@Andrews-MacBook-Pro-3:~/Software/arrow-rs$ datafusion-cli
    DataFusion CLI v51.0.0
    select * from foo:bar; 🤔 Invalid statement: SQL error: ParserError("Expected: end of statement, found: : at Line: 1, Column: 18")

    Any ideas here? Should we start with adding colon operator to sqlparser's GenericDialect? or should this be done in a custom dialect?

    I would recommend finding an existing dialect that supports it (hopefully one does) and then we can add it to that specific dialect as well as GenericDialect

  19. Samyak2 commented on Dec 4, 2025

    @Samyak2
    Contributor

    I would recommend finding an existing dialect that supports it (hopefully one does)

    SnowflakeDialect seems to support it:
    (in datafusion-sqlparser-rs)

    echo "select a:b;" | cargo run --example cli -- - --snowflake

    gives this:

    ...
                        projection: [
                            UnnamedExpr(
                                JsonAccess {
                                    value: Identifier(
                                        Ident {
                                            value: "a",
                                            quote_style: None,
                                            span: Span(Location(1,8)..Location(1,9)),
                                        },
                                    ),
                                    path: JsonPath {
                                        path: [
                                            Dot {
                                                key: "b",
                                                quoted: false,
                                            },
                                        ],
                                    },
                                },
                            ),
                        ]
    ...
    

    Update: GenericDialect also supports it - https://github.com/apache/datafusion-sqlparser-rs/blob/326f1118081807933160cb530e69dfdbf7a84b22/src/parser/mod.rs#L3759 . But echo "select a:b;" | cargo run --example cli -- - doesn't work. Seems to be a precedence thing, because adding this fn to impl Dialect for GenericDialect works (copied from SnowflakeDialect):

        fn get_next_precedence(
            &self,
            parser: &crate::parser::Parser,
        ) -> Option<Result<u8, crate::parser::ParserError>> {
            let token = parser.peek_token();
            match token.token {
                Token::Colon => Some(Ok(self.prec_value(Precedence::DoubleColon))),
                _ => None,
            }
        }

    I'll try to get this into a proper PR.

  20. Samyak2 commented on Dec 4, 2025

    @Samyak2
    Contributor

    I have a draft PR here to support : in GenericDialect - apache/datafusion-sqlparser-rs#2124.
    A couple of tests are failing, but I want to know if my approach is correct.

  21. Samyak2 commented on Jan 1, 2026

    @Samyak2
    Contributor

    I have created a separate issue for : operator: #26

    BTW, the PR in parser is ready for review - apache/datafusion-sqlparser-rs#2124. Not sure who needs to be tagged there, so I'm putting it out here.

  22. sdf-jkl commented on Sep 24, 2026

    @sdf-jkl
    Collaborator

    superseded by #69

  23. unpinned this issue on Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions