feat: add a custom ExprPlanner to support colon operator by Samyak2 · Pull Request #57 · datafusion-contrib/datafusion-variant

Samyak2 · 2026-03-30T18:53:58Z

Closes #26

Add a custom ExprPlanner that handles : as an operator and re-writes it to variant_get.

Closes datafusion-contrib#26 - Add a custom `ExprPlanner` that handles `:` as an operator and re-writes it to `variant_get`.

Samyak2 · 2026-03-30T18:56:42Z

TODO:

This handles the 2-arg version. The 3-arg version (with cast) is yet to be converted
Tests

friendlymatthew

Thanks @Samyak2

Can we add some tests as well please

friendlymatthew · 2026-03-30T19:53:58Z

+#[derive(Debug)]
+pub struct VariantExprPlanner;


Can we add some comments here that explain what this does

Done. Please take a look

Samyak2 · 2026-03-31T08:17:06Z

There seems to a bug in the sql parser. These are failing:

select variant_pretty(json_to_variant(json_str):[1]) from json_data;

and

select variant_pretty(json_to_variant(json_str):['name']) from json_data;

fails with:

SQL error: ParserError("Expected: variant object key name, found: [ at Line: 1, Column: 49")

Samyak2 · 2026-03-31T08:46:59Z

Another problem. Cast does not get parsed as a BinaryOperator. It's a separate variant of Expr: https://docs.rs/sqlparser/0.61.0/sqlparser/ast/enum.Expr.html#variant.Cast

ExprPlanner currently does not support custom parsing for cast.

So we won't be able to support 3-arg version like variant_col:name::string, etc.

sdf-jkl · 2026-04-03T16:35:35Z

There seems to a bug in the sql parser. These are failing:

select variant_pretty(json_to_variant(json_str):[1]) from json_data;

and

select variant_pretty(json_to_variant(json_str):['name']) from json_data;

fails with:

SQL error: ParserError("Expected: variant object key name, found: [ at Line: 1, Column: 49")

Is this a bug though? It seems like a Databricks dialect specifics.

Another problem. Cast does not get parsed as a BinaryOperator. It's a separate variant of Expr: https://docs.rs/sqlparser/0.61.0/sqlparser/ast/enum.Expr.html#variant.Cast

ExprPlanner currently does not support custom parsing for cast.

So we won't be able to support 3-arg version like variant_col:name::string, etc.

Another issue to work on in sqlparser-rs?

Samyak2 · 2026-04-10T15:24:54Z

Is this a bug though? It seems like a Databricks dialect specifics.

I don't see any reason not to support variant_col:['name'] when variant_col:something['name'] is supported.

Another issue to work on in sqlparser-rs?

This would be something to add in datafusion's ExprPlanner. We need something like ExprPlanner::plan_cast.

Both of these are not blocking issues. I think we can merge without them. What do you think?

sdf-jkl · 2026-04-13T16:04:23Z

Sounds good to me. @friendlymatthew?

alamb · 2026-09-11T18:38:23Z

@sdf-jkl also posted to the mailing list here: https://lists.apache.org/thread/bg71srlcd84y4gv42cqzdo3j6mj3t93f

sdf-jkl

Thanks @Samyak2!

There are still some cases that might not work as expected in parity with Spark:

{
  "name": "Alice",
  "field-name": "hyphen",
  "field name": "space",
  "a]b": "bracket"
}

Expression	Spark	This crate + #57
v:`name`	"Alice"	SQL NULL
v:`field-name`	"hyphen"	SQL NULL
v:`field name`	"space"	SQL NULL
`v:['a]b']`	Expected "bracket" from parser logic; runtime verification pending	Path-parser error

But I'd defer this to later work for syntax hardening

alamb · 2026-09-24T16:26:02Z

amazing!

sdf-jkl · 2026-09-24T16:44:49Z

I added the syntax support SLT suite here #68

Samyak2 · 2026-09-24T18:38:57Z

Thank you for merging!

Sorry, I had not been keeping up with the progress around variant, but looks like this crate is finally picking up steam again 🚀

alamb · 2026-09-24T19:37:37Z

Thank you for merging!

Sorry, I had not been keeping up with the progress around variant, but looks like this crate is finally picking up steam again 🚀

Indeed, we have recruited @sdf-jkl (well, really he volunteered) and I have a good feeling about the future of this crate

feat: add a custom ExprPlanner to support colon operator

1203b50

Closes datafusion-contrib#26 - Add a custom `ExprPlanner` that handles `:` as an operator and re-writes it to `variant_get`.

Samyak2 mentioned this pull request Mar 30, 2026

wip: use ExprPlanner to convert colon op to variant_get #31

Closed

friendlymatthew reviewed Mar 30, 2026

View reviewed changes

samyak-e6 added 2 commits March 31, 2026 13:28

doc: add docstring to explain VariantExprPlanner

e715023

test: add sqllogictests for simple variant_get

b364520

test: more tests

233006b

Samyak2 marked this pull request as ready for review March 31, 2026 08:47

Samyak2 mentioned this pull request Jun 5, 2026

colon : operator - initial implementation #26

Closed

4 tasks done

alamb mentioned this pull request Sep 11, 2026

[DISCUSSION] Adopt datafusion-functions-json and datafusion-variant into core repo apache/datafusion#21301

Open

Merge main into colon-operator

62008a3

sdf-jkl approved these changes Sep 24, 2026

View reviewed changes

sdf-jkl merged commit 0b58831 into datafusion-contrib:main Sep 24, 2026
3 checks passed

sdf-jkl mentioned this pull request Sep 24, 2026

Epic: Apache Spark Variant compatibility #69

Open

18 of 27 tasks

		#[derive(Debug)]
		pub struct VariantExprPlanner;

Conversation

Samyak2 commented Mar 30, 2026

Uh oh!

Samyak2 commented Mar 30, 2026

Uh oh!

friendlymatthew left a comment

Choose a reason for hiding this comment

Uh oh!

friendlymatthew Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Samyak2 Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Samyak2 commented Mar 31, 2026

Uh oh!

Samyak2 commented Mar 31, 2026

Uh oh!

sdf-jkl commented Apr 3, 2026

Uh oh!

Samyak2 commented Apr 10, 2026

Uh oh!

sdf-jkl commented Apr 13, 2026

Uh oh!

alamb commented Sep 11, 2026

Uh oh!

sdf-jkl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alamb commented Sep 24, 2026

Uh oh!

sdf-jkl commented Sep 24, 2026

Uh oh!

Samyak2 commented Sep 24, 2026

Uh oh!

alamb commented Sep 24, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

sdf-jkl left a comment •

edited

Loading