Repository navigation
colon : operator - initial implementation #26
Description
Context
Discussion started here - #2 (comment) (see next few comments too).
The colon operator is a shorthand for variant_get. For example, variant_get(column, '$.field', 'int') can be written as column:field::int. The part after the colon is a JSON path.
Implementation
- The first major blocker was the fact that the parser did not understand
:as an operator.- It is supported in the
snowflakedialect ofdatafusion-sqlparser-rs, but datafusion uses thegenericdialect. - Some initial exploration was done here: epic: Initial implementation of this crate #2 (comment)
- I have a PR for this in the parser here: GenericDialect: support colon operator for JsonAccess apache/datafusion-sqlparser-rs#2124
- It is supported in the
- (Update: added on 1st March 2026) Make datafusion parse
JsonAccessfrom sqlparser-rs as a binary expr: feat: parseJsonAccessas a binary operator, addOperator::Colonapache/datafusion#20628 - Use
ExprPlannerto allow:as a custom operator from this crate. See: epic: Initial implementation of this crate #2 (comment)Need to wait for datafusion v53 release for the above changes to come in. I have a draft PR here: wip: use ExprPlanner to convert colon op to variant_get #31- PR open: feat: add a custom ExprPlanner to support colon operator #57
- The actual implementation of the operator - which would simply call the same function as
variant_get.
References
- Databricks documentation on
:- https://docs.databricks.com/aws/en/sql/language-manual/functions/colonsign (note: their colon works on JSON strings too, but we would be focusing on variant in this case) - A blog post I had authored on variant: https://www.e6data.com/blog/faster-json-sql-variant-data-type
Reacted by Matthew Kim
Activity
Samyak2 commentedon Jan 16, 2026
Update: apache/datafusion-sqlparser-rs#2124 is merged
sdf-jkl commentedon Feb 27, 2026
Hi @Samyak2, how should we proceed with the next step?
Samyak2 commentedon Feb 28, 2026
The next step would be to use
ExprPlannerto recognize the colon syntax. I'm working on that.The blocker here is that datafusion v52 still uses an older v0.60 of datafusion-sqlparser-rs (which doesn't include apache/datafusion-sqlparser-rs#2124). The update was done recently (in apache/datafusion#20177) and should be available in the next v53 release of datafusion.
sdf-jkl commentedon Feb 28, 2026
Thanks! Let's wait for the v53 release.
feat: parse JsonAccess as a custom binary operator
feat: parse JsonAccess as a custom binary operator
feat: parse JsonAccess as a custom binary operator
Samyak2 commentedon Mar 1, 2026
I tried using
ExprPlannerfrom datafusion main to convert colon op tovariant_get- #31But I ran into a problem. sqlparser-rs exposes colon operator as a special
JsonAccessmethod, not as a simple binary expr. I had to convert it in datafusion so that anExprPlannercan intercept it. Please take a look: apache/datafusion#20628feat: parse `JsonAccess` as a binary operator, add `Operator::Colon` (#…
[branch-53] feat: parse `JsonAccess` as a binary operator, add `Opera…
[branch-53] feat: parse `JsonAccess` as a binary operator, add `Opera…
feat: parse `JsonAccess` as a binary operator, add `Operator::Colon` (a…
feat: add a custom ExprPlanner to support colon operator
feat: add a custom ExprPlanner to support colon operator (datafusion-…