You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
These operations can be performed using the existing functions, so I am currently unblocked for my immediate use case but having these functions built in would be convenient.
Additional context
We currently have the following regexp functions implemented. The source is in datafusion/functions/src/regex/mod.rs
I would personally recommend we start implementing them in datafusion as that will avoid the need to wait for coordinated releases of arrow-rs, and then port backupstream to arrow-rs as a follow on step.
@xinlifoobar I suspect there will be several other contribtuors interested in helping out and learning during the process. If we have a good example to follow the work would be straightforward to scale I think
One way to do this might be:
You implement one of these functions in a PR, along with good docs, tests, etc
Then we can file additional tickets for the other functions, linking to your first implementation
Based on prior conversations it sounds like the group is most interested in making sure we are supporting Postgresql so I think adding this is a very good idea. We can also have regexp_substr as an alias.
It is a good idea to support these functions, especially for regexp_matches, Thank you.
Adding regexp_matches would be trivial - it's essentially the current regexp_match but the 'g' (global) flag is always present (whereas with the regexp_match it's disallowed)
I see your point for the postgres - since it essentially does either substring from this character index to that OR regex matching, I think it would probably add more confusion than value to support in exactly the way they do. However I'm far from a SQL expert.
Activity
xinlifoobar commentedon Aug 16, 2024
I could work on this. The only concern is whether we implement the regexp function in this project or in
arrow-rs.Hey @alamb, would you prefer implement function in
arrow-rsdirectly or we put them indatafusionand port later?alamb commentedon Aug 17, 2024
Thanks @xinlifoobar
I would personally recommend we start implementing them in datafusion as that will avoid the need to wait for coordinated releases of arrow-rs, and then port backupstream to arrow-rs as a follow on step.
alamb commentedon Aug 17, 2024
@xinlifoobar I suspect there will be several other contribtuors interested in helping out and learning during the process. If we have a good example to follow the work would be straightforward to scale I think
One way to do this might be:
nrc commentedon Aug 20, 2024
Related to this,
substringin Postgres supports regex matching (see https://www.postgresql.org/docs/current/functions-matching.html), would it be reasonable for DataFusion to also support it?The currently accepted argument types are:
Postgres's regex substring takes a string, a pattern, and an escape character, so I don't think there would be a conflict.
Omega359 commentedon Aug 20, 2024
Spark's version of this is https://spark.apache.org/docs/latest/api/sql/#regexp_substr
timsaucer commentedon Aug 20, 2024
Based on prior conversations it sounds like the group is most interested in making sure we are supporting Postgresql so I think adding this is a very good idea. We can also have
regexp_substras an alias.hengfeiyang commentedon Oct 18, 2024
It is a good idea to support these functions, especially for
regexp_matches, Thank you.Omega359 commentedon Oct 18, 2024
Adding
regexp_matcheswould be trivial - it's essentially the currentregexp_matchbut the 'g' (global) flag is always present (whereas with theregexp_matchit's disallowed)Omega359 commentedon Oct 22, 2024
I took a look at the syntax for that pg function, and frankly it's awful. Personally I think that is a function best ignored.
timsaucer commentedon Oct 22, 2024
I see your point for the postgres - since it essentially does either substring from this character index to that OR regex matching, I think it would probably add more confusion than value to support in exactly the way they do. However I'm far from a SQL expert.
osipovartem commentedon Jan 23, 2025
I can implement regexp_substr()
can I just create a pull request or do I need to be added to some access list?
Omega359 commentedon Jan 23, 2025
A PR should be fine. You should have a contributor agreement in place as well.
alamb commentedon Jan 23, 2025
👋 @osipovartem -- thank you!
Note this is not a requirement and we merge many PRs from contributors without one on place
Best practice would be to have one in place, however.