Skip to content

Commit 463d804

Browse files
pchintarcloud-fan
authored andcommitted
[SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace
### What changes were proposed in this pull request? Spark SQL already supports the 4-argument form of `regexp_replace`: ```sql regexp_replace(str, regexp, replacement, position) ``` However, the corresponding Scala, PySpark, and Spark Connect APIs currently expose only the 3-argument variants. This PR exposes the existing 4-argument functionality through: * Scala API (`functions.regexp_replace`) * PySpark API (`functions.regexp_replace`) * Spark Connect API (`functions.regexp_replace`) and adds corresponding Scala, PySpark, and Connect test coverage. ### Why are the changes needed? The underlying SQL functionality already exists and is available through SQL expressions, but it is not accessible through the public Scala, PySpark, and Spark Connect APIs. This creates an inconsistency between SQL and programmatic interfaces. Exposing the optional `position` argument aligns the public APIs with existing SQL functionality. ### Does this PR introduce *any* user-facing change? Yes. Users can now specify the optional `position` argument through the Scala, PySpark, and Spark Connect APIs. Before: ```scala regexp_replace(col("s"), "(\\d+)", "--") ``` After: ```scala regexp_replace(col("s"), "(\\d+)", "--", 5) ``` Similarly, PySpark users can now call: ```python F.regexp_replace("s", r"(\d+)", "--", 5) ``` ### How was this patch tested? Added test coverage in: * `StringFunctionsSuite` * `FunctionsTests.test_regexp_replace` * `SparkConnectFunctionTests.test_string_functions_multi_args` Verified with: ```bash ./build/sbt "sql-api/compile" ./build/sbt "sql/Test/compile" ./build/sbt \ "sql/testOnly org.apache.spark.sql.StringFunctionsSuite -- -z \"regex_replace / regex_extract\"" python3.11 -m pytest \ python/pyspark/sql/tests/test_functions.py::FunctionsTests::test_regexp_replace -v python3.11 -m pytest \ python/pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_multi_args -v ``` ### Was this patch authored or co-authored using generative AI tooling? Developed with assistance from GPT-5. Closes #56240 from pchintar/regexp_replace. Authored-by: pchintar <[email protected]> Signed-off-by: Wenchen Fan <[email protected]> (cherry picked from commit d35df2e) Signed-off-by: Wenchen Fan <[email protected]>
1 parent db649d9 commit 463d804

6 files changed

Lines changed: 82 additions & 6 deletions

File tree

‎python/pyspark/sql/connect/functions/builtin.py‎

Lines changed: 19 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2815,9 +2815,26 @@ def regexp_extract_all(
28152815

28162816

28172817
def regexp_replace(
2818-
string: "ColumnOrName", pattern: Union[str, Column], replacement: Union[str, Column]
2818+
string: "ColumnOrName",
2819+
pattern: Union[str, Column],
2820+
replacement: Union[str, Column],
2821+
position: Optional[Union[int, Column]] = None,
28192822
) -> Column:
2820-
return _invoke_function_over_columns("regexp_replace", string, lit(pattern), lit(replacement))
2823+
if position is None:
2824+
return _invoke_function_over_columns(
2825+
"regexp_replace",
2826+
string,
2827+
lit(pattern),
2828+
lit(replacement),
2829+
)
2830+
else:
2831+
return _invoke_function_over_columns(
2832+
"regexp_replace",
2833+
string,
2834+
lit(pattern),
2835+
lit(replacement),
2836+
lit(position),
2837+
)
28212838

28222839

28232840
regexp_replace.__doc__ = pysparkfuncs.regexp_replace.__doc__

‎python/pyspark/sql/functions/builtin.py‎

Lines changed: 30 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -16350,14 +16350,19 @@ def regexp_extract_all(
1635016350

1635116351
@_try_remote_functions
1635216352
def regexp_replace(
16353-
string: "ColumnOrName", pattern: Union[str, Column], replacement: Union[str, Column]
16353+
string: "ColumnOrName",
16354+
pattern: Union[str, Column],
16355+
replacement: Union[str, Column],
16356+
position: Optional[Union[int, Column]] = None,
1635416357
) -> Column:
1635516358
r"""Replace all substrings of the specified string value that match regexp with replacement.
1635616359

1635716360
.. versionadded:: 1.5.0
1635816361

1635916362
.. versionchanged:: 3.4.0
1636016363
Supports Spark Connect.
16364+
.. versionchanged:: 4.3.0
16365+
Supports the `position` parameter.
1636116366

1636216367
Parameters
1636316368
----------
@@ -16367,6 +16372,8 @@ def regexp_replace(
1636716372
column object or str containing the regexp pattern
1636816373
replacement : :class:`~pyspark.sql.Column` or str
1636916374
column object or str containing the replacement
16375+
position : :class:`~pyspark.sql.Column` or int, optional
16376+
position to start replacement. The first position is 1.
1637016377

1637116378
Returns
1637216379
-------
@@ -16404,8 +16411,29 @@ def regexp_replace(
1640416411
+-------+-------+-----------+--------------------------------------------+
1640516412
|100-200| (\d+)| --| -----|
1640616413
+-------+-------+-----------+--------------------------------------------+
16414+
16415+
Example 3: Replaces substrings starting from the specified position.
16416+
For the input string "100-200", position 5 starts replacement after "100-".
16417+
16418+
>>> df.select(sf.regexp_replace("str", r"(\d+)", "--", 5).alias("d")).show()
16419+
+------+
16420+
| d|
16421+
+------+
16422+
|100---|
16423+
+------+
1640716424
"""
16408-
return _invoke_function_over_columns("regexp_replace", string, lit(pattern), lit(replacement))
16425+
if position is None:
16426+
return _invoke_function_over_columns(
16427+
"regexp_replace", string, lit(pattern), lit(replacement)
16428+
)
16429+
else:
16430+
return _invoke_function_over_columns(
16431+
"regexp_replace",
16432+
string,
16433+
lit(pattern),
16434+
lit(replacement),
16435+
lit(position),
16436+
)
1640916437

1641016438

1641116439
@_try_remote_functions

‎python/pyspark/sql/tests/connect/test_connect_function.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2215,6 +2215,10 @@ def test_string_functions_multi_args(self):
22152215
cdf.select(CF.regexp_replace(cdf.b, "(a+)(b)?(c)", "--")).toPandas(),
22162216
sdf.select(SF.regexp_replace(sdf.b, "(a+)(b)?(c)", "--")).toPandas(),
22172217
)
2218+
self.assert_eq(
2219+
cdf.select(CF.regexp_replace(cdf.b, "(a+)(b)?(c)", "--", 2)).toPandas(),
2220+
sdf.select(SF.regexp_replace(sdf.b, "(a+)(b)?(c)", "--", 2)).toPandas(),
2221+
)
22182222
self.assert_eq(
22192223
cdf.select(CF.translate(cdf.b, "abc", "xyz")).toPandas(),
22202224
sdf.select(SF.translate(sdf.b, "abc", "xyz")).toPandas(),

‎python/pyspark/sql/tests/test_functions.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3278,11 +3278,15 @@ def test_regexp_replace(self):
32783278
df = self.spark.createDataFrame(
32793279
[("100-200", r"(\d+)", "--")], ["str", "pattern", "replacement"]
32803280
)
3281+
32813282
self.assertTrue(
32823283
all(
32833284
df.select(
32843285
F.regexp_replace("str", r"(\d+)", "--") == "-----",
32853286
F.regexp_replace("str", F.col("pattern"), F.col("replacement")) == "-----",
3287+
F.regexp_replace("str", r"(\d+)", "--", 5) == "100---",
3288+
F.regexp_replace("str", F.col("pattern"), F.col("replacement"), F.lit(5))
3289+
== "100---",
32863290
).first()
32873291
)
32883292
)

‎sql/api/src/main/scala/org/apache/spark/sql/functions.scala‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5392,6 +5392,16 @@ object functions {
53925392
def regexp_replace(e: Column, pattern: String, replacement: String): Column =
53935393
regexp_replace(e, lit(pattern), lit(replacement))
53945394

5395+
/**
5396+
* Replace all substrings of the specified string value that match regexp with rep, starting at
5397+
* the specified position `pos`.
5398+
*
5399+
* @group string_funcs
5400+
* @since 4.3.0
5401+
*/
5402+
def regexp_replace(e: Column, pattern: String, replacement: String, pos: Int): Column =
5403+
regexp_replace(e, lit(pattern), lit(replacement), lit(pos))
5404+
53955405
/**
53965406
* Replace all substrings of the specified string value that match regexp with rep.
53975407
*
@@ -5401,6 +5411,16 @@ object functions {
54015411
def regexp_replace(e: Column, pattern: Column, replacement: Column): Column =
54025412
Column.fn("regexp_replace", e, pattern, replacement)
54035413

5414+
/**
5415+
* Replace all substrings of the specified string value that match regexp with rep, starting at
5416+
* the specified position `pos`.
5417+
*
5418+
* @group string_funcs
5419+
* @since 4.3.0
5420+
*/
5421+
def regexp_replace(e: Column, pattern: Column, replacement: Column, pos: Column): Column =
5422+
Column.fn("regexp_replace", e, pattern, replacement, pos)
5423+
54045424
/**
54055425
* Returns the substring that matches the regular expression `regexp` within the string `str`.
54065426
* If the regular expression is not found, the result is null.

‎sql/core/src/test/scala/org/apache/spark/sql/StringFunctionsSuite.scala‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -188,9 +188,12 @@ class StringFunctionsSuite extends SharedSparkSession {
188188
df.select(
189189
regexp_replace($"a", "(\\d+)", "num"),
190190
regexp_replace($"a", $"b", $"c"),
191+
regexp_replace($"a", "(\\d+)", "num", 5),
192+
regexp_replace($"a", $"b", $"c", lit(5)),
191193
regexp_extract($"a", "(\\d+)-(\\d+)", 1)),
192-
Row("num-num", "300", "100") :: Row("num-num", "400", "100") ::
193-
Row("num-num", "400-400", "100") :: Nil)
194+
Row("num-num", "300", "100-num", "100-200", "100") ::
195+
Row("num-num", "400", "100-num", "100-200", "100") ::
196+
Row("num-num", "400-400", "100-num", "100-400", "100") :: Nil)
194197

195198
// for testing the mutable state of the expression in code gen.
196199
// This is a hack way to enable the codegen, thus the codegen is enable by default,

0 commit comments

Comments
 (0)