Repository navigation
Commit 463d804
[SPARK-57190][SQL] Fix API inconsistency for 4-argument regexp_replace
### What changes were proposed in this pull request?
Spark SQL already supports the 4-argument form of `regexp_replace`:
```sql
regexp_replace(str, regexp, replacement, position)
```
However, the corresponding Scala, PySpark, and Spark Connect APIs currently expose only the 3-argument variants.
This PR exposes the existing 4-argument functionality through:
* Scala API (`functions.regexp_replace`)
* PySpark API (`functions.regexp_replace`)
* Spark Connect API (`functions.regexp_replace`)
and adds corresponding Scala, PySpark, and Connect test coverage.
### Why are the changes needed?
The underlying SQL functionality already exists and is available through SQL expressions, but it is not accessible through the public Scala, PySpark, and Spark Connect APIs.
This creates an inconsistency between SQL and programmatic interfaces. Exposing the optional `position` argument aligns the public APIs with existing SQL functionality.
### Does this PR introduce *any* user-facing change?
Yes.
Users can now specify the optional `position` argument through the Scala, PySpark, and Spark Connect APIs.
Before:
```scala
regexp_replace(col("s"), "(\\d+)", "--")
```
After:
```scala
regexp_replace(col("s"), "(\\d+)", "--", 5)
```
Similarly, PySpark users can now call:
```python
F.regexp_replace("s", r"(\d+)", "--", 5)
```
### How was this patch tested?
Added test coverage in:
* `StringFunctionsSuite`
* `FunctionsTests.test_regexp_replace`
* `SparkConnectFunctionTests.test_string_functions_multi_args`
Verified with:
```bash
./build/sbt "sql-api/compile"
./build/sbt "sql/Test/compile"
./build/sbt \
"sql/testOnly org.apache.spark.sql.StringFunctionsSuite -- -z \"regex_replace / regex_extract\""
python3.11 -m pytest \
python/pyspark/sql/tests/test_functions.py::FunctionsTests::test_regexp_replace -v
python3.11 -m pytest \
python/pyspark/sql/tests/connect/test_connect_function.py::SparkConnectFunctionTests::test_string_functions_multi_args -v
```
### Was this patch authored or co-authored using generative AI tooling?
Developed with assistance from GPT-5.
Closes #56240 from pchintar/regexp_replace.
Authored-by: pchintar <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit d35df2e)
Signed-off-by: Wenchen Fan <[email protected]>1 parent db649d9 commit 463d804
6 files changed
Lines changed: 82 additions & 6 deletions
File tree
- python/pyspark/sql
- connect/functions
- functions
- tests
- connect
- sql
- api/src/main/scala/org/apache/spark/sql
- core/src/test/scala/org/apache/spark/sql
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2815 | 2815 | | |
2816 | 2816 | | |
2817 | 2817 | | |
2818 | | - | |
| 2818 | + | |
| 2819 | + | |
| 2820 | + | |
| 2821 | + | |
2819 | 2822 | | |
2820 | | - | |
| 2823 | + | |
| 2824 | + | |
| 2825 | + | |
| 2826 | + | |
| 2827 | + | |
| 2828 | + | |
| 2829 | + | |
| 2830 | + | |
| 2831 | + | |
| 2832 | + | |
| 2833 | + | |
| 2834 | + | |
| 2835 | + | |
| 2836 | + | |
| 2837 | + | |
2821 | 2838 | | |
2822 | 2839 | | |
2823 | 2840 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16350 | 16350 | | |
16351 | 16351 | | |
16352 | 16352 | | |
16353 | | - | |
| 16353 | + | |
| 16354 | + | |
| 16355 | + | |
| 16356 | + | |
16354 | 16357 | | |
16355 | 16358 | | |
16356 | 16359 | | |
16357 | 16360 | | |
16358 | 16361 | | |
16359 | 16362 | | |
16360 | 16363 | | |
| 16364 | + | |
| 16365 | + | |
16361 | 16366 | | |
16362 | 16367 | | |
16363 | 16368 | | |
| |||
16367 | 16372 | | |
16368 | 16373 | | |
16369 | 16374 | | |
| 16375 | + | |
| 16376 | + | |
16370 | 16377 | | |
16371 | 16378 | | |
16372 | 16379 | | |
| |||
16404 | 16411 | | |
16405 | 16412 | | |
16406 | 16413 | | |
| 16414 | + | |
| 16415 | + | |
| 16416 | + | |
| 16417 | + | |
| 16418 | + | |
| 16419 | + | |
| 16420 | + | |
| 16421 | + | |
| 16422 | + | |
| 16423 | + | |
16407 | 16424 | | |
16408 | | - | |
| 16425 | + | |
| 16426 | + | |
| 16427 | + | |
| 16428 | + | |
| 16429 | + | |
| 16430 | + | |
| 16431 | + | |
| 16432 | + | |
| 16433 | + | |
| 16434 | + | |
| 16435 | + | |
| 16436 | + | |
16409 | 16437 | | |
16410 | 16438 | | |
16411 | 16439 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2215 | 2215 | | |
2216 | 2216 | | |
2217 | 2217 | | |
| 2218 | + | |
| 2219 | + | |
| 2220 | + | |
| 2221 | + | |
2218 | 2222 | | |
2219 | 2223 | | |
2220 | 2224 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3278 | 3278 | | |
3279 | 3279 | | |
3280 | 3280 | | |
| 3281 | + | |
3281 | 3282 | | |
3282 | 3283 | | |
3283 | 3284 | | |
3284 | 3285 | | |
3285 | 3286 | | |
| 3287 | + | |
| 3288 | + | |
| 3289 | + | |
3286 | 3290 | | |
3287 | 3291 | | |
3288 | 3292 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
5392 | 5392 | | |
5393 | 5393 | | |
5394 | 5394 | | |
| 5395 | + | |
| 5396 | + | |
| 5397 | + | |
| 5398 | + | |
| 5399 | + | |
| 5400 | + | |
| 5401 | + | |
| 5402 | + | |
| 5403 | + | |
| 5404 | + | |
5395 | 5405 | | |
5396 | 5406 | | |
5397 | 5407 | | |
| |||
5401 | 5411 | | |
5402 | 5412 | | |
5403 | 5413 | | |
| 5414 | + | |
| 5415 | + | |
| 5416 | + | |
| 5417 | + | |
| 5418 | + | |
| 5419 | + | |
| 5420 | + | |
| 5421 | + | |
| 5422 | + | |
| 5423 | + | |
5404 | 5424 | | |
5405 | 5425 | | |
5406 | 5426 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
188 | 188 | | |
189 | 189 | | |
190 | 190 | | |
| 191 | + | |
| 192 | + | |
191 | 193 | | |
192 | | - | |
193 | | - | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
194 | 197 | | |
195 | 198 | | |
196 | 199 | | |
| |||
0 commit comments