Repository navigation
Commit 9d88020
[SPARK-51112][CONNECT] Avoid using pyarrow's
### What changes were proposed in this pull request?
When the `pyarrow` table is empty, avoid calling the `to_pandas` method due to potential segfault failures. Instead, an empty pandas dataframe is created manually.
### Why are the changes needed?
Consider the following code:
```python
from pyspark.sql.types import StructField, ArrayType, StringType, StructType, IntegerType
import faulthandler
faulthandler.enable()
spark = SparkSession.builder \
.remote("sc://localhost:15002") \
.getOrCreate()
sp_df = spark.createDataFrame(
data = [],
schema=StructType(
[
StructField(
name='b_int',
dataType=IntegerType(),
nullable=False,
),
StructField(
name='b',
dataType=ArrayType(ArrayType(StringType(), True), True),
nullable=True,
),
]
)
)
print(sp_df)
print('Spark dataframe generated.')
print(sp_df.toPandas())
print('Pandas dataframe generated.')
```
Executing this may lead to a segfault when the line `sp_df.toPandas()` is run.
Example:
```
Thread 0x00000001f1904f40 (most recent call first):
File "/Users/venkata.gudesa/spark/test_env/lib/python3.13/site-packages/pyarrow/pandas_compat.py", line 808 in table_to_dataframe
File "/Users/venkata.gudesa/spark/test_env/lib/python3.13/site-packages/pyspark/sql/connect/client/core.py", line 949 in to_pandas
File "/Users/venkata.gudesa/spark/test_env/lib/python3.13/site-packages/pyspark/sql/connect/dataframe.py", line 1857 in toPandas
File "<python-input-3>", line 1 in <module>
File "/opt/homebrew/Cellar/python3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/code.py", line 92 in runcode
File "/opt/homebrew/Cellar/python3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/_pyrepl/console.py", line 205 in runsource
File "/opt/homebrew/Cellar/python3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/code.py", line 313 in push
File "/opt/homebrew/Cellar/python3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/_pyrepl/simple_interact.py", line 160 in run_multiline_interactive_console
File "/opt/homebrew/Cellar/python3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/_pyrepl/main.py", line 59 in interactive_console
File "/opt/homebrew/Cellar/python3.13/3.13.0_1/Frameworks/Python.framework/Versions/3.13/lib/python3.13/_pyrepl/__main__.py", line 6 in <module>
File "<frozen runpy>", line 88 in _run_code
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
New unit test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #49834 from vicennial/SPARK-51112.
Lead-authored-by: vicennial <[email protected]>
Co-authored-by: Venkata Sai Akhil Gudesa <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>to_pandas on an empty table1 parent c91fdd4 commit 9d88020
3 files changed
Lines changed: 73 additions & 27 deletions
File tree
- python/pyspark/sql
- connect/client
- tests
- connect
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
933 | 933 | | |
934 | 934 | | |
935 | 935 | | |
936 | | - | |
937 | | - | |
938 | | - | |
939 | | - | |
940 | | - | |
941 | | - | |
942 | | - | |
943 | | - | |
944 | | - | |
945 | | - | |
946 | | - | |
947 | | - | |
948 | | - | |
949 | | - | |
950 | | - | |
951 | | - | |
952 | | - | |
953 | | - | |
954 | | - | |
955 | | - | |
956 | | - | |
957 | | - | |
958 | | - | |
959 | | - | |
960 | | - | |
961 | | - | |
962 | | - | |
| 936 | + | |
| 937 | + | |
| 938 | + | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
| 943 | + | |
| 944 | + | |
| 945 | + | |
| 946 | + | |
| 947 | + | |
| 948 | + | |
| 949 | + | |
| 950 | + | |
| 951 | + | |
| 952 | + | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
963 | 969 | | |
964 | 970 | | |
965 | 971 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
45 | 48 | | |
46 | 49 | | |
47 | 50 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| |||
35 | 37 | | |
36 | 38 | | |
37 | 39 | | |
| 40 | + | |
38 | 41 | | |
39 | 42 | | |
40 | 43 | | |
| |||
289 | 292 | | |
290 | 293 | | |
291 | 294 | | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
292 | 329 | | |
293 | 330 | | |
294 | 331 | | |
| |||
0 commit comments