Skip to content

Commit 8796909

Browse files
Make the experimental Spark API work as advertised against PySpark 4.2.0 and 3.5.9 (duckdb#686)
The experimental Spark API (`duckdb.experimental.spark`) had drifted a long way from what it claims to be, a drop-in subset of PySpark on DuckDB. A clean install of 1.5.6 could not even import it: `_globals.py` and `_typing.py` imported `typing_extensions` and `numpy` at module level, neither of which the `duckdb` package declares (duckdb#653). Beyond that, I graded the module against real Spark 4.2.0 on 64 typical snippets and 47 edge cases, and found that 18 of the snippets raised and 12 of the edge cases returned a different value than Spark without any error. Joins with repeated column names were the worst: after a second join, or a `withColumn` or `drop`, both columns of a repeated name silently carried the values of the first one. This PR makes the module do what it says, with a fixed compatibility target: PySpark 4.2.0 and 3.5.9. Where the two versions differ, the module follows 4.2.0 unless it cannot. The rule for everything else is simple. Behavior is mapped mechanically onto DuckDB where that gives Spark's answer; where it does not, the module raises rather than returning a different value; a small number of cases where DuckDB's answer is deliberately kept are documented (38 of them, which I will put on the Spark API docs page). The module does not re-implement Spark semantics: an earlier version of this work had a CSV reader that parsed values the way Spark does, and I took it out again in favor of DuckDB's casts and type detection. What changed, by area: - Imports: no module-level dependency on anything the `duckdb` package does not declare. pandas, pyarrow and pytz stay optional at the call sites that need them. A test imports the module with each optional package blocked. - Names: every function and expression gets Spark's column name, and a frame keeps Spark's names separate from DuckDB's, so repeated names (`id, id`) survive instead of becoming `id, id_1`. Columns of a join input are referenced by position, and every name resolution checks for ambiguity and raises `AMBIGUOUS_REFERENCE` where Spark does. - Values: integer casts truncate toward zero like Spark; set operations match struct fields by position; timestamps are collected as epoch microseconds and converted with Spark's own rules, so `collect()`, `toPandas()` and `show()` agree with Spark in any process or session time zone and no longer need pytz; dozens of function-level differences in NULL handling, `sequence`, `regexp_replace`, padding, binary `concat`, and so on. - UDFs: `register` returns a usable callable, a UDF named like a builtin neither shadows nor is replaced by it, struct arguments arrive as `Row`s. - Readers and writers: Spark's save modes (a plain `write.csv()` no longer overwrites silently), `load()` by file extension and with a schema (it returned all-NULL rows for a CSV with a schema), `_c0` column names, `nullValue`, header checks against the schema, and option flags that are either honored or raise instead of being accepted and ignored. - Input: `createDataFrame` verifies rows and pandas input against the schema the way Spark does; a list of dicts works (duckdb#183); a statement without a result in `spark.sql()` gives a frame with an empty schema (duckdb#428; the rest of that issue needs relations without columns in the engine). - Errors: Spark's error classes for the cases above, with seven classes added to the module's table from Spark 4.2.0's definitions. Four binding bugs came out of this work and are fixed in the same change, all confirmed new: - `StarExpression(exclude=[ColumnExpression("b", "id")])` dropped the qualifier and excluded every `id`; the name parts now map to column, table, schema and catalog, and an empty name or more than four parts raise `ValueError`. - `ColumnExpression("s.a.b.c")` with a single dotted string kept only the first, the second to last and the last part on this branch (1.5.6 raised instead), so it bound another struct field. It now keeps every part, as the multi-argument form does. - `duckdb.struct_type({})` refused the empty struct, a check the engine dropped long ago. - A NaN inside a list, struct, map or typed `Value` parameter became NULL, because the nested conversions dropped the `nan_as_null` flag the parameter path sets. Four open community PRs addressed parts of this, and I reimplemented their fixes rather than merging them, because the surrounding code changed under them: duckdb#552 (`show()` signature and the UDF return type bug), duckdb#388 (inference for `createDataFrame` from dicts), duckdb#378 (`struct()` of a list) and duckdb#626 (path quoting in `load()`). The authors are credited: wavebyrd's and Raaif-Yousuf's commits are cherry-picked, and all four are co-authors of the main commit. What this does not do: the module's coverage of PySpark is unchanged, about a third of the API, with no `Window`; missing API raises `AttributeError` or `ContributionsAcceptedError` as before. The engine's relational API has a bug that this module contains with an extra projection rather than fixes: an aggregate, filter or projection over a `limit`, `order` or `distinct` of a join relation merges into the join's own SELECT beneath the modifier, so `join.limit(1).aggregate("count(*)")` gives the unlimited count. I will file that separately. Evidence. The Spark tests now run in three modes: against the module, against real PySpark 4.2.0 and against real PySpark 3.5.9 (`USE_ACTUAL_SPARK=true`), plus a run in a non-UTC time zone. On this tree: 1813, 1757 and 1745 tests pass, 0 fail (before: 289 tests, with 11 and 15 failures on the two Spark versions). PySpark's own function doctests, run against the module with the names aliased, pass fully for 111 of 185 functions (3 before). The 64 snippets graded against Spark 4.2.0: 48 match, 16 raise, 0 differ silently (45, 18 and silent differences before). The full `tests/fast` suite passes (8089 tests). The binding fixes were checked under AddressSanitizer and LeakSanitizer against an unmodified build, with no new report. Resolves duckdb#653 and duckdb#183, partly duckdb#428.
2 parents 16c769b + caa0755 commit 8796909

53 files changed

Lines changed: 14841 additions & 1169 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎duckdb/experimental/spark/_globals.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ def foo(arg=pyducdkb.spark._NoValue):
3333

3434
__ALL__ = ["_NoValue"]
3535

36-
from typing_extensions import Self
36+
from typing import Self
3737

3838
# Disallow reloading this module so as to preserve the identities of the
3939
# classes defined here.

‎duckdb/experimental/spark/_typing.py‎

Lines changed: 1 addition & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,7 @@
1717
# under the License.
1818

1919
from collections.abc import Callable, Iterable, Sized
20-
from typing import Literal, TypeVar
21-
22-
from typing_extensions import Protocol, Self
23-
24-
from numpy import float32, float64, int32, int64, ndarray
20+
from typing import Literal, Protocol, Self, TypeVar
2521

2622
F = TypeVar("F", bound=Callable)
2723
T_co = TypeVar("T_co", covariant=True)
@@ -43,5 +39,3 @@ class SizedIterable(Protocol, Sized, Iterable[T_co]): ...
4339

4440

4541
S = TypeVar("S", bound=SupportsOrdering)
46-
47-
NumberOrArray = TypeVar("NumberOrArray", float, int, complex, int32, int64, float32, float64, ndarray)

‎duckdb/experimental/spark/errors/error_classes.py‎

Lines changed: 116 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,12 @@
1818

1919
ERROR_CLASSES_JSON = """
2020
{
21+
"AMBIGUOUS_REFERENCE" : {
22+
"message" : [
23+
"Reference <name> is ambiguous, could be: <referenceNames>."
24+
],
25+
"sqlState" : "42704"
26+
},
2127
"APPLICATION_NAME_NOT_SET" : {
2228
"message" : [
2329
"An application name must be set in your configuration."
@@ -106,22 +112,26 @@
106112
"CANNOT_INFER_EMPTY_SCHEMA": {
107113
"message": [
108114
"Can not infer schema from empty dataset."
109-
]
115+
],
116+
"sqlState": "42K09"
110117
},
111118
"CANNOT_INFER_SCHEMA_FOR_TYPE": {
112119
"message": [
113120
"Can not infer schema for type: `<data_type>`."
114-
]
121+
],
122+
"sqlState": "42K09"
115123
},
116124
"CANNOT_INFER_TYPE_FOR_FIELD": {
117125
"message": [
118126
"Unable to infer the type of the field `<field_name>`."
119-
]
127+
],
128+
"sqlState": "42K09"
120129
},
121130
"CANNOT_MERGE_TYPE": {
122131
"message": [
123132
"Can not merge type `<data_type1>` and `<data_type2>`."
124-
]
133+
],
134+
"sqlState": "42K09"
125135
},
126136
"CANNOT_OPEN_SOCKET": {
127137
"message": [
@@ -148,6 +158,12 @@
148158
"returnType can not be specified when `<arg_name>` is a user-defined function, but got <return_type>."
149159
]
150160
},
161+
"COLUMN_ALREADY_EXISTS" : {
162+
"message" : [
163+
"The column <columnName> already exists. Choose another name or rename the existing column."
164+
],
165+
"sqlState" : "42711"
166+
},
151167
"COLUMN_IN_LIST": {
152168
"message": [
153169
"`<func_name>` does not allow a Column in a list."
@@ -230,11 +246,23 @@
230246
"Duplicated field names in Arrow Struct are not allowed, got <field_names>"
231247
]
232248
},
249+
"FIELD_NOT_FOUND" : {
250+
"message" : [
251+
"No such struct field <fieldName> in <fields>."
252+
],
253+
"sqlState" : "42704"
254+
},
233255
"HIGHER_ORDER_FUNCTION_SHOULD_RETURN_COLUMN" : {
234256
"message" : [
235257
"Function `<func_name>` should return Column, got <return_type>."
236258
]
237259
},
260+
"INCOMPATIBLE_COLUMN_TYPE" : {
261+
"message" : [
262+
"<operator> can only be performed on tables with compatible column types. The <columnOrdinalNumber> column of the <tableOrdinalNumber> table is <dataType1> type which is not compatible with <dataType2> at the same column of the first table.<hint>."
263+
],
264+
"sqlState" : "42825"
265+
},
238266
"INCORRECT_CONF_FOR_PROFILE" : {
239267
"message" : [
240268
"`spark.python.profile` or `spark.python.profile.memory` configuration",
@@ -286,6 +314,12 @@
286314
"Pandas UDF should return StructType for <eval_type>, got <return_type>."
287315
]
288316
},
317+
"INVALID_SAVE_MODE" : {
318+
"message" : [
319+
"The specified save mode <mode> is invalid. Valid save modes include \\"append\\", \\"overwrite\\", \\"ignore\\", \\"error\\", \\"errorifexists\\", and \\"default\\"."
320+
],
321+
"sqlState" : "42000"
322+
},
289323
"INVALID_TIMEOUT_TIMESTAMP" : {
290324
"message" : [
291325
"Timeout timestamp (<timestamp>) cannot be earlier than the current watermark (<watermark>)."
@@ -371,6 +405,12 @@
371405
"A master URL must be set in your configuration."
372406
]
373407
},
408+
"MISSING_GROUP_BY" : {
409+
"message" : [
410+
"The query does not include a GROUP BY clause. Add GROUP BY or turn it into the window functions using OVER clauses."
411+
],
412+
"sqlState" : "42803"
413+
},
374414
"MISSING_LIBRARY_FOR_PROFILER" : {
375415
"message" : [
376416
"Install the 'memory_profiler' library in the cluster to enable memory profiling."
@@ -626,6 +666,12 @@
626666
"No active Spark session found. Please create a new Spark session before running the code."
627667
]
628668
},
669+
"NUM_COLUMNS_MISMATCH" : {
670+
"message" : [
671+
"<operator> can only be performed on inputs with the same number of columns, but the first input has <firstNumColumns> columns and the <invalidOrdinalNum> input has <invalidNumColumns> columns."
672+
],
673+
"sqlState" : "42826"
674+
},
629675
"ONLY_ALLOWED_FOR_SINGLE_COLUMN" : {
630676
"message" : [
631677
"Argument `<arg_name>` can only be provided for a single column."
@@ -636,6 +682,18 @@
636682
"Only a single trigger is allowed."
637683
]
638684
},
685+
"PARSE_SYNTAX_ERROR" : {
686+
"message" : [
687+
"Syntax error at or near <error><hint>."
688+
],
689+
"sqlState" : "42601"
690+
},
691+
"PATH_ALREADY_EXISTS" : {
692+
"message" : [
693+
"Path <outputPath> already exists. Set mode as \\"overwrite\\" to overwrite the existing path."
694+
],
695+
"sqlState" : "42K04"
696+
},
639697
"PIPE_FUNCTION_EXITED" : {
640698
"message" : [
641699
"Pipe function `<func_name>` exited with error code <error_code>."
@@ -797,11 +855,41 @@
797855
"Unknown response: <response>."
798856
]
799857
},
858+
"UNRESOLVED_COLUMN" : {
859+
"message" : [
860+
"A column, variable, or function parameter with name <objectName> cannot be resolved."
861+
],
862+
"sub_class" : {
863+
"WITHOUT_SUGGESTION" : {
864+
"message" : [
865+
""
866+
]
867+
},
868+
"WITH_SUGGESTION" : {
869+
"message" : [
870+
"Did you mean one of the following? [<proposal>]."
871+
]
872+
}
873+
},
874+
"sqlState" : "42703"
875+
},
876+
"UNRESOLVED_COLUMN_AMONG_FIELD_NAMES" : {
877+
"message" : [
878+
"Cannot resolve column name \\"<colName>\\" among (<fieldNames>)."
879+
],
880+
"sqlState" : "42703"
881+
},
800882
"UNSUPPORTED_DATA_TYPE" : {
801883
"message" : [
802884
"Unsupported DataType `<data_type>`."
803885
]
804886
},
887+
"UNSUPPORTED_DATATYPE" : {
888+
"message" : [
889+
"Unsupported data type <typeName>."
890+
],
891+
"sqlState" : "0A000"
892+
},
805893
"UNSUPPORTED_DATA_TYPE_FOR_ARROW" : {
806894
"message" : [
807895
"Single data type <data_type> is not supported with Arrow."
@@ -817,6 +905,25 @@
817905
"<data_type> is only supported with pyarrow 2.0.0 and above."
818906
]
819907
},
908+
"UNSUPPORTED_DATA_TYPE_FOR_DATASOURCE" : {
909+
"message" : [
910+
"The <format> datasource doesn't support the column <columnName> of the type <columnType>."
911+
],
912+
"sqlState" : "0A000"
913+
},
914+
"UNSUPPORTED_FEATURE" : {
915+
"message" : [
916+
"The feature is not supported:"
917+
],
918+
"sub_class" : {
919+
"SET_OPERATION_ON_MAP_TYPE" : {
920+
"message" : [
921+
"Cannot have MAP type columns in DataFrame which calls set operations (INTERSECT, EXCEPT, etc.), but the type of column <colName> is <dataType>."
922+
]
923+
}
924+
},
925+
"sqlState" : "0A000"
926+
},
820927
"UNSUPPORTED_JOIN_TYPE" : {
821928
"message" : [
822929
"Unsupported join type: <join_type>. Supported join types include: \\"inner\\", \\"outer\\", \\"full\\", \\"fullouter\\", \\"full_outer\\", \\"leftouter\\", \\"left\\", \\"left_outer\\", \\"rightouter\\", \\"right\\", \\"right_outer\\", \\"leftsemi\\", \\"left_semi\\", \\"semi\\", \\"leftanti\\", \\"left_anti\\", \\"anti\\", \\"cross\\"."
@@ -911,6 +1018,11 @@
9111018
"message": [
9121019
"Index must be non-zero."
9131020
]
1021+
},
1022+
"_LEGACY_ERROR_TEMP_1323" : {
1023+
"message" : [
1024+
"\\"<colName>\\" is not a numeric column. Aggregation function can only be applied on a numeric column."
1025+
]
9141026
}
9151027
}
9161028
"""

‎duckdb/experimental/spark/errors/exceptions/base.py‎

Lines changed: 15 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,18 @@ def __init__( # noqa: D107
3131
self.error_class = error_class
3232
self.message_parameters = message_parameters
3333

34+
def getCondition(self) -> str | None:
35+
"""Returns an error condition, the name Spark 4 uses for an error class.
36+
37+
.. versionadded:: 4.0.0
38+
39+
See Also:
40+
--------
41+
:meth:`PySparkException.getMessageParameters`
42+
:meth:`PySparkException.getSqlState`
43+
"""
44+
return self.error_class
45+
3446
def getErrorClass(self) -> str | None:
3547
"""Returns an error class as a string.
3648
@@ -55,10 +67,8 @@ def getMessageParameters(self) -> dict[str, str] | None:
5567
"""
5668
return self.message_parameters
5769

58-
def getSqlState(self) -> None:
59-
"""Returns an SQLSTATE as a string.
60-
61-
Errors generated in Python have no SQLSTATE, so it always returns None.
70+
def getSqlState(self) -> str | None:
71+
"""Returns an SQLSTATE as a string, or None when the error class has none.
6272
6373
.. versionadded:: 3.4.0
6474
@@ -67,7 +77,7 @@ def getSqlState(self) -> None:
6777
:meth:`PySparkException.getErrorClass`
6878
:meth:`PySparkException.getMessageParameters`
6979
"""
70-
return None
80+
return self.error_reader.get_sqlstate(self.error_class)
7181

7282
def __str__(self) -> str: # noqa: D105
7383
if self.getErrorClass() is not None:

‎duckdb/experimental/spark/errors/utils.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,12 @@ def get_error_message(self, error_class: str, message_parameters: dict[str, str]
3838

3939
return message_template.translate(table).format(**message_parameters)
4040

41+
def get_sqlstate(self, error_class: str | None) -> str | None:
42+
"""Returns the SQLSTATE of an error class, or None when it has none."""
43+
if error_class is None:
44+
return None
45+
return self.error_info_map.get(error_class.split(".")[0], {}).get("sqlState")
46+
4147
def get_message_template(self, error_class: str) -> str:
4248
"""Returns the message template for corresponding error class from error_classes.py.
4349

‎duckdb/experimental/spark/sql/__init__.py‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,5 +3,6 @@
33
from .dataframe import DataFrame
44
from .readwriter import DataFrameWriter
55
from .session import SparkSession
6+
from .types import Row
67

7-
__all__ = ["Catalog", "DataFrame", "DataFrameWriter", "RuntimeConfig", "SparkSession"]
8+
__all__ = ["Catalog", "DataFrame", "DataFrameWriter", "Row", "RuntimeConfig", "SparkSession"]

‎duckdb/experimental/spark/sql/_typing.py‎

Lines changed: 2 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -16,21 +16,10 @@
1616
# specific language governing permissions and limitations
1717
# under the License.
1818

19-
from collections.abc import Callable
20-
from typing import (
21-
Any,
22-
TypeVar,
23-
)
24-
25-
try:
26-
from typing import Literal, Protocol
27-
except ImportError:
28-
from typing import Literal
29-
30-
from typing_extensions import Protocol
31-
3219
import datetime
3320
import decimal
21+
from collections.abc import Callable
22+
from typing import Any, Literal, Protocol, TypeVar
3423

3524
from .._typing import PrimitiveType
3625
from . import types

0 commit comments

Comments
 (0)