Repository navigation
[Python] PySpark-API: struct(List[Column]) not working #17189
Description
What happens?
From PySpark docs: pyspark.sql.functions.struct
struct() should accept a list of Columns as input.
In DuckDB this does not work:
InvalidInputException: Invalid Input Error: Expected argument of type Expression, received '<class 'list'>' instead
DuckDB currently only accepts plain arguments (unpacked list).
To Reproduce
input (from docs)
from duckdb.experimental.spark.sql import SparkSession as session
from duckdb.experimental.spark.sql import functions as F
spark = session.builder.getOrCreate()
df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
df.select(F.struct([df.age, df.name]).alias("struct")).collect()output
---------------------------------------------------------------------------
InvalidInputException Traceback (most recent call last)
Cell In[30], line 1
----> 1 df.select(F.struct([df.age, df.name]).alias("struct")).collect()
File ...\.venv\lib\site-packages\duckdb\experimental\spark\sql\functions.py:108, in struct(*cols)
106 def struct(*cols: Column) -> Column:
107 return Column(
--> 108 FunctionExpression("struct_pack", *[_inner_expr_or_val(x) for x in cols])
109 )
InvalidInputException: Invalid Input Error: Expected argument of type Expression, received '<class 'list'>' instead
workaround: unpacking the list via * asterisk operator
from duckdb.experimental.spark.sql import SparkSession as session
from duckdb.experimental.spark.sql import functions as F
spark = session.builder.getOrCreate()
df = spark.createDataFrame([("Alice", 2), ("Bob", 5)], ("name", "age"))
df.select(F.struct(*[df.age, df.name]).alias("struct")).collect()output
[Row(struct={'age': 2, 'name': 'Alice'}),
Row(struct={'age': 5, 'name': 'Bob'})]
OS:
win_amd64
DuckDB Version:
1.2.2 (duckdb-1.2.2-cp310-cp310-win_amd64.whl)
DuckDB Client:
Python
Hardware:
No response
Full Name:
Martin Bode
Affiliation:
N/A
What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.
I have tested with a stable release
Did you include all relevant data sets for reproducing the issue?
Yes
Did you include all code required to reproduce the issue?
- Yes, I have
Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?
- Yes, I have
Activity
shuyouZZ commentedon Jun 12, 2025
@keen85 I have submitted the PR #17896 to remain consistent with the pyspark API
keen85 commentedon Jun 12, 2025
@shuyouZZ thanks a lot!
I'd like to work on the PySpark API myself; however I did not manage to setup my local dev environment.
Can you give me a short intro, what I need to do?
I already created a discussion here: #17628 but so far no one replied 😞
shuyouZZ commentedon Jun 12, 2025
The following are the build steps in my local environment.
Just rebuild the duckdb pkg if change the source code.
github-actions commentedon Jun 13, 2026
This issue is stale because it has been open 365 days with no activity. Remove stale label or comment or this will be closed in 30 days.
github-actions commentedon Jul 14, 2026
This issue was closed because it has been stale for 30 days with no activity.