Repository navigation
Commit bd1ba25
committed
fix: support createDataFrame with list of dicts in Spark API
Port schema inference from duckdb/duckdb#18051 to fix duckdb#183.
When calling spark.createDataFrame([{"col": value}, ...]), the Spark API
now infers the schema from dict keys, matching PySpark behavior.
Changes:
- Add _type_mappings, _array_type_mappings, _has_nulltype, _merge_type,
_infer_type, and _infer_schema functions to types.py
- Update session.py to handle dict rows in _combine_data_and_schema and
add schema inference branch in createDataFrame for list[dict]
- Add _inferSchemaFromList method to SparkSession
- Fix test_struct_column to use inferred field names instead of col0/col1
- Add test_dataframe_from_list_dicts test case1 parent 3ecd608 commit bd1ba25
4 files changed
Lines changed: 394 additions & 20 deletions
File tree
- duckdb/experimental/spark/sql
- tests/fast/spark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | 1 | | |
2 | 2 | | |
3 | | - | |
| 3 | + | |
| 4 | + | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
| |||
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | | - | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
20 | 21 | | |
21 | | - | |
| 22 | + | |
22 | 23 | | |
23 | 24 | | |
24 | 25 | | |
| |||
38 | 39 | | |
39 | 40 | | |
40 | 41 | | |
41 | | - | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
42 | 47 | | |
43 | 48 | | |
44 | 49 | | |
| |||
150 | 155 | | |
151 | 156 | | |
152 | 157 | | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
153 | 161 | | |
154 | 162 | | |
155 | 163 | | |
| |||
188 | 196 | | |
189 | 197 | | |
190 | 198 | | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
191 | 246 | | |
192 | 247 | | |
193 | 248 | | |
| |||
0 commit comments