Repository navigation
Conversation
jayhan94
force-pushed
the
jayhan/infer_schema_like_pyspark
branch
from
June 25, 2025 06:31
4cb8294 to
e4a23ef
Compare
jayhan94
commented
Jun 25, 2025
| _INTERVAL_DAYTIME = re.compile(r"interval (day|hour|minute|second)( to (day|hour|minute|second))?") | ||
|
|
||
|
|
||
| # Mapping Python types to Spark SQL DataType |
Contributor
Author
There was a problem hiding this comment.
changes of this file is nearly copied from pyspark and only make a few changes
jayhan94
commented
Jun 25, 2025
| df = df.toDF(*names) | ||
| return df | ||
|
|
||
| def _inferSchemaFromList( |
Contributor
Author
There was a problem hiding this comment.
this function is copied from pyspark
jayhan94
marked this pull request as ready for review
June 25, 2025 06:36
jayhan94
changed the title
Contributor
Author
|
Could anyone help review to push this forward? @Mytherin @evertlammerts |
Member
|
The Python API was moved here: https://github.com/duckdb/duckdb-python |
mariotaddeucci
added a commit
to mariotaddeucci/duckdb-python
that referenced
this pull request
Mar 19, 2026
Port schema inference from duckdb/duckdb#18051 to fix duckdb#183. When calling spark.createDataFrame([{"col": value}, ...]), the Spark API now infers the schema from dict keys, matching PySpark behavior. Changes: - Add _type_mappings, _array_type_mappings, _has_nulltype, _merge_type, _infer_type, and _infer_schema functions to types.py - Update session.py to handle dict rows in _combine_data_and_schema and add schema inference branch in createDataFrame for list[dict] - Add _inferSchemaFromList method to SparkSession - Fix test_struct_column to use inferred field names instead of col0/col1 - Add test_dataframe_from_list_dicts test case
mariotaddeucci
added a commit
to mariotaddeucci/duckdb-python
that referenced
this pull request
Mar 19, 2026
Port schema inference from duckdb/duckdb#18051 to fix duckdb#183. When calling spark.createDataFrame([{"col": value}, ...]), the Spark API now infers the schema from dict keys, matching PySpark behavior. Changes: - Add _type_mappings, _array_type_mappings, _has_nulltype, _merge_type, _infer_type, and _infer_schema functions to types.py - Update session.py to handle dict rows in _combine_data_and_schema and add schema inference branch in createDataFrame for list[dict] - Add _inferSchemaFromList method to SparkSession - Fix test_struct_column to use inferred field names instead of col0/col1 - Add test_dataframe_from_list_dicts test case
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close #17187