Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
4.0.1
Description
Currently, Spark does not support NullType columns in Parquet files. This is blocked at the ParquetFileFormat level, which throws an exception if there's an attempt to read or write NullType column. Parquet has UNKNOWN logical type annotation, which can be used for columns whose values are always null. This can be used to write NullType columns into Parquet files, and when reading from Parquet files that already has this annotation.
Attachments
Issue Links
- links to