Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-54220

NullType/VOID/UNKNOWN Type Support in Parquet

    XMLWordPrintableJSON

Details

    Description

      Currently, Spark does not support NullType columns in Parquet files. This is blocked at the ParquetFileFormat level, which throws an exception if there's an attempt to read or write NullType column. Parquet has UNKNOWN logical type annotation, which can be used for columns whose values are always null. This can be used to write NullType columns into Parquet files, and when reading from Parquet files that already has this annotation.

      Attachments

        Issue Links

          Activity

            People

              ziyaza Ziya Mukhtarov
              ziyaza Ziya Mukhtarov
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: