Add vectorized Parquet reader support for nanosecond-precision timestamp types

XMLWordPrintableJSON

      This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond precision).

      Problem

      Nanosecond Parquet READ only works via the row-based reader. TimestampNanosParquetOps.isBatchReadSupported (datasources/parquet/TimestampNanosParquetOps.scala ~L54-58) stays at the trait default false, and ParquetUtils.isBatchReadSupportedForSchema requires every column to support batch read, so any nanosecond column forces the whole file onto ParquetRowConverter even when vectorized reading is enabled. ParquetVectorUpdaterFactory (~L140-161) only has MICROS/MILLIS timestamp updaters. Correct but slow; Parquet WRITE + non-vectorized READ are already done (SPARK-57102).

      Goal

      Read TIMESTAMP(NANOS) columns through the vectorized path, materializing the 16-byte TimestampNanosVal (epoch micros + nanosecond remainder) into the columnar batch.

      Scope

      Implement a nanosecond updater in ParquetVectorUpdaterFactory / VectorizedColumnReader; flip isBatchReadSupported for the nanosecond ops; ensure the columnar layout matches the nanosecond ColumnVector (added under SPARK-57100).

      Acceptance criteria

      • Nanosecond Parquet files read correctly with spark.sql.parquet.enableVectorizedReader=true, matching the row-reader results; no fallback for nanosecond-only schemas.

      Testing

      ParquetTimestampNanosSuite (with the vectorized reader on), ParquetQuerySuite.

      Dependencies

      None - independent (columnar layout resolved in SPARK-57100).

            Assignee:
            Anupam Yadav
            Reporter:
            Max Gekk
            Max Gekk Max Gekk
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: