-
Type:
Sub-task
-
Resolution: Fixed
-
Priority:
Major
-
Affects Version/s: 4.3.0
-
Component/s: SQL
This sub-task is part of the umbrella SPARK-56822 (timestamps with nanosecond precision).
Problem
Nanosecond Parquet READ only works via the row-based reader. TimestampNanosParquetOps.isBatchReadSupported (datasources/parquet/TimestampNanosParquetOps.scala ~L54-58) stays at the trait default false, and ParquetUtils.isBatchReadSupportedForSchema requires every column to support batch read, so any nanosecond column forces the whole file onto ParquetRowConverter even when vectorized reading is enabled. ParquetVectorUpdaterFactory (~L140-161) only has MICROS/MILLIS timestamp updaters. Correct but slow; Parquet WRITE + non-vectorized READ are already done (SPARK-57102).
Goal
Read TIMESTAMP(NANOS) columns through the vectorized path, materializing the 16-byte TimestampNanosVal (epoch micros + nanosecond remainder) into the columnar batch.
Scope
Implement a nanosecond updater in ParquetVectorUpdaterFactory / VectorizedColumnReader; flip isBatchReadSupported for the nanosecond ops; ensure the columnar layout matches the nanosecond ColumnVector (added under SPARK-57100).
Acceptance criteria
- Nanosecond Parquet files read correctly with spark.sql.parquet.enableVectorizedReader=true, matching the row-reader results; no fallback for nanosecond-only schemas.
Testing
ParquetTimestampNanosSuite (with the vectorized reader on), ParquetQuerySuite.
Dependencies
None - independent (columnar layout resolved in SPARK-57100).
- links to