-
Type:
Improvement
-
Resolution: Unresolved
-
Priority:
Major
-
None
-
Affects Version/s: 4.4.0
-
Component/s: SQL
-
None
A dictionary id at or beyond the size of the dictionary fails with an ArrayIndexOutOfBoundsException from Dictionary.decodeTo*. With eager decoding it is thrown in VectorizedColumnReader.readBatch without the page or the column. With lazy dictionary decoding readBatch succeeds, and the exception is thrown later from ParquetDictionary in the consumer, e.g. Index 5 out of bounds for length 2 for a 2-entry INT32 dictionary. That is outside the error handling of FileScanRDD, so the error names neither the file nor the column, and spark.sql.files.ignoreCorruptFiles cannot skip it.
The id reader could check the ids against the dictionary size and throw a ParquetDecodingException with the column. An id is always below 2^bitWidth, so the check is only needed when the dictionary is smaller than that: once per RLE run, and once per group of 8 values for bit-packed runs. Since this is on the hot path, the change needs a benchmark.
Found in the review of SPARK-59854 (https://github.com/apache/spark/pull/59542).
- relates to
-
SPARK-59854 Report the column in decoding errors of the vectorized Parquet reader
-
- Resolved
-