-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Major
-
None
-
Affects Version/s: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
-
Component/s: SQL
-
None
Spark passes a Parquet dictionary page to parquet-java's Encoding.initDictionary without checking the declared dictionary size or the entry lengths against the bytes of the page. A corrupted dictionary page can then:
- Kill the executor. A huge positive num_values in the dictionary page header makes PlainValuesDictionary allocate new int[size], new Binary[size] and so on. For example, an INT32 dictionary page with num_values = 0x7FFFFFF0 throws OutOfMemoryError: Java heap space. The executor exits, every retry kills another executor, and spark.sql.files.ignoreCorruptFiles cannot skip the file.
- Return wrong data without an error. PlainBinaryDictionary does not check an entry's length against the page. In an UNCOMPRESSED column chunk the dictionary buffer is a slice of the buffer of the chunk, so an entry that claims more bytes than are left reads bytes of the next page. For example, a string dictionary 02 00 00 00 'a' 'b' 05 00 00 00 'c' 'd' returns cd followed by 3 bytes of the next page header, with both the vectorized and the non-vectorized reader. With a compressed chunk the same entry fails with an ArrayIndexOutOfBoundsException that names no column, and a truncated FIXED_LEN_BYTE_ARRAY dictionary fails with an ArrayIndexOutOfBoundsException in VectorizedColumnReader.readBatch.
Before initDictionary, the vectorized reader could reject a negative size or a size that cannot fit in the page (size * minEntryWidth > page size), check that every BINARY entry ends within the page and that FIXED_LEN_BYTE_ARRAY and INT96 entries fit (size * length), and throw a ParquetDecodingException that names the column. This runs once per column chunk. The non-vectorized reader builds the dictionary in parquet-java, so it needs the same check there.
Found in the review of SPARK-59854 (https://github.com/apache/spark/pull/59542).
- relates to
-
SPARK-59854 Report the column in decoding errors of the vectorized Parquet reader
-
- Resolved
-