Validate the dictionary page against its size when reading Parquet

XMLWordPrintableJSON

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Major
    • None
    • Affects Version/s: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
    • Component/s: SQL
    • None

      Spark passes a Parquet dictionary page to parquet-java's Encoding.initDictionary without checking the declared dictionary size or the entry lengths against the bytes of the page. A corrupted dictionary page can then:

      1. Kill the executor. A huge positive num_values in the dictionary page header makes PlainValuesDictionary allocate new int[size], new Binary[size] and so on. For example, an INT32 dictionary page with num_values = 0x7FFFFFF0 throws OutOfMemoryError: Java heap space. The executor exits, every retry kills another executor, and spark.sql.files.ignoreCorruptFiles cannot skip the file.
      2. Return wrong data without an error. PlainBinaryDictionary does not check an entry's length against the page. In an UNCOMPRESSED column chunk the dictionary buffer is a slice of the buffer of the chunk, so an entry that claims more bytes than are left reads bytes of the next page. For example, a string dictionary 02 00 00 00 'a' 'b' 05 00 00 00 'c' 'd' returns cd followed by 3 bytes of the next page header, with both the vectorized and the non-vectorized reader. With a compressed chunk the same entry fails with an ArrayIndexOutOfBoundsException that names no column, and a truncated FIXED_LEN_BYTE_ARRAY dictionary fails with an ArrayIndexOutOfBoundsException in VectorizedColumnReader.readBatch.

      Before initDictionary, the vectorized reader could reject a negative size or a size that cannot fit in the page (size * minEntryWidth > page size), check that every BINARY entry ends within the page and that FIXED_LEN_BYTE_ARRAY and INT96 entries fit (size * length), and throw a ParquetDecodingException that names the column. This runs once per column chunk. The non-vectorized reader builds the dictionary in parquet-java, so it needs the same check there.

      Found in the review of SPARK-59854 (https://github.com/apache/spark/pull/59542).

            Assignee:
            Unassigned
            Reporter:
            L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Created:
              Updated: