You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 50f367f
Browse filesBrowse the repository at this point in the historyBrowse files
[SPARK-59854][SQL] Report the column in decoding errors of the vectorized Parquet reader
### What changes were proposed in this pull request?
This is a follow-up of SPARK-59832 (#59117). That change added the page and the column to errors from initializing a data page in `VectorizedColumnReader`. This PR adds the column to the decoding errors that still do not report it:
- **Decoding a batch.** `readBatch` wraps a `ParquetDecodingException` thrown while decoding values or levels in a `ParquetDecodingException` that names the page and the column: `could not decode page <page> in col <column>: <original message>`. This covers the `ParquetDecodingException`s of the RLE, PLAIN, DELTA_BINARY_PACKED, DELTA_LENGTH_BYTE_ARRAY, DELTA_BYTE_ARRAY and BYTE_STREAM_SPLIT readers. Only `ParquetDecodingException` is caught, so other exceptions keep their types: `SchemaColumnConvertNotSupportedException` and `SparkUpgradeException`, which `FileDataSourceV2.attachFilePath` dispatches on, and the `RuntimeException` of a vector that cannot grow, which `ParquetStorageFilter.rethrowIfMustPropagate` dispatches on. The reader keeps only the description of the current page, not the page, because for codecs that decompress lazily the page holds the decompressor stream.
- **Reading a data page.** parquet-java decompresses ZSTD pages eagerly, inside `PageReader.readPage()`, which was outside every catch. Its `ParquetDecodingException` (`could not decompress page`) now gets the column. Only `ParquetDecodingException` is caught here, because `readPage()` also decrypts pages, and decryption errors are not decoding errors.
- **Reading the dictionary.** `readDictionaryPage()` and `Encoding.initDictionary` now share one `try`, so a dictionary page that cannot be read fails with `could not decode the dictionary for <column>` whether it is decompressed eagerly (ZSTD) or when `initDictionary` reads its bytes (SNAPPY, GZIP). Besides `IOException` and `ParquetDecodingException`, it wraps the three runtime exceptions a corrupted dictionary page or page header causes there: an `IndexOutOfBoundsException` when a BINARY dictionary ends within the length of an entry, a `NegativeArraySizeException` for a negative size, and an `UnsupportedOperationException` for an encoding that has no dictionary. Other runtime exceptions are not wrapped, so a bug in the reader is not reported as a corrupt dictionary.
- **DELTA_BINARY_PACKED.** `VectorizedDeltaBinaryPackedReader.initFromPage` checked the page value count and the mini block size with `checkArgument`, which throws an `IllegalArgumentException`. These checks now throw a `ParquetDecodingException`, like the block size checks next to them, and the mini block size is checked with integer arithmetic. A mini block bit width above 64, which failed with an `ArrayIndexOutOfBoundsException`, is now rejected with a `ParquetDecodingException` when the mini block is unpacked. The existing page wrappers and the new batch wrapper then add the page and the column.
- **An empty column chunk.** The constructor's `totalValueCount == 0` error, which a corrupt footer can cause, now names the column as well.
Each wrapper keeps the original message and adds a single layer to the cause chain.
These cases are not covered:
- An out-of-range dictionary id fails with an `ArrayIndexOutOfBoundsException` from `Dictionary.decodeTo*`, either in `readBatch` or, with lazy dictionary decoding, after `readBatch` returns. Reporting the column for it needs a check of every id against the dictionary size, so it is left to SPARK-60317.
- A dictionary page whose declared size or entry lengths do not fit in the page can end in an `OutOfMemoryError` or, for an uncompressed chunk, return bytes of the next page. That is left to SPARK-60316.
- A corrupt LZ4_RAW or LZ4 page fails with a `RuntimeException` of the codec library, aircompressor's `MalformedInputException` or lz4-java's `LZ4Exception`, which parquet-java does not turn into an `IOException`.
- A negative uncompressed size in the header of a compressed data page fails with a `NegativeArraySizeException`, in `readPage()` for ZSTD and LZ4_RAW and in `readPageV1` or `readPageV2` for the other codecs.
- With a pushed filter, parquet-java's `DictionaryFilter` reads dictionary pages while it filters row groups, before any `VectorizedColumnReader` exists, so a dictionary page that cannot be read there fails without the column.
### Why are the changes needed?
Without the column, a decoding error from a corrupted file does not say which column is affected. For example, a truncated RLE run fails with only `Corrupted RLE data: reading past the end of the encoded values`. When `VectorizedColumnReader` reads a SNAPPY or GZIP data page, a decompression error already reports the column, because the page is decompressed lazily inside the existing catch. ZSTD pages did not report it.
### Does this PR introduce _any_ user-facing change?
Yes, in error messages only. The decoding errors above now name the column, and the page for errors while decoding a batch. The corrupt dictionaries above now fail with a `ParquetDecodingException` that names the column instead of an `IndexOutOfBoundsException`, `NegativeArraySizeException` or `UnsupportedOperationException`, which is kept as the cause. The DELTA_BINARY_PACKED checks now throw a `ParquetDecodingException` instead of the `IllegalArgumentException` of the value count and mini block size checks, or the `ArrayIndexOutOfBoundsException` of a bit width above 64, and the page and the column are added to it. A ZSTD dictionary page that cannot be decompressed now fails with `could not decode the dictionary for <column>`. Valid files are not affected.
### How was this patch tested?
New tests:
- `ParquetVectorizedSuite`:
- A truncated RLE dictionary id section and truncated PLAIN values report the page and the column, with the original `ParquetDecodingException` as the cause.
- A truncated BINARY dictionary, a negative dictionary size and a dictionary page with a non-dictionary encoding report the column.
- A mini block bit width of 65 in a DELTA_BINARY_PACKED page reports the page and the column, like the truncated pages above, and an invalid DELTA_BINARY_PACKED header reports the column.
- A column chunk without values reports the column.
- `ParquetDeltaEncodingSuite`: mini blocks that are not a multiple of 8 values and a page with no values are rejected with a `ParquetDecodingException`.
- `ParquetIOSuite`: a ZSTD data page and a ZSTD dictionary page that cannot be decompressed report the column. The test shares a helper with the SPARK-59832 tests, including their timeout.
All new tests fail without this change. Existing suites also pass: `VectorizedRleValuesReaderSuite`, `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaEncodingSuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite`, `VectorizedPlainValuesReaderSuite`, `ParquetVectorUpdaterSuite` and `ParquetCompressionCodecPrecedenceSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
This pull request and its description were written by Isaac.
Closes#59542 from viirya/SPARK-59854.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 8c9aa48)
Signed-off-by: Dongjoon Hyun <[email protected]>
Copy file name to clipboardExpand all lines: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedDeltaBinaryPackedReader.java
Copy file name to clipboardExpand all lines: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetDeltaEncodingSuite.scala
+19Lines changed: 19 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -204,6 +204,25 @@ abstract class ParquetDeltaEncodingSuite[T] extends ParquetCompatibilityTest
204
204
reader.initFromPage(100, craftHeader(128, 4, -1))
205
205
}
206
206
assert(e3.getMessage.contains("total value count"))
207
+
208
+
// The mini blocks must have a multiple of 8 values: 128 values do not split into 3 mini
209
+
// blocks, and split into 32 mini blocks of 4 values.
0 commit comments