Skip to content

Commit 605deea

Browse files
committed
[SPARK-59832][SQL] Fail on truncated input in the vectorized RLE reader
### What changes were proposed in this pull request? This is a follow-up of SPARK-59812. It makes `VectorizedRleValuesReader`, which decodes definition/repetition levels, dictionary ids and RLE booleans, fail with a `ParquetDecodingException` when the encoded data ends early or a header is invalid. - `readNextGroup()` throws when there is no encoded data left, instead of returning false. Callers never read more values than the page declares, so running out of data always means a corrupt page. All the `if (currentCount == 0 && !readNextGroup()) break;` loop guards become `if (currentCount == 0) readNextGroup();`. With a bit width of 0, `initFromPage` already provides every value of the page, so reaching `readNextGroup()` also means reading past the end. - `readNextGroup()` skips runs of length 0, so `currentCount` is always positive when it returns. The existing group decoding moves unchanged into a `readGroup()` helper that `readNextGroup()` calls in a loop. The Parquet spec does not allow runs of length 0; skipping them keeps what the batch paths already did, so no file that could be read before fails because of them. - For a bit-packed run, check that `numGroups * 8` does not overflow and that `numGroups * bitWidth` bytes are left before allocating the value buffer. - `initFromPage` rejects a level length that is cut off, negative or larger than the bytes left in the page, and a dictionary id bit width above 32. - `VectorizedColumnReader` now also adds the page or dictionary and the column to a `ParquetDecodingException` thrown while initializing a page (in `readPageV1` and `readPageV2`) or decoding the dictionary page, as it already did for an `IOException`, so a corrupt length still reports the column. It wraps it in a new `ParquetDecodingException` that includes the original message, so the exception type stays the same. - A read error inside a run header or an RLE value (e.g. the data ends in the middle of a header) now also starts with "Corrupted RLE data". - A dictionary-encoded page with no data at all (not even the bit width byte) no longer decodes every value as id 0. Reading from it fails, like parquet-java's `DictionaryValuesReader` ("Attempt to read from empty page"). An all-null page, which reads no ids, still works. The added checks run once per run or page, not per value. ### Why are the changes needed? SPARK-34863 made `readNextGroup()` return false at the end of the stream. Before that, it called `in.read()`, which throws an `EOFException` that was wrapped in a `ParquetDecodingException`. Since then, on a corrupt page: - The batch read and skip loops stop early, so dictionary ids are only partially read or skipped, and the query returns wrong results without an error. - For definition and repetition levels, `readBatch` and `readBatchRepeated` return without making progress, and `VectorizedColumnReader.readBatch` loops forever because `rowsToReadInBatch` and `valuesToReadInPage` never change. - For repeated columns, when only the definition levels end early, the missing definition levels are silently left as 0, which produces wrong nested values. - `readBooleans` ignores the return value, and loops forever at the end of the stream. - `readInteger` ignores the return value and decrements `currentCount` below 0, returning the last run's value again. It also returns the value of a run of length 0 instead of skipping it. - A bit-packed header with a large group count overflows `numGroups * 8` and `numGroups * bitWidth` (the latter was added by SPARK-56895). `in.slice` with a negative length moves a `SingleBufferInputStream` backwards, and a large count allocates up to an `int[2^31 - 8]` buffer before noticing that the bytes are missing. - In Parquet 1.18.1, `SingleBufferInputStream.sliceBuffers` only rejects `length > remaining`, so a negative level length moves the stream backwards and the next section of the page is decoded from the wrong offset. When the encoded data ends early, the non-vectorized reader already fails: parquet-java's `RunLengthBitPackingHybridDecoder` throws "Reading past RLE/BitPacking stream.". This PR does not change how a last bit-packed run that is shorter than its padding is handled. Some non-compliant writers produce it (e.g. Polars before pola-rs/polars#13883). The vectorized reader already rejected it before this change, because `in.slice` threw an `EOFException`, while parquet-java and Arrow C++ (apache/arrow#47992) accept it when the bytes cover the values that are read. Accepting it is left for a follow-up, SPARK-59853. A bit-packed run whose bytes are all present is still not bounded by the number of values in the page, so a corrupt header can still make the reader allocate a buffer much larger than the page needs. Rejecting such runs would also fail non-compliant pages that decode correctly today (e.g. 9 values written as a run of 3 groups), so bounding them is left for a follow-up. ### Does this PR introduce _any_ user-facing change? Only for corrupt or non-compliant Parquet files: reading a page whose levels, dictionary ids or RLE booleans end early, or have an invalid length, bit width or bit-packed header, now fails with a `ParquetDecodingException` instead of returning wrong values, spinning forever, or allocating a buffer for a bit-packed run whose bytes are missing. Valid files are not affected. With `spark.sql.files.ignoreCorruptFiles=true`, the rest of such a file is now skipped. A few of these pages could be read before and give the right result by chance, and now fail: - Dictionary ids (or RLE boolean values) that end early, when every row after the cut is skipped by column index filtering, e.g. the rest of the page before a row range in a later page. The skip stopped silently at the end of the data. This was possible since 3.3.0 (SPARK-34863); 3.2 failed the same way as this PR. - A dictionary-encoded page whose data is empty (no bit width byte, which the spec requires), when all its non-null rows are skipped, or when the dictionary has a single entry, so that decoding every id as 0 happened to be right. This was the behavior since the vectorized reader was added in 2.0.0. Selecting rows after the cut, or a dictionary with more than one entry, returns wrong values for the same pages without this PR. With `spark.sql.files.ignoreCorruptFiles=true`, such a file is no longer read in full: from the batch that reaches such a page, the rest of the file, including later row groups, is skipped with only a warning in the log. ### How was this patch tested? New unit tests in `VectorizedRleValuesReaderSuite`. The ones on hand-built dictionary id, boolean and length sections (9 of 14) run both on a single buffer and on the same data split into 1-byte buffers, so that every part of the data longer than 1 byte, including a slice of it, is read from a `MultiBufferInputStream`. Negative tests check the expected message, and the ones that read values run on a separate thread with a timeout, since some looped forever before this change. - Truncated dictionary ids on `readIntegers`, `skipIntegers` and `readInteger`; truncated booleans on `readBooleans`; truncated definition levels on `readBatch` (with and without definition level output, and with row ranges that skip past the truncation); truncated repetition levels on `readBatchRepeated`; truncated definition levels on `readBatchRepeated` (with and without row ranges); reading past a page with bit width 0. - Cut-off, negative and oversized (up to `Int.MaxValue`) level lengths, and dictionary id bit widths above 32. - Bit-packed runs that are truncated, that overflow `currentCount` to 0 or a negative value, or that would allocate a huge buffer. - Data ending inside a run header or an RLE value. - A dictionary-encoded page with no data, and runs of length 0. - Positive cases: reads and skips that end exactly at the end of the encoded values for mixed RLE and bit-packed dictionary ids and booleans, with a level length that covers exactly the rest of the page. New end-to-end tests in `ParquetIOSuite` corrupt a written file: a shortened definition level run fails instead of hanging, and an oversized length of the definition levels (V1 page) or of RLE boolean values (V2 page) reports the page and column once. A new test in `ParquetVectorizedSuite` checks that a corrupted dictionary page reports the column. All the new negative tests fail without this change. Existing suites also pass: `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite`, `VectorizedPlainValuesReaderSuite` and `ParquetVectorUpdaterSuite`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #59117 from viirya/SPARK-59832-rle-truncated. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent de8bfbe commit 605deea

6 files changed

Lines changed: 530 additions & 29 deletions

File tree

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
import org.apache.parquet.column.values.RequiresPreviousReader;
3434
import org.apache.parquet.column.values.ValuesReader;
3535
import org.apache.parquet.filter2.columnindex.RowRanges;
36+
import org.apache.parquet.io.ParquetDecodingException;
3637
import org.apache.parquet.schema.LogicalTypeAnnotation;
3738
import org.apache.parquet.schema.LogicalTypeAnnotation.DateLogicalTypeAnnotation;
3839
import org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
@@ -153,6 +154,9 @@ public VectorizedColumnReader(
153154
this.isCurrentPageDictionaryEncoded = true;
154155
} catch (IOException e) {
155156
throw new IOException("could not decode the dictionary for " + descriptor, e);
157+
} catch (ParquetDecodingException e) {
158+
throw new ParquetDecodingException(
159+
"could not decode the dictionary for " + descriptor + ": " + e.getMessage(), e);
156160
}
157161
} else {
158162
this.dictionary = null;
@@ -364,9 +368,7 @@ private void initDataReader(
364368
if (dataEncoding.usesDictionary()) {
365369
this.dataColumn = null;
366370
if (dictionary == null) {
367-
throw new IOException(
368-
"could not read page in col " + descriptor +
369-
" as the dictionary was missing for encoding " + dataEncoding);
371+
throw new IOException("the dictionary was missing for encoding " + dataEncoding);
370372
}
371373
@SuppressWarnings("deprecation")
372374
Encoding plainDict = Encoding.PLAIN_DICTIONARY; // var to allow warning suppression
@@ -381,11 +383,9 @@ private void initDataReader(
381383
this.isCurrentPageDictionaryEncoded = false;
382384
}
383385

384-
try {
385-
dataColumn.initFromPage(pageValueCount, in);
386-
} catch (IOException e) {
387-
throw new IOException("could not read page in col " + descriptor, e);
388-
}
386+
// readPageV1 and readPageV2 add the page and the column to an IOException or a
387+
// ParquetDecodingException thrown here.
388+
dataColumn.initFromPage(pageValueCount, in);
389389
// for PARQUET-246 (See VectorizedDeltaByteArrayReader.setPreviousValues)
390390
if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
391391
previousReader instanceof RequiresPreviousReader) {
@@ -455,6 +455,9 @@ private int readPageV1(DataPageV1 page) throws IOException {
455455
return pageValueCount;
456456
} catch (IOException e) {
457457
throw new IOException("could not read page " + page + " in col " + descriptor, e);
458+
} catch (ParquetDecodingException e) {
459+
throw new ParquetDecodingException(
460+
"could not read page " + page + " in col " + descriptor + ": " + e.getMessage(), e);
458461
}
459462
}
460463

@@ -475,6 +478,9 @@ private int readPageV2(DataPageV2 page) throws IOException {
475478
return pageValueCount;
476479
} catch (IOException e) {
477480
throw new IOException("could not read page " + page + " in col " + descriptor, e);
481+
} catch (ParquetDecodingException e) {
482+
throw new ParquetDecodingException(
483+
"could not read page " + page + " in col " + descriptor + ": " + e.getMessage(), e);
478484
}
479485
}
480486
}

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java‎

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ private enum MODE {
6262
private int bytesWidth;
6363
private BytePacker packer;
6464

65+
// True if the dictionary id section of the page is empty, not even holding the bit width.
66+
private boolean emptyIdSection;
67+
6568
// Current decoding mode and values
6669
private MODE mode;
6770
private int currentCount;
@@ -96,16 +99,35 @@ public VectorizedRleValuesReader(int bitWidth, boolean readLength) {
9699
@Override
97100
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
98101
this.in = in;
102+
this.emptyIdSection = false;
99103
if (fixedWidth) {
100104
// Initialize for repetition and definition levels
101105
if (readLength) {
106+
if (in.available() < 4) {
107+
throw new ParquetDecodingException("Corrupted RLE data: the 4-byte length is cut off, " +
108+
"only " + in.available() + " bytes are left in the page");
109+
}
102110
int length = readIntLittleEndian();
111+
if (length < 0 || length > in.available()) {
112+
throw new ParquetDecodingException("Corrupted RLE data: invalid length " + length +
113+
", " + in.available() + " bytes are left in the page");
114+
}
103115
this.in = in.sliceStream(length);
104116
}
105117
} else {
106118
// Initialize for values
107119
if (in.available() > 0) {
108-
init(in.read());
120+
int width = in.read();
121+
if (width > 32) {
122+
throw new ParquetDecodingException("Corrupted RLE data: invalid bit width " + width);
123+
}
124+
init(width);
125+
} else {
126+
// No encoded values (e.g. an all-null page). Do not treat this as a bit width of 0,
127+
// which would decode every value as 0; any read fails in `readNextGroup` instead.
128+
this.emptyIdSection = true;
129+
this.currentCount = 0;
130+
return;
109131
}
110132
}
111133
if (bitWidth == 0) {
@@ -208,7 +230,7 @@ private void readBatchInternal(
208230
int leftInPage = state.valuesToReadInPage;
209231

210232
while (leftInBatch > 0 && leftInPage > 0) {
211-
if (currentCount == 0 && !readNextGroup()) break;
233+
if (currentCount == 0) readNextGroup();
212234
int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount));
213235

214236
long rangeStart = state.currentRangeStart();
@@ -318,7 +340,7 @@ private void readBatchInternalWithDefLevels(
318340
int leftInPage = state.valuesToReadInPage;
319341

320342
while (leftInBatch > 0 && leftInPage > 0) {
321-
if (currentCount == 0 && !readNextGroup()) break;
343+
if (currentCount == 0) readNextGroup();
322344
int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount));
323345

324346
long rangeStart = state.currentRangeStart();
@@ -431,7 +453,7 @@ public void readBatchRepeatedInternal(
431453
values, nulls, valuesReused, valueReader, updater);
432454

433455
while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
434-
if (currentCount == 0 && !readNextGroup()) break;
456+
if (currentCount == 0) readNextGroup();
435457

436458
// Values to read in the current RLE/PACKED block, must be <= what's left in the page
437459
int valuesLeftInBlock = Math.min(leftInPage, currentCount);
@@ -637,7 +659,7 @@ private void readValues(
637659
int n = total;
638660
int initialValueOffset = state.valueOffset;
639661
while (n > 0) {
640-
if (currentCount == 0 && !readNextGroup()) break;
662+
if (currentCount == 0) readNextGroup();
641663
int num = Math.min(n, this.currentCount);
642664
readValuesN(num, state, defLevels, values, nulls, valueReader, updater);
643665
state.levelOffset += num;
@@ -731,7 +753,7 @@ private void skipValues(
731753
VectorizedValuesReader valuesReader,
732754
ParquetVectorUpdater updater) {
733755
while (n > 0) {
734-
if (currentCount == 0 && !readNextGroup()) break;
756+
if (currentCount == 0) readNextGroup();
735757
int num = Math.min(n, this.currentCount);
736758
switch (mode) {
737759
case RLE -> {
@@ -764,7 +786,7 @@ private void skipValues(
764786
public void readIntegers(int total, WritableColumnVector c, int rowId) {
765787
int left = total;
766788
while (left > 0) {
767-
if (currentCount == 0 && !readNextGroup()) break;
789+
if (currentCount == 0) readNextGroup();
768790
int n = Math.min(left, this.currentCount);
769791
switch (mode) {
770792
case RLE -> c.putInts(rowId, n, currentValue);
@@ -957,7 +979,6 @@ private int readIntLittleEndian() throws IOException {
957979
*/
958980
private int readIntLittleEndianPaddedOnBitWidth() throws IOException {
959981
return switch (bytesWidth) {
960-
case 0 -> 0;
961982
case 1 -> in.read();
962983
case 2 -> {
963984
int ch2 = in.read();
@@ -976,14 +997,30 @@ private int readIntLittleEndianPaddedOnBitWidth() throws IOException {
976997
}
977998

978999
/**
979-
* Reads the next group. Returns false if no more group available.
1000+
* Reads the next non-empty group, so that `currentCount` is positive on return.
1001+
* <p>
1002+
* Callers never read more values than the page declares, so running out of encoded data here
1003+
* means the page is corrupted. With a bit width of 0, `initFromPage` already provides all the
1004+
* values of the page, so reaching here also means reading past the end.
9801005
*/
981-
private boolean readNextGroup() {
982-
if (in.available() <= 0) {
983-
currentCount = 0;
984-
return false;
1006+
private void readNextGroup() {
1007+
if (emptyIdSection) {
1008+
throw new ParquetDecodingException(
1009+
"Corrupted RLE data: the dictionary id section is empty, not even the bit width");
9851010
}
1011+
do {
1012+
if (bitWidth == 0 || in.available() <= 0) {
1013+
throw new ParquetDecodingException(
1014+
"Corrupted RLE data: reading past the end of the encoded values");
1015+
}
1016+
readGroup();
1017+
} while (this.currentCount == 0);
1018+
}
9861019

1020+
/**
1021+
* Reads the next group, which may be empty.
1022+
*/
1023+
private void readGroup() {
9871024
try {
9881025
int header = readUnsignedVarInt();
9891026
this.mode = (header & 1) == 0 ? MODE.RLE : MODE.PACKED;
@@ -994,6 +1031,17 @@ private boolean readNextGroup() {
9941031
}
9951032
case PACKED -> {
9961033
int numGroups = header >>> 1;
1034+
// Validate before allocating the buffer, so a corrupted header can neither overflow
1035+
// `currentCount` nor make us allocate more than the remaining bytes can fill.
1036+
if (numGroups > Integer.MAX_VALUE / 8) {
1037+
throw new ParquetDecodingException(
1038+
"Corrupted RLE data: bit-packed run of " + numGroups + " groups is too long");
1039+
}
1040+
if ((long) numGroups * bitWidth > in.available()) {
1041+
throw new ParquetDecodingException("Corrupted RLE data: bit-packed run of " +
1042+
numGroups + " groups needs " + (long) numGroups * bitWidth + " bytes, but only " +
1043+
in.available() + " bytes are left");
1044+
}
9971045
this.currentCount = numGroups * 8;
9981046

9991047
if (this.currentBuffer.length < this.currentCount) {
@@ -1014,10 +1062,8 @@ private boolean readNextGroup() {
10141062
}
10151063
}
10161064
} catch (IOException e) {
1017-
throw new ParquetDecodingException("Failed to read from input stream", e);
1065+
throw new ParquetDecodingException("Corrupted RLE data: failed to read from input stream", e);
10181066
}
1019-
1020-
return true;
10211067
}
10221068

10231069
/**
@@ -1026,7 +1072,7 @@ private boolean readNextGroup() {
10261072
private void skipValues(int n) {
10271073
int left = n;
10281074
while (left > 0) {
1029-
if (this.currentCount == 0 && !readNextGroup()) break;
1075+
if (this.currentCount == 0) readNextGroup();
10301076
int num = Math.min(left, this.currentCount);
10311077
switch (mode) {
10321078
case RLE -> {}

‎sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.spark.sql.execution.datasources.parquet
1919

2020
import java.math.{BigDecimal => JBigDecimal}
21+
import java.nio.file.Files
2122
import java.time.{LocalDateTime, LocalTime}
2223
import java.util.Locale
2324

@@ -27,16 +28,21 @@ import scala.reflect.ClassTag
2728
import scala.reflect.runtime.universe.TypeTag
2829

2930
import com.google.common.primitives.UnsignedLong
31+
import org.apache.commons.lang3.exception.ExceptionUtils
3032
import org.apache.hadoop.fs.{FileSystem, Path}
3133
import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}
34+
import org.apache.parquet.bytes.BytesUtils
3235
import org.apache.parquet.column.{Encoding, ParquetProperties}
3336
import org.apache.parquet.column.ParquetProperties.WriterVersion.PARQUET_1_0
3437
import org.apache.parquet.example.data.Group
3538
import org.apache.parquet.example.data.simple.{SimpleGroup, SimpleGroupFactory}
3639
import org.apache.parquet.hadoop._
3740
import org.apache.parquet.hadoop.example.ExampleParquetWriter
41+
import org.apache.parquet.io.ParquetDecodingException
3842
import org.apache.parquet.io.api.Binary
3943
import org.apache.parquet.schema.{MessageType, MessageTypeParser}
44+
import org.scalatest.concurrent.{Signaler, ThreadSignaler}
45+
import org.scalatest.time.{Seconds, Span}
4046

4147
import org.apache.spark.{SPARK_VERSION_SHORT, SparkException, TestUtils}
4248
import org.apache.spark.rdd.RDD
@@ -2480,6 +2486,89 @@ class ParquetIOSuite extends ParquetTest with SharedSparkSession {
24802486
}
24812487
}
24822488
}
2489+
2490+
/**
2491+
* Writes 100 rows of `column` to a single uncompressed data page, lets `corrupt` modify the
2492+
* RLE section that starts with `03 00 00 00 C8 01 01` (the 4-byte length 3, then one RLE run
2493+
* with header 100 << 1 and value 1), and returns the error of reading the file with the
2494+
* vectorized reader. For a nullable INT32 column that is the definition levels; for a
2495+
* BOOLEAN column written as a V2 page it is the RLE-encoded values.
2496+
*/
2497+
private def readCorruptedRleSection(
2498+
column: String,
2499+
writerVersion: String = "PARQUET_1_0")(
2500+
corrupt: (Array[Byte], Int) => Unit): Throwable = {
2501+
var error: Throwable = null
2502+
withTempPath { dir =>
2503+
val path = dir.getCanonicalPath
2504+
spark.range(100).selectExpr(column).coalesce(1)
2505+
.write
2506+
.option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
2507+
.option(ParquetOutputFormat.WRITER_VERSION, writerVersion)
2508+
.option("compression", "uncompressed")
2509+
.parquet(path)
2510+
val file = dir.listFiles().filter(_.getName.endsWith(".parquet")).head
2511+
val bytes = Files.readAllBytes(file.toPath)
2512+
val section = Array[Byte](3, 0, 0, 0, 0xC8.toByte, 0x01, 1)
2513+
val offset = bytes.indexOfSlice(section)
2514+
assert(offset >= 0 && bytes.indexOfSlice(section, offset + 1) < 0)
2515+
corrupt(bytes, offset)
2516+
Files.write(file.toPath, bytes)
2517+
dir.listFiles().filter(_.getName.endsWith(".crc")).foreach(_.delete())
2518+
2519+
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true") {
2520+
// Before SPARK-59832 a truncated read never returned, so run it with a timeout. The test
2521+
// thread waits interruptibly for the job, so it can be signaled directly.
2522+
implicit val signaler: Signaler = ThreadSignaler
2523+
error = intercept[SparkException] {
2524+
failAfter(Span(60, Seconds))(spark.read.parquet(path).collect())
2525+
}
2526+
}
2527+
}
2528+
error
2529+
}
2530+
2531+
private def causes(e: Throwable): Seq[Throwable] =
2532+
ExceptionUtils.getThrowableList(e).asScala.toSeq
2533+
2534+
/** Sets the 4-byte length at `offset` to Int.MaxValue, far more than the rest of the page. */
2535+
private def setMaxLength(bytes: Array[Byte], offset: Int): Unit =
2536+
BytesUtils.intToBytes(Int.MaxValue).copyToArray(bytes, offset)
2537+
2538+
/**
2539+
* Checks that the page error is reported once with the column, directly on top of the
2540+
* `ParquetDecodingException` about the invalid length.
2541+
*/
2542+
private def assertInvalidLengthReportsColumn(e: Throwable, column: String): Unit = {
2543+
val pageErrors =
2544+
causes(e).filter(c => Option(c.getMessage).exists(_.startsWith("could not read page")))
2545+
assert(pageErrors.size == 1, e)
2546+
assert(pageErrors.head.getMessage.contains(s"in col [$column]"), e)
2547+
assert(pageErrors.head.getCause.isInstanceOf[ParquetDecodingException], e)
2548+
assert(pageErrors.head.getCause.getMessage.contains(
2549+
s"Corrupted RLE data: invalid length ${Int.MaxValue}"), e)
2550+
}
2551+
2552+
test("SPARK-59832: vectorized reader fails on truncated definition levels") {
2553+
// Shorten the run to 64 values (header 64 << 1 = varint 0x80 0x01).
2554+
val e = readCorruptedRleSection("if(id < 0, null, cast(id as int)) as a") {
2555+
(bytes, offset) => bytes(offset + 4) = 0x80.toByte
2556+
}
2557+
assert(causes(e).exists {
2558+
case c: ParquetDecodingException => c.getMessage.contains("Corrupted RLE data")
2559+
case _ => false
2560+
}, e)
2561+
}
2562+
2563+
test("SPARK-59832: invalid definition level length reports the column") {
2564+
val e = readCorruptedRleSection("if(id < 0, null, cast(id as int)) as a")(setMaxLength)
2565+
assertInvalidLengthReportsColumn(e, "a")
2566+
}
2567+
2568+
test("SPARK-59832: invalid length of RLE booleans in a V2 page reports the column") {
2569+
val e = readCorruptedRleSection("id >= 0 as b", writerVersion = "PARQUET_2_0")(setMaxLength)
2570+
assertInvalidLengthReportsColumn(e, "b")
2571+
}
24832572
}
24842573

24852574
class JobCommitFailureParquetOutputCommitter(outputPath: Path, context: TaskAttemptContext)

0 commit comments

Comments
 (0)