Skip to content

Commit 45f1cc8

Browse files
viiryadongjoon-hyun
authored andcommitted
[SPARK-59832][4.X][SQL] Fail on truncated input in the vectorized RLE reader
### What changes were proposed in this pull request? Backport of #59117 to branch-4.x. The cherry-pick conflicted only on code that SPARK-59620 added to master, which is not on this branch: an import in `VectorizedColumnReader` and `VectorizedRleValuesReaderSuite`, and a doc comment next to the new `readBatchRepeated` bridge in `ParquetTestAccess`. They are left out; otherwise the change is the same as on master. This commit also applies cleanly to branch-4.3, so it can be cherry-picked there when merging. The same change was built and tested on branch-4.3. ### Why are the changes needed? Same as #59117. On a corrupt page, the vectorized RLE reader, which decodes definition/repetition levels, dictionary ids and RLE booleans, silently returned wrong values or looped forever when the encoded data ended early, and a corrupt bit-packed header could overflow `currentCount` or allocate a huge buffer. The reader now fails with a `ParquetDecodingException`. ### Does this PR introduce _any_ user-facing change? Only for corrupt or non-compliant Parquet files, as described in #59117: such pages now fail with a `ParquetDecodingException` instead of returning wrong values or hanging, and with `spark.sql.files.ignoreCorruptFiles=true` the rest of such a file is skipped. A few of these pages could be read before and gave the right result by chance; #59117 lists them. ### How was this patch tested? The 18 tests added in #59117 pass on this branch, as do the existing tests in `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite`, `VectorizedPlainValuesReaderSuite` and `ParquetVectorUpdaterSuite`. ``` build/sbt 'sql/testOnly *VectorizedRleValuesReaderSuite *ParquetEncodingSuite *ParquetVectorizedSuite *ParquetIOSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #59481 from viirya/SPARK-59832-branch-4.x. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 212ff5a commit 45f1cc8

6 files changed

Lines changed: 530 additions & 29 deletions

File tree

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
import org.apache.parquet.column.page.*;
3333
import org.apache.parquet.column.values.RequiresPreviousReader;
3434
import org.apache.parquet.column.values.ValuesReader;
35+
import org.apache.parquet.io.ParquetDecodingException;
3536
import org.apache.parquet.schema.LogicalTypeAnnotation;
3637
import org.apache.parquet.schema.LogicalTypeAnnotation.DateLogicalTypeAnnotation;
3738
import org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
@@ -132,6 +133,9 @@ public VectorizedColumnReader(
132133
this.isCurrentPageDictionaryEncoded = true;
133134
} catch (IOException e) {
134135
throw new IOException("could not decode the dictionary for " + descriptor, e);
136+
} catch (ParquetDecodingException e) {
137+
throw new ParquetDecodingException(
138+
"could not decode the dictionary for " + descriptor + ": " + e.getMessage(), e);
135139
}
136140
} else {
137141
this.dictionary = null;
@@ -343,9 +347,7 @@ private void initDataReader(
343347
if (dataEncoding.usesDictionary()) {
344348
this.dataColumn = null;
345349
if (dictionary == null) {
346-
throw new IOException(
347-
"could not read page in col " + descriptor +
348-
" as the dictionary was missing for encoding " + dataEncoding);
350+
throw new IOException("the dictionary was missing for encoding " + dataEncoding);
349351
}
350352
@SuppressWarnings("deprecation")
351353
Encoding plainDict = Encoding.PLAIN_DICTIONARY; // var to allow warning suppression
@@ -360,11 +362,9 @@ private void initDataReader(
360362
this.isCurrentPageDictionaryEncoded = false;
361363
}
362364

363-
try {
364-
dataColumn.initFromPage(pageValueCount, in);
365-
} catch (IOException e) {
366-
throw new IOException("could not read page in col " + descriptor, e);
367-
}
365+
// readPageV1 and readPageV2 add the page and the column to an IOException or a
366+
// ParquetDecodingException thrown here.
367+
dataColumn.initFromPage(pageValueCount, in);
368368
// for PARQUET-246 (See VectorizedDeltaByteArrayReader.setPreviousValues)
369369
if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
370370
previousReader instanceof RequiresPreviousReader) {
@@ -434,6 +434,9 @@ private int readPageV1(DataPageV1 page) throws IOException {
434434
return pageValueCount;
435435
} catch (IOException e) {
436436
throw new IOException("could not read page " + page + " in col " + descriptor, e);
437+
} catch (ParquetDecodingException e) {
438+
throw new ParquetDecodingException(
439+
"could not read page " + page + " in col " + descriptor + ": " + e.getMessage(), e);
437440
}
438441
}
439442

@@ -454,6 +457,9 @@ private int readPageV2(DataPageV2 page) throws IOException {
454457
return pageValueCount;
455458
} catch (IOException e) {
456459
throw new IOException("could not read page " + page + " in col " + descriptor, e);
460+
} catch (ParquetDecodingException e) {
461+
throw new ParquetDecodingException(
462+
"could not read page " + page + " in col " + descriptor + ": " + e.getMessage(), e);
457463
}
458464
}
459465
}

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java‎

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ private enum MODE {
6262
private int bytesWidth;
6363
private BytePacker packer;
6464

65+
// True if the dictionary id section of the page is empty, not even holding the bit width.
66+
private boolean emptyIdSection;
67+
6568
// Current decoding mode and values
6669
private MODE mode;
6770
private int currentCount;
@@ -96,16 +99,35 @@ public VectorizedRleValuesReader(int bitWidth, boolean readLength) {
9699
@Override
97100
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
98101
this.in = in;
102+
this.emptyIdSection = false;
99103
if (fixedWidth) {
100104
// Initialize for repetition and definition levels
101105
if (readLength) {
106+
if (in.available() < 4) {
107+
throw new ParquetDecodingException("Corrupted RLE data: the 4-byte length is cut off, " +
108+
"only " + in.available() + " bytes are left in the page");
109+
}
102110
int length = readIntLittleEndian();
111+
if (length < 0 || length > in.available()) {
112+
throw new ParquetDecodingException("Corrupted RLE data: invalid length " + length +
113+
", " + in.available() + " bytes are left in the page");
114+
}
103115
this.in = in.sliceStream(length);
104116
}
105117
} else {
106118
// Initialize for values
107119
if (in.available() > 0) {
108-
init(in.read());
120+
int width = in.read();
121+
if (width > 32) {
122+
throw new ParquetDecodingException("Corrupted RLE data: invalid bit width " + width);
123+
}
124+
init(width);
125+
} else {
126+
// No encoded values (e.g. an all-null page). Do not treat this as a bit width of 0,
127+
// which would decode every value as 0; any read fails in `readNextGroup` instead.
128+
this.emptyIdSection = true;
129+
this.currentCount = 0;
130+
return;
109131
}
110132
}
111133
if (bitWidth == 0) {
@@ -208,7 +230,7 @@ private void readBatchInternal(
208230
int leftInPage = state.valuesToReadInPage;
209231

210232
while (leftInBatch > 0 && leftInPage > 0) {
211-
if (currentCount == 0 && !readNextGroup()) break;
233+
if (currentCount == 0) readNextGroup();
212234
int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount));
213235

214236
long rangeStart = state.currentRangeStart();
@@ -318,7 +340,7 @@ private void readBatchInternalWithDefLevels(
318340
int leftInPage = state.valuesToReadInPage;
319341

320342
while (leftInBatch > 0 && leftInPage > 0) {
321-
if (currentCount == 0 && !readNextGroup()) break;
343+
if (currentCount == 0) readNextGroup();
322344
int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount));
323345

324346
long rangeStart = state.currentRangeStart();
@@ -431,7 +453,7 @@ public void readBatchRepeatedInternal(
431453
values, nulls, valuesReused, valueReader, updater);
432454

433455
while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
434-
if (currentCount == 0 && !readNextGroup()) break;
456+
if (currentCount == 0) readNextGroup();
435457

436458
// Values to read in the current RLE/PACKED block, must be <= what's left in the page
437459
int valuesLeftInBlock = Math.min(leftInPage, currentCount);
@@ -637,7 +659,7 @@ private void readValues(
637659
int n = total;
638660
int initialValueOffset = state.valueOffset;
639661
while (n > 0) {
640-
if (currentCount == 0 && !readNextGroup()) break;
662+
if (currentCount == 0) readNextGroup();
641663
int num = Math.min(n, this.currentCount);
642664
readValuesN(num, state, defLevels, values, nulls, valueReader, updater);
643665
state.levelOffset += num;
@@ -731,7 +753,7 @@ private void skipValues(
731753
VectorizedValuesReader valuesReader,
732754
ParquetVectorUpdater updater) {
733755
while (n > 0) {
734-
if (currentCount == 0 && !readNextGroup()) break;
756+
if (currentCount == 0) readNextGroup();
735757
int num = Math.min(n, this.currentCount);
736758
switch (mode) {
737759
case RLE -> {
@@ -764,7 +786,7 @@ private void skipValues(
764786
public void readIntegers(int total, WritableColumnVector c, int rowId) {
765787
int left = total;
766788
while (left > 0) {
767-
if (currentCount == 0 && !readNextGroup()) break;
789+
if (currentCount == 0) readNextGroup();
768790
int n = Math.min(left, this.currentCount);
769791
switch (mode) {
770792
case RLE -> c.putInts(rowId, n, currentValue);
@@ -957,7 +979,6 @@ private int readIntLittleEndian() throws IOException {
957979
*/
958980
private int readIntLittleEndianPaddedOnBitWidth() throws IOException {
959981
return switch (bytesWidth) {
960-
case 0 -> 0;
961982
case 1 -> in.read();
962983
case 2 -> {
963984
int ch2 = in.read();
@@ -976,14 +997,30 @@ private int readIntLittleEndianPaddedOnBitWidth() throws IOException {
976997
}
977998

978999
/**
979-
* Reads the next group. Returns false if no more group available.
1000+
* Reads the next non-empty group, so that `currentCount` is positive on return.
1001+
* <p>
1002+
* Callers never read more values than the page declares, so running out of encoded data here
1003+
* means the page is corrupted. With a bit width of 0, `initFromPage` already provides all the
1004+
* values of the page, so reaching here also means reading past the end.
9801005
*/
981-
private boolean readNextGroup() {
982-
if (in.available() <= 0) {
983-
currentCount = 0;
984-
return false;
1006+
private void readNextGroup() {
1007+
if (emptyIdSection) {
1008+
throw new ParquetDecodingException(
1009+
"Corrupted RLE data: the dictionary id section is empty, not even the bit width");
9851010
}
1011+
do {
1012+
if (bitWidth == 0 || in.available() <= 0) {
1013+
throw new ParquetDecodingException(
1014+
"Corrupted RLE data: reading past the end of the encoded values");
1015+
}
1016+
readGroup();
1017+
} while (this.currentCount == 0);
1018+
}
9861019

1020+
/**
1021+
* Reads the next group, which may be empty.
1022+
*/
1023+
private void readGroup() {
9871024
try {
9881025
int header = readUnsignedVarInt();
9891026
this.mode = (header & 1) == 0 ? MODE.RLE : MODE.PACKED;
@@ -994,6 +1031,17 @@ private boolean readNextGroup() {
9941031
}
9951032
case PACKED -> {
9961033
int numGroups = header >>> 1;
1034+
// Validate before allocating the buffer, so a corrupted header can neither overflow
1035+
// `currentCount` nor make us allocate more than the remaining bytes can fill.
1036+
if (numGroups > Integer.MAX_VALUE / 8) {
1037+
throw new ParquetDecodingException(
1038+
"Corrupted RLE data: bit-packed run of " + numGroups + " groups is too long");
1039+
}
1040+
if ((long) numGroups * bitWidth > in.available()) {
1041+
throw new ParquetDecodingException("Corrupted RLE data: bit-packed run of " +
1042+
numGroups + " groups needs " + (long) numGroups * bitWidth + " bytes, but only " +
1043+
in.available() + " bytes are left");
1044+
}
9971045
this.currentCount = numGroups * 8;
9981046

9991047
if (this.currentBuffer.length < this.currentCount) {
@@ -1014,10 +1062,8 @@ private boolean readNextGroup() {
10141062
}
10151063
}
10161064
} catch (IOException e) {
1017-
throw new ParquetDecodingException("Failed to read from input stream", e);
1065+
throw new ParquetDecodingException("Corrupted RLE data: failed to read from input stream", e);
10181066
}
1019-
1020-
return true;
10211067
}
10221068

10231069
/**
@@ -1026,7 +1072,7 @@ private boolean readNextGroup() {
10261072
private void skipValues(int n) {
10271073
int left = n;
10281074
while (left > 0) {
1029-
if (this.currentCount == 0 && !readNextGroup()) break;
1075+
if (this.currentCount == 0) readNextGroup();
10301076
int num = Math.min(left, this.currentCount);
10311077
switch (mode) {
10321078
case RLE -> {}

‎sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
package org.apache.spark.sql.execution.datasources.parquet
1919

2020
import java.math.{BigDecimal => JBigDecimal}
21+
import java.nio.file.Files
2122
import java.time.{LocalDateTime, LocalTime}
2223
import java.util.Locale
2324

@@ -27,16 +28,21 @@ import scala.reflect.ClassTag
2728
import scala.reflect.runtime.universe.TypeTag
2829

2930
import com.google.common.primitives.UnsignedLong
31+
import org.apache.commons.lang3.exception.ExceptionUtils
3032
import org.apache.hadoop.fs.{FileSystem, Path}
3133
import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}
34+
import org.apache.parquet.bytes.BytesUtils
3235
import org.apache.parquet.column.{Encoding, ParquetProperties}
3336
import org.apache.parquet.column.ParquetProperties.WriterVersion.PARQUET_1_0
3437
import org.apache.parquet.example.data.Group
3538
import org.apache.parquet.example.data.simple.{SimpleGroup, SimpleGroupFactory}
3639
import org.apache.parquet.hadoop._
3740
import org.apache.parquet.hadoop.example.ExampleParquetWriter
41+
import org.apache.parquet.io.ParquetDecodingException
3842
import org.apache.parquet.io.api.Binary
3943
import org.apache.parquet.schema.{MessageType, MessageTypeParser}
44+
import org.scalatest.concurrent.{Signaler, ThreadSignaler}
45+
import org.scalatest.time.{Seconds, Span}
4046

4147
import org.apache.spark.{SPARK_VERSION_SHORT, SparkException, TestUtils}
4248
import org.apache.spark.rdd.RDD
@@ -2480,6 +2486,89 @@ class ParquetIOSuite extends ParquetTest with SharedSparkSession {
24802486
}
24812487
}
24822488
}
2489+
2490+
/**
2491+
* Writes 100 rows of `column` to a single uncompressed data page, lets `corrupt` modify the
2492+
* RLE section that starts with `03 00 00 00 C8 01 01` (the 4-byte length 3, then one RLE run
2493+
* with header 100 << 1 and value 1), and returns the error of reading the file with the
2494+
* vectorized reader. For a nullable INT32 column that is the definition levels; for a
2495+
* BOOLEAN column written as a V2 page it is the RLE-encoded values.
2496+
*/
2497+
private def readCorruptedRleSection(
2498+
column: String,
2499+
writerVersion: String = "PARQUET_1_0")(
2500+
corrupt: (Array[Byte], Int) => Unit): Throwable = {
2501+
var error: Throwable = null
2502+
withTempPath { dir =>
2503+
val path = dir.getCanonicalPath
2504+
spark.range(100).selectExpr(column).coalesce(1)
2505+
.write
2506+
.option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
2507+
.option(ParquetOutputFormat.WRITER_VERSION, writerVersion)
2508+
.option("compression", "uncompressed")
2509+
.parquet(path)
2510+
val file = dir.listFiles().filter(_.getName.endsWith(".parquet")).head
2511+
val bytes = Files.readAllBytes(file.toPath)
2512+
val section = Array[Byte](3, 0, 0, 0, 0xC8.toByte, 0x01, 1)
2513+
val offset = bytes.indexOfSlice(section)
2514+
assert(offset >= 0 && bytes.indexOfSlice(section, offset + 1) < 0)
2515+
corrupt(bytes, offset)
2516+
Files.write(file.toPath, bytes)
2517+
dir.listFiles().filter(_.getName.endsWith(".crc")).foreach(_.delete())
2518+
2519+
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true") {
2520+
// Before SPARK-59832 a truncated read never returned, so run it with a timeout. The test
2521+
// thread waits interruptibly for the job, so it can be signaled directly.
2522+
implicit val signaler: Signaler = ThreadSignaler
2523+
error = intercept[SparkException] {
2524+
failAfter(Span(60, Seconds))(spark.read.parquet(path).collect())
2525+
}
2526+
}
2527+
}
2528+
error
2529+
}
2530+
2531+
private def causes(e: Throwable): Seq[Throwable] =
2532+
ExceptionUtils.getThrowableList(e).asScala.toSeq
2533+
2534+
/** Sets the 4-byte length at `offset` to Int.MaxValue, far more than the rest of the page. */
2535+
private def setMaxLength(bytes: Array[Byte], offset: Int): Unit =
2536+
BytesUtils.intToBytes(Int.MaxValue).copyToArray(bytes, offset)
2537+
2538+
/**
2539+
* Checks that the page error is reported once with the column, directly on top of the
2540+
* `ParquetDecodingException` about the invalid length.
2541+
*/
2542+
private def assertInvalidLengthReportsColumn(e: Throwable, column: String): Unit = {
2543+
val pageErrors =
2544+
causes(e).filter(c => Option(c.getMessage).exists(_.startsWith("could not read page")))
2545+
assert(pageErrors.size == 1, e)
2546+
assert(pageErrors.head.getMessage.contains(s"in col [$column]"), e)
2547+
assert(pageErrors.head.getCause.isInstanceOf[ParquetDecodingException], e)
2548+
assert(pageErrors.head.getCause.getMessage.contains(
2549+
s"Corrupted RLE data: invalid length ${Int.MaxValue}"), e)
2550+
}
2551+
2552+
test("SPARK-59832: vectorized reader fails on truncated definition levels") {
2553+
// Shorten the run to 64 values (header 64 << 1 = varint 0x80 0x01).
2554+
val e = readCorruptedRleSection("if(id < 0, null, cast(id as int)) as a") {
2555+
(bytes, offset) => bytes(offset + 4) = 0x80.toByte
2556+
}
2557+
assert(causes(e).exists {
2558+
case c: ParquetDecodingException => c.getMessage.contains("Corrupted RLE data")
2559+
case _ => false
2560+
}, e)
2561+
}
2562+
2563+
test("SPARK-59832: invalid definition level length reports the column") {
2564+
val e = readCorruptedRleSection("if(id < 0, null, cast(id as int)) as a")(setMaxLength)
2565+
assertInvalidLengthReportsColumn(e, "a")
2566+
}
2567+
2568+
test("SPARK-59832: invalid length of RLE booleans in a V2 page reports the column") {
2569+
val e = readCorruptedRleSection("id >= 0 as b", writerVersion = "PARQUET_2_0")(setMaxLength)
2570+
assertInvalidLengthReportsColumn(e, "b")
2571+
}
24832572
}
24842573

24852574
class JobCommitFailureParquetOutputCommitter(outputPath: Path, context: TaskAttemptContext)

0 commit comments

Comments
 (0)