Skip to content

Commit eabee9e

Browse files
committed
[SPARK-59832][4.1][SQL] Fail on truncated input in the vectorized RLE reader
### What changes were proposed in this pull request? Backport of #59117 to branch-4.1. This branch does not have `VectorizedRleValuesReaderSuite`, `ParquetTestAccess` or `VectorizedRleValuesReaderTestUtils`, so this PR adds a `VectorizedRleValuesReaderSuite` with the 14 tests from #59117 and the helpers they use. It creates and drives the package-private `ParquetReadState` directly instead of through the `ParquetTestAccess` bridge. It also does not have SPARK-56895, so `readGroup` still slices a bit-packed run one group at a time; the new checks run before that loop, as on master. Tests that came into the conflicts as context but are not on this branch are left out (the SPARK-57828 test in `ParquetIOSuite` and the SPARK-59831 test in `ParquetVectorizedSuite`), and `ParquetVectorizedSuite` now imports `BytesInput`, `Encoding` and `Statistics` for the new dictionary page test. ### Why are the changes needed? Same as #59117. On a corrupt page, the vectorized RLE reader, which decodes definition/repetition levels, dictionary ids and RLE booleans, silently returned wrong values or looped forever when the encoded data ended early, and a corrupt bit-packed header could overflow `currentCount` or allocate a huge buffer. The reader now fails with a `ParquetDecodingException`. ### Does this PR introduce _any_ user-facing change? Only for corrupt or non-compliant Parquet files, as described in #59117: such pages now fail with a `ParquetDecodingException` instead of returning wrong values or hanging, and with `spark.sql.files.ignoreCorruptFiles=true` the rest of such a file is skipped. A few of these pages could be read before and gave the right result by chance; #59117 lists them. ### How was this patch tested? The 18 tests added in #59117 pass on this branch, as do the existing tests in `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite` and `VectorizedPlainValuesReaderSuite`. Without the fix, the 13 negative tests in `VectorizedRleValuesReaderSuite` fail, as on master. ``` build/sbt 'sql/testOnly *VectorizedRleValuesReaderSuite *ParquetEncodingSuite *ParquetVectorizedSuite *ParquetIOSuite' ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #59483 from viirya/SPARK-59832-branch-4.1. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent 40e335f commit eabee9e

5 files changed

Lines changed: 607 additions & 26 deletions

File tree

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@
3232
import org.apache.parquet.column.page.*;
3333
import org.apache.parquet.column.values.RequiresPreviousReader;
3434
import org.apache.parquet.column.values.ValuesReader;
35+
import org.apache.parquet.io.ParquetDecodingException;
3536
import org.apache.parquet.schema.LogicalTypeAnnotation;
3637
import org.apache.parquet.schema.LogicalTypeAnnotation.DateLogicalTypeAnnotation;
3738
import org.apache.parquet.schema.LogicalTypeAnnotation.DecimalLogicalTypeAnnotation;
@@ -129,6 +130,9 @@ public VectorizedColumnReader(
129130
this.isCurrentPageDictionaryEncoded = true;
130131
} catch (IOException e) {
131132
throw new IOException("could not decode the dictionary for " + descriptor, e);
133+
} catch (ParquetDecodingException e) {
134+
throw new ParquetDecodingException(
135+
"could not decode the dictionary for " + descriptor + ": " + e.getMessage(), e);
132136
}
133137
} else {
134138
this.dictionary = null;
@@ -331,9 +335,7 @@ private void initDataReader(
331335
if (dataEncoding.usesDictionary()) {
332336
this.dataColumn = null;
333337
if (dictionary == null) {
334-
throw new IOException(
335-
"could not read page in col " + descriptor +
336-
" as the dictionary was missing for encoding " + dataEncoding);
338+
throw new IOException("the dictionary was missing for encoding " + dataEncoding);
337339
}
338340
@SuppressWarnings("deprecation")
339341
Encoding plainDict = Encoding.PLAIN_DICTIONARY; // var to allow warning suppression
@@ -348,11 +350,9 @@ private void initDataReader(
348350
this.isCurrentPageDictionaryEncoded = false;
349351
}
350352

351-
try {
352-
dataColumn.initFromPage(pageValueCount, in);
353-
} catch (IOException e) {
354-
throw new IOException("could not read page in col " + descriptor, e);
355-
}
353+
// readPageV1 and readPageV2 add the page and the column to an IOException or a
354+
// ParquetDecodingException thrown here.
355+
dataColumn.initFromPage(pageValueCount, in);
356356
// for PARQUET-246 (See VectorizedDeltaByteArrayReader.setPreviousValues)
357357
if (CorruptDeltaByteArrays.requiresSequentialReads(writerVersion, dataEncoding) &&
358358
previousReader instanceof RequiresPreviousReader) {
@@ -409,6 +409,9 @@ private int readPageV1(DataPageV1 page) throws IOException {
409409
return pageValueCount;
410410
} catch (IOException e) {
411411
throw new IOException("could not read page " + page + " in col " + descriptor, e);
412+
} catch (ParquetDecodingException e) {
413+
throw new ParquetDecodingException(
414+
"could not read page " + page + " in col " + descriptor + ": " + e.getMessage(), e);
412415
}
413416
}
414417

@@ -429,6 +432,9 @@ private int readPageV2(DataPageV2 page) throws IOException {
429432
return pageValueCount;
430433
} catch (IOException e) {
431434
throw new IOException("could not read page " + page + " in col " + descriptor, e);
435+
} catch (ParquetDecodingException e) {
436+
throw new ParquetDecodingException(
437+
"could not read page " + page + " in col " + descriptor + ": " + e.getMessage(), e);
432438
}
433439
}
434440
}

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java‎

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,9 @@ private enum MODE {
6262
private int bytesWidth;
6363
private BytePacker packer;
6464

65+
// True if the dictionary id section of the page is empty, not even holding the bit width.
66+
private boolean emptyIdSection;
67+
6568
// Current decoding mode and values
6669
private MODE mode;
6770
private int currentCount;
@@ -96,16 +99,35 @@ public VectorizedRleValuesReader(int bitWidth, boolean readLength) {
9699
@Override
97100
public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOException {
98101
this.in = in;
102+
this.emptyIdSection = false;
99103
if (fixedWidth) {
100104
// Initialize for repetition and definition levels
101105
if (readLength) {
106+
if (in.available() < 4) {
107+
throw new ParquetDecodingException("Corrupted RLE data: the 4-byte length is cut off, " +
108+
"only " + in.available() + " bytes are left in the page");
109+
}
102110
int length = readIntLittleEndian();
111+
if (length < 0 || length > in.available()) {
112+
throw new ParquetDecodingException("Corrupted RLE data: invalid length " + length +
113+
", " + in.available() + " bytes are left in the page");
114+
}
103115
this.in = in.sliceStream(length);
104116
}
105117
} else {
106118
// Initialize for values
107119
if (in.available() > 0) {
108-
init(in.read());
120+
int width = in.read();
121+
if (width > 32) {
122+
throw new ParquetDecodingException("Corrupted RLE data: invalid bit width " + width);
123+
}
124+
init(width);
125+
} else {
126+
// No encoded values (e.g. an all-null page). Do not treat this as a bit width of 0,
127+
// which would decode every value as 0; any read fails in `readNextGroup` instead.
128+
this.emptyIdSection = true;
129+
this.currentCount = 0;
130+
return;
109131
}
110132
}
111133
if (bitWidth == 0) {
@@ -208,7 +230,7 @@ private void readBatchInternal(
208230
int leftInPage = state.valuesToReadInPage;
209231

210232
while (leftInBatch > 0 && leftInPage > 0) {
211-
if (currentCount == 0 && !readNextGroup()) break;
233+
if (currentCount == 0) readNextGroup();
212234
int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount));
213235

214236
long rangeStart = state.currentRangeStart();
@@ -282,7 +304,7 @@ private void readBatchInternalWithDefLevels(
282304
int leftInPage = state.valuesToReadInPage;
283305

284306
while (leftInBatch > 0 && leftInPage > 0) {
285-
if (currentCount == 0 && !readNextGroup()) break;
307+
if (currentCount == 0) readNextGroup();
286308
int n = Math.min(leftInBatch, Math.min(leftInPage, this.currentCount));
287309

288310
long rangeStart = state.currentRangeStart();
@@ -395,7 +417,7 @@ public void readBatchRepeatedInternal(
395417
values, nulls, valuesReused, valueReader, updater);
396418

397419
while ((leftInBatch > 0 || !state.lastListCompleted) && leftInPage > 0) {
398-
if (currentCount == 0 && !readNextGroup()) break;
420+
if (currentCount == 0) readNextGroup();
399421

400422
// Values to read in the current RLE/PACKED block, must be <= what's left in the page
401423
int valuesLeftInBlock = Math.min(leftInPage, currentCount);
@@ -601,7 +623,7 @@ private void readValues(
601623
int n = total;
602624
int initialValueOffset = state.valueOffset;
603625
while (n > 0) {
604-
if (currentCount == 0 && !readNextGroup()) break;
626+
if (currentCount == 0) readNextGroup();
605627
int num = Math.min(n, this.currentCount);
606628
readValuesN(num, state, defLevels, values, nulls, valueReader, updater);
607629
state.levelOffset += num;
@@ -660,7 +682,7 @@ private void skipValues(
660682
VectorizedValuesReader valuesReader,
661683
ParquetVectorUpdater updater) {
662684
while (n > 0) {
663-
if (currentCount == 0 && !readNextGroup()) break;
685+
if (currentCount == 0) readNextGroup();
664686
int num = Math.min(n, this.currentCount);
665687
switch (mode) {
666688
case RLE -> {
@@ -693,7 +715,7 @@ private void skipValues(
693715
public void readIntegers(int total, WritableColumnVector c, int rowId) {
694716
int left = total;
695717
while (left > 0) {
696-
if (currentCount == 0 && !readNextGroup()) break;
718+
if (currentCount == 0) readNextGroup();
697719
int n = Math.min(left, this.currentCount);
698720
switch (mode) {
699721
case RLE -> c.putInts(rowId, n, currentValue);
@@ -876,7 +898,6 @@ private int readIntLittleEndian() throws IOException {
876898
*/
877899
private int readIntLittleEndianPaddedOnBitWidth() throws IOException {
878900
return switch (bytesWidth) {
879-
case 0 -> 0;
880901
case 1 -> in.read();
881902
case 2 -> {
882903
int ch2 = in.read();
@@ -895,14 +916,30 @@ private int readIntLittleEndianPaddedOnBitWidth() throws IOException {
895916
}
896917

897918
/**
898-
* Reads the next group. Returns false if no more group available.
919+
* Reads the next non-empty group, so that `currentCount` is positive on return.
920+
* <p>
921+
* Callers never read more values than the page declares, so running out of encoded data here
922+
* means the page is corrupted. With a bit width of 0, `initFromPage` already provides all the
923+
* values of the page, so reaching here also means reading past the end.
899924
*/
900-
private boolean readNextGroup() {
901-
if (in.available() <= 0) {
902-
currentCount = 0;
903-
return false;
925+
private void readNextGroup() {
926+
if (emptyIdSection) {
927+
throw new ParquetDecodingException(
928+
"Corrupted RLE data: the dictionary id section is empty, not even the bit width");
904929
}
930+
do {
931+
if (bitWidth == 0 || in.available() <= 0) {
932+
throw new ParquetDecodingException(
933+
"Corrupted RLE data: reading past the end of the encoded values");
934+
}
935+
readGroup();
936+
} while (this.currentCount == 0);
937+
}
905938

939+
/**
940+
* Reads the next group, which may be empty.
941+
*/
942+
private void readGroup() {
906943
try {
907944
int header = readUnsignedVarInt();
908945
this.mode = (header & 1) == 0 ? MODE.RLE : MODE.PACKED;
@@ -913,6 +950,17 @@ private boolean readNextGroup() {
913950
}
914951
case PACKED -> {
915952
int numGroups = header >>> 1;
953+
// Validate before allocating the buffer, so a corrupted header can neither overflow
954+
// `currentCount` nor make us allocate more than the remaining bytes can fill.
955+
if (numGroups > Integer.MAX_VALUE / 8) {
956+
throw new ParquetDecodingException(
957+
"Corrupted RLE data: bit-packed run of " + numGroups + " groups is too long");
958+
}
959+
if ((long) numGroups * bitWidth > in.available()) {
960+
throw new ParquetDecodingException("Corrupted RLE data: bit-packed run of " +
961+
numGroups + " groups needs " + (long) numGroups * bitWidth + " bytes, but only " +
962+
in.available() + " bytes are left");
963+
}
916964
this.currentCount = numGroups * 8;
917965

918966
if (this.currentBuffer.length < this.currentCount) {
@@ -929,10 +977,8 @@ private boolean readNextGroup() {
929977
}
930978
}
931979
} catch (IOException e) {
932-
throw new ParquetDecodingException("Failed to read from input stream", e);
980+
throw new ParquetDecodingException("Corrupted RLE data: failed to read from input stream", e);
933981
}
934-
935-
return true;
936982
}
937983

938984
/**
@@ -941,7 +987,7 @@ private boolean readNextGroup() {
941987
private void skipValues(int n) {
942988
int left = n;
943989
while (left > 0) {
944-
if (this.currentCount == 0 && !readNextGroup()) break;
990+
if (this.currentCount == 0) readNextGroup();
945991
int num = Math.min(left, this.currentCount);
946992
switch (mode) {
947993
case RLE -> {}

‎sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetIOSuite.scala‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
package org.apache.spark.sql.execution.datasources.parquet
1919

20+
import java.nio.file.Files
2021
import java.time.{LocalDateTime, LocalTime}
2122
import java.util.Locale
2223

@@ -26,16 +27,21 @@ import scala.reflect.ClassTag
2627
import scala.reflect.runtime.universe.TypeTag
2728

2829
import com.google.common.primitives.UnsignedLong
30+
import org.apache.commons.lang3.exception.ExceptionUtils
2931
import org.apache.hadoop.fs.{FileSystem, Path}
3032
import org.apache.hadoop.mapreduce.{JobContext, TaskAttemptContext}
33+
import org.apache.parquet.bytes.BytesUtils
3134
import org.apache.parquet.column.{Encoding, ParquetProperties}
3235
import org.apache.parquet.column.ParquetProperties.WriterVersion.PARQUET_1_0
3336
import org.apache.parquet.example.data.Group
3437
import org.apache.parquet.example.data.simple.{SimpleGroup, SimpleGroupFactory}
3538
import org.apache.parquet.hadoop._
3639
import org.apache.parquet.hadoop.example.ExampleParquetWriter
40+
import org.apache.parquet.io.ParquetDecodingException
3741
import org.apache.parquet.io.api.Binary
3842
import org.apache.parquet.schema.{MessageType, MessageTypeParser}
43+
import org.scalatest.concurrent.{Signaler, ThreadSignaler}
44+
import org.scalatest.time.{Seconds, Span}
3945

4046
import org.apache.spark.{SPARK_VERSION_SHORT, SparkException, TestUtils}
4147
import org.apache.spark.rdd.RDD
@@ -1917,6 +1923,89 @@ class ParquetIOSuite extends QueryTest with ParquetTest with SharedSparkSession
19171923
}
19181924
}
19191925
}
1926+
1927+
/**
1928+
* Writes 100 rows of `column` to a single uncompressed data page, lets `corrupt` modify the
1929+
* RLE section that starts with `03 00 00 00 C8 01 01` (the 4-byte length 3, then one RLE run
1930+
* with header 100 << 1 and value 1), and returns the error of reading the file with the
1931+
* vectorized reader. For a nullable INT32 column that is the definition levels; for a
1932+
* BOOLEAN column written as a V2 page it is the RLE-encoded values.
1933+
*/
1934+
private def readCorruptedRleSection(
1935+
column: String,
1936+
writerVersion: String = "PARQUET_1_0")(
1937+
corrupt: (Array[Byte], Int) => Unit): Throwable = {
1938+
var error: Throwable = null
1939+
withTempPath { dir =>
1940+
val path = dir.getCanonicalPath
1941+
spark.range(100).selectExpr(column).coalesce(1)
1942+
.write
1943+
.option(ParquetOutputFormat.ENABLE_DICTIONARY, "false")
1944+
.option(ParquetOutputFormat.WRITER_VERSION, writerVersion)
1945+
.option("compression", "uncompressed")
1946+
.parquet(path)
1947+
val file = dir.listFiles().filter(_.getName.endsWith(".parquet")).head
1948+
val bytes = Files.readAllBytes(file.toPath)
1949+
val section = Array[Byte](3, 0, 0, 0, 0xC8.toByte, 0x01, 1)
1950+
val offset = bytes.indexOfSlice(section)
1951+
assert(offset >= 0 && bytes.indexOfSlice(section, offset + 1) < 0)
1952+
corrupt(bytes, offset)
1953+
Files.write(file.toPath, bytes)
1954+
dir.listFiles().filter(_.getName.endsWith(".crc")).foreach(_.delete())
1955+
1956+
withSQLConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key -> "true") {
1957+
// Before SPARK-59832 a truncated read never returned, so run it with a timeout. The test
1958+
// thread waits interruptibly for the job, so it can be signaled directly.
1959+
implicit val signaler: Signaler = ThreadSignaler
1960+
error = intercept[SparkException] {
1961+
failAfter(Span(60, Seconds))(spark.read.parquet(path).collect())
1962+
}
1963+
}
1964+
}
1965+
error
1966+
}
1967+
1968+
private def causes(e: Throwable): Seq[Throwable] =
1969+
ExceptionUtils.getThrowableList(e).asScala.toSeq
1970+
1971+
/** Sets the 4-byte length at `offset` to Int.MaxValue, far more than the rest of the page. */
1972+
private def setMaxLength(bytes: Array[Byte], offset: Int): Unit =
1973+
BytesUtils.intToBytes(Int.MaxValue).copyToArray(bytes, offset)
1974+
1975+
/**
1976+
* Checks that the page error is reported once with the column, directly on top of the
1977+
* `ParquetDecodingException` about the invalid length.
1978+
*/
1979+
private def assertInvalidLengthReportsColumn(e: Throwable, column: String): Unit = {
1980+
val pageErrors =
1981+
causes(e).filter(c => Option(c.getMessage).exists(_.startsWith("could not read page")))
1982+
assert(pageErrors.size == 1, e)
1983+
assert(pageErrors.head.getMessage.contains(s"in col [$column]"), e)
1984+
assert(pageErrors.head.getCause.isInstanceOf[ParquetDecodingException], e)
1985+
assert(pageErrors.head.getCause.getMessage.contains(
1986+
s"Corrupted RLE data: invalid length ${Int.MaxValue}"), e)
1987+
}
1988+
1989+
test("SPARK-59832: vectorized reader fails on truncated definition levels") {
1990+
// Shorten the run to 64 values (header 64 << 1 = varint 0x80 0x01).
1991+
val e = readCorruptedRleSection("if(id < 0, null, cast(id as int)) as a") {
1992+
(bytes, offset) => bytes(offset + 4) = 0x80.toByte
1993+
}
1994+
assert(causes(e).exists {
1995+
case c: ParquetDecodingException => c.getMessage.contains("Corrupted RLE data")
1996+
case _ => false
1997+
}, e)
1998+
}
1999+
2000+
test("SPARK-59832: invalid definition level length reports the column") {
2001+
val e = readCorruptedRleSection("if(id < 0, null, cast(id as int)) as a")(setMaxLength)
2002+
assertInvalidLengthReportsColumn(e, "a")
2003+
}
2004+
2005+
test("SPARK-59832: invalid length of RLE booleans in a V2 page reports the column") {
2006+
val e = readCorruptedRleSection("id >= 0 as b", writerVersion = "PARQUET_2_0")(setMaxLength)
2007+
assertInvalidLengthReportsColumn(e, "b")
2008+
}
19202009
}
19212010

19222011
class JobCommitFailureParquetOutputCommitter(outputPath: Path, context: TaskAttemptContext)

0 commit comments

Comments
 (0)