Skip to content

Commit f868de6

Browse files
committed
[SPARK-59831][SQL] Validate bounds in the vectorized BYTE_STREAM_SPLIT reader
### What changes were proposed in this pull request? Follow-up of SPARK-59812 (#59087), which made the vectorized PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY readers fail with a `ParquetDecodingException` on corrupt pages. This does the same for `VectorizedByteStreamSplitValuesReader` (added by SPARK-56894), which did not check that reads and skips stay within the page: - `initFromPage` rejects a page whose length is not a multiple of the value width, and a page with more encoded values than the page value count. parquet-java 1.18.1's `ByteStreamSplitValuesReader` rejects both. For a column without definition levels (max definition level 0), where every row has a value, the page value count must match the encoded values exactly, for both v1 and v2 pages. For other columns it includes nulls and is only an upper bound. - Batch reads (`readIntegers`, `readLongs`, `readFloats`, `readDoubles`, their widening/narrowing variants, `readBytes`, `readShorts`, `readBinary`, `readFixedLenByteArray`) and all skips check once per batch that the page has enough values left. The per-value loops are unchanged. - Single-value reads (`readInteger`, `readLong`, `readFloat`, `readDouble`, `readByte`, `readShort`, `readBinary(len)`) check that a value is left. - A negative count to read or skip, which can only come from the caller, throws an `IllegalArgumentException`. So does `readBinary(int len)` with a `len` other than the value width, before consuming a value. Its callers always pass the type length of the column, so this is not reachable today, but it used to fail with an `ArrayIndexOutOfBoundsException` for a larger `len` and return a prefix of the value for a smaller one. ### Why are the changes needed? On a corrupt page: - The skip methods only did `offset += total`, so skipping past the end of the page was never detected. - Reading past the end (or reading after such a skip) failed with an `ArrayIndexOutOfBoundsException` instead of a `ParquetDecodingException`. - A page whose length is not a multiple of the value width was accepted. BYTE_STREAM_SPLIT stores no value count, and the Parquet spec allows no padding in a data page, so the stream boundaries cannot be derived for such a page: for example, a page that lost a byte has every stream start at the wrong offset, so every value is decoded from the wrong bytes. In addition, reads past the last whole value silently returned values assembled from the trailing bytes instead of failing. - For a required column, a page with fewer encoded values than its value count failed only when a read reached the end of the page, after returning values decoded with the wrong stream stride. For a column with definition levels, a data section that is off by whole values is still not detected when the page is initialized, as in parquet-java, and its values are decoded with the wrong stride. With fewer encoded values than non-null values, a read or skip past the last encoded value fails. With more, the page is not detected; checking that all encoded values are consumed at the end of each page is left as a follow-up. The exact count of `DataPageV2` (`num_values - num_nulls`) is not used, because neither parquet-java's read path nor Spark reads `num_nulls` today, so enforcing it could reject files that read correctly now. ### Does this PR introduce _any_ user-facing change? Only for corrupt Parquet files: reading such a BYTE_STREAM_SPLIT page with the vectorized reader now fails with a `ParquetDecodingException` instead of an `ArrayIndexOutOfBoundsException` or silently wrong values. Valid files are not affected. For a required column, a page with fewer encoded values than its value count now fails when the page is initialized, so a query that reads only the first rows of such a page (e.g. with `LIMIT` or column-index row ranges) fails instead of returning values decoded with the wrong stride. The vectorized BYTE_STREAM_SPLIT reader is not in a released version yet. ### How was this patch tested? New tests in `ParquetByteStreamSplitEncodingSuite`, run for INT32, INT64, FLOAT and DOUBLE, and for FIXED_LEN_BYTE_ARRAY of width 5: - Negative: every applicable batch read and skip past the end of the page, after a skip, and with a count that would overflow `offset + total`; a negative count; every single-value read past the end; page lengths with 1 to W-1 extra or missing bytes; more encoded values than the page value count; fewer encoded values than an exact page value count; reads and skips on an empty page. - Positive: every batch read and skip ending exactly at the end of the page, checking the values read, followed by empty reads and skips; reading the last value after a skip; fewer encoded values than the page value count (nullable columns); an exact page value count that matches. End-to-end tests: - `ParquetVectorizedSuite`: a required BYTE_STREAM_SPLIT page whose value count is one more than its encoded values, written with v1 and v2 pages to an in-memory page store and read through `VectorizedParquetRecordReader`, fails when the page is initialized. This covers the wiring of the exact check in `VectorizedColumnReader`. - `ParquetEncodingSuite`: a round trip with v1 and v2 pages, more than one page per column, and required, optional and array columns of all BYTE_STREAM_SPLIT types. - `ParquetColumnIndexSuite`: unaligned BYTE_STREAM_SPLIT pages with v1 and v2 pages, required and nullable columns, so that rows are skipped within the BYTE_STREAM_SPLIT pages. The test asserts the encodings and that the page boundaries of the filter column fall inside the pages of the unaligned columns. All the new negative tests fail without this change. Making the skips throw fails the `ParquetColumnIndexSuite` test for both page versions, making the exact count check throw fails the `ParquetEncodingSuite` round trip for both, passing `false` for the exact check in `VectorizedColumnReader` fails the `ParquetVectorizedSuite` test, and rewriting the remaining count check as `offset + total > valueCount` fails the overflow test. Existing suites also pass: `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite` and `ParquetIOSuite`. The single-value reads add a bounds check per value. An A/B microbenchmark of master's reader against this one (per-value `readInteger`, `readLong`, `readDouble`, `readBinary(12)` and batch `readIntegers`, `readDoubles`, 1M values, JDK 21) showed no measurable difference. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) Closes #59118 from viirya/SPARK-59831-bss-bounds. Authored-by: Liang-Chi Hsieh <[email protected]> Signed-off-by: Liang-Chi Hsieh <[email protected]>
1 parent 6ed637f commit f868de6

6 files changed

Lines changed: 594 additions & 39 deletions

File tree

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedByteStreamSplitValuesReader.java‎

Lines changed: 102 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
import java.nio.ByteBuffer;
2121

2222
import org.apache.parquet.bytes.ByteBufferInputStream;
23+
import org.apache.parquet.io.ParquetDecodingException;
2324
import org.apache.parquet.io.api.Binary;
2425

2526
import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
@@ -50,7 +51,10 @@ public class VectorizedByteStreamSplitValuesReader
5051
/** Width of each value in bytes (4 for FLOAT/INT32, 8 for DOUBLE/INT64). */
5152
private final int typeWidth;
5253

53-
/** Total number of values in the current page. */
54+
/**
55+
* Number of values encoded in the current page. For a column with definition levels this
56+
* excludes nulls, so it can be smaller than the page value count.
57+
*/
5458
private int valueCount;
5559

5660
/** Raw encoded page data: W streams of valueCount bytes each. */
@@ -59,8 +63,20 @@ public class VectorizedByteStreamSplitValuesReader
5963
/** Current read position (number of values consumed so far). */
6064
private int offset;
6165

66+
/**
67+
* Whether the page value count is exactly the number of encoded values. This holds for a
68+
* column without definition levels (max definition level 0), where every row has a value.
69+
* Otherwise the page value count includes nulls and is only an upper bound.
70+
*/
71+
private final boolean exactValueCount;
72+
6273
public VectorizedByteStreamSplitValuesReader(int typeWidth) {
74+
this(typeWidth, false);
75+
}
76+
77+
public VectorizedByteStreamSplitValuesReader(int typeWidth, boolean exactValueCount) {
6378
this.typeWidth = typeWidth;
79+
this.exactValueCount = exactValueCount;
6480
}
6581

6682
@Override
@@ -69,7 +85,30 @@ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOExce
6985
// but valueCount includes nulls. Use the actual bytes available in the stream
7086
// to derive the real number of encoded values.
7187
int totalBytes = in.available();
72-
this.valueCount = totalBytes / typeWidth;
88+
// The page stores no value count and the Parquet spec allows no padding in a data
89+
// page, so the page length must be an exact multiple of the value width. Otherwise
90+
// the page is truncated or has trailing bytes, and the stream boundaries derived from
91+
// its length may be wrong.
92+
if (totalBytes % typeWidth != 0) {
93+
throw new ParquetDecodingException("Corrupted BYTE_STREAM_SPLIT page: page data length " +
94+
totalBytes + " is not a multiple of the value width " + typeWidth);
95+
}
96+
int encodedValues = totalBytes / typeWidth;
97+
// Without definition levels the page value count must match the encoded values exactly.
98+
if (exactValueCount && encodedValues != valueCount) {
99+
throw new ParquetDecodingException("Corrupted BYTE_STREAM_SPLIT page: " + encodedValues +
100+
" encoded values do not match the page value count " + valueCount);
101+
}
102+
// Otherwise the page value count includes nulls and is only an upper bound. A page with
103+
// more encoded values than that is rejected below, but a page that is off by whole values
104+
// within the bound is not detected here, and its values are decoded with the wrong stream
105+
// stride. With fewer encoded values than non-null rows, a read or skip past the last
106+
// encoded value fails. With more, the page is not detected.
107+
if (encodedValues > valueCount) {
108+
throw new ParquetDecodingException("Corrupted BYTE_STREAM_SPLIT page: " + encodedValues +
109+
" encoded values exceed the page value count " + valueCount);
110+
}
111+
this.valueCount = encodedValues;
73112
this.offset = 0;
74113
this.pageData = new byte[totalBytes];
75114
// Read the entire page into pageData. ByteBufferInputStream.slice() returns a
@@ -81,6 +120,28 @@ public void initFromPage(int valueCount, ByteBufferInputStream in) throws IOExce
81120

82121
// --------------- helpers ---------------
83122

123+
/**
124+
* Checks that the page has at least {@code total} values left. Batch reads and skips call
125+
* this once per batch, so the per-value loops need no bound check of their own.
126+
*/
127+
private void checkRemaining(int total) {
128+
if (total < 0) {
129+
throw new IllegalArgumentException("Cannot read or skip a negative number of values: " +
130+
total);
131+
}
132+
if (total > valueCount - offset) {
133+
throw new ParquetDecodingException("Corrupted BYTE_STREAM_SPLIT page: cannot read or " +
134+
"skip " + total + " values, only " + (valueCount - offset) + " of " + valueCount +
135+
" values are left");
136+
}
137+
}
138+
139+
/** Returns the index of the next value for a single-value read and advances past it. */
140+
private int nextIndex() {
141+
checkRemaining(1);
142+
return offset++;
143+
}
144+
84145
/** Assembles a single 4-byte little-endian int from 4 streams at the given index. */
85146
private int assembleInt(int idx) {
86147
return (pageData[idx] & 0xFF)
@@ -115,38 +176,44 @@ public short readShort() {
115176

116177
@Override
117178
public int readInteger() {
118-
return assembleInt(offset++);
179+
return assembleInt(nextIndex());
119180
}
120181

121182
@Override
122183
public long readLong() {
123-
return assembleLong(offset++);
184+
return assembleLong(nextIndex());
124185
}
125186

126187
@Override
127188
public float readFloat() {
128-
return Float.intBitsToFloat(assembleInt(offset++));
189+
return Float.intBitsToFloat(assembleInt(nextIndex()));
129190
}
130191

131192
@Override
132193
public double readDouble() {
133-
return Double.longBitsToDouble(assembleLong(offset++));
194+
return Double.longBitsToDouble(assembleLong(nextIndex()));
134195
}
135196

136197
@Override
137198
public Binary readBinary(int len) {
138-
byte[] result = new byte[len];
139-
for (int b = 0; b < len; b++) {
140-
result[b] = pageData[b * valueCount + offset];
199+
// Each value has one byte in each of the typeWidth streams.
200+
if (len != typeWidth) {
201+
throw new IllegalArgumentException(
202+
"Cannot read a value of length " + len + " from values of width " + typeWidth);
203+
}
204+
byte[] result = new byte[typeWidth];
205+
int idx = nextIndex();
206+
for (int b = 0; b < typeWidth; b++) {
207+
result[b] = pageData[b * valueCount + idx];
141208
}
142-
offset++;
143209
return Binary.fromConstantByteArray(result);
144210
}
145211

146212
// --------------- batch reads ---------------
147213

148214
@Override
149215
public void readBytes(int total, WritableColumnVector c, int rowId) {
216+
checkRemaining(total);
150217
for (int i = 0; i < total; i++) {
151218
c.putByte(rowId + i, (byte) assembleInt(offset + i));
152219
}
@@ -155,6 +222,7 @@ public void readBytes(int total, WritableColumnVector c, int rowId) {
155222

156223
@Override
157224
public void readShorts(int total, WritableColumnVector c, int rowId) {
225+
checkRemaining(total);
158226
for (int i = 0; i < total; i++) {
159227
c.putShort(rowId + i, (short) assembleInt(offset + i));
160228
}
@@ -163,6 +231,7 @@ public void readShorts(int total, WritableColumnVector c, int rowId) {
163231

164232
@Override
165233
public void readIntegers(int total, WritableColumnVector c, int rowId) {
234+
checkRemaining(total);
166235
for (int i = 0; i < total; i++) {
167236
c.putInt(rowId + i, assembleInt(offset + i));
168237
}
@@ -171,6 +240,7 @@ public void readIntegers(int total, WritableColumnVector c, int rowId) {
171240

172241
@Override
173242
public void readIntegersAsLongs(int total, WritableColumnVector c, int rowId) {
243+
checkRemaining(total);
174244
for (int i = 0; i < total; i++) {
175245
c.putLong(rowId + i, assembleInt(offset + i));
176246
}
@@ -179,6 +249,7 @@ public void readIntegersAsLongs(int total, WritableColumnVector c, int rowId) {
179249

180250
@Override
181251
public void readIntegersAsDoubles(int total, WritableColumnVector c, int rowId) {
252+
checkRemaining(total);
182253
for (int i = 0; i < total; i++) {
183254
c.putDouble(rowId + i, (double) assembleInt(offset + i));
184255
}
@@ -187,6 +258,7 @@ public void readIntegersAsDoubles(int total, WritableColumnVector c, int rowId)
187258

188259
@Override
189260
public void readLongs(int total, WritableColumnVector c, int rowId) {
261+
checkRemaining(total);
190262
for (int i = 0; i < total; i++) {
191263
c.putLong(rowId + i, assembleLong(offset + i));
192264
}
@@ -195,6 +267,7 @@ public void readLongs(int total, WritableColumnVector c, int rowId) {
195267

196268
@Override
197269
public void readLongsAsInts(int total, WritableColumnVector c, int rowId) {
270+
checkRemaining(total);
198271
for (int i = 0; i < total; i++) {
199272
c.putInt(rowId + i, (int) assembleLong(offset + i));
200273
}
@@ -203,6 +276,7 @@ public void readLongsAsInts(int total, WritableColumnVector c, int rowId) {
203276

204277
@Override
205278
public void readFloats(int total, WritableColumnVector c, int rowId) {
279+
checkRemaining(total);
206280
for (int i = 0; i < total; i++) {
207281
c.putFloat(rowId + i, Float.intBitsToFloat(assembleInt(offset + i)));
208282
}
@@ -211,6 +285,7 @@ public void readFloats(int total, WritableColumnVector c, int rowId) {
211285

212286
@Override
213287
public void readFloatsAsDoubles(int total, WritableColumnVector c, int rowId) {
288+
checkRemaining(total);
214289
for (int i = 0; i < total; i++) {
215290
c.putDouble(rowId + i, (double) Float.intBitsToFloat(assembleInt(offset + i)));
216291
}
@@ -219,6 +294,7 @@ public void readFloatsAsDoubles(int total, WritableColumnVector c, int rowId) {
219294

220295
@Override
221296
public void readDoubles(int total, WritableColumnVector c, int rowId) {
297+
checkRemaining(total);
222298
for (int i = 0; i < total; i++) {
223299
c.putDouble(rowId + i, Double.longBitsToDouble(assembleLong(offset + i)));
224300
}
@@ -227,6 +303,7 @@ public void readDoubles(int total, WritableColumnVector c, int rowId) {
227303

228304
@Override
229305
public void readBinary(int total, WritableColumnVector c, int rowId) {
306+
checkRemaining(total);
230307
// Reuse a single scratch buffer to avoid per-value byte[] + Binary allocation.
231308
byte[] scratch = new byte[typeWidth];
232309
for (int i = 0; i < total; i++) {
@@ -244,30 +321,36 @@ public void readFixedLenByteArray(int total, int len, WritableColumnVector c, in
244321
}
245322

246323
// --------------- skip methods ---------------
247-
// All types share the same page layout, so skipping is just advancing the offset.
324+
// All types share the same page layout, so skipping is just advancing the offset
325+
// after checking the page has enough values left.
248326
// skipBooleans is not overridden: BSS never encodes booleans; the base class throws.
249327

328+
private void skipValues(int total) {
329+
checkRemaining(total);
330+
offset += total;
331+
}
332+
250333
@Override
251-
public void skipBytes(int total) { offset += total; }
334+
public void skipBytes(int total) { skipValues(total); }
252335

253336
@Override
254-
public void skipShorts(int total) { offset += total; }
337+
public void skipShorts(int total) { skipValues(total); }
255338

256339
@Override
257-
public void skipIntegers(int total) { offset += total; }
340+
public void skipIntegers(int total) { skipValues(total); }
258341

259342
@Override
260-
public void skipLongs(int total) { offset += total; }
343+
public void skipLongs(int total) { skipValues(total); }
261344

262345
@Override
263-
public void skipFloats(int total) { offset += total; }
346+
public void skipFloats(int total) { skipValues(total); }
264347

265348
@Override
266-
public void skipDoubles(int total) { offset += total; }
349+
public void skipDoubles(int total) { skipValues(total); }
267350

268351
@Override
269-
public void skipBinary(int total) { offset += total; }
352+
public void skipBinary(int total) { skipValues(total); }
270353

271354
@Override
272-
public void skipFixedLenByteArray(int total, int len) { offset += total; }
355+
public void skipFixedLenByteArray(int total, int len) { skipValues(total); }
273356
}

‎sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -389,7 +389,8 @@ private ValuesReader getValuesReader(Encoding encoding) {
389389
default -> throw new SparkUnsupportedOperationException(
390390
"_LEGACY_ERROR_TEMP_3190", Map.of("typeName", typeName.toString()));
391391
};
392-
yield new VectorizedByteStreamSplitValuesReader(typeWidth);
392+
yield new VectorizedByteStreamSplitValuesReader(
393+
typeWidth, this.descriptor.getMaxDefinitionLevel() == 0);
393394
}
394395
case RLE -> {
395396
PrimitiveType.PrimitiveTypeName typeName =

0 commit comments

Comments
 (0)