Repository navigation
Commit f868de6
committed
[SPARK-59831][SQL] Validate bounds in the vectorized BYTE_STREAM_SPLIT reader
### What changes were proposed in this pull request?
Follow-up of SPARK-59812 (#59087), which made the vectorized PLAIN, DELTA_LENGTH_BYTE_ARRAY and DELTA_BYTE_ARRAY readers fail with a `ParquetDecodingException` on corrupt pages. This does the same for `VectorizedByteStreamSplitValuesReader` (added by SPARK-56894), which did not check that reads and skips stay within the page:
- `initFromPage` rejects a page whose length is not a multiple of the value width, and a page with more encoded values than the page value count. parquet-java 1.18.1's `ByteStreamSplitValuesReader` rejects both. For a column without definition levels (max definition level 0), where every row has a value, the page value count must match the encoded values exactly, for both v1 and v2 pages. For other columns it includes nulls and is only an upper bound.
- Batch reads (`readIntegers`, `readLongs`, `readFloats`, `readDoubles`, their widening/narrowing variants, `readBytes`, `readShorts`, `readBinary`, `readFixedLenByteArray`) and all skips check once per batch that the page has enough values left. The per-value loops are unchanged.
- Single-value reads (`readInteger`, `readLong`, `readFloat`, `readDouble`, `readByte`, `readShort`, `readBinary(len)`) check that a value is left.
- A negative count to read or skip, which can only come from the caller, throws an `IllegalArgumentException`. So does `readBinary(int len)` with a `len` other than the value width, before consuming a value. Its callers always pass the type length of the column, so this is not reachable today, but it used to fail with an `ArrayIndexOutOfBoundsException` for a larger `len` and return a prefix of the value for a smaller one.
### Why are the changes needed?
On a corrupt page:
- The skip methods only did `offset += total`, so skipping past the end of the page was never detected.
- Reading past the end (or reading after such a skip) failed with an `ArrayIndexOutOfBoundsException` instead of a `ParquetDecodingException`.
- A page whose length is not a multiple of the value width was accepted. BYTE_STREAM_SPLIT stores no value count, and the Parquet spec allows no padding in a data page, so the stream boundaries cannot be derived for such a page: for example, a page that lost a byte has every stream start at the wrong offset, so every value is decoded from the wrong bytes. In addition, reads past the last whole value silently returned values assembled from the trailing bytes instead of failing.
- For a required column, a page with fewer encoded values than its value count failed only when a read reached the end of the page, after returning values decoded with the wrong stream stride.
For a column with definition levels, a data section that is off by whole values is still not detected when the page is initialized, as in parquet-java, and its values are decoded with the wrong stride. With fewer encoded values than non-null values, a read or skip past the last encoded value fails. With more, the page is not detected; checking that all encoded values are consumed at the end of each page is left as a follow-up. The exact count of `DataPageV2` (`num_values - num_nulls`) is not used, because neither parquet-java's read path nor Spark reads `num_nulls` today, so enforcing it could reject files that read correctly now.
### Does this PR introduce _any_ user-facing change?
Only for corrupt Parquet files: reading such a BYTE_STREAM_SPLIT page with the vectorized reader now fails with a `ParquetDecodingException` instead of an `ArrayIndexOutOfBoundsException` or silently wrong values. Valid files are not affected. For a required column, a page with fewer encoded values than its value count now fails when the page is initialized, so a query that reads only the first rows of such a page (e.g. with `LIMIT` or column-index row ranges) fails instead of returning values decoded with the wrong stride. The vectorized BYTE_STREAM_SPLIT reader is not in a released version yet.
### How was this patch tested?
New tests in `ParquetByteStreamSplitEncodingSuite`, run for INT32, INT64, FLOAT and DOUBLE, and for FIXED_LEN_BYTE_ARRAY of width 5:
- Negative: every applicable batch read and skip past the end of the page, after a skip, and with a count that would overflow `offset + total`; a negative count; every single-value read past the end; page lengths with 1 to W-1 extra or missing bytes; more encoded values than the page value count; fewer encoded values than an exact page value count; reads and skips on an empty page.
- Positive: every batch read and skip ending exactly at the end of the page, checking the values read, followed by empty reads and skips; reading the last value after a skip; fewer encoded values than the page value count (nullable columns); an exact page value count that matches.
End-to-end tests:
- `ParquetVectorizedSuite`: a required BYTE_STREAM_SPLIT page whose value count is one more than its encoded values, written with v1 and v2 pages to an in-memory page store and read through `VectorizedParquetRecordReader`, fails when the page is initialized. This covers the wiring of the exact check in `VectorizedColumnReader`.
- `ParquetEncodingSuite`: a round trip with v1 and v2 pages, more than one page per column, and required, optional and array columns of all BYTE_STREAM_SPLIT types.
- `ParquetColumnIndexSuite`: unaligned BYTE_STREAM_SPLIT pages with v1 and v2 pages, required and nullable columns, so that rows are skipped within the BYTE_STREAM_SPLIT pages. The test asserts the encodings and that the page boundaries of the filter column fall inside the pages of the unaligned columns.
All the new negative tests fail without this change. Making the skips throw fails the `ParquetColumnIndexSuite` test for both page versions, making the exact count check throw fails the `ParquetEncodingSuite` round trip for both, passing `false` for the exact check in `VectorizedColumnReader` fails the `ParquetVectorizedSuite` test, and rewriting the remaining count check as `offset + total > valueCount` fails the overflow test. Existing suites also pass: `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite` and `ParquetIOSuite`.
The single-value reads add a bounds check per value. An A/B microbenchmark of master's reader against this one (per-value `readInteger`, `readLong`, `readDouble`, `readBinary(12)` and batch `readIntegers`, `readDoubles`, 1M values, JDK 21) showed no measurable difference.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59118 from viirya/SPARK-59831-bss-bounds.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>1 parent 6ed637f commit f868de6
6 files changed
Lines changed: 594 additions & 39 deletions
File tree
- sql/core/src
- main/java/org/apache/spark/sql/execution/datasources/parquet
- test/scala/org/apache/spark/sql/execution/datasources/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
50 | 51 | | |
51 | 52 | | |
52 | 53 | | |
53 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
54 | 58 | | |
55 | 59 | | |
56 | 60 | | |
| |||
59 | 63 | | |
60 | 64 | | |
61 | 65 | | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
62 | 73 | | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
63 | 78 | | |
| 79 | + | |
64 | 80 | | |
65 | 81 | | |
66 | 82 | | |
| |||
69 | 85 | | |
70 | 86 | | |
71 | 87 | | |
72 | | - | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
73 | 112 | | |
74 | 113 | | |
75 | 114 | | |
| |||
81 | 120 | | |
82 | 121 | | |
83 | 122 | | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
84 | 145 | | |
85 | 146 | | |
86 | 147 | | |
| |||
115 | 176 | | |
116 | 177 | | |
117 | 178 | | |
118 | | - | |
| 179 | + | |
119 | 180 | | |
120 | 181 | | |
121 | 182 | | |
122 | 183 | | |
123 | | - | |
| 184 | + | |
124 | 185 | | |
125 | 186 | | |
126 | 187 | | |
127 | 188 | | |
128 | | - | |
| 189 | + | |
129 | 190 | | |
130 | 191 | | |
131 | 192 | | |
132 | 193 | | |
133 | | - | |
| 194 | + | |
134 | 195 | | |
135 | 196 | | |
136 | 197 | | |
137 | 198 | | |
138 | | - | |
139 | | - | |
140 | | - | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
141 | 208 | | |
142 | | - | |
143 | 209 | | |
144 | 210 | | |
145 | 211 | | |
146 | 212 | | |
147 | 213 | | |
148 | 214 | | |
149 | 215 | | |
| 216 | + | |
150 | 217 | | |
151 | 218 | | |
152 | 219 | | |
| |||
155 | 222 | | |
156 | 223 | | |
157 | 224 | | |
| 225 | + | |
158 | 226 | | |
159 | 227 | | |
160 | 228 | | |
| |||
163 | 231 | | |
164 | 232 | | |
165 | 233 | | |
| 234 | + | |
166 | 235 | | |
167 | 236 | | |
168 | 237 | | |
| |||
171 | 240 | | |
172 | 241 | | |
173 | 242 | | |
| 243 | + | |
174 | 244 | | |
175 | 245 | | |
176 | 246 | | |
| |||
179 | 249 | | |
180 | 250 | | |
181 | 251 | | |
| 252 | + | |
182 | 253 | | |
183 | 254 | | |
184 | 255 | | |
| |||
187 | 258 | | |
188 | 259 | | |
189 | 260 | | |
| 261 | + | |
190 | 262 | | |
191 | 263 | | |
192 | 264 | | |
| |||
195 | 267 | | |
196 | 268 | | |
197 | 269 | | |
| 270 | + | |
198 | 271 | | |
199 | 272 | | |
200 | 273 | | |
| |||
203 | 276 | | |
204 | 277 | | |
205 | 278 | | |
| 279 | + | |
206 | 280 | | |
207 | 281 | | |
208 | 282 | | |
| |||
211 | 285 | | |
212 | 286 | | |
213 | 287 | | |
| 288 | + | |
214 | 289 | | |
215 | 290 | | |
216 | 291 | | |
| |||
219 | 294 | | |
220 | 295 | | |
221 | 296 | | |
| 297 | + | |
222 | 298 | | |
223 | 299 | | |
224 | 300 | | |
| |||
227 | 303 | | |
228 | 304 | | |
229 | 305 | | |
| 306 | + | |
230 | 307 | | |
231 | 308 | | |
232 | 309 | | |
| |||
244 | 321 | | |
245 | 322 | | |
246 | 323 | | |
247 | | - | |
| 324 | + | |
| 325 | + | |
248 | 326 | | |
249 | 327 | | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
250 | 333 | | |
251 | | - | |
| 334 | + | |
252 | 335 | | |
253 | 336 | | |
254 | | - | |
| 337 | + | |
255 | 338 | | |
256 | 339 | | |
257 | | - | |
| 340 | + | |
258 | 341 | | |
259 | 342 | | |
260 | | - | |
| 343 | + | |
261 | 344 | | |
262 | 345 | | |
263 | | - | |
| 346 | + | |
264 | 347 | | |
265 | 348 | | |
266 | | - | |
| 349 | + | |
267 | 350 | | |
268 | 351 | | |
269 | | - | |
| 352 | + | |
270 | 353 | | |
271 | 354 | | |
272 | | - | |
| 355 | + | |
273 | 356 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
389 | 389 | | |
390 | 390 | | |
391 | 391 | | |
392 | | - | |
| 392 | + | |
| 393 | + | |
393 | 394 | | |
394 | 395 | | |
395 | 396 | | |
| |||
0 commit comments