Repository navigation
Commit 95fd662
[SPARK-59832][4.X][SQL] Fail on truncated input in the vectorized RLE reader
### What changes were proposed in this pull request?
Backport of #59117 to branch-4.x.
The cherry-pick conflicted only on code that SPARK-59620 added to master, which is not on this branch: an import in `VectorizedColumnReader` and `VectorizedRleValuesReaderSuite`, and a doc comment next to the new `readBatchRepeated` bridge in `ParquetTestAccess`. They are left out; otherwise the change is the same as on master.
This commit also applies cleanly to branch-4.3, so it can be cherry-picked there when merging. The same change was built and tested on branch-4.3.
### Why are the changes needed?
Same as #59117. On a corrupt page, the vectorized RLE reader, which decodes definition/repetition levels, dictionary ids and RLE booleans, silently returned wrong values or looped forever when the encoded data ended early, and a corrupt bit-packed header could overflow `currentCount` or allocate a huge buffer. The reader now fails with a `ParquetDecodingException`.
### Does this PR introduce _any_ user-facing change?
Only for corrupt or non-compliant Parquet files, as described in #59117: such pages now fail with a `ParquetDecodingException` instead of returning wrong values or hanging, and with `spark.sql.files.ignoreCorruptFiles=true` the rest of such a file is skipped. A few of these pages could be read before and gave the right result by chance; #59117 lists them.
### How was this patch tested?
The 18 tests added in #59117 pass on this branch, as do the existing tests in `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite`, `VectorizedPlainValuesReaderSuite` and `ParquetVectorUpdaterSuite`.
```
build/sbt 'sql/testOnly *VectorizedRleValuesReaderSuite *ParquetEncodingSuite *ParquetVectorizedSuite *ParquetIOSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59481 from viirya/SPARK-59832-branch-4.x.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 45f1cc8)
Signed-off-by: Dongjoon Hyun <[email protected]>1 parent 66920b0 commit 95fd662
6 files changed
Lines changed: 530 additions & 29 deletions
File tree
- sql/core/src
- main/java/org/apache/spark/sql/execution/datasources/parquet
- test/scala/org/apache/spark/sql/execution/datasources/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
132 | 133 | | |
133 | 134 | | |
134 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
135 | 139 | | |
136 | 140 | | |
137 | 141 | | |
| |||
343 | 347 | | |
344 | 348 | | |
345 | 349 | | |
346 | | - | |
347 | | - | |
348 | | - | |
| 350 | + | |
349 | 351 | | |
350 | 352 | | |
351 | 353 | | |
| |||
360 | 362 | | |
361 | 363 | | |
362 | 364 | | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
368 | 368 | | |
369 | 369 | | |
370 | 370 | | |
| |||
434 | 434 | | |
435 | 435 | | |
436 | 436 | | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
437 | 440 | | |
438 | 441 | | |
439 | 442 | | |
| |||
454 | 457 | | |
455 | 458 | | |
456 | 459 | | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
457 | 463 | | |
458 | 464 | | |
459 | 465 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
65 | 68 | | |
66 | 69 | | |
67 | 70 | | |
| |||
96 | 99 | | |
97 | 100 | | |
98 | 101 | | |
| 102 | + | |
99 | 103 | | |
100 | 104 | | |
101 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
102 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
103 | 115 | | |
104 | 116 | | |
105 | 117 | | |
106 | 118 | | |
107 | 119 | | |
108 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
109 | 131 | | |
110 | 132 | | |
111 | 133 | | |
| |||
208 | 230 | | |
209 | 231 | | |
210 | 232 | | |
211 | | - | |
| 233 | + | |
212 | 234 | | |
213 | 235 | | |
214 | 236 | | |
| |||
318 | 340 | | |
319 | 341 | | |
320 | 342 | | |
321 | | - | |
| 343 | + | |
322 | 344 | | |
323 | 345 | | |
324 | 346 | | |
| |||
431 | 453 | | |
432 | 454 | | |
433 | 455 | | |
434 | | - | |
| 456 | + | |
435 | 457 | | |
436 | 458 | | |
437 | 459 | | |
| |||
637 | 659 | | |
638 | 660 | | |
639 | 661 | | |
640 | | - | |
| 662 | + | |
641 | 663 | | |
642 | 664 | | |
643 | 665 | | |
| |||
731 | 753 | | |
732 | 754 | | |
733 | 755 | | |
734 | | - | |
| 756 | + | |
735 | 757 | | |
736 | 758 | | |
737 | 759 | | |
| |||
764 | 786 | | |
765 | 787 | | |
766 | 788 | | |
767 | | - | |
| 789 | + | |
768 | 790 | | |
769 | 791 | | |
770 | 792 | | |
| |||
957 | 979 | | |
958 | 980 | | |
959 | 981 | | |
960 | | - | |
961 | 982 | | |
962 | 983 | | |
963 | 984 | | |
| |||
976 | 997 | | |
977 | 998 | | |
978 | 999 | | |
979 | | - | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
980 | 1005 | | |
981 | | - | |
982 | | - | |
983 | | - | |
984 | | - | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
985 | 1010 | | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
986 | 1019 | | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
987 | 1024 | | |
988 | 1025 | | |
989 | 1026 | | |
| |||
994 | 1031 | | |
995 | 1032 | | |
996 | 1033 | | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
997 | 1045 | | |
998 | 1046 | | |
999 | 1047 | | |
| |||
1014 | 1062 | | |
1015 | 1063 | | |
1016 | 1064 | | |
1017 | | - | |
| 1065 | + | |
1018 | 1066 | | |
1019 | | - | |
1020 | | - | |
1021 | 1067 | | |
1022 | 1068 | | |
1023 | 1069 | | |
| |||
1026 | 1072 | | |
1027 | 1073 | | |
1028 | 1074 | | |
1029 | | - | |
| 1075 | + | |
1030 | 1076 | | |
1031 | 1077 | | |
1032 | 1078 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
| 34 | + | |
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
37 | 40 | | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
| 44 | + | |
| 45 | + | |
40 | 46 | | |
41 | 47 | | |
42 | 48 | | |
| |||
2410 | 2416 | | |
2411 | 2417 | | |
2412 | 2418 | | |
| 2419 | + | |
| 2420 | + | |
| 2421 | + | |
| 2422 | + | |
| 2423 | + | |
| 2424 | + | |
| 2425 | + | |
| 2426 | + | |
| 2427 | + | |
| 2428 | + | |
| 2429 | + | |
| 2430 | + | |
| 2431 | + | |
| 2432 | + | |
| 2433 | + | |
| 2434 | + | |
| 2435 | + | |
| 2436 | + | |
| 2437 | + | |
| 2438 | + | |
| 2439 | + | |
| 2440 | + | |
| 2441 | + | |
| 2442 | + | |
| 2443 | + | |
| 2444 | + | |
| 2445 | + | |
| 2446 | + | |
| 2447 | + | |
| 2448 | + | |
| 2449 | + | |
| 2450 | + | |
| 2451 | + | |
| 2452 | + | |
| 2453 | + | |
| 2454 | + | |
| 2455 | + | |
| 2456 | + | |
| 2457 | + | |
| 2458 | + | |
| 2459 | + | |
| 2460 | + | |
| 2461 | + | |
| 2462 | + | |
| 2463 | + | |
| 2464 | + | |
| 2465 | + | |
| 2466 | + | |
| 2467 | + | |
| 2468 | + | |
| 2469 | + | |
| 2470 | + | |
| 2471 | + | |
| 2472 | + | |
| 2473 | + | |
| 2474 | + | |
| 2475 | + | |
| 2476 | + | |
| 2477 | + | |
| 2478 | + | |
| 2479 | + | |
| 2480 | + | |
| 2481 | + | |
| 2482 | + | |
| 2483 | + | |
| 2484 | + | |
| 2485 | + | |
| 2486 | + | |
| 2487 | + | |
| 2488 | + | |
| 2489 | + | |
| 2490 | + | |
| 2491 | + | |
| 2492 | + | |
| 2493 | + | |
| 2494 | + | |
| 2495 | + | |
| 2496 | + | |
| 2497 | + | |
| 2498 | + | |
| 2499 | + | |
| 2500 | + | |
| 2501 | + | |
2413 | 2502 | | |
2414 | 2503 | | |
2415 | 2504 | | |
| |||
0 commit comments