Repository navigation
Commit 45f1cc8
[SPARK-59832][4.X][SQL] Fail on truncated input in the vectorized RLE reader
### What changes were proposed in this pull request?
Backport of #59117 to branch-4.x.
The cherry-pick conflicted only on code that SPARK-59620 added to master, which is not on this branch: an import in `VectorizedColumnReader` and `VectorizedRleValuesReaderSuite`, and a doc comment next to the new `readBatchRepeated` bridge in `ParquetTestAccess`. They are left out; otherwise the change is the same as on master.
This commit also applies cleanly to branch-4.3, so it can be cherry-picked there when merging. The same change was built and tested on branch-4.3.
### Why are the changes needed?
Same as #59117. On a corrupt page, the vectorized RLE reader, which decodes definition/repetition levels, dictionary ids and RLE booleans, silently returned wrong values or looped forever when the encoded data ended early, and a corrupt bit-packed header could overflow `currentCount` or allocate a huge buffer. The reader now fails with a `ParquetDecodingException`.
### Does this PR introduce _any_ user-facing change?
Only for corrupt or non-compliant Parquet files, as described in #59117: such pages now fail with a `ParquetDecodingException` instead of returning wrong values or hanging, and with `spark.sql.files.ignoreCorruptFiles=true` the rest of such a file is skipped. A few of these pages could be read before and gave the right result by chance; #59117 lists them.
### How was this patch tested?
The 18 tests added in #59117 pass on this branch, as do the existing tests in `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite`, `VectorizedPlainValuesReaderSuite` and `ParquetVectorUpdaterSuite`.
```
build/sbt 'sql/testOnly *VectorizedRleValuesReaderSuite *ParquetEncodingSuite *ParquetVectorizedSuite *ParquetIOSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59481 from viirya/SPARK-59832-branch-4.x.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>1 parent 212ff5a commit 45f1cc8
6 files changed
Lines changed: 530 additions & 29 deletions
File tree
- sql/core/src
- main/java/org/apache/spark/sql/execution/datasources/parquet
- test/scala/org/apache/spark/sql/execution/datasources/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
132 | 133 | | |
133 | 134 | | |
134 | 135 | | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
135 | 139 | | |
136 | 140 | | |
137 | 141 | | |
| |||
343 | 347 | | |
344 | 348 | | |
345 | 349 | | |
346 | | - | |
347 | | - | |
348 | | - | |
| 350 | + | |
349 | 351 | | |
350 | 352 | | |
351 | 353 | | |
| |||
360 | 362 | | |
361 | 363 | | |
362 | 364 | | |
363 | | - | |
364 | | - | |
365 | | - | |
366 | | - | |
367 | | - | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
368 | 368 | | |
369 | 369 | | |
370 | 370 | | |
| |||
434 | 434 | | |
435 | 435 | | |
436 | 436 | | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
437 | 440 | | |
438 | 441 | | |
439 | 442 | | |
| |||
454 | 457 | | |
455 | 458 | | |
456 | 459 | | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
457 | 463 | | |
458 | 464 | | |
459 | 465 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
65 | 68 | | |
66 | 69 | | |
67 | 70 | | |
| |||
96 | 99 | | |
97 | 100 | | |
98 | 101 | | |
| 102 | + | |
99 | 103 | | |
100 | 104 | | |
101 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
102 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
103 | 115 | | |
104 | 116 | | |
105 | 117 | | |
106 | 118 | | |
107 | 119 | | |
108 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
109 | 131 | | |
110 | 132 | | |
111 | 133 | | |
| |||
208 | 230 | | |
209 | 231 | | |
210 | 232 | | |
211 | | - | |
| 233 | + | |
212 | 234 | | |
213 | 235 | | |
214 | 236 | | |
| |||
318 | 340 | | |
319 | 341 | | |
320 | 342 | | |
321 | | - | |
| 343 | + | |
322 | 344 | | |
323 | 345 | | |
324 | 346 | | |
| |||
431 | 453 | | |
432 | 454 | | |
433 | 455 | | |
434 | | - | |
| 456 | + | |
435 | 457 | | |
436 | 458 | | |
437 | 459 | | |
| |||
637 | 659 | | |
638 | 660 | | |
639 | 661 | | |
640 | | - | |
| 662 | + | |
641 | 663 | | |
642 | 664 | | |
643 | 665 | | |
| |||
731 | 753 | | |
732 | 754 | | |
733 | 755 | | |
734 | | - | |
| 756 | + | |
735 | 757 | | |
736 | 758 | | |
737 | 759 | | |
| |||
764 | 786 | | |
765 | 787 | | |
766 | 788 | | |
767 | | - | |
| 789 | + | |
768 | 790 | | |
769 | 791 | | |
770 | 792 | | |
| |||
957 | 979 | | |
958 | 980 | | |
959 | 981 | | |
960 | | - | |
961 | 982 | | |
962 | 983 | | |
963 | 984 | | |
| |||
976 | 997 | | |
977 | 998 | | |
978 | 999 | | |
979 | | - | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
980 | 1005 | | |
981 | | - | |
982 | | - | |
983 | | - | |
984 | | - | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
985 | 1010 | | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
| 1016 | + | |
| 1017 | + | |
| 1018 | + | |
986 | 1019 | | |
| 1020 | + | |
| 1021 | + | |
| 1022 | + | |
| 1023 | + | |
987 | 1024 | | |
988 | 1025 | | |
989 | 1026 | | |
| |||
994 | 1031 | | |
995 | 1032 | | |
996 | 1033 | | |
| 1034 | + | |
| 1035 | + | |
| 1036 | + | |
| 1037 | + | |
| 1038 | + | |
| 1039 | + | |
| 1040 | + | |
| 1041 | + | |
| 1042 | + | |
| 1043 | + | |
| 1044 | + | |
997 | 1045 | | |
998 | 1046 | | |
999 | 1047 | | |
| |||
1014 | 1062 | | |
1015 | 1063 | | |
1016 | 1064 | | |
1017 | | - | |
| 1065 | + | |
1018 | 1066 | | |
1019 | | - | |
1020 | | - | |
1021 | 1067 | | |
1022 | 1068 | | |
1023 | 1069 | | |
| |||
1026 | 1072 | | |
1027 | 1073 | | |
1028 | 1074 | | |
1029 | | - | |
| 1075 | + | |
1030 | 1076 | | |
1031 | 1077 | | |
1032 | 1078 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
27 | 28 | | |
28 | 29 | | |
29 | 30 | | |
| 31 | + | |
30 | 32 | | |
31 | 33 | | |
| 34 | + | |
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
37 | 40 | | |
| 41 | + | |
38 | 42 | | |
39 | 43 | | |
| 44 | + | |
| 45 | + | |
40 | 46 | | |
41 | 47 | | |
42 | 48 | | |
| |||
2480 | 2486 | | |
2481 | 2487 | | |
2482 | 2488 | | |
| 2489 | + | |
| 2490 | + | |
| 2491 | + | |
| 2492 | + | |
| 2493 | + | |
| 2494 | + | |
| 2495 | + | |
| 2496 | + | |
| 2497 | + | |
| 2498 | + | |
| 2499 | + | |
| 2500 | + | |
| 2501 | + | |
| 2502 | + | |
| 2503 | + | |
| 2504 | + | |
| 2505 | + | |
| 2506 | + | |
| 2507 | + | |
| 2508 | + | |
| 2509 | + | |
| 2510 | + | |
| 2511 | + | |
| 2512 | + | |
| 2513 | + | |
| 2514 | + | |
| 2515 | + | |
| 2516 | + | |
| 2517 | + | |
| 2518 | + | |
| 2519 | + | |
| 2520 | + | |
| 2521 | + | |
| 2522 | + | |
| 2523 | + | |
| 2524 | + | |
| 2525 | + | |
| 2526 | + | |
| 2527 | + | |
| 2528 | + | |
| 2529 | + | |
| 2530 | + | |
| 2531 | + | |
| 2532 | + | |
| 2533 | + | |
| 2534 | + | |
| 2535 | + | |
| 2536 | + | |
| 2537 | + | |
| 2538 | + | |
| 2539 | + | |
| 2540 | + | |
| 2541 | + | |
| 2542 | + | |
| 2543 | + | |
| 2544 | + | |
| 2545 | + | |
| 2546 | + | |
| 2547 | + | |
| 2548 | + | |
| 2549 | + | |
| 2550 | + | |
| 2551 | + | |
| 2552 | + | |
| 2553 | + | |
| 2554 | + | |
| 2555 | + | |
| 2556 | + | |
| 2557 | + | |
| 2558 | + | |
| 2559 | + | |
| 2560 | + | |
| 2561 | + | |
| 2562 | + | |
| 2563 | + | |
| 2564 | + | |
| 2565 | + | |
| 2566 | + | |
| 2567 | + | |
| 2568 | + | |
| 2569 | + | |
| 2570 | + | |
| 2571 | + | |
2483 | 2572 | | |
2484 | 2573 | | |
2485 | 2574 | | |
| |||
0 commit comments