Repository navigation
Commit eabee9e
committed
[SPARK-59832][4.1][SQL] Fail on truncated input in the vectorized RLE reader
### What changes were proposed in this pull request?
Backport of #59117 to branch-4.1.
This branch does not have `VectorizedRleValuesReaderSuite`, `ParquetTestAccess` or `VectorizedRleValuesReaderTestUtils`, so this PR adds a `VectorizedRleValuesReaderSuite` with the 14 tests from #59117 and the helpers they use. It creates and drives the package-private `ParquetReadState` directly instead of through the `ParquetTestAccess` bridge.
It also does not have SPARK-56895, so `readGroup` still slices a bit-packed run one group at a time; the new checks run before that loop, as on master. Tests that came into the conflicts as context but are not on this branch are left out (the SPARK-57828 test in `ParquetIOSuite` and the SPARK-59831 test in `ParquetVectorizedSuite`), and `ParquetVectorizedSuite` now imports `BytesInput`, `Encoding` and `Statistics` for the new dictionary page test.
### Why are the changes needed?
Same as #59117. On a corrupt page, the vectorized RLE reader, which decodes definition/repetition levels, dictionary ids and RLE booleans, silently returned wrong values or looped forever when the encoded data ended early, and a corrupt bit-packed header could overflow `currentCount` or allocate a huge buffer. The reader now fails with a `ParquetDecodingException`.
### Does this PR introduce _any_ user-facing change?
Only for corrupt or non-compliant Parquet files, as described in #59117: such pages now fail with a `ParquetDecodingException` instead of returning wrong values or hanging, and with `spark.sql.files.ignoreCorruptFiles=true` the rest of such a file is skipped. A few of these pages could be read before and gave the right result by chance; #59117 lists them.
### How was this patch tested?
The 18 tests added in #59117 pass on this branch, as do the existing tests in `ParquetEncodingSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetDeltaByteArrayEncodingSuite`, `ParquetDeltaLengthByteArrayEncodingSuite` and `VectorizedPlainValuesReaderSuite`. Without the fix, the 13 negative tests in `VectorizedRleValuesReaderSuite` fail, as on master.
```
build/sbt 'sql/testOnly *VectorizedRleValuesReaderSuite *ParquetEncodingSuite *ParquetVectorizedSuite *ParquetIOSuite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
Closes #59483 from viirya/SPARK-59832-branch-4.1.
Authored-by: Liang-Chi Hsieh <[email protected]>
Signed-off-by: Liang-Chi Hsieh <[email protected]>1 parent 40e335f commit eabee9e
5 files changed
Lines changed: 607 additions & 26 deletions
File tree
- sql/core/src
- main/java/org/apache/spark/sql/execution/datasources/parquet
- test/scala/org/apache/spark/sql/execution/datasources/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| 35 | + | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| |||
129 | 130 | | |
130 | 131 | | |
131 | 132 | | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
132 | 136 | | |
133 | 137 | | |
134 | 138 | | |
| |||
331 | 335 | | |
332 | 336 | | |
333 | 337 | | |
334 | | - | |
335 | | - | |
336 | | - | |
| 338 | + | |
337 | 339 | | |
338 | 340 | | |
339 | 341 | | |
| |||
348 | 350 | | |
349 | 351 | | |
350 | 352 | | |
351 | | - | |
352 | | - | |
353 | | - | |
354 | | - | |
355 | | - | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
356 | 356 | | |
357 | 357 | | |
358 | 358 | | |
| |||
409 | 409 | | |
410 | 410 | | |
411 | 411 | | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
412 | 415 | | |
413 | 416 | | |
414 | 417 | | |
| |||
429 | 432 | | |
430 | 433 | | |
431 | 434 | | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
432 | 438 | | |
433 | 439 | | |
434 | 440 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
65 | 68 | | |
66 | 69 | | |
67 | 70 | | |
| |||
96 | 99 | | |
97 | 100 | | |
98 | 101 | | |
| 102 | + | |
99 | 103 | | |
100 | 104 | | |
101 | 105 | | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
102 | 110 | | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
103 | 115 | | |
104 | 116 | | |
105 | 117 | | |
106 | 118 | | |
107 | 119 | | |
108 | | - | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
109 | 131 | | |
110 | 132 | | |
111 | 133 | | |
| |||
208 | 230 | | |
209 | 231 | | |
210 | 232 | | |
211 | | - | |
| 233 | + | |
212 | 234 | | |
213 | 235 | | |
214 | 236 | | |
| |||
282 | 304 | | |
283 | 305 | | |
284 | 306 | | |
285 | | - | |
| 307 | + | |
286 | 308 | | |
287 | 309 | | |
288 | 310 | | |
| |||
395 | 417 | | |
396 | 418 | | |
397 | 419 | | |
398 | | - | |
| 420 | + | |
399 | 421 | | |
400 | 422 | | |
401 | 423 | | |
| |||
601 | 623 | | |
602 | 624 | | |
603 | 625 | | |
604 | | - | |
| 626 | + | |
605 | 627 | | |
606 | 628 | | |
607 | 629 | | |
| |||
660 | 682 | | |
661 | 683 | | |
662 | 684 | | |
663 | | - | |
| 685 | + | |
664 | 686 | | |
665 | 687 | | |
666 | 688 | | |
| |||
693 | 715 | | |
694 | 716 | | |
695 | 717 | | |
696 | | - | |
| 718 | + | |
697 | 719 | | |
698 | 720 | | |
699 | 721 | | |
| |||
876 | 898 | | |
877 | 899 | | |
878 | 900 | | |
879 | | - | |
880 | 901 | | |
881 | 902 | | |
882 | 903 | | |
| |||
895 | 916 | | |
896 | 917 | | |
897 | 918 | | |
898 | | - | |
| 919 | + | |
| 920 | + | |
| 921 | + | |
| 922 | + | |
| 923 | + | |
899 | 924 | | |
900 | | - | |
901 | | - | |
902 | | - | |
903 | | - | |
| 925 | + | |
| 926 | + | |
| 927 | + | |
| 928 | + | |
904 | 929 | | |
| 930 | + | |
| 931 | + | |
| 932 | + | |
| 933 | + | |
| 934 | + | |
| 935 | + | |
| 936 | + | |
| 937 | + | |
905 | 938 | | |
| 939 | + | |
| 940 | + | |
| 941 | + | |
| 942 | + | |
906 | 943 | | |
907 | 944 | | |
908 | 945 | | |
| |||
913 | 950 | | |
914 | 951 | | |
915 | 952 | | |
| 953 | + | |
| 954 | + | |
| 955 | + | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
| 960 | + | |
| 961 | + | |
| 962 | + | |
| 963 | + | |
916 | 964 | | |
917 | 965 | | |
918 | 966 | | |
| |||
929 | 977 | | |
930 | 978 | | |
931 | 979 | | |
932 | | - | |
| 980 | + | |
933 | 981 | | |
934 | | - | |
935 | | - | |
936 | 982 | | |
937 | 983 | | |
938 | 984 | | |
| |||
941 | 987 | | |
942 | 988 | | |
943 | 989 | | |
944 | | - | |
| 990 | + | |
945 | 991 | | |
946 | 992 | | |
947 | 993 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
| |||
26 | 27 | | |
27 | 28 | | |
28 | 29 | | |
| 30 | + | |
29 | 31 | | |
30 | 32 | | |
| 33 | + | |
31 | 34 | | |
32 | 35 | | |
33 | 36 | | |
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
| 40 | + | |
37 | 41 | | |
38 | 42 | | |
| 43 | + | |
| 44 | + | |
39 | 45 | | |
40 | 46 | | |
41 | 47 | | |
| |||
1917 | 1923 | | |
1918 | 1924 | | |
1919 | 1925 | | |
| 1926 | + | |
| 1927 | + | |
| 1928 | + | |
| 1929 | + | |
| 1930 | + | |
| 1931 | + | |
| 1932 | + | |
| 1933 | + | |
| 1934 | + | |
| 1935 | + | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
| 1940 | + | |
| 1941 | + | |
| 1942 | + | |
| 1943 | + | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
| 1950 | + | |
| 1951 | + | |
| 1952 | + | |
| 1953 | + | |
| 1954 | + | |
| 1955 | + | |
| 1956 | + | |
| 1957 | + | |
| 1958 | + | |
| 1959 | + | |
| 1960 | + | |
| 1961 | + | |
| 1962 | + | |
| 1963 | + | |
| 1964 | + | |
| 1965 | + | |
| 1966 | + | |
| 1967 | + | |
| 1968 | + | |
| 1969 | + | |
| 1970 | + | |
| 1971 | + | |
| 1972 | + | |
| 1973 | + | |
| 1974 | + | |
| 1975 | + | |
| 1976 | + | |
| 1977 | + | |
| 1978 | + | |
| 1979 | + | |
| 1980 | + | |
| 1981 | + | |
| 1982 | + | |
| 1983 | + | |
| 1984 | + | |
| 1985 | + | |
| 1986 | + | |
| 1987 | + | |
| 1988 | + | |
| 1989 | + | |
| 1990 | + | |
| 1991 | + | |
| 1992 | + | |
| 1993 | + | |
| 1994 | + | |
| 1995 | + | |
| 1996 | + | |
| 1997 | + | |
| 1998 | + | |
| 1999 | + | |
| 2000 | + | |
| 2001 | + | |
| 2002 | + | |
| 2003 | + | |
| 2004 | + | |
| 2005 | + | |
| 2006 | + | |
| 2007 | + | |
| 2008 | + | |
1920 | 2009 | | |
1921 | 2010 | | |
1922 | 2011 | | |
| |||
0 commit comments