Repository navigation
Commit c4cd513
authored
fix(io): keep null lists of structs when projecting Arrow data (apache#4065)
The projection visitor rebuilds a list of structs from its offsets alone,
which cannot represent a null list, so every null list<struct<...>> was
turned into an empty list. On the write path this is persisted into the
Parquet file and the null is lost for every reader.
Rebuild the list from the original validity and offsets buffers instead,
which also keeps the original array offset so sliced batches stay correct.
Closes apache#38331 parent 03e80a7 commit c4cd513
3 files changed
Lines changed: 46 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2198 | 2198 | | |
2199 | 2199 | | |
2200 | 2200 | | |
2201 | | - | |
| 2201 | + | |
| 2202 | + | |
| 2203 | + | |
| 2204 | + | |
| 2205 | + | |
| 2206 | + | |
| 2207 | + | |
| 2208 | + | |
| 2209 | + | |
2202 | 2210 | | |
2203 | 2211 | | |
2204 | 2212 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1002 | 1002 | | |
1003 | 1003 | | |
1004 | 1004 | | |
1005 | | - | |
1006 | | - | |
1007 | | - | |
1008 | | - | |
| 1005 | + | |
1009 | 1006 | | |
1010 | 1007 | | |
1011 | 1008 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
3946 | 3946 | | |
3947 | 3947 | | |
3948 | 3948 | | |
| 3949 | + | |
| 3950 | + | |
| 3951 | + | |
| 3952 | + | |
| 3953 | + | |
| 3954 | + | |
| 3955 | + | |
| 3956 | + | |
| 3957 | + | |
| 3958 | + | |
| 3959 | + | |
| 3960 | + | |
| 3961 | + | |
| 3962 | + | |
| 3963 | + | |
| 3964 | + | |
| 3965 | + | |
| 3966 | + | |
| 3967 | + | |
| 3968 | + | |
| 3969 | + | |
| 3970 | + | |
| 3971 | + | |
| 3972 | + | |
| 3973 | + | |
| 3974 | + | |
| 3975 | + | |
| 3976 | + | |
| 3977 | + | |
| 3978 | + | |
| 3979 | + | |
| 3980 | + | |
| 3981 | + | |
| 3982 | + | |
| 3983 | + | |
3949 | 3984 | | |
3950 | 3985 | | |
3951 | 3986 | | |
| |||
0 commit comments