Repository navigation
Commit 6df3d03
committed
fix(io): keep null lists of structs when projecting Arrow data
The projection visitor rebuilds a list of structs from its offsets alone,
which cannot represent a null list, so every null list<struct<...>> was
turned into an empty list. On the write path this is persisted into the
Parquet file and the null is lost for every reader.
Rebuild the list from the original validity and offsets buffers instead,
which also keeps the original array offset so sliced batches stay correct.
Closes apache#38331 parent 1310a19 commit 6df3d03
3 files changed
Lines changed: 46 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2183 | 2183 | | |
2184 | 2184 | | |
2185 | 2185 | | |
2186 | | - | |
| 2186 | + | |
| 2187 | + | |
| 2188 | + | |
| 2189 | + | |
| 2190 | + | |
| 2191 | + | |
| 2192 | + | |
| 2193 | + | |
| 2194 | + | |
2187 | 2195 | | |
2188 | 2196 | | |
2189 | 2197 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1001 | 1001 | | |
1002 | 1002 | | |
1003 | 1003 | | |
1004 | | - | |
1005 | | - | |
1006 | | - | |
1007 | | - | |
| 1004 | + | |
1008 | 1005 | | |
1009 | 1006 | | |
1010 | 1007 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
3813 | 3813 | | |
3814 | 3814 | | |
3815 | 3815 | | |
| 3816 | + | |
| 3817 | + | |
| 3818 | + | |
| 3819 | + | |
| 3820 | + | |
| 3821 | + | |
| 3822 | + | |
| 3823 | + | |
| 3824 | + | |
| 3825 | + | |
| 3826 | + | |
| 3827 | + | |
| 3828 | + | |
| 3829 | + | |
| 3830 | + | |
| 3831 | + | |
| 3832 | + | |
| 3833 | + | |
| 3834 | + | |
| 3835 | + | |
| 3836 | + | |
| 3837 | + | |
| 3838 | + | |
| 3839 | + | |
| 3840 | + | |
| 3841 | + | |
| 3842 | + | |
| 3843 | + | |
| 3844 | + | |
| 3845 | + | |
| 3846 | + | |
| 3847 | + | |
| 3848 | + | |
| 3849 | + | |
| 3850 | + | |
3816 | 3851 | | |
3817 | 3852 | | |
3818 | 3853 | | |
| |||
0 commit comments