Repository navigation
Commit f136c36
committed
fix(arrow): address final review on nested field-id promotion
- Add a reader-level test that writes an old nested schema to Parquet and
reads it back through an evolved one, pinning that parquet-rs attaches
field ids to struct children, list elements, and map keys and values
- Reject duplicate source field ids and nested sources read as a
non-nested type when the plan is built
- Report non-struct map entries as DataInvalid
- Pin the id-less same-type reorder fallback with a test until #1845
- Assert map data types, since map array equality skips field names
- Point at #3378 where the projection mask skips all-missing structs1 parent ee881aa commit f136c36
2 files changed
Lines changed: 195 additions & 12 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
221 | 221 | | |
222 | 222 | | |
223 | 223 | | |
| 224 | + | |
| 225 | + | |
224 | 226 | | |
225 | 227 | | |
226 | 228 | | |
| |||
2046 | 2048 | | |
2047 | 2049 | | |
2048 | 2050 | | |
| 2051 | + | |
| 2052 | + | |
| 2053 | + | |
| 2054 | + | |
| 2055 | + | |
| 2056 | + | |
| 2057 | + | |
| 2058 | + | |
| 2059 | + | |
| 2060 | + | |
| 2061 | + | |
| 2062 | + | |
| 2063 | + | |
| 2064 | + | |
| 2065 | + | |
| 2066 | + | |
| 2067 | + | |
| 2068 | + | |
| 2069 | + | |
| 2070 | + | |
| 2071 | + | |
| 2072 | + | |
| 2073 | + | |
| 2074 | + | |
| 2075 | + | |
| 2076 | + | |
| 2077 | + | |
| 2078 | + | |
| 2079 | + | |
| 2080 | + | |
| 2081 | + | |
| 2082 | + | |
| 2083 | + | |
| 2084 | + | |
| 2085 | + | |
| 2086 | + | |
| 2087 | + | |
| 2088 | + | |
| 2089 | + | |
| 2090 | + | |
| 2091 | + | |
| 2092 | + | |
| 2093 | + | |
| 2094 | + | |
| 2095 | + | |
| 2096 | + | |
| 2097 | + | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
| 2102 | + | |
| 2103 | + | |
| 2104 | + | |
| 2105 | + | |
| 2106 | + | |
| 2107 | + | |
| 2108 | + | |
| 2109 | + | |
| 2110 | + | |
| 2111 | + | |
| 2112 | + | |
| 2113 | + | |
| 2114 | + | |
| 2115 | + | |
| 2116 | + | |
| 2117 | + | |
| 2118 | + | |
| 2119 | + | |
| 2120 | + | |
| 2121 | + | |
| 2122 | + | |
| 2123 | + | |
| 2124 | + | |
| 2125 | + | |
| 2126 | + | |
| 2127 | + | |
| 2128 | + | |
| 2129 | + | |
| 2130 | + | |
| 2131 | + | |
| 2132 | + | |
| 2133 | + | |
| 2134 | + | |
| 2135 | + | |
| 2136 | + | |
| 2137 | + | |
| 2138 | + | |
| 2139 | + | |
| 2140 | + | |
| 2141 | + | |
| 2142 | + | |
| 2143 | + | |
| 2144 | + | |
| 2145 | + | |
| 2146 | + | |
| 2147 | + | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
| 2163 | + | |
| 2164 | + | |
| 2165 | + | |
| 2166 | + | |
| 2167 | + | |
| 2168 | + | |
| 2169 | + | |
| 2170 | + | |
| 2171 | + | |
| 2172 | + | |
2049 | 2173 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
283 | 283 | | |
284 | 284 | | |
285 | 285 | | |
286 | | - | |
287 | | - | |
288 | | - | |
289 | | - | |
290 | | - | |
| 286 | + | |
| 287 | + | |
291 | 288 | | |
292 | 289 | | |
293 | 290 | | |
294 | 291 | | |
295 | | - | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
296 | 295 | | |
297 | 296 | | |
298 | 297 | | |
| |||
307 | 306 | | |
308 | 307 | | |
309 | 308 | | |
310 | | - | |
311 | | - | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
312 | 313 | | |
313 | 314 | | |
314 | | - | |
315 | | - | |
316 | | - | |
317 | | - | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
318 | 319 | | |
319 | 320 | | |
320 | 321 | | |
| |||
1633 | 1634 | | |
1634 | 1635 | | |
1635 | 1636 | | |
| 1637 | + | |
1636 | 1638 | | |
1637 | 1639 | | |
1638 | 1640 | | |
| |||
1694 | 1696 | | |
1695 | 1697 | | |
1696 | 1698 | | |
| 1699 | + | |
| 1700 | + | |
| 1701 | + | |
| 1702 | + | |
| 1703 | + | |
| 1704 | + | |
| 1705 | + | |
| 1706 | + | |
| 1707 | + | |
| 1708 | + | |
| 1709 | + | |
| 1710 | + | |
| 1711 | + | |
| 1712 | + | |
| 1713 | + | |
| 1714 | + | |
| 1715 | + | |
| 1716 | + | |
| 1717 | + | |
| 1718 | + | |
| 1719 | + | |
| 1720 | + | |
| 1721 | + | |
| 1722 | + | |
1697 | 1723 | | |
1698 | 1724 | | |
1699 | 1725 | | |
| |||
1896 | 1922 | | |
1897 | 1923 | | |
1898 | 1924 | | |
| 1925 | + | |
| 1926 | + | |
1899 | 1927 | | |
1900 | 1928 | | |
1901 | 1929 | | |
| 1930 | + | |
| 1931 | + | |
| 1932 | + | |
| 1933 | + | |
| 1934 | + | |
| 1935 | + | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
| 1940 | + | |
| 1941 | + | |
| 1942 | + | |
| 1943 | + | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
| 1950 | + | |
| 1951 | + | |
| 1952 | + | |
| 1953 | + | |
| 1954 | + | |
| 1955 | + | |
| 1956 | + | |
| 1957 | + | |
| 1958 | + | |
| 1959 | + | |
| 1960 | + | |
1902 | 1961 | | |
1903 | 1962 | | |
1904 | 1963 | | |
| |||
0 commit comments