Repository navigation
Commit bee5811
committed
Preserve field metadata when coercing INT96 timestamps
Backport of the fix on main to the 53.1.0 release, for pinning via
[patch.crates-io] until upstream lands it. Upstream issue:
apache#24786
`coerce_int96_to_resolution` rebuilds every struct, list and map field with
`Field::new_struct`, `Field::new_list` and `Field::new`, which start from empty
metadata. Leaf fields go through `field_with_new_type`, which clones the field,
so only container fields are affected, and only when the file holds an INT96
column: the same file read without `coerce_int96` keeps its metadata.
`PARQUET:field_id` lives in that metadata, and formats that identify a column
by id rather than by name depend on it. Delta Lake column mapping is one: the
read schema names columns `col-<id>` and pairs them with the file by field id,
so a struct column that loses its id is resolved by neither id nor name. It
reads back as NULL, or errors if the table declares the column NOT NULL.
Carry `current_field.metadata()` across in the three constructors.1 parent eae7bf4 commit bee5811
1 file changed
Lines changed: 56 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
816 | 816 | | |
817 | 817 | | |
818 | 818 | | |
819 | | - | |
| 819 | + | |
| 820 | + | |
820 | 821 | | |
821 | 822 | | |
822 | 823 | | |
| |||
849 | 850 | | |
850 | 851 | | |
851 | 852 | | |
852 | | - | |
| 853 | + | |
| 854 | + | |
853 | 855 | | |
854 | 856 | | |
855 | 857 | | |
| |||
879 | 881 | | |
880 | 882 | | |
881 | 883 | | |
882 | | - | |
| 884 | + | |
| 885 | + | |
883 | 886 | | |
884 | 887 | | |
885 | 888 | | |
| |||
1778 | 1781 | | |
1779 | 1782 | | |
1780 | 1783 | | |
| 1784 | + | |
1781 | 1785 | | |
1782 | 1786 | | |
1783 | 1787 | | |
| |||
1992 | 1996 | | |
1993 | 1997 | | |
1994 | 1998 | | |
| 1999 | + | |
| 2000 | + | |
| 2001 | + | |
| 2002 | + | |
| 2003 | + | |
| 2004 | + | |
| 2005 | + | |
| 2006 | + | |
| 2007 | + | |
| 2008 | + | |
| 2009 | + | |
| 2010 | + | |
| 2011 | + | |
| 2012 | + | |
| 2013 | + | |
| 2014 | + | |
| 2015 | + | |
| 2016 | + | |
| 2017 | + | |
| 2018 | + | |
| 2019 | + | |
| 2020 | + | |
| 2021 | + | |
| 2022 | + | |
| 2023 | + | |
| 2024 | + | |
| 2025 | + | |
| 2026 | + | |
| 2027 | + | |
| 2028 | + | |
| 2029 | + | |
| 2030 | + | |
| 2031 | + | |
| 2032 | + | |
| 2033 | + | |
| 2034 | + | |
| 2035 | + | |
| 2036 | + | |
| 2037 | + | |
| 2038 | + | |
| 2039 | + | |
| 2040 | + | |
| 2041 | + | |
| 2042 | + | |
| 2043 | + | |
| 2044 | + | |
| 2045 | + | |
| 2046 | + | |
| 2047 | + | |
1995 | 2048 | | |
0 commit comments