Repository navigation
Commit a9b61ab
authored
fix(parquet): remap sorting columns for partitioned writes (#24211)
## Which issue does this PR close?
- Closes #24210.
## Rationale for this change
When Hive partition columns are not kept in Parquet files,
`FileSinkConfig::output_schema()` still describes the sink input while
`get_writer_schema(&conf)` removes those partition columns.
`ParquetFormat::create_writer_physical_plan` currently converts the
input ordering directly to Parquet `sorting_columns`, so its column
indices can reference removed columns or the wrong positions in the
written schema. Reading that metadata can then panic in
`SchemaDescriptor::column` with an out-of-bounds index.
The execution ordering and the Parquet footer ordering have different
schema domains: `DataSinkExec` must retain the original input ordering,
while `ParquetSink` metadata must use indices from the actual writer
schema.
## What changes are included in this PR?
- Derive Parquet `sorting_columns` from both the input schema and
`get_writer_schema(&conf)`.
- Omit ordering keys that are removed from the written file, such as
Hive partition columns.
- Remap retained ordering keys to their writer-schema indices while
preserving sort direction and null ordering.
- Keep the original `order_requirements` unchanged for `DataSinkExec`.
- Avoid writing an empty `sorting_columns` list when every ordering key
is removed.
- Add unit and end-to-end regression coverage for partitioned Parquet
writes.
## Are these changes tested?
Yes. The new tests cover the writer-schema remapping directly and verify
the resulting Parquet footer after a partitioned write.
## Are there any user-facing changes?
There are no public API changes. Newly written partitioned Parquet files
now contain valid `sorting_columns` metadata based on their physical
file schema. Existing files with invalid metadata are unchanged and must
be rewritten separately.1 parent d5bd10d commit a9b61ab
3 files changed
Lines changed: 177 additions & 10 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
| 53 | + | |
53 | 54 | | |
54 | 55 | | |
55 | 56 | | |
| |||
534 | 535 | | |
535 | 536 | | |
536 | 537 | | |
| 538 | + | |
537 | 539 | | |
538 | 540 | | |
539 | 541 | | |
540 | 542 | | |
541 | | - | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
542 | 550 | | |
543 | 551 | | |
544 | 552 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
| |||
952 | 953 | | |
953 | 954 | | |
954 | 955 | | |
955 | | - | |
| 956 | + | |
| 957 | + | |
| 958 | + | |
| 959 | + | |
956 | 960 | | |
957 | 961 | | |
958 | | - | |
| 962 | + | |
| 963 | + | |
| 964 | + | |
959 | 965 | | |
960 | 966 | | |
961 | 967 | | |
| |||
964 | 970 | | |
965 | 971 | | |
966 | 972 | | |
967 | | - | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
968 | 987 | | |
969 | | - | |
970 | | - | |
| 988 | + | |
971 | 989 | | |
972 | 990 | | |
973 | 991 | | |
974 | | - | |
| 992 | + | |
975 | 993 | | |
976 | 994 | | |
977 | 995 | | |
978 | | - | |
| 996 | + | |
979 | 997 | | |
980 | 998 | | |
981 | 999 | | |
982 | 1000 | | |
983 | | - | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
984 | 1004 | | |
985 | 1005 | | |
| 1006 | + | |
| 1007 | + | |
986 | 1008 | | |
987 | | - | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
988 | 1016 | | |
989 | 1017 | | |
990 | 1018 | | |
| |||
1058 | 1086 | | |
1059 | 1087 | | |
1060 | 1088 | | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
1061 | 1140 | | |
1062 | 1141 | | |
1063 | 1142 | | |
| |||
0 commit comments