Repository navigation
Commit 863f11d
authored
fix: reject a file without field ids at any depth whether or not id matching is on (apache#6116)
* fix: reject a file without field ids at any depth whether or not id matching is on
Spark's `ParquetReadSupport.getRequestedSchema` raises when the requested
schema carries a Parquet field id at any depth and the file carries none at
any depth, unless `spark.sql.parquet.fieldId.read.ignoreMissing` is set. The
check runs on every read, for both readers, and does not consult
`spark.sql.parquet.fieldId.read.enabled`.
The native scan ran that check only with the read flag on, only over root
fields, and only inside the name remap, which a case-sensitive session with
the flag off never reaches. A read schema with ids over a file without ids
returned rows where Spark raises, and a file whose ids sit only on nested
fields was rejected where Spark reads it and null-fills the unmatched fields.
The check now runs at the top of the expression adapter factory with a
recursive predicate on both schemas. Id matching itself stays gated on the
read flag and root ids, as Spark gates it per struct level. The Iceberg scan
opts out, since its reader resolves columns by id and supplies the ids itself.
* fix: answer both halves of the missing-field-id check from what Spark reads
Spark checks the pruned required schema for ids and the raw Parquet schema
for their absence. The requested half now comes from `required_schema` at
plan time, since the logical schema DataFusion hands the adapter is the full
read schema, so an id on a column the query never projects no longer raises.
The file half moves into the eager page index reader, which walks the raw
schema from the footer. The Arrow schema the adapter sees has lost container
metadata after INT96 coercion and never carries ids on repeated list or key
value groups, so a file whose only id sat on a struct holding a timestamp
raised where Spark reads. The JNI error conversion unwraps the Parquet
external error so the Java side keeps the same exception, and the Iceberg
scan no longer needs an opt-out since it does not use that reader.
Tests cover the pruned projection, the timestamp struct, an id only on a list
or key value group, a directory mixing files with and without ids, nested
schema pruning, id zero, count(*), and the exception class on each version.
* test: find the missing field ids error anywhere in the cause chain
On Spark 4.0 and 4.1 the RuntimeException Spark raises for a read schema with
field ids over a file without any arrives wrapped once, in the FAILED_READ_FILE
SparkException that collect raises, and the helper assumed one more layer above
it. It now walks the cause chain and requires a plain RuntimeException carrying
Spark's message, for Spark and for Comet alike, so the layer count no longer
matters.
* fix: let the serde say whether the file must carry field ids
The requested side of the missing field ids check is Spark's own hasFieldIds
over the required schema, so the serde now sends one flag, require_field_ids,
in place of ignore_missing_field_id, and the native Arrow walk is gone. The
reader factory takes that one flag through a builder and the parquet options
carry nothing for it. The error names the file, and the module doc leads with
the lasting reason the check reads the raw footer: ids on repeated groups and
on the message root never reach Arrow.
The Rust scan tests that had Scala twins are gone, the repeated group cases
moved to Scala where they compare against Spark, and the Scala tests fold into
the existing port and one pruned schema test with Spark's own assertion form.1 parent bc4be39 commit 863f11d
10 files changed
Lines changed: 496 additions & 73 deletions
File tree
- native
- common/src
- core/src
- execution
- parquet
- jni-bridge/src
- proto/src/proto
- spark/src
- main/scala/org/apache/comet/serde/operator
- test/scala/org/apache/comet/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
233 | 233 | | |
234 | 234 | | |
235 | 235 | | |
236 | | - | |
237 | | - | |
238 | | - | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
239 | 240 | | |
240 | | - | |
| 241 | + | |
241 | 242 | | |
242 | 243 | | |
243 | 244 | | |
| |||
353 | 354 | | |
354 | 355 | | |
355 | 356 | | |
356 | | - | |
| 357 | + | |
357 | 358 | | |
358 | 359 | | |
359 | 360 | | |
| |||
604 | 605 | | |
605 | 606 | | |
606 | 607 | | |
| 608 | + | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
607 | 613 | | |
608 | 614 | | |
609 | 615 | | |
| |||
711 | 717 | | |
712 | 718 | | |
713 | 719 | | |
714 | | - | |
715 | | - | |
716 | | - | |
717 | | - | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
718 | 725 | | |
719 | 726 | | |
720 | 727 | | |
| |||
814 | 821 | | |
815 | 822 | | |
816 | 823 | | |
817 | | - | |
818 | | - | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
819 | 827 | | |
820 | 828 | | |
821 | 829 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1743 | 1743 | | |
1744 | 1744 | | |
1745 | 1745 | | |
1746 | | - | |
| 1746 | + | |
1747 | 1747 | | |
1748 | 1748 | | |
1749 | 1749 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
46 | | - | |
47 | | - | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
48 | 62 | | |
49 | 63 | | |
50 | 64 | | |
| |||
58 | 72 | | |
59 | 73 | | |
60 | 74 | | |
| 75 | + | |
61 | 76 | | |
62 | 77 | | |
63 | 78 | | |
| |||
77 | 92 | | |
78 | 93 | | |
79 | 94 | | |
80 | | - | |
| 95 | + | |
81 | 96 | | |
82 | 97 | | |
83 | 98 | | |
| |||
163 | 178 | | |
164 | 179 | | |
165 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
166 | 184 | | |
167 | 185 | | |
168 | 186 | | |
| |||
191 | 209 | | |
192 | 210 | | |
193 | 211 | | |
| 212 | + | |
194 | 213 | | |
195 | 214 | | |
196 | 215 | | |
197 | 216 | | |
198 | 217 | | |
199 | 218 | | |
200 | 219 | | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
201 | 238 | | |
202 | 239 | | |
203 | 240 | | |
| |||
225 | 262 | | |
226 | 263 | | |
227 | 264 | | |
| 265 | + | |
228 | 266 | | |
229 | 267 | | |
230 | 268 | | |
| |||
240 | 278 | | |
241 | 279 | | |
242 | 280 | | |
| 281 | + | |
243 | 282 | | |
244 | 283 | | |
245 | 284 | | |
| |||
439 | 478 | | |
440 | 479 | | |
441 | 480 | | |
| 481 | + | |
442 | 482 | | |
443 | 483 | | |
444 | 484 | | |
| |||
498 | 538 | | |
499 | 539 | | |
500 | 540 | | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
501 | 553 | | |
502 | 554 | | |
503 | 555 | | |
| |||
833 | 885 | | |
834 | 886 | | |
835 | 887 | | |
836 | | - | |
| 888 | + | |
837 | 889 | | |
838 | 890 | | |
839 | 891 | | |
| |||
992 | 1044 | | |
993 | 1045 | | |
994 | 1046 | | |
| 1047 | + | |
| 1048 | + | |
| 1049 | + | |
| 1050 | + | |
| 1051 | + | |
| 1052 | + | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
| 1060 | + | |
| 1061 | + | |
| 1062 | + | |
| 1063 | + | |
| 1064 | + | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
| 1075 | + | |
| 1076 | + | |
| 1077 | + | |
| 1078 | + | |
| 1079 | + | |
| 1080 | + | |
| 1081 | + | |
| 1082 | + | |
| 1083 | + | |
| 1084 | + | |
| 1085 | + | |
| 1086 | + | |
| 1087 | + | |
| 1088 | + | |
| 1089 | + | |
| 1090 | + | |
| 1091 | + | |
| 1092 | + | |
| 1093 | + | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
| 1111 | + | |
| 1112 | + | |
| 1113 | + | |
| 1114 | + | |
| 1115 | + | |
| 1116 | + | |
| 1117 | + | |
| 1118 | + | |
| 1119 | + | |
| 1120 | + | |
| 1121 | + | |
| 1122 | + | |
| 1123 | + | |
| 1124 | + | |
| 1125 | + | |
| 1126 | + | |
| 1127 | + | |
| 1128 | + | |
| 1129 | + | |
| 1130 | + | |
| 1131 | + | |
| 1132 | + | |
| 1133 | + | |
| 1134 | + | |
| 1135 | + | |
| 1136 | + | |
| 1137 | + | |
| 1138 | + | |
| 1139 | + | |
| 1140 | + | |
| 1141 | + | |
| 1142 | + | |
| 1143 | + | |
| 1144 | + | |
| 1145 | + | |
| 1146 | + | |
| 1147 | + | |
| 1148 | + | |
| 1149 | + | |
| 1150 | + | |
| 1151 | + | |
| 1152 | + | |
| 1153 | + | |
| 1154 | + | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
995 | 1186 | | |
996 | 1187 | | |
997 | 1188 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
83 | 83 | | |
84 | 84 | | |
85 | 85 | | |
86 | | - | |
| 86 | + | |
87 | 87 | | |
88 | 88 | | |
89 | 89 | | |
| |||
101 | 101 | | |
102 | 102 | | |
103 | 103 | | |
104 | | - | |
105 | 104 | | |
106 | 105 | | |
107 | 106 | | |
| |||
194 | 193 | | |
195 | 194 | | |
196 | 195 | | |
197 | | - | |
| 196 | + | |
| 197 | + | |
198 | 198 | | |
199 | 199 | | |
200 | 200 | | |
| |||
0 commit comments