Repository navigation
Commit b1130d3
committed
fix: reject BINARY-as-DECIMAL, re-ignore parquet-mr permissive tests (apache#4351, apache#4352)
Two follow-ups to 80836b1, which unignored four `ParquetTypeWideningSuite`
test groups in `dev/diffs/4.1.1.diff` and surfaced a remaining schema-adapter
gap on top.
Schema adapter (apache#4351): a Parquet BINARY column without a
`DecimalLogicalTypeAnnotation` was silently allowed through when the requested
schema was `DecimalType`. The cast then fell through and Arrow's
`RecordBatch::try_new` raised a generic `column types must match schema types`
error as `CometNativeException` instead of the Spark-equivalent
`SchemaColumnConvertNotSupportedException`. Drop `Decimal128/Decimal256` from
the BINARY/STRING allowed-target match in `replace_with_spark_cast`, and add
the same rejection in `wrap_all_type_mismatches` so the fallback path doesn't
construct a `CometCastColumnExpr` for a cast it can't actually perform. New
unit test `parquet_binary_read_as_decimal_errors`. With this fix, Spark's
SPARK-34212 in `ParquetQuerySuite` passes without an ignore tag.
Test diffs (apache#4352): re-add `IgnoreCometNativeDataFusion` to the five test
groups in `dev/diffs/4.1.1.diff`'s `ParquetTypeWideningSuite` whose failure
isn't a rejection-logic bug but a parquet-mr-only behavior assertion
(`expectError = vectorized` paths and the explicit `overflows with parquet-mr`
test). Same annotation added to the previously-unannotated overflow test in
`dev/diffs/4.0.2.diff`. The schema-adapter rejection is correct; Comet just
has no parquet-mr-equivalent backend that produces silent overflow on the
non-vectorized path.
Docs: new `Schema Mismatch Handling` subsection in the parquet scan compat
guide. Explains that these gaps only apply to explicit schemas or schema
evolution (not plain `spark.read.parquet(path)`), notes cross-version
differences, and lists the one remaining gap (apache#4316: missing file path in
`ParquetSchemaConvert` errors).1 parent 80836b1 commit b1130d3
4 files changed
Lines changed: 195 additions & 17 deletions
File tree
- dev/diffs
- docs/source/user-guide/latest/compatibility
- native/core/src/parquet
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2895 | 2895 | | |
2896 | 2896 | | |
2897 | 2897 | | |
2898 | | - | |
| 2898 | + | |
2899 | 2899 | | |
2900 | 2900 | | |
2901 | 2901 | | |
| |||
2987 | 2987 | | |
2988 | 2988 | | |
2989 | 2989 | | |
| 2990 | + | |
| 2991 | + | |
| 2992 | + | |
| 2993 | + | |
| 2994 | + | |
| 2995 | + | |
| 2996 | + | |
| 2997 | + | |
| 2998 | + | |
| 2999 | + | |
2990 | 3000 | | |
2991 | 3001 | | |
2992 | 3002 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3036 | 3036 | | |
3037 | 3037 | | |
3038 | 3038 | | |
3039 | | - | |
| 3039 | + | |
3040 | 3040 | | |
3041 | 3041 | | |
| 3042 | + | |
| 3043 | + | |
| 3044 | + | |
| 3045 | + | |
| 3046 | + | |
| 3047 | + | |
| 3048 | + | |
| 3049 | + | |
| 3050 | + | |
3042 | 3051 | | |
3043 | 3052 | | |
3044 | 3053 | | |
| |||
3069 | 3078 | | |
3070 | 3079 | | |
3071 | 3080 | | |
| 3081 | + | |
| 3082 | + | |
| 3083 | + | |
| 3084 | + | |
| 3085 | + | |
| 3086 | + | |
| 3087 | + | |
| 3088 | + | |
| 3089 | + | |
| 3090 | + | |
| 3091 | + | |
| 3092 | + | |
| 3093 | + | |
| 3094 | + | |
| 3095 | + | |
| 3096 | + | |
| 3097 | + | |
| 3098 | + | |
| 3099 | + | |
| 3100 | + | |
| 3101 | + | |
| 3102 | + | |
| 3103 | + | |
| 3104 | + | |
| 3105 | + | |
| 3106 | + | |
| 3107 | + | |
| 3108 | + | |
| 3109 | + | |
| 3110 | + | |
| 3111 | + | |
| 3112 | + | |
| 3113 | + | |
| 3114 | + | |
| 3115 | + | |
| 3116 | + | |
| 3117 | + | |
| 3118 | + | |
| 3119 | + | |
| 3120 | + | |
| 3121 | + | |
| 3122 | + | |
| 3123 | + | |
| 3124 | + | |
| 3125 | + | |
| 3126 | + | |
| 3127 | + | |
| 3128 | + | |
| 3129 | + | |
| 3130 | + | |
| 3131 | + | |
| 3132 | + | |
| 3133 | + | |
| 3134 | + | |
| 3135 | + | |
| 3136 | + | |
| 3137 | + | |
| 3138 | + | |
| 3139 | + | |
| 3140 | + | |
3072 | 3141 | | |
3073 | 3142 | | |
3074 | 3143 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
94 | 124 | | |
95 | 125 | | |
96 | 126 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
548 | 548 | | |
549 | 549 | | |
550 | 550 | | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
551 | 584 | | |
552 | 585 | | |
553 | 586 | | |
| |||
584 | 617 | | |
585 | 618 | | |
586 | 619 | | |
587 | | - | |
588 | | - | |
589 | | - | |
590 | | - | |
591 | | - | |
592 | | - | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
593 | 630 | | |
594 | 631 | | |
595 | 632 | | |
596 | 633 | | |
597 | 634 | | |
598 | | - | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
599 | 640 | | |
600 | 641 | | |
601 | 642 | | |
602 | 643 | | |
603 | 644 | | |
604 | | - | |
605 | | - | |
606 | | - | |
607 | | - | |
608 | | - | |
609 | | - | |
| 645 | + | |
610 | 646 | | |
611 | 647 | | |
612 | 648 | | |
| |||
1104 | 1140 | | |
1105 | 1141 | | |
1106 | 1142 | | |
1107 | | - | |
1108 | | - | |
| 1143 | + | |
| 1144 | + | |
1109 | 1145 | | |
1110 | 1146 | | |
1111 | 1147 | | |
| |||
1173 | 1209 | | |
1174 | 1210 | | |
1175 | 1211 | | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
| 1240 | + | |
| 1241 | + | |
| 1242 | + | |
| 1243 | + | |
| 1244 | + | |
1176 | 1245 | | |
1177 | 1246 | | |
1178 | 1247 | | |
| |||
0 commit comments