Skip to content

Commit 80836b1

Browse files
committed
fix: reject Parquet conversions Spark rejects (#4297, #4343, #4344)
Extend the native_datafusion schema adapter to mirror the rejections in `ParquetVectorUpdaterFactory.getUpdater` / `isDecimalTypeMatched`: - Integer -> decimal where the requested decimal cannot represent the source integer type's range (#4344). INT32 sources require `precision - scale >= 10`, INT64 sources `>= 20`. - Decimal -> decimal where `scaleIncrease < 0` OR `precisionIncrease < scaleIncrease` (#4343). Generalises the prior scale-only narrowing check to also reject precision narrowing and scale widening that overflows the integer side. - Primitive numeric / date / timestamp conversions Spark rejects on every supported version (#4297): `long -> int`, `double -> float`, `float|double -> int*`, `int|long -> float`, `int -> timestamp`, `long -> date|timestamp`, `date -> timestamp(LTZ)`, and `timestamp -> date`. Deferred to runtime via `RejectOnNonEmpty` so empty Parquet files pass through (SPARK-26709). `spark_catalog_name` now returns String so it can format `decimal(p,s)` per Spark's `catalogString()`. `parquet_primitive_name` gains decimal / date / timestamp arms so error messages report the underlying Parquet primitive (INT32/INT64/FIXED_LEN_BYTE_ARRAY) instead of UNKNOWN. Coverage: 18 new Rust unit tests across the three rejection classes, plus expanded JVM regression tests in ParquetReadSuite. The four Spark SQL tests pointing at #4343/#4344/#4297 in dev/diffs/4.1.1.diff are unignored (no umbrella-#3720 references remain). The compat doc collapses the type-promotion bullet, leaving only the TimestampLTZ-as-TimestampNTZ pre-Spark-4.0 caveat.
1 parent e7269c9 commit 80836b1

4 files changed

Lines changed: 694 additions & 157 deletions

File tree

‎dev/diffs/4.1.1.diff‎

Lines changed: 5 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -2870,18 +2870,10 @@ index 3072657a095..599d169cf8a 100644
28702870
checkAnswer(
28712871
// "fruit" column in this file is encoded using DELTA_LENGTH_BYTE_ARRAY.
28722872
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
2873-
index c530dc0d3df..cf073150fd1 100644
2873+
index c530dc0d3df..d23069689da 100644
28742874
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
28752875
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetQuerySuite.scala
2876-
@@ -27,6 +27,7 @@ import org.apache.parquet.hadoop.ParquetOutputFormat
2877-
2878-
import org.apache.spark.{DebugFilesystem, SparkConf, SparkException}
2879-
import org.apache.spark.sql._
2880-
+import org.apache.spark.sql.IgnoreCometNativeDataFusion
2881-
import org.apache.spark.sql.catalyst.{InternalRow, TableIdentifier}
2882-
import org.apache.spark.sql.catalyst.expressions.SpecificInternalRow
2883-
import org.apache.spark.sql.catalyst.util.ArrayData
2884-
@@ -996,7 +997,11 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
2876+
@@ -996,7 +996,11 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
28852877
Seq(Some("A"), Some("A"), None).toDF().repartition(1)
28862878
.write.parquet(path.getAbsolutePath)
28872879
val df = spark.read.parquet(path.getAbsolutePath)
@@ -2894,17 +2886,7 @@ index c530dc0d3df..cf073150fd1 100644
28942886
}
28952887
}
28962888
}
2897-
@@ -1042,7 +1047,8 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
2898-
testMigration(fromTsType = "TIMESTAMP_MICROS", toTsType = "INT96")
2899-
}
2900-
2901-
- test("SPARK-34212 Parquet should read decimals correctly") {
2902-
+ test("SPARK-34212 Parquet should read decimals correctly",
2903-
+ IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/4343")) {
2904-
def readParquet(schema: String, path: File): DataFrame = {
2905-
spark.read.schema(schema).parquet(path.toString)
2906-
}
2907-
@@ -1060,7 +1066,8 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
2889+
@@ -1060,7 +1064,8 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
29082890
checkAnswer(readParquet(schema2, path), df)
29092891
}
29102892

@@ -2914,7 +2896,7 @@ index c530dc0d3df..cf073150fd1 100644
29142896
val schema1 = "a DECIMAL(3, 2), b DECIMAL(18, 3), c DECIMAL(37, 3)"
29152897
checkAnswer(readParquet(schema1, path), df)
29162898
val schema2 = "a DECIMAL(3, 0), b DECIMAL(18, 1), c DECIMAL(37, 1)"
2917-
@@ -1084,7 +1091,8 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
2899+
@@ -1084,7 +1089,8 @@ abstract class ParquetQuerySuite extends QueryTest with ParquetTest with SharedS
29182900
val df = sql(s"SELECT 1 a, 123456 b, ${Int.MaxValue.toLong * 10} c, CAST('1.2' AS BINARY) d")
29192901
df.write.parquet(path.toString)
29202902

@@ -3054,18 +3036,9 @@ index 56076175d60..5872d9962cc 100644
30543036
val e = testSchemaMismatch(dir.getCanonicalPath, vectorizedReaderEnabled = true)
30553037
assert(e.getCause.isInstanceOf[SchemaColumnConvertNotSupportedException])
30563038
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala
3057-
index 09ed6955a51..6b5be93cffd 100644
3039+
index 09ed6955a51..236a4e99824 100644
30583040
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala
30593041
+++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetTypeWideningSuite.scala
3060-
@@ -24,7 +24,7 @@ import org.apache.parquet.format.converter.ParquetMetadataConverter
3061-
import org.apache.parquet.hadoop.{ParquetFileReader, ParquetOutputFormat}
3062-
3063-
import org.apache.spark.SparkException
3064-
-import org.apache.spark.sql.{DataFrame, QueryTest, Row}
3065-
+import org.apache.spark.sql.{DataFrame, IgnoreCometNativeDataFusion, QueryTest, Row}
3066-
import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
3067-
import org.apache.spark.sql.execution.datasources.SchemaColumnConvertNotSupportedException
3068-
import org.apache.spark.sql.functions.col
30693042
@@ -65,7 +65,9 @@ class ParquetTypeWideningSuite
30703043
withClue(
30713044
s"with dictionary encoding '$dictionaryEnabled' with timestamp rebase mode " +
@@ -3096,56 +3069,6 @@ index 09ed6955a51..6b5be93cffd 100644
30963069
)
30973070
}
30983071
test(s"parquet widening conversion $fromType -> $toType") {
3099-
@@ -231,7 +234,8 @@ class ParquetTypeWideningSuite
3100-
(Seq("2020-01-01", "2020-01-02", "1312-02-27"), DateType, TimestampType)
3101-
)
3102-
}
3103-
- test(s"unsupported parquet conversion $fromType -> $toType") {
3104-
+ test(s"unsupported parquet conversion $fromType -> $toType",
3105-
+ IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/4297")) {
3106-
checkAllParquetReaders(values, fromType, toType, expectError = true)
3107-
}
3108-
3109-
@@ -257,7 +261,8 @@ class ParquetTypeWideningSuite
3110-
(Seq("1", "2"), LongType, DecimalType(LongDecimal.precision, 1))
3111-
)
3112-
}
3113-
- test(s"unsupported parquet conversion $fromType -> $toType") {
3114-
+ test(s"unsupported parquet conversion $fromType -> $toType",
3115-
+ IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/4344")) {
3116-
checkAllParquetReaders(values, fromType, toType,
3117-
expectError =
3118-
// parquet-mr allows reading decimals into a smaller precision decimal type without
3119-
@@ -271,7 +276,8 @@ class ParquetTypeWideningSuite
3120-
(Seq("2020-01-01", "2020-01-02", "1312-02-27"), TimestampNTZType, DateType))
3121-
outputTimestampType <- ParquetOutputTimestampType.values
3122-
}
3123-
- test(s"unsupported parquet timestamp conversion $fromType ($outputTimestampType) -> $toType") {
3124-
+ test(s"unsupported parquet timestamp conversion $fromType ($outputTimestampType) -> $toType",
3125-
+ IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/4297")) {
3126-
withSQLConf(
3127-
SQLConf.PARQUET_OUTPUT_TIMESTAMP_TYPE.key -> outputTimestampType.toString,
3128-
SQLConf.PARQUET_INT96_REBASE_MODE_IN_WRITE.key -> LegacyBehaviorPolicy.CORRECTED.toString
3129-
@@ -291,7 +297,8 @@ class ParquetTypeWideningSuite
3130-
Seq(7 -> 5, 10 -> 5, 20 -> 5, 12 -> 10, 20 -> 10, 22 -> 20)
3131-
}
3132-
test(
3133-
- s"parquet decimal precision change Decimal($fromPrecision, 2) -> Decimal($toPrecision, 2)") {
3134-
+ s"parquet decimal precision change Decimal($fromPrecision, 2) -> Decimal($toPrecision, 2)",
3135-
+ IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/4343")) {
3136-
checkAllParquetReaders(
3137-
values = Seq("1.23", "10.34"),
3138-
fromType = DecimalType(fromPrecision, 2),
3139-
@@ -322,7 +329,8 @@ class ParquetTypeWideningSuite
3140-
Seq((5, 2) -> (6, 4), (10, 4) -> (12, 7), (20, 5) -> (22, 8))
3141-
}
3142-
test(s"parquet decimal precision and scale change Decimal($fromPrecision, $fromScale) -> " +
3143-
- s"Decimal($toPrecision, $toScale)"
3144-
+ s"Decimal($toPrecision, $toScale)",
3145-
+ IgnoreCometNativeDataFusion("https://github.com/apache/datafusion-comet/issues/4343")
3146-
) {
3147-
checkAllParquetReaders(
3148-
values = Seq("1.23", "10.34"),
31493072
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala
31503073
index 1cc6d3afbee..8275727fbb4 100644
31513074
--- a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetVariantShreddingSuite.scala

‎docs/source/user-guide/latest/compatibility/scans.md‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -91,20 +91,6 @@ without falling back to Spark:
9191
([SPARK-47447](https://issues.apache.org/jira/browse/SPARK-47447)) and Comet matches Spark's behavior.
9292
See [#4219](https://github.com/apache/datafusion-comet/issues/4219).
9393

94-
- Unsupported Parquet type conversions. Spark's vectorized reader rejects several classes of read-schema
95-
type promotion that the `native_datafusion` scan currently accepts. In each case Comet returns
96-
truncated, overflowed, or otherwise wrong values instead of raising
97-
`SchemaColumnConvertNotSupportedException`:
98-
- **Primitive numeric and date/timestamp narrowing.** Long → Int, Double → Float, Float → Long,
99-
Long → Double, Int → Float, Int → Timestamp[NTZ], Date → Timestamp, Timestamp[NTZ] → Date.
100-
See [#4297](https://github.com/apache/datafusion-comet/issues/4297).
101-
- **Decimal-to-decimal precision/scale narrowing.** Reading e.g. `DECIMAL(18, 2)` as
102-
`DECIMAL(3, 0)` (or any other case where the requested decimal cannot represent the file's
103-
values). See [#4343](https://github.com/apache/datafusion-comet/issues/4343).
104-
- **Integer-to-decimal narrowing.** Reading `Byte` / `Short` / `Int` / `Long` as a `DECIMAL(p, s)`
105-
whose precision cannot hold the integer's range.
106-
See [#4344](https://github.com/apache/datafusion-comet/issues/4344).
107-
10894
## `native_iceberg_compat` Limitations
10995

11096
The `native_iceberg_compat` scan has the following additional limitation that may produce incorrect results

0 commit comments

Comments
 (0)