Repository navigation
native_datafusion more permissive than Spark 3.x when reading Parquet TimestampNTZ columns #4219
Description
Describe the bug
Spark 3.x does not allow reading Parquet timestamp columns as TimestampNTZ due to the ambiguity. Spark 4.0 relaxed this restriction.
See https://issues.apache.org/jira/browse/SPARK-47447 for details.
The behavior of native_datafusion scan is correct when running with Spark 4.0+ but incorrect when running with Spark 3.x because it returns a value rather than throwing an exception.
The value returned is correct per Spark 4 definition, so this is not a data correctness issue per se, but Comet is more permissive with Spark 3.x
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Activity
test: expect exception for INT96→NTZ read on pre-Spark 4
add suite to CI, update spark diff issue links for INT96→NTZ
fix: format type-promotion error message to match Spark
fix: unignore passing 3720 tests across all spark version diffs