-
Notifications
You must be signed in to change notification settings - Fork 286
Closed
Description
Describe the bug
Currently native_datafusion/native_iceberg_compat scans are case-sensitive, which may be inconsistent with vanilla spark.
test case:
test("test V1 parquet scan uses native_iceberg_compat -- case insensitive") {
withTempPath { path =>
spark.range(10).toDF("a").write.parquet(path.toString)
Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(scanMode => {
withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
sql("create table test (A long) using parquet options (path '" + path + "')")
val df = sql("select A from test")
checkSparkAnswer(df)
}
})
}
}
error:
== Results ==
!== Correct Answer - 10 == == Spark Answer - 10 ==
struct<A:bigint> struct<A:bigint>
![0] [null]
![1] [null]
![2] [null]
![3] [null]
![4] [null]
![5] [null]
![6] [null]
![7] [null]
![8] [null]
![9] [null]
Steps to reproduce
No response
Expected behavior
No response
Additional context
No response
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working