Skip to content

native_datafusion/native_iceberg_compat scans case sensitive #1574

@wForget

Description

@wForget

Describe the bug

Currently native_datafusion/native_iceberg_compat scans are case-sensitive, which may be inconsistent with vanilla spark.

test case:

  test("test V1 parquet scan uses native_iceberg_compat -- case insensitive") {
    withTempPath { path =>
      spark.range(10).toDF("a").write.parquet(path.toString)
      Seq(CometConf.SCAN_NATIVE_DATAFUSION, CometConf.SCAN_NATIVE_ICEBERG_COMPAT).foreach(scanMode => {
        withSQLConf(CometConf.COMET_NATIVE_SCAN_IMPL.key -> scanMode) {
          sql("create table test (A long) using parquet options (path '" + path + "')")
          val df = sql("select A from test")
          checkSparkAnswer(df)
        }
      })
    }
  }

error:

== Results ==
!== Correct Answer - 10 ==   == Spark Answer - 10 ==
 struct<A:bigint>            struct<A:bigint>
![0]                         [null]
![1]                         [null]
![2]                         [null]
![3]                         [null]
![4]                         [null]
![5]                         [null]
![6]                         [null]
![7]                         [null]
![8]                         [null]
![9]                         [null]

Steps to reproduce

No response

Expected behavior

No response

Additional context

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions