Enable auto scan mode for Spark 4.0.0 #1967

What is the problem the feature request solves?

For the initial Spark 4.0.0 support, we disabled the auto scan mode:

  private def selectScan(scanExec: FileSourceScanExec, partitionSchema: StructType): String = {

    val fallbackReasons = new ListBuffer[String]()

    if (CometSparkSessionExtensions.isSpark40Plus) {
      fallbackReasons += s"$SCAN_NATIVE_ICEBERG_COMPAT  is not implemented for Spark 4.0.0"
    }

We should now remove this limitation. However, some other issues need resolving first:

• The native_iceberg_compat reader fails at runtime with Unsupported data type: string collate UTF8_LCASE, so we need to fall back to native_comet for now if we see collation types. This error is thrown in Utils.toArrowType.
  • Related issue: [Spark 4.0] Add string collation support #2190

Describe the potential solution

No response

Additional context

No response