Remove native_comet scan implementation #2177

What is the problem the feature request solves?

We can't do this right away because the Iceberg integration depends on native_comet for now, but we should start working towards removing the native_comet scan implementation.

Motivation:

• Reduce the amount of code to maintain by reducing the number of scan implementations from 3 to 2
• This is a major step towards Implement zero-copy FFI transfer of batches from JVM to Native #2171

The first step will be to review all of the tests where we currently explicitly set the scan implementation to native_comet. For each test we need to consider:

• Can we update the test such that it passes when using the default scan implementation?
• Is the test specific to native_comet and can therefore just be removed (when we stop supporting native_comet)?
• Does the test fail with the default scan because there is a bug or missing feature? If so, can we just fall back to Spark?

We need to implement the following before removing native_comet:

• Enable auto scan mode for Spark 4.0.0 #1967
• Deprecate native_comet scan (remove from auto scan mode) #2186
• [EPIC] Update Iceberg integration to use native_iceberg_compat scan #2189
• [Spark 4.0] Add string collation support #2190
• [iceberg] Add support to native_iceberg_compat for reading encrypted Parquet files #2202
• Add hdfs experimental support for native iceberg_compat reader #1407
• Improve integration of hadoop s3a and comet #1829
• Tracking fs-hdfs issues #2034

Describe the potential solution

No response

Additional context

No response