Deprecate native_comet scan (remove from auto scan mode)

What is the problem the feature request solves?

As a step towards removing native_comet scan support, I propose that we stop falling back to this scan when in auto mode and fall back to Spark instead.

The main case where we currently fall back to native_comet is for byte and short fields when COMET_SCAN_ALLOW_INCOMPATIBLE is false.

Describe the potential solution

No response

Additional context

No response

A quick experiment shows that there are relatively few tests that fail (or need updating) when we remove native_comet from auto mode:

PR tests

fix: native Unsafe row accessors return incorrect results

Spark SQL core-2

Filters should be pushed down for vectorized Parquet reader at row group level
filter pushdown - StringPredicate
SPARK-17091: Convert IN predicate to Parquet filter push-down
SPARK-34562: Bloom filter push down
view-with-default-collation.sql

Spark SQL hive-1

SPARK-30201 HiveOutputWriter standardOI should use ObjectInspectorCopyOption.DEFAULT

Spark SQL hive-3

Change column collation

Navigation Menu

Repository navigation

Deprecate `native_comet` scan (remove from `auto` scan mode) #2186

What is the problem the feature request solves?

Describe the potential solution

Additional context

PR tests

Spark SQL core-2

Spark SQL hive-1

Spark SQL hive-3

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Deprecate native_comet scan (remove from auto scan mode) #2186

Description

What is the problem the feature request solves?

Describe the potential solution

Additional context

Activity

andygrove commented on Aug 19, 2025

PR tests

Spark SQL core-2

Spark SQL hive-1

Spark SQL hive-3

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Deprecate `native_comet` scan (remove from `auto` scan mode) #2186