You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 01fc37a
Browse filesBrowse the repository at this point in the historyBrowse files
fix: don't treat planner-inserted casts as Parquet schema conversions (#7159)
* fix: don't treat planner-inserted casts as Parquet schema conversions
The native Parquet scan's expression adapter rewrites every DataFusion
`CastExpr` in a pushed-down predicate as though it were a file-versus-table
schema conversion, and applies Spark's Parquet read rules to it. A predicate
can also contain casts that have nothing to do with the file schema: Comet's
planner builds them with `CastExpr::new` to coerce scalar-function arguments
to the types the function requires. `substring(s, 1, 2)` is one: Spark sends
Int32 literals, `SparkSubstring` takes Int64 positions, so the planner wraps
them in `CAST(1 AS Int64)` and `CAST(2 AS Int64)`.
On Spark 3.x (`allow_type_promotion = false`) the adapter reports that cast as
a disallowed `INT32 -> bigint` Parquet promotion. The check is deferred to
runtime, so it only fires when the predicate is evaluated against a decoded
batch, i.e. with `spark.comet.parquet.rowFilterPushdown.enabled=true`:
Error evaluating filter predicate: External(ParquetSchemaConvert {
file_path: "", column: "[]", physical_type: "INT32", spark_type: "bigint" })
The file's schema matches the requested schema exactly. The empty column name
is the tell: the cast is not over a column.
`replace_with_spark_cast` now leaves a cast alone unless it reconciles a file
column with its table-schema type (`is_schema_adaptation_cast`). DataFusion's
default adapter builds those with `CastExpr::new_with_target_field`, passing
the table schema's field, so the target carries the column's name; the
planner's `CastExpr::new` casts have an unnamed target field.
Casts the planner inserted therefore stay DataFusion `CastExpr`s on every
Spark version, instead of being rewritten into Spark `Cast`s on Spark 4.x.
* fix: find schema-adaptation casts by explicit target, not by name
Spark allows a Parquet column named "", so an empty target name does not
mark a planner cast: an INT64 column named "" read as int skipped the
read-conversion check and returned its values where Spark raises
SchemaColumnConvertNotSupportedException.
DataFusion's default adapter builds its schema casts with
CastExpr::new_with_target_field, which sets the explicit-target flag that
CastExpr::has_explicit_metadata() returns, also for an empty name, and
with_new_children keeps it. Comet's planner builds the casts it puts in a
predicate with CastExpr::new, which doesn't set it. replace_with_spark_cast
now checks that flag, and is_schema_adaptation_cast is gone.
Tests: an INT64 column named "" read as int is rejected under both
type-promotion settings, the naming-convention test is replaced by one that
pins the explicit-target flag, and the three substring filter tests are now
one table-driven test on the existing scan_parquet_file helper, which still
fails with the original INT32 -> bigint error without the fix.
0 commit comments