Skip to content

Commit 01fc37a

Browse files
authored
fix: don't treat planner-inserted casts as Parquet schema conversions (#7159)
* fix: don't treat planner-inserted casts as Parquet schema conversions The native Parquet scan's expression adapter rewrites every DataFusion `CastExpr` in a pushed-down predicate as though it were a file-versus-table schema conversion, and applies Spark's Parquet read rules to it. A predicate can also contain casts that have nothing to do with the file schema: Comet's planner builds them with `CastExpr::new` to coerce scalar-function arguments to the types the function requires. `substring(s, 1, 2)` is one: Spark sends Int32 literals, `SparkSubstring` takes Int64 positions, so the planner wraps them in `CAST(1 AS Int64)` and `CAST(2 AS Int64)`. On Spark 3.x (`allow_type_promotion = false`) the adapter reports that cast as a disallowed `INT32 -> bigint` Parquet promotion. The check is deferred to runtime, so it only fires when the predicate is evaluated against a decoded batch, i.e. with `spark.comet.parquet.rowFilterPushdown.enabled=true`: Error evaluating filter predicate: External(ParquetSchemaConvert { file_path: "", column: "[]", physical_type: "INT32", spark_type: "bigint" }) The file's schema matches the requested schema exactly. The empty column name is the tell: the cast is not over a column. `replace_with_spark_cast` now leaves a cast alone unless it reconciles a file column with its table-schema type (`is_schema_adaptation_cast`). DataFusion's default adapter builds those with `CastExpr::new_with_target_field`, passing the table schema's field, so the target carries the column's name; the planner's `CastExpr::new` casts have an unnamed target field. Casts the planner inserted therefore stay DataFusion `CastExpr`s on every Spark version, instead of being rewritten into Spark `Cast`s on Spark 4.x. * fix: find schema-adaptation casts by explicit target, not by name Spark allows a Parquet column named "", so an empty target name does not mark a planner cast: an INT64 column named "" read as int skipped the read-conversion check and returned its values where Spark raises SchemaColumnConvertNotSupportedException. DataFusion's default adapter builds its schema casts with CastExpr::new_with_target_field, which sets the explicit-target flag that CastExpr::has_explicit_metadata() returns, also for an empty name, and with_new_children keeps it. Comet's planner builds the casts it puts in a predicate with CastExpr::new, which doesn't set it. replace_with_spark_cast now checks that flag, and is_schema_adaptation_cast is gone. Tests: an INT64 column named "" read as int is rejected under both type-promotion settings, the naming-convention test is replaced by one that pins the explicit-target flag, and the three substring filter tests are now one table-driven test on the existing scan_parquet_file helper, which still fails with the original INT32 -> bigint error without the fix.
1 parent 92be8e8 commit 01fc37a

1 file changed

Lines changed: 209 additions & 0 deletions

File tree

‎native/core/src/parquet/schema_adapter.rs‎

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1410,13 +1410,25 @@ impl SparkPhysicalExprAdapter {
14101410
}
14111411

14121412
/// Replace CastExpr (DataFusion's cast) with Spark's Cast expression.
1413+
///
1414+
/// Only the casts that reconcile a file column with its table-schema type are Parquet read
1415+
/// conversions, which Spark's vectorized reader validates. DataFusion's default adapter builds
1416+
/// them with `CastExpr::new_with_target_field`, so `has_explicit_metadata()` is true, also for a
1417+
/// column named `""`. Any other `CastExpr` is part of the predicate itself and is left as the
1418+
/// planner built it. Comet's planner uses `CastExpr::new`, for example to coerce `substring`'s
1419+
/// Int32 literals to Int64, and applying the read rules to that cast rejected valid queries as
1420+
/// a disallowed `INT32 -> bigint` promotion on Spark 3.x.
14131421
fn replace_with_spark_cast(
14141422
&self,
14151423
expr: Arc<dyn PhysicalExpr>,
14161424
) -> DataFusionResult<Transformed<Arc<dyn PhysicalExpr>>> {
14171425
// Check for CastExpr and replace with spark_expr::Cast
14181426
if let Some(cast) = expr.downcast_ref::<datafusion::physical_expr::expressions::CastExpr>()
14191427
{
1428+
if !cast.has_explicit_metadata() {
1429+
return Ok(Transformed::no(expr));
1430+
}
1431+
14201432
let child = Arc::clone(cast.expr());
14211433
let target_type = cast.target_field().data_type();
14221434

@@ -2099,6 +2111,36 @@ pub(crate) mod test {
20992111
Ok(err.to_string())
21002112
}
21012113

2114+
/// Spark allows a Parquet column named `""`, and reading one goes through the same conversion
2115+
/// check as any other column: an `INT64` column named `""` read as `int` is rejected, whether or
2116+
/// not type promotion is allowed, even when every value would fit.
2117+
#[tokio::test]
2118+
async fn empty_named_column_long_read_as_int_errors() -> Result<(), DataFusionError> {
2119+
let file_schema = Arc::new(Schema::new(vec![Field::new("", DataType::Int64, true)]));
2120+
let batch = RecordBatch::try_new(
2121+
Arc::clone(&file_schema),
2122+
vec![Arc::new(Int64Array::from(vec![1i64, 2]))],
2123+
)?;
2124+
let required_schema = Arc::new(Schema::new(vec![Field::new("", DataType::Int32, true)]));
2125+
for allow_type_promotion in [false, true] {
2126+
let mut options = default_options();
2127+
options.allow_type_promotion = allow_type_promotion;
2128+
let mut stream = scan_parquet(&batch, Arc::clone(&required_schema), options)?;
2129+
let msg = match stream.next().await {
2130+
Some(Err(e)) => e.to_string(),
2131+
other => panic!(
2132+
"allow_type_promotion={allow_type_promotion}: expected ParquetSchemaConvert, \
2133+
got {other:?}"
2134+
),
2135+
};
2136+
assert!(
2137+
msg.contains("Found: INT64") && msg.contains("Expected: int"),
2138+
"allow_type_promotion={allow_type_promotion}: {msg}"
2139+
);
2140+
}
2141+
Ok(())
2142+
}
2143+
21022144
/// `INT64 -> INT32` truncates to the lower 32 bits in DataFusion's cast.
21032145
/// Spark's vectorized reader rejects this. See #4297.
21042146
#[tokio::test]
@@ -2796,6 +2838,173 @@ pub(crate) mod test {
27962838
Ok(())
27972839
}
27982840

2841+
/// `replace_with_spark_cast` tells the casts DataFusion's default adapter inserts apart from the
2842+
/// casts Comet's planner puts in a predicate by `CastExpr::has_explicit_metadata()`. This pins
2843+
/// what that relies on: `new_with_target_field` sets it, also for a field named `""`, which
2844+
/// Spark allows as a column name. `CastExpr::new` doesn't set it, and `with_new_children` keeps
2845+
/// it.
2846+
#[test]
2847+
fn schema_casts_are_told_apart_by_explicit_target() -> Result<(), DataFusionError> {
2848+
use datafusion::physical_expr::expressions::CastExpr;
2849+
2850+
// Built by the planner, from a bare type.
2851+
let planner_cast = CastExpr::new(
2852+
Arc::new(Literal::new(ScalarValue::Int32(Some(1)))),
2853+
DataType::Int64,
2854+
None,
2855+
);
2856+
assert!(!planner_cast.has_explicit_metadata());
2857+
2858+
// Built by the default adapter, from the table schema's field.
2859+
for name in ["a", ""] {
2860+
let adapter_cast: Arc<dyn PhysicalExpr> = Arc::new(CastExpr::new_with_target_field(
2861+
Arc::new(Column::new(name, 0)),
2862+
Arc::new(Field::new(name, DataType::Int64, true)),
2863+
None,
2864+
));
2865+
let rebuilt = Arc::clone(&adapter_cast)
2866+
.with_new_children(vec![Arc::new(Column::new(name, 0))])?;
2867+
for cast in [adapter_cast, rebuilt] {
2868+
let cast = cast.downcast_ref::<CastExpr>().expect("a CastExpr");
2869+
assert!(cast.has_explicit_metadata(), "target field {name:?}");
2870+
}
2871+
}
2872+
Ok(())
2873+
}
2874+
2875+
/// The pushed-down `substring(s, 1, 2) = 'ab'` data filter over `schema`, the way
2876+
/// `CometNativeScan` sends it: built by the real planner from the proto `CometSubstring` emits,
2877+
/// with Int32 literals for the position and length. `SparkSubstring` takes Int64 positions, so
2878+
/// the planner coerces those literals with `CastExpr::new`.
2879+
fn substring_filter(schema: SchemaRef) -> Result<Arc<dyn PhysicalExpr>, DataFusionError> {
2880+
use crate::execution::planner::PhysicalPlanner;
2881+
use datafusion::common::tree_node::{TreeNode, TreeNodeRecursion};
2882+
use datafusion::logical_expr::ScalarUDF;
2883+
use datafusion::physical_expr::expressions::CastExpr;
2884+
use datafusion::prelude::SessionContext;
2885+
use datafusion_comet_proto::spark_expression::{self, expr::ExprStruct, literal, Expr};
2886+
use datafusion_spark::function::string::substring::SparkSubstring;
2887+
2888+
let session_ctx = Arc::new(SessionContext::new());
2889+
session_ctx.register_udf(ScalarUDF::new_from_impl(SparkSubstring::default()));
2890+
let planner = PhysicalPlanner::new(Arc::clone(&session_ctx), 0);
2891+
2892+
let proto_type = |type_id: i32| spark_expression::DataType {
2893+
type_id,
2894+
type_info: None,
2895+
};
2896+
let proto_expr = |expr_struct: ExprStruct| Expr {
2897+
expr_struct: Some(expr_struct),
2898+
query_context: None,
2899+
expr_id: None,
2900+
};
2901+
let int_literal = |value: i32| {
2902+
proto_expr(ExprStruct::Literal(spark_expression::Literal {
2903+
value: Some(literal::Value::IntVal(value)),
2904+
datatype: Some(proto_type(3)), // Int32
2905+
is_null: false,
2906+
}))
2907+
};
2908+
let substring = proto_expr(ExprStruct::ScalarFunc(spark_expression::ScalarFunc {
2909+
func: "substring".to_string(),
2910+
args: vec![
2911+
proto_expr(ExprStruct::Bound(spark_expression::BoundReference {
2912+
index: 0,
2913+
datatype: Some(proto_type(7)), // String
2914+
})),
2915+
int_literal(1),
2916+
int_literal(2),
2917+
],
2918+
return_type: None,
2919+
fail_on_error: false,
2920+
null_short_circuit: false,
2921+
}));
2922+
let predicate = planner.create_expr(
2923+
&proto_expr(ExprStruct::Eq(Box::new(spark_expression::BinaryExpr {
2924+
left: Some(Box::new(substring)),
2925+
right: Some(Box::new(proto_expr(ExprStruct::Literal(
2926+
spark_expression::Literal {
2927+
value: Some(literal::Value::StringVal("ab".to_string())),
2928+
datatype: Some(proto_type(7)),
2929+
is_null: false,
2930+
},
2931+
)))),
2932+
}))),
2933+
schema,
2934+
)?;
2935+
2936+
// Guard the premise: the planner really does leave DataFusion `CastExpr`s in the predicate.
2937+
let mut casts = 0;
2938+
predicate.apply(|e| {
2939+
if e.downcast_ref::<CastExpr>().is_some() {
2940+
casts += 1;
2941+
}
2942+
Ok(TreeNodeRecursion::Continue)
2943+
})?;
2944+
assert!(casts >= 2, "expected planner-inserted casts in {predicate}");
2945+
Ok(predicate)
2946+
}
2947+
2948+
/// The casts Comet's planner puts in a pushed-down predicate are not Parquet read conversions.
2949+
/// The file's schema equals the requested schema, so there is no schema evolution of any kind.
2950+
#[tokio::test]
2951+
async fn pushed_substring_filter_keeps_planner_casts() -> Result<(), DataFusionError> {
2952+
let schema = Arc::new(Schema::new(vec![Field::new("s", DataType::Utf8, true)]));
2953+
let batch = RecordBatch::try_new(
2954+
Arc::clone(&schema),
2955+
vec![Arc::new(StringArray::from(vec![
2956+
"abc", "xyz", "abd", "qrs",
2957+
]))],
2958+
)?;
2959+
let filename = get_temp_filename().to_str().unwrap().to_string();
2960+
let mut writer = ArrowWriter::try_new(File::create(&filename)?, Arc::clone(&schema), None)?;
2961+
writer.write(&batch)?;
2962+
writer.close()?;
2963+
let predicate = substring_filter(Arc::clone(&schema))?;
2964+
2965+
let every_row: &[&str] = &["abc", "xyz", "abd", "qrs"];
2966+
let matching: &[&str] = &["abc", "abd"];
2967+
for (row_filters, allow_type_promotion, expected) in [
2968+
// Control: without row-level filter pushdown (the default) the predicate is only used
2969+
// for row-group and page pruning, so the scan returns every row and leaves the
2970+
// filtering to the Spark `Filter` above it.
2971+
(false, false, every_row),
2972+
// With `spark.comet.parquet.rowFilterPushdown.enabled=true` the predicate is evaluated
2973+
// on every decoded batch. On Spark 3.x (`allow_type_promotion = false`) the planner's
2974+
// Int32 -> Int64 coercion was taken for a disallowed `INT32 -> bigint` promotion, which
2975+
// failed the scan with `ParquetSchemaConvert { column: "[]", .. }`.
2976+
(true, false, matching),
2977+
// Control: Spark 4.x allows that promotion, so this case passed before the fix too.
2978+
(true, true, matching),
2979+
] {
2980+
let mut options = default_options();
2981+
options.allow_type_promotion = allow_type_promotion;
2982+
let mut stream = scan_parquet_file(
2983+
filename.clone(),
2984+
Arc::clone(&schema),
2985+
options,
2986+
Some(Arc::clone(&predicate)),
2987+
row_filters,
2988+
)?;
2989+
let mut values: Vec<String> = vec![];
2990+
while let Some(batch) = stream.next().await {
2991+
let column = arrow::compute::cast(batch?.column(0), &DataType::Utf8)?;
2992+
values.extend(
2993+
column
2994+
.as_string::<i32>()
2995+
.iter()
2996+
.flatten()
2997+
.map(str::to_string),
2998+
);
2999+
}
3000+
assert_eq!(
3001+
values, expected,
3002+
"row_filters={row_filters} allow_type_promotion={allow_type_promotion}"
3003+
);
3004+
}
3005+
Ok(())
3006+
}
3007+
27993008
#[tokio::test]
28003009
async fn parquet_roundtrip_unsigned_int() -> Result<(), DataFusionError> {
28013010
let file_schema = Arc::new(Schema::new(vec![Field::new("id", DataType::UInt32, false)]));

0 commit comments

Comments
 (0)