Skip to content

Commit 124291e

Browse files
paleolimbotalambtimsaucer
authored
Align metadata propagation through Physical and Logical casts (apache#23169)
## Which issue does this PR close? - Closes apache#22079 - Closes apache#24724 ## Rationale for this change The logical `Expr::Cast` and `Expr::TryCast` have a `FieldRef` target that was added in apache#18136 so that logical casts can express a cast to an extension type. In combination with a SQL type planner ( apache#20676 ) and an optimizer rule, this enabled casting to/from extension types with custom semantics to actually occur. The ability to do this was reverted by apache#20836 (which removed the original test) and I am not sure that ability ever made it into a release. When investigating this issue, it became clear the logical and physical cast behaviour had diverged with respect to the target field. ## What changes are included in this PR? This PR strips specific metadata keys (extension name and extension metadata) when propagating metadata from the source of a cast to the target (because doing so may result in an invalid destination field that consumers could reject), and propagates all metadata from the (logical) cast target field (e.g., so that a cast to an extension type represented by the cast target field will have a `to_field()` that communicates the extension type). For the physical cast, this behaviour is replicated exactly (I hope). Note that actually casting to an extension type can be implemented with an optimizer rule, planner, or by the mechanism I have in the works in apache#21071 . ## Are these changes tested? Yes ## Are there any user-facing changes? It was in practice not common to create a `Expr::Cast` with field metadata internally and thus I don't think users will see metadata changes from the inclusion of metadata from the target field. I would be surprised if stripping the extension name/metadata from the source was disruptive (it was more likely to have caused errors). Superceeds an earlier but similar attempt ( apache#22162 ). --------- Co-authored-by: Andrew Lamb <[email protected]> Co-authored-by: Tim Saucer <[email protected]>
1 parent ac78aa0 commit 124291e

14 files changed

Lines changed: 1069 additions & 154 deletions

File tree

‎Cargo.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎datafusion-examples/examples/custom_data_source/custom_file_casts.rs‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -188,10 +188,11 @@ impl PhysicalExprAdapter for CustomCastsPhysicalExprAdapter {
188188
if let Some(cast) = expr.downcast_ref::<CastExpr>() {
189189
let input_data_type =
190190
cast.expr().data_type(&self.physical_file_schema)?;
191-
let output_data_type = cast.target_field().data_type();
191+
let output_field = cast.target_field();
192192
if !cast.is_bigger_cast(&input_data_type) {
193193
return not_impl_err!(
194-
"Unsupported CAST from {input_data_type} to {output_data_type}"
194+
"Unsupported CAST from {input_data_type} to {}",
195+
output_field.data_type()
195196
);
196197
}
197198
}

‎datafusion/expr/src/expr_schema.rs‎

Lines changed: 152 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ use crate::{
3333
use arrow::compute::can_cast_types;
3434
use arrow::datatypes::FieldRef;
3535
use arrow::datatypes::{DataType, Field};
36+
use arrow_schema::extension::{EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY};
3637
use datafusion_common::datatype::FieldExt;
3738
use datafusion_common::{
3839
Column, DataFusionError, ExprSchema, Result, ScalarValue, Spans, TableReference,
@@ -71,18 +72,46 @@ pub trait ExprSchemable {
7172
-> Result<(DataType, bool)>;
7273
}
7374

74-
/// Derives the output field for a cast expression from the source field.
75+
/// Derives the output field for a cast expression from the source and target fields.
76+
///
77+
/// Metadata handling:
78+
/// - Type-only casts (i.e., target_field == DataType::SomeDataType.into_nullable_field())
79+
/// propagate non extension-type metadata from the source. This is for backward compatibility
80+
/// (casts have propagated source metadata for many if not all previous versions), recognizing
81+
/// that the return type of `<some extension type>::<some non extension type>` should have the
82+
/// return type of `<some non extension type>` (e.g., casting arrow.json to utf8).
83+
/// - All other casts preserve target metadata exactly. This ensures in particular that output
84+
/// metadata when casting to an extension type contains the extension information in the
85+
/// output field. Callers that wish to have some mix of source and target metadata can use
86+
/// Alias or construct an output field themselves (whose metadata will be used directly).
87+
///
7588
/// For `TryCast`, `force_nullable` is `true` since a failed cast returns NULL.
7689
fn cast_output_field(
7790
source_field: &FieldRef,
78-
target_type: &DataType,
91+
target_field: &FieldRef,
7992
force_nullable: bool,
8093
) -> Arc<Field> {
94+
// Check if this is a "type-only" cast (target_field == DataType::X.into_nullable_field())
95+
let is_type_only = target_field.name().is_empty()
96+
&& target_field.is_nullable()
97+
&& target_field.metadata().is_empty();
98+
99+
let metadata = if is_type_only {
100+
// Type-only cast: propagate source metadata, stripping extension type keys
101+
let mut meta = source_field.metadata().clone();
102+
meta.remove(EXTENSION_TYPE_NAME_KEY);
103+
meta.remove(EXTENSION_TYPE_METADATA_KEY);
104+
meta
105+
} else {
106+
// Explicit target field: use target metadata exactly
107+
target_field.metadata().clone()
108+
};
109+
81110
let mut f = source_field
82111
.as_ref()
83112
.clone()
84-
.with_data_type(target_type.clone())
85-
.with_metadata(source_field.metadata().clone());
113+
.with_data_type(target_field.data_type().clone())
114+
.with_metadata(metadata);
86115
if force_nullable {
87116
f = f.with_nullable(true);
88117
}
@@ -478,7 +507,8 @@ impl ExprSchemable for Expr {
478507
/// - **Aliases**: Merge underlying expr metadata with alias-specific metadata, preferring the alias metadata
479508
/// - **Binary expressions**: field metadata is empty
480509
/// - **Boolean expressions**: field metadata is empty
481-
/// - **Cast expressions**: determined by the input expression's field metadata handling
510+
/// - **Cast expressions**: Type-only casts pass through source metadata (stripping extension
511+
/// type keys); casts with explicit target fields use target metadata exactly
482512
/// - **Scalar functions**: Generate metadata via function's [`return_field_from_args`] method,
483513
/// with the default implementation returning empty field metadata
484514
/// - **Aggregate functions**: Generate metadata via function's [`return_field`] method,
@@ -623,20 +653,16 @@ impl ExprSchemable for Expr {
623653
func.return_field_from_args(args)
624654
}
625655
// _ => Ok((self.get_type(schema)?, self.nullable(schema)?)),
626-
Expr::Cast(Cast { expr, field }) => {
627-
expr.to_field(schema).map(|(_table_ref, src)| {
628-
cast_output_field(&src, field.data_type(), false)
629-
})
630-
}
656+
Expr::Cast(Cast { expr, field }) => expr
657+
.to_field(schema)
658+
.map(|(_table_ref, src)| cast_output_field(&src, field, false)),
631659
Expr::Placeholder(Placeholder {
632660
id: _,
633661
field: Some(field),
634662
}) => Ok(Arc::clone(field).renamed(&schema_name)),
635-
Expr::TryCast(TryCast { expr, field }) => {
636-
expr.to_field(schema).map(|(_table_ref, src)| {
637-
cast_output_field(&src, field.data_type(), true)
638-
})
639-
}
663+
Expr::TryCast(TryCast { expr, field }) => expr
664+
.to_field(schema)
665+
.map(|(_table_ref, src)| cast_output_field(&src, field, true)),
640666
Expr::LambdaVariable(LambdaVariable {
641667
field: Some(field), ..
642668
}) => Ok(Arc::clone(field).renamed(&schema_name)),
@@ -1427,4 +1453,115 @@ mod tests {
14271453

14281454
assert_eq!(meta, expr.metadata(&schema).unwrap());
14291455
}
1456+
1457+
#[test]
1458+
fn test_cast_and_try_cast_extension_type_metadata() {
1459+
use crate::expr::{Cast, TryCast};
1460+
use arrow_schema::extension::{
1461+
EXTENSION_TYPE_METADATA_KEY, EXTENSION_TYPE_NAME_KEY,
1462+
};
1463+
1464+
// Helper to build either Cast or TryCast expression
1465+
fn make_cast_expr(
1466+
expr: Expr,
1467+
target_field: FieldRef,
1468+
use_try_cast: bool,
1469+
) -> Expr {
1470+
if use_try_cast {
1471+
Expr::TryCast(TryCast {
1472+
expr: Box::new(expr),
1473+
field: target_field,
1474+
})
1475+
} else {
1476+
Expr::Cast(Cast {
1477+
expr: Box::new(expr),
1478+
field: target_field,
1479+
})
1480+
}
1481+
}
1482+
1483+
// Run the same test logic for both Cast and TryCast
1484+
for use_try_cast in [false, true] {
1485+
let cast_name = if use_try_cast { "TryCast" } else { "Cast" };
1486+
1487+
// Create a schema with a field that has extension type metadata
1488+
let mut source_meta = HashMap::new();
1489+
source_meta.insert(
1490+
EXTENSION_TYPE_NAME_KEY.to_string(),
1491+
"arrow.uuid".to_string(),
1492+
);
1493+
source_meta.insert("custom_key".to_string(), "custom_value".to_string());
1494+
1495+
let source_field = Field::new("foo", DataType::FixedSizeBinary(16), false)
1496+
.with_metadata(source_meta);
1497+
1498+
let schema = MockExprSchema::new()
1499+
.with_data_type(DataType::FixedSizeBinary(16))
1500+
.with_metadata(FieldMetadata::from(source_field.metadata().clone()));
1501+
1502+
// Test 1: Cast to a type without extension metadata strips extension metadata
1503+
// but preserves non-extension metadata
1504+
let cast_expr = make_cast_expr(
1505+
col("foo"),
1506+
Arc::new(Field::new("", DataType::Utf8, true)),
1507+
use_try_cast,
1508+
);
1509+
1510+
let (_, result_field) = cast_expr.to_field(&schema).unwrap();
1511+
assert!(
1512+
result_field
1513+
.metadata()
1514+
.get(EXTENSION_TYPE_NAME_KEY)
1515+
.is_none(),
1516+
"{cast_name}: Extension type name should be stripped when target has no extension metadata"
1517+
);
1518+
assert_eq!(
1519+
result_field.metadata().get("custom_key"),
1520+
Some(&"custom_value".to_string()),
1521+
"{cast_name}: Non-extension metadata should be preserved"
1522+
);
1523+
if use_try_cast {
1524+
assert!(
1525+
result_field.is_nullable(),
1526+
"TryCast result should be nullable"
1527+
);
1528+
}
1529+
1530+
// Test 2: Cast to a field with explicit metadata uses target metadata exactly
1531+
let mut target_meta = HashMap::new();
1532+
target_meta.insert(
1533+
EXTENSION_TYPE_NAME_KEY.to_string(),
1534+
"arrow.json".to_string(),
1535+
);
1536+
target_meta.insert(EXTENSION_TYPE_METADATA_KEY.to_string(), "{}".to_string());
1537+
1538+
let target_field =
1539+
Field::new("", DataType::Utf8, true).with_metadata(target_meta);
1540+
1541+
let cast_expr =
1542+
make_cast_expr(col("foo"), Arc::new(target_field), use_try_cast);
1543+
1544+
let (_, result_field) = cast_expr.to_field(&schema).unwrap();
1545+
assert_eq!(
1546+
result_field.metadata().get(EXTENSION_TYPE_NAME_KEY),
1547+
Some(&"arrow.json".to_string()),
1548+
"{cast_name}: Extension type name should come from target field"
1549+
);
1550+
assert_eq!(
1551+
result_field.metadata().get(EXTENSION_TYPE_METADATA_KEY),
1552+
Some(&"{}".to_string()),
1553+
"{cast_name}: Extension type metadata should come from target field"
1554+
);
1555+
assert!(
1556+
result_field.metadata().get("custom_key").is_none(),
1557+
"{cast_name}: Source metadata should NOT propagate when target has explicit metadata"
1558+
);
1559+
if use_try_cast {
1560+
assert!(
1561+
result_field.is_nullable(),
1562+
"TryCast result should be nullable"
1563+
);
1564+
}
1565+
}
1566+
}
14301567
}

‎datafusion/functions/src/core/arrow_cast.rs‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,8 @@ use datafusion_common::{
2727

2828
use datafusion_expr::simplify::{ExprSimplifyResult, SimplifyContext};
2929
use datafusion_expr::{
30-
Coercion, ColumnarValue, Documentation, Expr, ReturnFieldArgs, ScalarFunctionArgs,
31-
ScalarUDFImpl, Signature, TypeSignatureClass, Volatility,
30+
Coercion, ColumnarValue, Documentation, Expr, ExprSchemable, ReturnFieldArgs,
31+
ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignatureClass, Volatility,
3232
};
3333
use datafusion_macros::user_doc;
3434

@@ -161,9 +161,27 @@ impl ScalarUDFImpl for ArrowCastFunc {
161161
let [source_arg, type_arg] = take_function_args(self.name(), args)?;
162162
let target_type = data_type_from_type_arg(self.name(), &type_arg)?;
163163
let source_type = info.get_data_type(&source_arg)?;
164+
165+
// We can skip the cast only if:
166+
// 1. The source and target types are the same
167+
// 2. The source has no extension metadata that needs to be stripped
164168
let new_expr = if source_type == target_type {
165-
// the argument's data type is already the correct type
166-
source_arg
169+
// Check if source has extension metadata
170+
let source_field = source_arg.to_field(info.schema())?;
171+
let has_extension_metadata = source_field
172+
.1
173+
.metadata()
174+
.contains_key("ARROW:extension:name");
175+
if has_extension_metadata {
176+
// Need to create a cast to strip extension metadata
177+
Expr::Cast(datafusion_expr::Cast {
178+
expr: Box::new(source_arg),
179+
field: target_type.into_nullable_field_ref(),
180+
})
181+
} else {
182+
// the argument's data type is already the correct type
183+
source_arg
184+
}
167185
} else {
168186
// Use an actual cast to get the correct type
169187
Expr::Cast(datafusion_expr::Cast {

‎datafusion/functions/src/core/arrow_try_cast.rs‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,8 +26,8 @@ use datafusion_common::{
2626

2727
use datafusion_expr::simplify::{ExprSimplifyResult, SimplifyContext};
2828
use datafusion_expr::{
29-
Coercion, ColumnarValue, Documentation, Expr, ReturnFieldArgs, ScalarFunctionArgs,
30-
ScalarUDFImpl, Signature, TypeSignatureClass, Volatility,
29+
Coercion, ColumnarValue, Documentation, Expr, ExprSchemable, ReturnFieldArgs,
30+
ScalarFunctionArgs, ScalarUDFImpl, Signature, TypeSignatureClass, Volatility,
3131
};
3232
use datafusion_macros::user_doc;
3333

@@ -134,8 +134,26 @@ impl ScalarUDFImpl for ArrowTryCastFunc {
134134
let target_type = data_type_from_type_arg(self.name(), &type_arg)?;
135135

136136
let source_type = info.get_data_type(&source_arg)?;
137+
138+
// We can skip the cast only if:
139+
// 1. The source and target types are the same
140+
// 2. The source has no extension metadata that needs to be stripped
137141
let new_expr = if source_type == target_type {
138-
source_arg
142+
// Check if source has extension metadata
143+
let source_field = source_arg.to_field(info.schema())?;
144+
let has_extension_metadata = source_field
145+
.1
146+
.metadata()
147+
.contains_key("ARROW:extension:name");
148+
if has_extension_metadata {
149+
// Need to create a try_cast to strip extension metadata
150+
Expr::TryCast(datafusion_expr::TryCast {
151+
expr: Box::new(source_arg),
152+
field: target_type.into_nullable_field_ref(),
153+
})
154+
} else {
155+
source_arg
156+
}
139157
} else {
140158
Expr::TryCast(datafusion_expr::TryCast {
141159
expr: Box::new(source_arg),

‎datafusion/physical-expr-adapter/src/rewrite.rs‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,9 @@ mod tests {
319319
assert_eq!(source.name(), "__datafusion_file_row_index");
320320
assert_eq!(source.index(), 2);
321321

322+
// The row index column is at index 2, beyond the user-visible schema.
323+
// When the source column lookup fails, the field name is empty. The
324+
// correct field name would be provided by a parent projection/alias.
322325
let input_schema = Schema::new(vec![
323326
Field::new("value", DataType::Int64, true),
324327
Field::new("__datafusion_file_row_index", DataType::Int64, false)
@@ -328,9 +331,11 @@ mod tests {
328331
)])),
329332
]);
330333
let return_field = expr.return_field(&input_schema)?;
331-
assert_eq!(return_field.name(), "file_row_index");
334+
// Field name is empty because column index 2 is beyond the schema
335+
assert_eq!(return_field.name(), "");
332336
assert_eq!(return_field.data_type(), &DataType::Int64);
333337
assert!(return_field.is_nullable());
338+
// Exact target field does not preserve source metadata
334339
assert!(return_field.metadata().is_empty());
335340
Ok(())
336341
}

‎datafusion/physical-expr/Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@ proto = [
5151

5252
[dependencies]
5353
arrow = { workspace = true }
54+
arrow-schema = { workspace = true }
5455
datafusion-common = { workspace = true }
5556
datafusion-expr = { workspace = true }
5657
datafusion-expr-common = { workspace = true }

0 commit comments

Comments
 (0)