Skip to content

Commit c882119

Browse files
sunchaoadriangb
authored andcommitted
fix: preserve explicit Struct casts and evolved field semantics (apache#24680)
## Which issue does this PR close? Closes apache#24679. Follow-up to apache#24125 and apache#24530. ## Rationale for this change When Parquet files have different schemas, DataFusion may need to convert a file's Struct column to the table's logical Struct type. If a query only reads `s.x`, struct-cast narrowing can save work by converting just `x` instead of every field in `s`. That optimization must preserve the meaning of the original expression. The earlier narrowing rule could hide an explicit cast's error, change the result's nullability, or make an entirely null input fail during decimal conversion. ### Selecting one field must not hide an explicit cast error For `s = {x: 1, y: 'bad'}`, consider this physical expression: ```sql get_field(CAST(s AS STRUCT<x INT, y INT>), 'x') ``` The explicit cast asks to convert both fields. It must fail because `'bad'` cannot become an integer, even though the caller only uses `x`. Narrowing it to `CAST(get_field(s, 'x') AS INT)` instead returns `1`: the conversion of `y`, and its error, have disappeared. This failure is reproduced directly at the physical-expression adapter boundary. The distinction is whether a cast was inserted to reconcile file and table schemas or was already part of the query; the two can have the same shape but different obligations. ### An entirely null Struct should not acquire a decimal-conversion error Consider a Parquet batch with this schema evolution: ```text File schema: s: Struct<x: Utf8> Table schema: s: Struct<x: Decimal128(10, -1)> Batch values: s is NULL for every row ``` DataFusion's whole-Struct conversion returns nulls without converting its children. If the rewrite extracts `x` first, however, it invokes string-to-decimal conversion on an array of null strings. Arrow rejects the negative scale while setting up that conversion, before looking at any values. A query that should return nulls now fails despite there being no non-null value to convert. The same problem occurs inside a selected container. For example, evolving `s.x` from `List<Utf8>` to `List<Decimal128(10, -1)>` can fail even when every parent Struct is null. Checking only whether `x` itself is a decimal misses the conversion inside the List. This case also reproduces through a Parquet scan with filter pushdown: `WHERE get_field(s, 'x') IS NULL` should select every row in the batch, not fail while preparing a decimal conversion. There is a related schema-contract problem. A required child `x` inside a nullable logical parent `s` still gives a nullable result for `s.x`. Rebuilding an expression from the child's Field alone can lose that inherited nullability, including through nested parents. ## What changes are included in this PR? The adapter now distinguishes casts it introduces for schema adaptation from casts already present in the query. It can continue narrowing generated conversions where that is safe while preserving explicit Struct casts and their errors. Narrowing also preserves the original logical result Field, including metadata and nullability; matching scalar types alone do not establish an equivalent result. Cast tracking also avoids quadratic lookup work for large expressions. For the covered decimal changes, the conversion stays inside its Struct ancestors so DataFusion's existing casting code retains control over the all-null shortcut. The decision looks through container value types too, so selecting a List, Map, or Dictionary does not hide a decimal conversion from the check. The cast target keeps only the selected field path, excluding conversions for unselected siblings. Matching types still narrow normally, and existing container-to-Struct conversion paths are preserved. This extends the existing fix without duplicating Arrow's conversion-validation rules or changing the underlying casting implementation. Unwrapping a Dictionary whose values are Structs retains its existing all-null decimal limitation; the built-in Parquet reader does not produce that source shape. The Parquet reader must then be able to evaluate the retained expression after schema adaptation. This matters when the query plan has already delegated a predicate to the reader: retaining a cast must not prevent that filter from executing. The runtime allowance is specifically for field access through a Struct-to-Struct cast of a column. The reader preserves the cast and reuses projection's existing read clipping to decode only the leaves named by the cast target, with a full-root fallback when clipping is unsafe. The target, rather than the final field selection, determines which conversions must run: an explicit cast that names `y` still reads and converts `y` even if the query selects only `x`. Filter cost estimates use the same selected leaves. Planning remains conservative about explicit casts, keeping a residual filter for them. ## Are these changes tested? The Rust regressions cover explicit cast errors, nullability inherited from parents, decimal conversions on entirely null Structs, unselected sibling conversion errors, and execution through Parquet filters. Container cases include the List families, Maps, Dictionaries, and deeper nesting, with checks that matching types and existing container-to-Struct conversions keep their previous behavior. The existing `schema_evolution_nested.slt` now adds SQL coverage using generated Parquet files: an explicit Struct cast whose unselected sibling must still fail, all-null scalar and List decimal projections, and null predicates with filter pushdown both enabled and disabled. Statistics and pruning shortcuts are disabled for the filter cases. These SQL tests supplement the Rust tests, which also check adapter provenance, logical Field metadata/nullability, expression shape, and Arrow types that are difficult to express in SQL. The explicit-cast SQL case is an integration control; the direct adapter test exercises the cast-provenance regression. The retained-cast read-plan regression now uses three physical siblings. It verifies that an unused sibling is pruned, a sibling named by the cast still raises its conversion error, the projected physical schema is correct, and the estimated compressed bytes count only the selected leaves. A target containing just the selected field also decodes and evaluates successfully. Local validation for [ec4b3f2](apache@ec4b3f2) used the tracked `Cargo.lock` unchanged, with `--locked` for the Cargo test and all-feature Clippy commands: - The required extended workspace suite passed **10,795 Rust tests** (eight existing ignores) and **all 505 SQL logic files**. - All **107 CLI tests** passed. - The adapter and Parquet datasource crates also passed separately: **291 unit tests and eight doctests**, with five existing doctest ignores. - `cargo fmt --all`, full-workspace Clippy across all targets and features with warnings denied, and the complete `./dev/rust_lint.sh` suite passed. The strengthened read-plan regression was also checked with the old full-root collection behavior restored: it failed because it selected leaves `[1, 2, 3]` instead of `[1, 2]`. Restoring the fix made it pass again. <details> <summary>Existing benchmark comparison</summary> The existing benchmarks were compared against `main` at `ee59f628b44eb80e8a4f126288632ef51fda5dc2`, using `release-nonlto`, freshly built workspace artifacts for each revision, and the checked-in Criterion settings. Variants ran sequentially after builds and tests had stopped. | Existing benchmark case | Result | | --- | --- | | `parquet_struct_filter_pushdown`: `select_id/with_pushdown` | No change detected (p = 0.83; 100 samples per variant). | | `parquet_nested_schema_pruning`: `top_level_struct/select_struct_narrow_schema` | PR/main mean-time changes were +4.15%, -2.96%, and -2.84% across three repeats, including reversed execution order (10 samples per variant). The direction changed, so no stable timing conclusion is claimed. | Scan-byte counts were identical in every projection run. These cases cover ordinary field-filter pushdown and the existing projection-clipping path; neither directly measures retained-cast row-filter speedup. The `main` snapshot also contains unrelated commits, so this comparison does not isolate the follow-up patch. </details> ## Are there any user-facing changes? The intended changes preserve explicit-cast errors, logical field metadata and nullability, and the covered all-null decimal behavior. Ordinary field pruning remains enabled. Retained-cast Parquet filters can also avoid reading sibling leaves outside the cast target; required conversions are preserved, with a full-root fallback where necessary. There are no public API or dependency changes. General changes to container-to-Struct conversion semantics, generic `get_field` behavior under null parents, and masking of encoded arrays remain outside this PR's scope. AI assistance: Codex assisted with the implementation, regression tests, PR text, and the stated local checks and source reviews. The row-filter clipping follow-up incorporates Adrian Garcia Badaracco's [recommended change](pydantic@9746f3f), with additional regression coverage. --------- Co-authored-by: Adrian Garcia Badaracco <[email protected]>
1 parent e4c72c5 commit c882119

5 files changed

Lines changed: 1251 additions & 72 deletions

File tree

‎datafusion/core/tests/parquet/expr_adapter.rs‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -789,6 +789,212 @@ async fn test_physical_expr_adapter_with_non_null_defaults() {
789789
assert_batches_eq!(expected, &batches);
790790
}
791791

792+
#[tokio::test]
793+
async fn test_explicit_struct_cast_projection_preserves_sibling_errors() -> Result<()> {
794+
let physical_fields: Fields = vec![
795+
Field::new("x", DataType::Int32, true),
796+
Field::new("y", DataType::Utf8, true),
797+
]
798+
.into();
799+
let batch = RecordBatch::try_from_iter(vec![(
800+
"s",
801+
Arc::new(StructArray::new(
802+
physical_fields,
803+
vec![
804+
Arc::new(Int32Array::from(vec![1])) as ArrayRef,
805+
Arc::new(StringArray::from(vec!["bad"])) as ArrayRef,
806+
],
807+
None,
808+
)) as ArrayRef,
809+
)])?;
810+
let table_schema = Arc::new(Schema::new(vec![Field::new(
811+
"s",
812+
DataType::Struct(
813+
vec![
814+
Field::new("x", DataType::Int64, true),
815+
Field::new("y", DataType::Utf8, true),
816+
]
817+
.into(),
818+
),
819+
true,
820+
)]));
821+
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
822+
write_parquet(batch, Arc::clone(&store), "explicit_cast/data.parquet").await;
823+
let ctx = test_context();
824+
register_memory_listing_table(&ctx, store, "memory:///explicit_cast/", table_schema)
825+
.await;
826+
827+
// The file requires schema adaptation for x. The explicit SQL cast also
828+
// converts y, and selecting x must not hide that invalid conversion.
829+
let error = ctx
830+
.sql("SELECT get_field(CAST(s AS STRUCT<x BIGINT, y INT>), 'x') FROM t")
831+
.await?
832+
.collect()
833+
.await
834+
.unwrap_err()
835+
.to_string();
836+
datafusion_common::assert_contains!(error, "While casting struct field 'y'");
837+
Ok(())
838+
}
839+
840+
#[tokio::test]
841+
async fn test_all_null_struct_decimal_cast_filter_pushdown() -> Result<()> {
842+
use arrow::array::new_null_array;
843+
use datafusion_physical_plan::{collect, displayable};
844+
845+
for (physical_type, logical_type) in [
846+
(DataType::Utf8, DataType::Decimal128(10, -1)),
847+
(
848+
DataType::new_list(DataType::Utf8, true),
849+
DataType::new_list(DataType::Decimal128(10, -1), true),
850+
),
851+
] {
852+
let physical_fields: Fields =
853+
vec![Field::new("x", physical_type.clone(), true)].into();
854+
let batch = RecordBatch::try_from_iter(vec![
855+
("row_id", Arc::new(Int32Array::from(vec![1, 2])) as ArrayRef),
856+
(
857+
"s",
858+
Arc::new(StructArray::new(
859+
physical_fields,
860+
vec![new_null_array(&physical_type, 2)],
861+
Some(NullBuffer::new_null(2)),
862+
)) as ArrayRef,
863+
),
864+
])?;
865+
let table_schema = Arc::new(Schema::new(vec![
866+
Field::new("row_id", DataType::Int32, false),
867+
Field::new(
868+
"s",
869+
DataType::Struct(vec![Field::new("x", logical_type, true)].into()),
870+
true,
871+
),
872+
]));
873+
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
874+
write_parquet(batch, Arc::clone(&store), "null_decimal/data.parquet").await;
875+
876+
for pushdown_filters in [false, true] {
877+
let mut config = SessionConfig::new()
878+
.with_collect_statistics(false)
879+
.with_parquet_pruning(false)
880+
.with_parquet_page_index_pruning(false);
881+
config.options_mut().execution.parquet.pushdown_filters = pushdown_filters;
882+
let ctx = SessionContext::new_with_config(config);
883+
register_memory_listing_table(
884+
&ctx,
885+
Arc::clone(&store),
886+
"memory:///null_decimal/",
887+
Arc::clone(&table_schema),
888+
)
889+
.await;
890+
891+
for (predicate, expected_rows) in [("IS NULL", 2), ("IS NOT NULL", 0)] {
892+
let plan = ctx
893+
.sql(&format!(
894+
"SELECT row_id FROM t WHERE get_field(s, 'x') {predicate}"
895+
))
896+
.await?
897+
.create_physical_plan()
898+
.await?;
899+
if pushdown_filters {
900+
let plan_text = displayable(plan.as_ref()).indent(false).to_string();
901+
assert!(
902+
!plan_text.contains("FilterExec"),
903+
"the scan must fully handle the filter: {plan_text}"
904+
);
905+
}
906+
let batches = collect(plan, ctx.task_ctx()).await?;
907+
assert_eq!(
908+
batches.iter().map(RecordBatch::num_rows).sum::<usize>(),
909+
expected_rows,
910+
"physical_type={physical_type:?}, pushdown_filters={pushdown_filters}, predicate={predicate}"
911+
);
912+
}
913+
}
914+
}
915+
Ok(())
916+
}
917+
918+
#[tokio::test]
919+
async fn test_evolved_decimal_ignores_unselected_sibling() -> Result<()> {
920+
let physical_fields: Fields = vec![
921+
Field::new("x", DataType::Int32, true),
922+
Field::new("y", DataType::Utf8, true),
923+
]
924+
.into();
925+
let batch = RecordBatch::try_from_iter(vec![
926+
("row_id", Arc::new(Int32Array::from(vec![1, 2])) as ArrayRef),
927+
(
928+
"s",
929+
Arc::new(StructArray::new(
930+
physical_fields,
931+
vec![
932+
Arc::new(Int32Array::from(vec![1, 0])),
933+
Arc::new(StringArray::from(vec!["bad", "bad"])),
934+
],
935+
None,
936+
)) as ArrayRef,
937+
),
938+
])?;
939+
let table_schema = Arc::new(Schema::new(vec![
940+
Field::new("row_id", DataType::Int32, false),
941+
Field::new(
942+
"s",
943+
DataType::Struct(
944+
vec![
945+
Field::new("x", DataType::Decimal128(10, 2), true),
946+
Field::new("y", DataType::Int32, true),
947+
]
948+
.into(),
949+
),
950+
true,
951+
),
952+
]));
953+
let store = Arc::new(InMemory::new()) as Arc<dyn ObjectStore>;
954+
write_parquet(batch, Arc::clone(&store), "decimal_sibling/data.parquet").await;
955+
956+
for pushdown_filters in [false, true] {
957+
let mut config = SessionConfig::new()
958+
.with_collect_statistics(false)
959+
.with_parquet_pruning(false)
960+
.with_parquet_page_index_pruning(false);
961+
config.options_mut().execution.parquet.pushdown_filters = pushdown_filters;
962+
let ctx = SessionContext::new_with_config(config);
963+
register_memory_listing_table(
964+
&ctx,
965+
Arc::clone(&store),
966+
"memory:///decimal_sibling/",
967+
Arc::clone(&table_schema),
968+
)
969+
.await;
970+
971+
// Adapting x must not evaluate the invalid conversion of y.
972+
for (sql, expected) in [
973+
(
974+
"SELECT get_field(s, 'x') AS x FROM t",
975+
vec![
976+
"+------+", "| x |", "+------+", "| 1.00 |", "| 0.00 |",
977+
"+------+",
978+
],
979+
),
980+
(
981+
"SELECT row_id FROM t WHERE get_field(s, 'x') > 0",
982+
vec![
983+
"+--------+",
984+
"| row_id |",
985+
"+--------+",
986+
"| 1 |",
987+
"+--------+",
988+
],
989+
),
990+
] {
991+
let batches = ctx.sql(sql).await?.collect().await?;
992+
assert_batches_eq!(expected, &batches);
993+
}
994+
}
995+
Ok(())
996+
}
997+
792998
#[tokio::test]
793999
async fn test_struct_schema_evolution_projection_and_filter() -> Result<()> {
7941000
use std::collections::HashMap;

0 commit comments

Comments
 (0)