Repository navigation
fix(core): allow safe_cast when merge widens a struct - #4917
Merged
Merged
Conversation
ahlag
requested review from
ethan-tyler,
ion-elgreco,
roeap and
rtyler
as code owners
October 20, 2026 11:57
Contributor
Author
|
@ion-elgreco |
Collaborator
|
@ahlag can you add a python test as well? |
ahlag
force-pushed
the
fix-merge-struct-safe-cast
branch
from
October 20, 2026 12:17
f8fbb90 to
29cc4a5
Compare
Contributor
Author
|
@ion-elgreco Added a Python test in 29cc4a5. It runs the struct-widening merge with error_on_type_mismatch True and False and checks the rows plus the change feed. Both fail on 1.6.6 and pass with this PR. |
ion-elgreco
force-pushed
the
fix-merge-struct-safe-cast
branch
from
October 20, 2026 13:30
29cc4a5 to
1eb4423
Compare
ion-elgreco
enabled auto-merge (squash)
October 20, 2026 13:31
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4917 +/- ##
==========================================
- Coverage 81.95% 81.93% -0.02%
==========================================
Files 200 200
Lines 75769 75695 -74
Branches 75769 75695 -74
==========================================
- Hits 62094 62020 -74
Misses 10042 10042
Partials 3633 3633 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
With safe_cast, a merge that widens a struct failed during planning with "Unsupported TRY_CAST from Struct(...)": DataFusion plans try_cast with arrow's can_cast_types, which requires the same struct fields. The merged type is a superset of the target and source types, so changed fields now use a strict cast in the CASE branches and the CDF preimage. Signed-off-by: ahlag <[email protected]>
Signed-off-by: ahlag <[email protected]>
ion-elgreco
force-pushed
the
fix-merge-struct-safe-cast
branch
from
October 20, 2026 16:29
1eb4423 to
ae94209
Compare
rtyler
added a commit
to rtyler/delta-rs
that referenced
this pull request
Nov 7, 2026
Follow-up to delta-io#4889, from @ion-elgreco's [comment](delta-io#4889 (comment)). With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a merge with `merge_schema=True` that widens a struct fails during planning: ``` Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8) ``` DataFusion's `cast` checks struct casts by field name, but `try_cast` still plans with arrow's `can_cast_types`, which needs the same fields (`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion `main`). The merged type is a superset of the target and source types, so this PR uses a strict `cast` for changed fields, in the CASE branches and in the CDF target preimage. This is the change Ion proposed. Tests: the struct-widening merge tests now run with and without `with_safe_cast(true)` (`rstest` `#[values(false, true)]`): - `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows` - `test_merge_with_schema_merge_struct_added_field_keeps_unset_column` - `test_merge_with_schema_merge_and_pre_existing_struct_added_column` (the source struct only has the new field) - new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF enabled, checks the preimage and postimage of the updated row. The four `safe_cast = true` cases fail on `main` with the error above. Testing (macOS arm64): - `cargo test -p deltalake-core --features datafusion --lib`: 1362 passed - `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features datafusion --tests --no-deps` reports nothing in `merge/mod.rs`. - Python bindings built from this branch: `tests/test_merge.py` 104 passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`, a source struct with only the new field, and a struct with CDF enabled all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake` 1.6.6, the first three drop the unmatched row's value to null and the CDF case fails with the same error. AI use: I used Claude Code to apply the change, write the tests, and run the test suites. I reviewed the diff and checked the DataFusion planning check and the test results myself. - follow-up to delta-io#4889 / delta-io#4887 - upstream: apache/datafusion#26528 (`try_cast` rejects struct casts that `cast` accepts) - DataFusion `try_cast` planning: https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs --------- Signed-off-by: ahlag <[email protected]> (cherry picked from commit 0a14212) Co-Authored-By: Devstral 2 <[email protected]> Signed-off-by: R. Tyler Croy <[email protected]> Co-Authored-By: R. Tyler Croy <[email protected]> Signed-off-by: R. Tyler Croy <[email protected]>
rtyler
pushed a commit
to rtyler/delta-rs
that referenced
this pull request
Nov 7, 2026
Follow-up to delta-io#4889, from @ion-elgreco's [comment](delta-io#4889 (comment)). With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a merge with `merge_schema=True` that widens a struct fails during planning: ``` Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8) ``` DataFusion's `cast` checks struct casts by field name, but `try_cast` still plans with arrow's `can_cast_types`, which needs the same fields (`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion `main`). The merged type is a superset of the target and source types, so this PR uses a strict `cast` for changed fields, in the CASE branches and in the CDF target preimage. This is the change Ion proposed. Tests: the struct-widening merge tests now run with and without `with_safe_cast(true)` (`rstest` `#[values(false, true)]`): - `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows` - `test_merge_with_schema_merge_struct_added_field_keeps_unset_column` - `test_merge_with_schema_merge_and_pre_existing_struct_added_column` (the source struct only has the new field) - new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF enabled, checks the preimage and postimage of the updated row. The four `safe_cast = true` cases fail on `main` with the error above. Testing (macOS arm64): - `cargo test -p deltalake-core --features datafusion --lib`: 1362 passed - `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features datafusion --tests --no-deps` reports nothing in `merge/mod.rs`. - Python bindings built from this branch: `tests/test_merge.py` 104 passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`, a source struct with only the new field, and a struct with CDF enabled all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake` 1.6.6, the first three drop the unmatched row's value to null and the CDF case fails with the same error. AI use: I used Claude Code to apply the change, write the tests, and run the test suites. I reviewed the diff and checked the DataFusion planning check and the test results myself. - follow-up to delta-io#4889 / delta-io#4887 - upstream: apache/datafusion#26528 (`try_cast` rejects struct casts that `cast` accepts) - DataFusion `try_cast` planning: https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs --------- Signed-off-by: ahlag <[email protected]> (cherry picked from commit 0a14212) Signed-off-by: R. Tyler Croy <[email protected]> Co-Authored-By: R. Tyler Croy <[email protected]> Co-Authored-By: Claude Haiku 5.5 <[email protected]>
rtyler
pushed a commit
that referenced
this pull request
Nov 7, 2026
Follow-up to #4889, from @ion-elgreco's [comment](#4889 (comment)). With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a merge with `merge_schema=True` that widens a struct fails during planning: ``` Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8) ``` DataFusion's `cast` checks struct casts by field name, but `try_cast` still plans with arrow's `can_cast_types`, which needs the same fields (`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion `main`). The merged type is a superset of the target and source types, so this PR uses a strict `cast` for changed fields, in the CASE branches and in the CDF target preimage. This is the change Ion proposed. Tests: the struct-widening merge tests now run with and without `with_safe_cast(true)` (`rstest` `#[values(false, true)]`): - `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows` - `test_merge_with_schema_merge_struct_added_field_keeps_unset_column` - `test_merge_with_schema_merge_and_pre_existing_struct_added_column` (the source struct only has the new field) - new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF enabled, checks the preimage and postimage of the updated row. The four `safe_cast = true` cases fail on `main` with the error above. Testing (macOS arm64): - `cargo test -p deltalake-core --features datafusion --lib`: 1362 passed - `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features datafusion --tests --no-deps` reports nothing in `merge/mod.rs`. - Python bindings built from this branch: `tests/test_merge.py` 104 passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`, a source struct with only the new field, and a struct with CDF enabled all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake` 1.6.6, the first three drop the unmatched row's value to null and the CDF case fails with the same error. AI use: I used Claude Code to apply the change, write the tests, and run the test suites. I reviewed the diff and checked the DataFusion planning check and the test results myself. - follow-up to #4889 / #4887 - upstream: apache/datafusion#26528 (`try_cast` rejects struct casts that `cast` accepts) - DataFusion `try_cast` planning: https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs --------- Signed-off-by: ahlag <[email protected]> (cherry picked from commit 0a14212) Signed-off-by: R. Tyler Croy <[email protected]> Co-Authored-By: R. Tyler Croy <[email protected]> Co-Authored-By: Claude Haiku 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up to #4889, from @ion-elgreco's comment.
With
safe_castenabled (error_on_type_mismatch=Falsein Python), a merge withmerge_schema=Truethat widens a struct fails during planning:DataFusion's
castchecks struct casts by field name, buttry_caststill plans with arrow'scan_cast_types, which needs the same fields (datafusion-physical-expr55.2try_cast.rs, also on DataFusionmain). The merged type is a superset of the target and source types, so this PR uses a strictcastfor changed fields, in the CASE branches and in the CDF target preimage. This is the change Ion proposed.Tests: the struct-widening merge tests now run with and without
with_safe_cast(true)(rstest#[values(false, true)]):test_merge_with_schema_merge_struct_added_field_keeps_copied_rowstest_merge_with_schema_merge_struct_added_field_keeps_unset_columntest_merge_with_schema_merge_and_pre_existing_struct_added_column(the source struct only has the new field)test_merge_with_schema_merge_struct_added_field_cdf: CDF enabled, checks the preimage and postimage of the updated row.The four
safe_cast = truecases fail onmainwith the error above.Testing (macOS arm64):
cargo test -p deltalake-core --features datafusion --lib: 1362 passedcargo fmt --check: clean.cargo clippy -p deltalake-core --features datafusion --tests --no-depsreports nothing inmerge/mod.rs.tests/test_merge.py104 passed. Witherror_on_type_mismatch=False, a struct, alist<struct>, a source struct with only the new field, and a struct with CDF enabled all merge, and the unmatched row keeps{x: 2, y: None}. Ondeltalake1.6.6, the first three drop the unmatched row's value to null and the CDF case fails with the same error.AI use: I used Claude Code to apply the change, write the tests, and run the test suites. I reviewed the diff and checked the DataFusion planning check and the test results myself.
Related Issue(s)
try_castrejects struct casts thatcastaccepts)Documentation
try_castplanning: https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs