Skip to content

fix(core): allow safe_cast when merge widens a struct - #4917

Merged
ion-elgreco merged 2 commits into
delta-io:mainfrom
ahlag:fix-merge-struct-safe-cast
Oct 20, 2026
Merged

ion-elgreco merged 2 commits into
delta-io:mainfrom
ahlag:fix-merge-struct-safe-cast

Conversation

@ahlag

@ahlag ahlag commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Description

Follow-up to #4889, from @ion-elgreco's comment.

With safe_cast enabled (error_on_type_mismatch=False in Python), a merge with merge_schema=True that widens a struct fails during planning:

Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8)

DataFusion's cast checks struct casts by field name, but try_cast still plans with arrow's can_cast_types, which needs the same fields (datafusion-physical-expr 55.2 try_cast.rs, also on DataFusion main). The merged type is a superset of the target and source types, so this PR uses a strict cast for changed fields, in the CASE branches and in the CDF target preimage. This is the change Ion proposed.

Tests: the struct-widening merge tests now run with and without with_safe_cast(true) (rstest #[values(false, true)]):

  • test_merge_with_schema_merge_struct_added_field_keeps_copied_rows
  • test_merge_with_schema_merge_struct_added_field_keeps_unset_column
  • test_merge_with_schema_merge_and_pre_existing_struct_added_column (the source struct only has the new field)
  • new test_merge_with_schema_merge_struct_added_field_cdf: CDF enabled, checks the preimage and postimage of the updated row.

The four safe_cast = true cases fail on main with the error above.

Testing (macOS arm64):

  • cargo test -p deltalake-core --features datafusion --lib: 1362 passed
  • cargo fmt --check: clean. cargo clippy -p deltalake-core --features datafusion --tests --no-deps reports nothing in merge/mod.rs.
  • Python bindings built from this branch: tests/test_merge.py 104 passed. With error_on_type_mismatch=False, a struct, a list<struct>, a source struct with only the new field, and a struct with CDF enabled all merge, and the unmatched row keeps {x: 2, y: None}. On deltalake 1.6.6, the first three drop the unmatched row's value to null and the CDF case fails with the same error.

AI use: I used Claude Code to apply the change, write the tests, and run the test suites. I reviewed the diff and checked the DataFusion planning check and the test results myself.

Related Issue(s)

Documentation

@ahlag

ahlag commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@ion-elgreco
Please let me know if this fix is what needed in #4889 (comment)

@ion-elgreco

Copy link
Copy Markdown
Collaborator

@ahlag can you add a python test as well?

@ahlag
ahlag force-pushed the fix-merge-struct-safe-cast branch from f8fbb90 to 29cc4a5 Compare October 20, 2026 12:17
@github-actions github-actions Bot added the binding/python Issues for the Python package label Oct 20, 2026
@ahlag

ahlag commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

@ion-elgreco Added a Python test in 29cc4a5. It runs the struct-widening merge with error_on_type_mismatch True and False and checks the rows plus the change feed. Both fail on 1.6.6 and pass with this PR.

@ion-elgreco ion-elgreco left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@ion-elgreco
ion-elgreco force-pushed the fix-merge-struct-safe-cast branch from 29cc4a5 to 1eb4423 Compare October 20, 2026 13:30
@ion-elgreco
ion-elgreco enabled auto-merge (squash) October 20, 2026 13:31
@codecov

codecov Bot commented Oct 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.93%. Comparing base (10a7c47) to head (ae94209).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4917      +/-   ##
==========================================
- Coverage   81.95%   81.93%   -0.02%     
==========================================
  Files         200      200              
  Lines       75769    75695      -74     
  Branches    75769    75695      -74     
==========================================
- Hits        62094    62020      -74     
  Misses      10042    10042              
  Partials     3633     3633              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ahlag added 2 commits October 20, 2026 18:29
With safe_cast, a merge that widens a struct failed during planning with
"Unsupported TRY_CAST from Struct(...)": DataFusion plans try_cast with
arrow's can_cast_types, which requires the same struct fields. The merged
type is a superset of the target and source types, so changed fields now
use a strict cast in the CASE branches and the CDF preimage.

Signed-off-by: ahlag <[email protected]>
@ion-elgreco
ion-elgreco force-pushed the fix-merge-struct-safe-cast branch from 1eb4423 to ae94209 Compare October 20, 2026 16:29
@ion-elgreco
ion-elgreco merged commit 0a14212 into delta-io:main Oct 20, 2026
35 checks passed
rtyler added a commit to rtyler/delta-rs that referenced this pull request Nov 7, 2026
Follow-up to delta-io#4889, from @ion-elgreco's
[comment](delta-io#4889 (comment)).

With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a
merge with `merge_schema=True` that widens a struct fails during
planning:

```
Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8)
```

DataFusion's `cast` checks struct casts by field name, but `try_cast`
still plans with arrow's `can_cast_types`, which needs the same fields
(`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion
`main`). The merged type is a superset of the target and source types,
so this PR uses a strict `cast` for changed fields, in the CASE branches
and in the CDF target preimage. This is the change Ion proposed.

Tests: the struct-widening merge tests now run with and without
`with_safe_cast(true)` (`rstest` `#[values(false, true)]`):
- `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows`
- `test_merge_with_schema_merge_struct_added_field_keeps_unset_column`
- `test_merge_with_schema_merge_and_pre_existing_struct_added_column`
(the source struct only has the new field)
- new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF
enabled, checks the preimage and postimage of the updated row.

The four `safe_cast = true` cases fail on `main` with the error above.

Testing (macOS arm64):
- `cargo test -p deltalake-core --features datafusion --lib`: 1362
passed
- `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features
datafusion --tests --no-deps` reports nothing in `merge/mod.rs`.
- Python bindings built from this branch: `tests/test_merge.py` 104
passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`,
a source struct with only the new field, and a struct with CDF enabled
all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake`
1.6.6, the first three drop the unmatched row's value to null and the
CDF case fails with the same error.

AI use: I used Claude Code to apply the change, write the tests, and run
the test suites. I reviewed the diff and checked the DataFusion planning
check and the test results myself.

- follow-up to delta-io#4889 / delta-io#4887
- upstream: apache/datafusion#26528 (`try_cast` rejects struct casts
that `cast` accepts)

- DataFusion `try_cast` planning:
https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs

---------

Signed-off-by: ahlag <[email protected]>

(cherry picked from commit 0a14212)

Co-Authored-By: Devstral 2 <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]>
Co-Authored-By: R. Tyler Croy <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]>
rtyler pushed a commit to rtyler/delta-rs that referenced this pull request Nov 7, 2026
Follow-up to delta-io#4889, from @ion-elgreco's
[comment](delta-io#4889 (comment)).

With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a
merge with `merge_schema=True` that widens a struct fails during
planning:

```
Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8)
```

DataFusion's `cast` checks struct casts by field name, but `try_cast`
still plans with arrow's `can_cast_types`, which needs the same fields
(`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion
`main`). The merged type is a superset of the target and source types,
so this PR uses a strict `cast` for changed fields, in the CASE branches
and in the CDF target preimage. This is the change Ion proposed.

Tests: the struct-widening merge tests now run with and without
`with_safe_cast(true)` (`rstest` `#[values(false, true)]`):
- `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows`
- `test_merge_with_schema_merge_struct_added_field_keeps_unset_column`
- `test_merge_with_schema_merge_and_pre_existing_struct_added_column`
(the source struct only has the new field)
- new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF
enabled, checks the preimage and postimage of the updated row.

The four `safe_cast = true` cases fail on `main` with the error above.

Testing (macOS arm64):
- `cargo test -p deltalake-core --features datafusion --lib`: 1362
passed
- `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features
datafusion --tests --no-deps` reports nothing in `merge/mod.rs`.
- Python bindings built from this branch: `tests/test_merge.py` 104
passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`,
a source struct with only the new field, and a struct with CDF enabled
all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake`
1.6.6, the first three drop the unmatched row's value to null and the
CDF case fails with the same error.

AI use: I used Claude Code to apply the change, write the tests, and run
the test suites. I reviewed the diff and checked the DataFusion planning
check and the test results myself.

- follow-up to delta-io#4889 / delta-io#4887
- upstream: apache/datafusion#26528 (`try_cast` rejects struct casts
that `cast` accepts)

- DataFusion `try_cast` planning:
https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs

---------

Signed-off-by: ahlag <[email protected]>

(cherry picked from commit 0a14212)

Signed-off-by: R. Tyler Croy <[email protected]>
Co-Authored-By: R. Tyler Croy <[email protected]>
Co-Authored-By: Claude Haiku 5.5 <[email protected]>
rtyler pushed a commit that referenced this pull request Nov 7, 2026
Follow-up to #4889, from @ion-elgreco's
[comment](#4889 (comment)).

With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a
merge with `merge_schema=True` that widens a struct fails during
planning:

```
Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8)
```

DataFusion's `cast` checks struct casts by field name, but `try_cast`
still plans with arrow's `can_cast_types`, which needs the same fields
(`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion
`main`). The merged type is a superset of the target and source types,
so this PR uses a strict `cast` for changed fields, in the CASE branches
and in the CDF target preimage. This is the change Ion proposed.

Tests: the struct-widening merge tests now run with and without
`with_safe_cast(true)` (`rstest` `#[values(false, true)]`):
- `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows`
- `test_merge_with_schema_merge_struct_added_field_keeps_unset_column`
- `test_merge_with_schema_merge_and_pre_existing_struct_added_column`
(the source struct only has the new field)
- new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF
enabled, checks the preimage and postimage of the updated row.

The four `safe_cast = true` cases fail on `main` with the error above.

Testing (macOS arm64):
- `cargo test -p deltalake-core --features datafusion --lib`: 1362
passed
- `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features
datafusion --tests --no-deps` reports nothing in `merge/mod.rs`.
- Python bindings built from this branch: `tests/test_merge.py` 104
passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`,
a source struct with only the new field, and a struct with CDF enabled
all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake`
1.6.6, the first three drop the unmatched row's value to null and the
CDF case fails with the same error.

AI use: I used Claude Code to apply the change, write the tests, and run
the test suites. I reviewed the diff and checked the DataFusion planning
check and the test results myself.

- follow-up to #4889 / #4887
- upstream: apache/datafusion#26528 (`try_cast` rejects struct casts
that `cast` accepts)

- DataFusion `try_cast` planning:
https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs

---------

Signed-off-by: ahlag <[email protected]>

(cherry picked from commit 0a14212)

Signed-off-by: R. Tyler Croy <[email protected]>
Co-Authored-By: R. Tyler Croy <[email protected]>
Co-Authored-By: Claude Haiku 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

binding/python Issues for the Python package binding/rust Issues for the Rust crate

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants