Repository navigation
Commit 0a14212
authored
fix(core): allow safe_cast when merge widens a struct (#4917)
# Description
Follow-up to #4889, from @ion-elgreco's
[comment](#4889 (comment)).
With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a
merge with `merge_schema=True` that widens a struct fails during
planning:
```
Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8)
```
DataFusion's `cast` checks struct casts by field name, but `try_cast`
still plans with arrow's `can_cast_types`, which needs the same fields
(`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion
`main`). The merged type is a superset of the target and source types,
so this PR uses a strict `cast` for changed fields, in the CASE branches
and in the CDF target preimage. This is the change Ion proposed.
Tests: the struct-widening merge tests now run with and without
`with_safe_cast(true)` (`rstest` `#[values(false, true)]`):
- `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows`
- `test_merge_with_schema_merge_struct_added_field_keeps_unset_column`
- `test_merge_with_schema_merge_and_pre_existing_struct_added_column`
(the source struct only has the new field)
- new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF
enabled, checks the preimage and postimage of the updated row.
The four `safe_cast = true` cases fail on `main` with the error above.
Testing (macOS arm64):
- `cargo test -p deltalake-core --features datafusion --lib`: 1362
passed
- `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features
datafusion --tests --no-deps` reports nothing in `merge/mod.rs`.
- Python bindings built from this branch: `tests/test_merge.py` 104
passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`,
a source struct with only the new field, and a struct with CDF enabled
all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake`
1.6.6, the first three drop the unmatched row's value to null and the
CDF case fails with the same error.
AI use: I used Claude Code to apply the change, write the tests, and run
the test suites. I reviewed the diff and checked the DataFusion planning
check and the test results myself.
# Related Issue(s)
- follow-up to #4889 / #4887
- upstream: apache/datafusion#26528 (`try_cast` rejects struct casts
that `cast` accepts)
# Documentation
- DataFusion `try_cast` planning:
https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs
---------
Signed-off-by: ahlag <[email protected]>1 parent 10a7c47 commit 0a14212
2 files changed
Lines changed: 109 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1366 | 1366 | | |
1367 | 1367 | | |
1368 | 1368 | | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
1369 | 1372 | | |
1370 | | - | |
| 1373 | + | |
1371 | 1374 | | |
1372 | 1375 | | |
1373 | 1376 | | |
| |||
1398 | 1401 | | |
1399 | 1402 | | |
1400 | 1403 | | |
1401 | | - | |
| 1404 | + | |
1402 | 1405 | | |
1403 | 1406 | | |
1404 | 1407 | | |
| |||
2045 | 2048 | | |
2046 | 2049 | | |
2047 | 2050 | | |
| 2051 | + | |
2048 | 2052 | | |
2049 | 2053 | | |
2050 | 2054 | | |
| |||
4239 | 4243 | | |
4240 | 4244 | | |
4241 | 4245 | | |
| 4246 | + | |
4242 | 4247 | | |
4243 | | - | |
| 4248 | + | |
| 4249 | + | |
| 4250 | + | |
4244 | 4251 | | |
4245 | 4252 | | |
4246 | 4253 | | |
| |||
4306 | 4313 | | |
4307 | 4314 | | |
4308 | 4315 | | |
| 4316 | + | |
4309 | 4317 | | |
4310 | 4318 | | |
4311 | 4319 | | |
| |||
4336 | 4344 | | |
4337 | 4345 | | |
4338 | 4346 | | |
4339 | | - | |
| 4347 | + | |
| 4348 | + | |
| 4349 | + | |
| 4350 | + | |
| 4351 | + | |
4340 | 4352 | | |
4341 | 4353 | | |
4342 | 4354 | | |
| |||
4352 | 4364 | | |
4353 | 4365 | | |
4354 | 4366 | | |
4355 | | - | |
| 4367 | + | |
4356 | 4368 | | |
4357 | 4369 | | |
4358 | 4370 | | |
| |||
4392 | 4404 | | |
4393 | 4405 | | |
4394 | 4406 | | |
| 4407 | + | |
4395 | 4408 | | |
4396 | 4409 | | |
4397 | 4410 | | |
| |||
4414 | 4427 | | |
4415 | 4428 | | |
4416 | 4429 | | |
| 4430 | + | |
4417 | 4431 | | |
4418 | | - | |
4419 | | - | |
| 4432 | + | |
| 4433 | + | |
| 4434 | + | |
| 4435 | + | |
4420 | 4436 | | |
4421 | 4437 | | |
4422 | 4438 | | |
| |||
4430 | 4446 | | |
4431 | 4447 | | |
4432 | 4448 | | |
| 4449 | + | |
4433 | 4450 | | |
4434 | | - | |
4435 | | - | |
| 4451 | + | |
| 4452 | + | |
| 4453 | + | |
| 4454 | + | |
4436 | 4455 | | |
4437 | 4456 | | |
4438 | 4457 | | |
| |||
4446 | 4465 | | |
4447 | 4466 | | |
4448 | 4467 | | |
| 4468 | + | |
| 4469 | + | |
| 4470 | + | |
| 4471 | + | |
| 4472 | + | |
| 4473 | + | |
| 4474 | + | |
| 4475 | + | |
| 4476 | + | |
| 4477 | + | |
| 4478 | + | |
| 4479 | + | |
| 4480 | + | |
| 4481 | + | |
| 4482 | + | |
| 4483 | + | |
| 4484 | + | |
| 4485 | + | |
| 4486 | + | |
| 4487 | + | |
| 4488 | + | |
| 4489 | + | |
| 4490 | + | |
| 4491 | + | |
| 4492 | + | |
| 4493 | + | |
| 4494 | + | |
| 4495 | + | |
| 4496 | + | |
| 4497 | + | |
| 4498 | + | |
| 4499 | + | |
| 4500 | + | |
| 4501 | + | |
| 4502 | + | |
| 4503 | + | |
4449 | 4504 | | |
4450 | 4505 | | |
4451 | 4506 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3320 | 3320 | | |
3321 | 3321 | | |
3322 | 3322 | | |
| 3323 | + | |
| 3324 | + | |
| 3325 | + | |
| 3326 | + | |
| 3327 | + | |
| 3328 | + | |
| 3329 | + | |
| 3330 | + | |
| 3331 | + | |
| 3332 | + | |
| 3333 | + | |
| 3334 | + | |
| 3335 | + | |
| 3336 | + | |
| 3337 | + | |
| 3338 | + | |
| 3339 | + | |
| 3340 | + | |
| 3341 | + | |
| 3342 | + | |
| 3343 | + | |
| 3344 | + | |
| 3345 | + | |
| 3346 | + | |
| 3347 | + | |
| 3348 | + | |
| 3349 | + | |
| 3350 | + | |
| 3351 | + | |
| 3352 | + | |
| 3353 | + | |
| 3354 | + | |
| 3355 | + | |
| 3356 | + | |
| 3357 | + | |
| 3358 | + | |
| 3359 | + | |
| 3360 | + | |
| 3361 | + | |
| 3362 | + | |
| 3363 | + | |
| 3364 | + | |
| 3365 | + | |
| 3366 | + | |
| 3367 | + | |
3323 | 3368 | | |
3324 | 3369 | | |
3325 | 3370 | | |
| |||
0 commit comments