Repository navigation
Commit b84a105
fix(core): allow safe_cast when merge widens a struct (delta-io#4917)
Follow-up to delta-io#4889, from @ion-elgreco's
[comment](delta-io#4889 (comment)).
With `safe_cast` enabled (`error_on_type_mismatch=False` in Python), a
merge with `merge_schema=True` that widens a struct fails during
planning:
```
Unsupported TRY_CAST from Struct("x": Int64) to Struct("x": Int64, "y": Utf8)
```
DataFusion's `cast` checks struct casts by field name, but `try_cast`
still plans with arrow's `can_cast_types`, which needs the same fields
(`datafusion-physical-expr` 55.2 `try_cast.rs`, also on DataFusion
`main`). The merged type is a superset of the target and source types,
so this PR uses a strict `cast` for changed fields, in the CASE branches
and in the CDF target preimage. This is the change Ion proposed.
Tests: the struct-widening merge tests now run with and without
`with_safe_cast(true)` (`rstest` `#[values(false, true)]`):
- `test_merge_with_schema_merge_struct_added_field_keeps_copied_rows`
- `test_merge_with_schema_merge_struct_added_field_keeps_unset_column`
- `test_merge_with_schema_merge_and_pre_existing_struct_added_column`
(the source struct only has the new field)
- new `test_merge_with_schema_merge_struct_added_field_cdf`: CDF
enabled, checks the preimage and postimage of the updated row.
The four `safe_cast = true` cases fail on `main` with the error above.
Testing (macOS arm64):
- `cargo test -p deltalake-core --features datafusion --lib`: 1362
passed
- `cargo fmt --check`: clean. `cargo clippy -p deltalake-core --features
datafusion --tests --no-deps` reports nothing in `merge/mod.rs`.
- Python bindings built from this branch: `tests/test_merge.py` 104
passed. With `error_on_type_mismatch=False`, a struct, a `list<struct>`,
a source struct with only the new field, and a struct with CDF enabled
all merge, and the unmatched row keeps `{x: 2, y: None}`. On `deltalake`
1.6.6, the first three drop the unmatched row's value to null and the
CDF case fails with the same error.
AI use: I used Claude Code to apply the change, write the tests, and run
the test suites. I reviewed the diff and checked the DataFusion planning
check and the test results myself.
- follow-up to delta-io#4889 / delta-io#4887
- upstream: apache/datafusion#26528 (`try_cast` rejects struct casts
that `cast` accepts)
- DataFusion `try_cast` planning:
https://github.com/apache/datafusion/blob/main/datafusion/physical-expr/src/expressions/try_cast.rs
---------
Signed-off-by: ahlag <[email protected]>
(cherry picked from commit 0a14212)
Co-Authored-By: Devstral 2 <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]>
Co-Authored-By: R. Tyler Croy <[email protected]>
Signed-off-by: R. Tyler Croy <[email protected]>1 parent 9e899c5 commit b84a105
2 files changed
Lines changed: 109 additions & 9 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1366 | 1366 | | |
1367 | 1367 | | |
1368 | 1368 | | |
| 1369 | + | |
| 1370 | + | |
| 1371 | + | |
1369 | 1372 | | |
1370 | | - | |
| 1373 | + | |
1371 | 1374 | | |
1372 | 1375 | | |
1373 | 1376 | | |
| |||
1398 | 1401 | | |
1399 | 1402 | | |
1400 | 1403 | | |
1401 | | - | |
| 1404 | + | |
1402 | 1405 | | |
1403 | 1406 | | |
1404 | 1407 | | |
| |||
2049 | 2052 | | |
2050 | 2053 | | |
2051 | 2054 | | |
| 2055 | + | |
2052 | 2056 | | |
2053 | 2057 | | |
2054 | 2058 | | |
| |||
4243 | 4247 | | |
4244 | 4248 | | |
4245 | 4249 | | |
| 4250 | + | |
4246 | 4251 | | |
4247 | | - | |
| 4252 | + | |
| 4253 | + | |
| 4254 | + | |
4248 | 4255 | | |
4249 | 4256 | | |
4250 | 4257 | | |
| |||
4310 | 4317 | | |
4311 | 4318 | | |
4312 | 4319 | | |
| 4320 | + | |
4313 | 4321 | | |
4314 | 4322 | | |
4315 | 4323 | | |
| |||
4340 | 4348 | | |
4341 | 4349 | | |
4342 | 4350 | | |
4343 | | - | |
| 4351 | + | |
| 4352 | + | |
| 4353 | + | |
| 4354 | + | |
| 4355 | + | |
4344 | 4356 | | |
4345 | 4357 | | |
4346 | 4358 | | |
| |||
4356 | 4368 | | |
4357 | 4369 | | |
4358 | 4370 | | |
4359 | | - | |
| 4371 | + | |
4360 | 4372 | | |
4361 | 4373 | | |
4362 | 4374 | | |
| |||
4396 | 4408 | | |
4397 | 4409 | | |
4398 | 4410 | | |
| 4411 | + | |
4399 | 4412 | | |
4400 | 4413 | | |
4401 | 4414 | | |
| |||
4418 | 4431 | | |
4419 | 4432 | | |
4420 | 4433 | | |
| 4434 | + | |
4421 | 4435 | | |
4422 | | - | |
4423 | | - | |
| 4436 | + | |
| 4437 | + | |
| 4438 | + | |
| 4439 | + | |
4424 | 4440 | | |
4425 | 4441 | | |
4426 | 4442 | | |
| |||
4434 | 4450 | | |
4435 | 4451 | | |
4436 | 4452 | | |
| 4453 | + | |
4437 | 4454 | | |
4438 | | - | |
4439 | | - | |
| 4455 | + | |
| 4456 | + | |
| 4457 | + | |
| 4458 | + | |
4440 | 4459 | | |
4441 | 4460 | | |
4442 | 4461 | | |
| |||
4450 | 4469 | | |
4451 | 4470 | | |
4452 | 4471 | | |
| 4472 | + | |
| 4473 | + | |
| 4474 | + | |
| 4475 | + | |
| 4476 | + | |
| 4477 | + | |
| 4478 | + | |
| 4479 | + | |
| 4480 | + | |
| 4481 | + | |
| 4482 | + | |
| 4483 | + | |
| 4484 | + | |
| 4485 | + | |
| 4486 | + | |
| 4487 | + | |
| 4488 | + | |
| 4489 | + | |
| 4490 | + | |
| 4491 | + | |
| 4492 | + | |
| 4493 | + | |
| 4494 | + | |
| 4495 | + | |
| 4496 | + | |
| 4497 | + | |
| 4498 | + | |
| 4499 | + | |
| 4500 | + | |
| 4501 | + | |
| 4502 | + | |
| 4503 | + | |
| 4504 | + | |
| 4505 | + | |
| 4506 | + | |
| 4507 | + | |
4453 | 4508 | | |
4454 | 4509 | | |
4455 | 4510 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
3320 | 3320 | | |
3321 | 3321 | | |
3322 | 3322 | | |
| 3323 | + | |
| 3324 | + | |
| 3325 | + | |
| 3326 | + | |
| 3327 | + | |
| 3328 | + | |
| 3329 | + | |
| 3330 | + | |
| 3331 | + | |
| 3332 | + | |
| 3333 | + | |
| 3334 | + | |
| 3335 | + | |
| 3336 | + | |
| 3337 | + | |
| 3338 | + | |
| 3339 | + | |
| 3340 | + | |
| 3341 | + | |
| 3342 | + | |
| 3343 | + | |
| 3344 | + | |
| 3345 | + | |
| 3346 | + | |
| 3347 | + | |
| 3348 | + | |
| 3349 | + | |
| 3350 | + | |
| 3351 | + | |
| 3352 | + | |
| 3353 | + | |
| 3354 | + | |
| 3355 | + | |
| 3356 | + | |
| 3357 | + | |
| 3358 | + | |
| 3359 | + | |
| 3360 | + | |
| 3361 | + | |
| 3362 | + | |
| 3363 | + | |
| 3364 | + | |
| 3365 | + | |
| 3366 | + | |
| 3367 | + | |
3323 | 3368 | | |
3324 | 3369 | | |
3325 | 3370 | | |
| |||
0 commit comments