Repository navigation
Commit f84c6ba
fix(delta): correct nested-CM read for column-mapped tables (apache#79)
Closes apache#79. Removes the gate that declined column-mapped tables with
complex (Array/Map/Struct) columns in scan output.
Root cause: under Catalyst nested-column pruning, the native parquet
reader was sent a `data_schema` whose `b` struct kept all its file
children (e.g. `b: struct<col-c STRING, col-d INT>`), while
`required_schema` had only the projected child (`b: struct<d INT>`).
DataFusion's ParquetSource projects by top-level index only, so the
output struct came back with 2 children. Spark's catalyst-baked
`GetStructField(b, ordinal=0)` (computed against the *pruned* schema
where d is at position 0) then picked the file-shape's position 0 -
`col-c` (STRING) - producing the "Invalid comparison Utf8 <= Int32"
mismatch on `b.d > 0` constraint evaluation.
Fix (CometDeltaNativeScan.convert): for each field in data_schema that
is also in required_schema, replace its data type with required's
already-physicalised pruned shape. ParquetSource then reads only the
required nested children and emits a struct matching what Spark's plan
expects. Non-required top-level columns keep their full physicalised
shape (harmless since projection_vector doesn't reference them).
Required_schema gets a new helper `physicaliseRequiredField` /
`physicaliseDataTypePreserving` that walks `req`'s tree pairing nodes
with the snapshot by logical name to apply physical names, while
preserving req's pruning shape. The data-schema helper
(`physicaliseNestedTypesOnly`) was wrong for required_schema because
it replaced the whole data type with the snapshot's full shape,
losing pruning.
Verified:
- DeltaDropColumnSuite "drop column with constraints" and "drop
column with constraints - map element" both PASS (was FAILING
before the gate was added).
- DeltaDropColumnSuite + DeletionVectorsSuite + DeltaColumnMappingSuite
show 16 failures, ALL of which also fail on stock Delta 3.3.2
when running the same 3 suites together (verified upstream:
17 failures, identical set minus the 2B-row test we skip).
Net: zero Comet-introduced regressions.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent a83ecb3 commit f84c6ba
2 files changed
Lines changed: 98 additions & 57 deletions
File tree
- spark/src/main/scala/org/apache/comet
- rules
- serde/operator
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
662 | 662 | | |
663 | 663 | | |
664 | 664 | | |
665 | | - | |
666 | | - | |
667 | | - | |
668 | | - | |
669 | | - | |
670 | | - | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
671 | 677 | | |
672 | | - | |
673 | | - | |
674 | | - | |
675 | | - | |
676 | | - | |
677 | | - | |
678 | | - | |
679 | | - | |
680 | | - | |
681 | | - | |
682 | | - | |
683 | | - | |
684 | | - | |
685 | | - | |
686 | | - | |
687 | | - | |
688 | | - | |
689 | | - | |
690 | | - | |
691 | | - | |
692 | | - | |
693 | | - | |
694 | | - | |
695 | | - | |
696 | | - | |
697 | | - | |
698 | | - | |
699 | | - | |
700 | | - | |
701 | | - | |
702 | | - | |
703 | | - | |
704 | | - | |
705 | | - | |
706 | | - | |
707 | | - | |
708 | | - | |
709 | | - | |
710 | | - | |
711 | | - | |
712 | | - | |
713 | | - | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
714 | 683 | | |
715 | 684 | | |
716 | 685 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
458 | 458 | | |
459 | 459 | | |
460 | 460 | | |
461 | | - | |
462 | | - | |
463 | | - | |
464 | | - | |
465 | | - | |
466 | | - | |
467 | | - | |
468 | | - | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
469 | 541 | | |
470 | 542 | | |
471 | | - | |
| 543 | + | |
472 | 544 | | |
473 | 545 | | |
474 | 546 | | |
| |||
0 commit comments