Repository navigation
Commit 9ebb273
Spark: Auto-set default name mapping in SparkTableUtil.importSparkTable
After restoring `NameMapping.empty()` as the safe default in `Parquet.ReadBuilder`
(commit 8881516), Spark tests that import external Parquet files via
`SparkTableUtil.importSparkTable` and then read them through Iceberg fail
field resolution: the imported files have no Iceberg field IDs and no
`schema.name-mapping.default` property is auto-set, so the strict empty
mapping returns null for every projected field.
Mirror the established pattern from `AddFilesProcedure.ensureNameMappingPresent`
and `BaseTableCreationSparkAction`: when the target Iceberg table does not
already have `schema.name-mapping.default` set, auto-derive a name mapping
from the target schema before importing. Subsequent reads via Iceberg pick
up the property automatically through `BaseReader`.
Applied identically to v3.5, v4.0, and v4.1.
This honors the deprecation guidance Russell Spitzer documented when
deprecating the NETFLIX_UNSAFE positional fallback ("Use name mapping
instead") and brings `importSparkTable` in line with `AddFilesProcedure`,
which has been doing this for some time.
Verified on Spark 4.1 via the full `org.apache.iceberg.spark.source.*`
test package (`-DtestParallelism=8`); previously-failing
`TestIcebergSourceHadoopTables.testTableWithInt96Timestamp` and
`TestIdentityPartitionData.testProjections` now pass with no
collateral failures elsewhere.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>1 parent 8da5062 commit 9ebb273
3 files changed
Lines changed: 48 additions & 0 deletions
File tree
- spark
- v3.5/spark/src/main/java/org/apache/iceberg/spark
- v4.0/spark/src/main/java/org/apache/iceberg/spark
- v4.1/spark/src/main/java/org/apache/iceberg/spark
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
70 | 71 | | |
71 | 72 | | |
72 | 73 | | |
| |||
582 | 583 | | |
583 | 584 | | |
584 | 585 | | |
| 586 | + | |
| 587 | + | |
585 | 588 | | |
586 | 589 | | |
587 | 590 | | |
| |||
659 | 662 | | |
660 | 663 | | |
661 | 664 | | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
662 | 678 | | |
663 | 679 | | |
664 | 680 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
67 | 67 | | |
68 | 68 | | |
69 | 69 | | |
| 70 | + | |
70 | 71 | | |
71 | 72 | | |
72 | 73 | | |
| |||
379 | 380 | | |
380 | 381 | | |
381 | 382 | | |
| 383 | + | |
| 384 | + | |
382 | 385 | | |
383 | 386 | | |
384 | 387 | | |
| |||
428 | 431 | | |
429 | 432 | | |
430 | 433 | | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
431 | 447 | | |
432 | 448 | | |
433 | 449 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| 69 | + | |
69 | 70 | | |
70 | 71 | | |
71 | 72 | | |
| |||
378 | 379 | | |
379 | 380 | | |
380 | 381 | | |
| 382 | + | |
| 383 | + | |
381 | 384 | | |
382 | 385 | | |
383 | 386 | | |
| |||
427 | 430 | | |
428 | 431 | | |
429 | 432 | | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
430 | 446 | | |
431 | 447 | | |
432 | 448 | | |
| |||
0 commit comments