Repository navigation
Commit 407bef6
authored
fix: read Hive's null partition placeholder as NULL (apache#26310)
## Which issue does this PR close?
- working towards apache#18083
## Rationale for this change
Hive compliant writers put the rows whose partition value is null into a
directory named `__HIVE_DEFAULT_PARTITION__` (Hive's
`hive.exec.default.partition.name`). `ListingTable` casts that directory
name straight to the partition column's type, so a table with a null
partition cannot be queried at all unless the partition column happens
to be a string:
```sql
> create external table t (id int, d date) stored as parquet partitioned by (d) location 'data/';
> select * from t;
Arrow error: Cast error: Cannot cast string '__HIVE_DEFAULT_PARTITION__' to value of Date32 type
```
For a string partition column the query runs, but the value comes back
as the literal `__HIVE_DEFAULT_PARTITION__` rather than NULL, and `WHERE
d IS NULL` never matches it because partition columns are declared
non-nullable. Hive-written tables are a common input, and both Spark and
Impala read these values as NULL.
We are not touching writes in this PR - just reads.
We also haven’t fixed the /date=/a.parquet problem here either.
## What changes are included in this PR?
1. A partition value equal to the placeholder is read as a typed NULL
rather than parsed
(`helpers::try_into_partitioned_file`).
2. `ListingTable` declares partition columns nullable, in both the table
schema and the file
source's `TableSchema`, so the NULL can be represented and `IS NULL`
selects the null partition.
3. A new option,
`datafusion.execution.listing_table_hive_default_partition_name`,
defaulting to
`__HIVE_DEFAULT_PARTITION__`, so a table written with a different
`hive.exec.default.partition.name` can be read. The empty name turns the
mapping off and reads the directory name as the value. The default is
also exposed as`datafusion_common::config::HIVE_DEFAULT_PARTITION_NAME`.
4. `filter_partitioned_file` no longer treats a NULL predicate result as
a match. That is reachable now that a partition value can be NULL
5. Claude caught this one not me: In the generated `configs.md`,
defaults containing `_` or `*` are shown as code. Markdown would
otherwise read `__HIVE_DEFAULT_PARTITION__` as emphasis and render it as
bold text; this also backticks the two existing such defaults,
`nulls_max` and `start_offset`.
## What is the testing strategy for this PR?
Unit tests and slt tests added.
## Are there any user-facing changes?
partition columns are now forced to be nullable
listing_table_hive_default_partition_name added1 parent b6b2e0e commit 407bef6
13 files changed
Lines changed: 673 additions & 45 deletions
File tree
- datafusion
- catalog-listing/src
- common/src
- core/tests/catalog_listing
- datasource/src
- write
- sqllogictest/test_files
- docs/source
- library-user-guide/upgrading
- user-guide
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
19 | | - | |
| 19 | + | |
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| |||
324 | 324 | | |
325 | 325 | | |
326 | 326 | | |
327 | | - | |
| 327 | + | |
| 328 | + | |
328 | 329 | | |
329 | | - | |
| 330 | + | |
| 331 | + | |
330 | 332 | | |
331 | 333 | | |
332 | 334 | | |
333 | 335 | | |
334 | 336 | | |
335 | 337 | | |
336 | 338 | | |
337 | | - | |
338 | | - | |
339 | | - | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
340 | 346 | | |
341 | 347 | | |
342 | 348 | | |
343 | 349 | | |
| 350 | + | |
344 | 351 | | |
345 | 352 | | |
346 | 353 | | |
| |||
354 | 361 | | |
355 | 362 | | |
356 | 363 | | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
357 | 367 | | |
358 | 368 | | |
359 | 369 | | |
| |||
413 | 423 | | |
414 | 424 | | |
415 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
416 | 432 | | |
417 | | - | |
| 433 | + | |
418 | 434 | | |
419 | 435 | | |
420 | 436 | | |
421 | 437 | | |
| 438 | + | |
422 | 439 | | |
423 | 440 | | |
424 | 441 | | |
| |||
495 | 512 | | |
496 | 513 | | |
497 | 514 | | |
| 515 | + | |
498 | 516 | | |
499 | 517 | | |
500 | 518 | | |
| |||
659 | 677 | | |
660 | 678 | | |
661 | 679 | | |
662 | | - | |
663 | | - | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
664 | 687 | | |
665 | 688 | | |
666 | 689 | | |
| |||
685 | 708 | | |
686 | 709 | | |
687 | 710 | | |
688 | | - | |
689 | | - | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
690 | 718 | | |
691 | 719 | | |
692 | 720 | | |
| |||
696 | 724 | | |
697 | 725 | | |
698 | 726 | | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
699 | 781 | | |
700 | 782 | | |
701 | 783 | | |
| |||
711 | 793 | | |
712 | 794 | | |
713 | 795 | | |
714 | | - | |
715 | | - | |
| 796 | + | |
| 797 | + | |
| 798 | + | |
| 799 | + | |
| 800 | + | |
| 801 | + | |
| 802 | + | |
716 | 803 | | |
717 | 804 | | |
718 | 805 | | |
| |||
732 | 819 | | |
733 | 820 | | |
734 | 821 | | |
735 | | - | |
736 | | - | |
| 822 | + | |
| 823 | + | |
| 824 | + | |
| 825 | + | |
| 826 | + | |
| 827 | + | |
| 828 | + | |
737 | 829 | | |
738 | 830 | | |
739 | 831 | | |
| |||
755 | 847 | | |
756 | 848 | | |
757 | 849 | | |
758 | | - | |
759 | | - | |
| 850 | + | |
| 851 | + | |
| 852 | + | |
| 853 | + | |
| 854 | + | |
| 855 | + | |
| 856 | + | |
760 | 857 | | |
761 | 858 | | |
762 | 859 | | |
| |||
786 | 883 | | |
787 | 884 | | |
788 | 885 | | |
789 | | - | |
790 | | - | |
| 886 | + | |
| 887 | + | |
| 888 | + | |
| 889 | + | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
791 | 893 | | |
792 | 894 | | |
793 | 895 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
241 | 241 | | |
242 | 242 | | |
243 | 243 | | |
244 | | - | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
245 | 247 | | |
246 | 248 | | |
247 | | - | |
| 249 | + | |
248 | 250 | | |
249 | 251 | | |
250 | 252 | | |
| |||
360 | 362 | | |
361 | 363 | | |
362 | 364 | | |
363 | | - | |
| 365 | + | |
| 366 | + | |
364 | 367 | | |
365 | 368 | | |
366 | 369 | | |
| |||
1120 | 1123 | | |
1121 | 1124 | | |
1122 | 1125 | | |
1123 | | - | |
| 1126 | + | |
1124 | 1127 | | |
1125 | 1128 | | |
1126 | 1129 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
886 | 887 | | |
887 | 888 | | |
888 | 889 | | |
| 890 | + | |
| 891 | + | |
| 892 | + | |
| 893 | + | |
889 | 894 | | |
890 | 895 | | |
891 | 896 | | |
| |||
1095 | 1100 | | |
1096 | 1101 | | |
1097 | 1102 | | |
| 1103 | + | |
| 1104 | + | |
| 1105 | + | |
| 1106 | + | |
| 1107 | + | |
| 1108 | + | |
| 1109 | + | |
| 1110 | + | |
1098 | 1111 | | |
1099 | 1112 | | |
1100 | 1113 | | |
| |||
2526 | 2539 | | |
2527 | 2540 | | |
2528 | 2541 | | |
2529 | | - | |
| 2542 | + | |
2530 | 2543 | | |
2531 | 2544 | | |
2532 | 2545 | | |
2533 | 2546 | | |
2534 | 2547 | | |
2535 | 2548 | | |
2536 | 2549 | | |
| 2550 | + | |
| 2551 | + | |
| 2552 | + | |
| 2553 | + | |
| 2554 | + | |
| 2555 | + | |
| 2556 | + | |
| 2557 | + | |
| 2558 | + | |
| 2559 | + | |
| 2560 | + | |
| 2561 | + | |
2537 | 2562 | | |
2538 | 2563 | | |
2539 | 2564 | | |
| |||
0 commit comments