Repository navigation
Commit 90540d3
authored
[fix](hive) invalidate stale file cache on partition refresh when partition cache misses (apache#65334)
Problem Summary:
Partition-level refresh / invalidation
(`HiveExternalMetaCache.invalidatePartitionCache`) returned early when
the target partition was absent from the partition metadata cache —
evicted under `max_hive_partition_cache_num` pressure, or simply never
loaded — and left that partition's file listing cache untouched.
The partition cache (`partition`) and the file listing cache (`file`)
are independent caches with independent size limits
(`max_hive_partition_cache_num` vs `max_external_file_cache_num`) and
evict separately, so the partition entry can be evicted while its file
listing survives. In that state a partition-level refresh cannot rebuild
the exact `FileCacheKey` (it needs the partition path / input format
that only the cached `HivePartition` carries), so the old code gave up
and the stale file listing remained. A later query would reload the
partition metadata and re-hit the same stale file listing — i.e. "the
partition exists, but Doris keeps seeing the old files/data".
Fix: when the exact `FileCacheKey` cannot be rebuilt, fall back to a
predicate-based invalidation on the file cache keyed by `table id +
partition values`, so the stale listing for that partition is cleared
without over-invalidating other partitions. This mirrors table-level
invalidation, which already invalidates the file cache by predicate.
The partition invalidation coordinator is also decoupled from
`ExternalTable` to `NameMapping` (mirroring the existing
`invalidateTableCache(NameMapping)`) so the path can be unit tested
directly.
This behavior has existed since partition-level invalidation was first
introduced in apache#15702 (the file cache was only invalidated inside the `if
(partition != null)` branch), and was later carried into the current
`HiveExternalMetaCache` by the apache#60937 cache-framework refactor.
### Release note
Fix a Hive external catalog consistency issue where a partition-level
refresh could leave a stale file listing cached when the partition
metadata cache had been evicted, causing queries to keep seeing outdated
files for that partition.1 parent bd10160 commit 90540d3
2 files changed
Lines changed: 85 additions & 18 deletions
File tree
- fe/fe-core/src
- main/java/org/apache/doris/datasource/hive
- test/java/org/apache/doris/datasource/hive
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
582 | 582 | | |
583 | 583 | | |
584 | 584 | | |
585 | | - | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
586 | 590 | | |
587 | 591 | | |
588 | 592 | | |
| |||
614 | 618 | | |
615 | 619 | | |
616 | 620 | | |
617 | | - | |
618 | | - | |
| 621 | + | |
619 | 622 | | |
620 | 623 | | |
621 | | - | |
622 | | - | |
623 | 624 | | |
624 | 625 | | |
625 | | - | |
| 626 | + | |
626 | 627 | | |
627 | 628 | | |
628 | 629 | | |
629 | 630 | | |
630 | | - | |
631 | | - | |
632 | | - | |
633 | | - | |
634 | | - | |
635 | | - | |
636 | | - | |
637 | | - | |
638 | | - | |
639 | | - | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
640 | 636 | | |
641 | 637 | | |
642 | 638 | | |
643 | 639 | | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
644 | 649 | | |
645 | 650 | | |
646 | 651 | | |
| |||
685 | 690 | | |
686 | 691 | | |
687 | 692 | | |
688 | | - | |
| 693 | + | |
689 | 694 | | |
690 | 695 | | |
691 | 696 | | |
| |||
772 | 777 | | |
773 | 778 | | |
774 | 779 | | |
775 | | - | |
| 780 | + | |
776 | 781 | | |
777 | 782 | | |
778 | 783 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
86 | 86 | | |
87 | 87 | | |
88 | 88 | | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
89 | 151 | | |
90 | 152 | | |
91 | 153 | | |
| |||
0 commit comments