Repository navigation
Commit 0944b63
committed
[perf](catalog) two-level cross-query cache for external partition derived views (#65829)
### What & why
The catalog-SPI cutover routes every external partition lookup through
the
connector SPI per query, dropping the cross-query partition caching the
legacy
fe-core path (CACHE-P1) provided: a repeated query on a partitioned
external
table re-enumerated the remote partition set **and** rebuilt the derived
partition view / `SortedPartitionRanges` from scratch every time. This
PR
restores that caching for the iceberg, paimon and hive connectors
without
breaking the one-way SPI isolation (`fe-connector → fe-core`) and
without
re-introducing the #65659 per-statement partition TOCTOU.
It uses two independent layers, both consulted only at the frozen
per-statement prune, so neither can serve data inconsistent with the
pinned
snapshot:
**Cache A — connector derived-partition-view cache (below the SPI
line).**
A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by
`(db, table, snapshotId, schemaId)`, modeled on the existing
`IcebergPartitionCache` pattern (contextual-only, manual-miss-load,
`CacheSpec`-driven config). It caches the *derived* view (display-name
rendering, null-sentinel normalization, `ConnectorPartitionInfo`
construction) — so a repeat query skips the BUILD, not only the
round-trip.
Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg
`session=user`
catalogs null the cache to keep the per-user authorization isolation
from
#65785; paimon and hive have no per-query session identity and build it
unconditionally.
**Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC
tables now
implement `SupportBinarySearchFilteringPartitions`, routing through the
same
`NereidsSortedPartitionsCacheManager` native tables use. The version
token is
the **frozen partition name-set** read from the same pinned snapshot the
scan
reads, so a cache hit can only return ranges built from the exact
partition
set the current statement prunes against — uniform across
snapshot-bearing
(iceberg/paimon) and snapshot-less (hive) connectors, with no generation
counter and no SPI change. `PruneFileScanPartition` consults it strictly
as a
fallback between the frozen `scan.getSelectedPartitions()` field and the
build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation
is
routed from
`ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so
REFRESH and metastore events drop both layers.
### How verified
Per-connector cache tests (iceberg / paimon / hive) assert a same-key
repeat
enumerates the remote seam exactly once, a distinct snapshot / name-set
re-enumerates, and REFRESH invalidation re-enumerates. fe-core
`NereidsSortedPartitionsCacheManagerExternalTest` and
`BinarySearchPartitionInconsistencyTest` prove a cache hit returns the
same
ranges, rebuilds on a partition-set change, and never breaks the #65659
within-query consistency invariant. The iceberg / paimon / hive
connector
suites and the fe-core partition-cache suites are green.
Part of the catalog-SPI migration tracked in #65185.1 parent 7ac3ca7 commit 0944b63
26 files changed
Lines changed: 3205 additions & 28 deletions
File tree
- fe
- fe-connector
- fe-connector-cache/src
- main/java/org/apache/doris/connector/cache
- test/java/org/apache/doris/connector/cache
- fe-connector-hive/src
- main/java/org/apache/doris/connector/hive
- test/java/org/apache/doris/connector/hive
- fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
- fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-core/src
- main/java/org/apache/doris
- datasource
- mvcc
- nereids/rules/rewrite
- test/java/org/apache/doris
- common/cache
- nereids/trees/plans/physical
- plan-doc
- designs
- plans
- regression-test/suites/external_table_p0/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
9 | 15 | | |
10 | 16 | | |
11 | 17 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
0 commit comments