Repository navigation
Commit 518d059
[perf](catalog) fe-connector-iceberg: cache the partition-view scan across queries (PERF-02)
For a partitioned table, every query pays an analysis-phase PARTITIONS
metadata-table scan (IcebergPartitionUtils.loadRawPartitions ->
partitionsTable.newScan().useSnapshot(id).planFiles()+rows()), which the iceberg
SDK materializes by reading EVERY data+delete manifest of the snapshot
(O(#manifests) remote IO, independent of query selectivity). It ran once per
query (memoized only within the statement) and 4~6 times per MTMV refresh, with
no cross-query cache -- the partition-info half of the legacy
IcebergExternalMetaCache that the SPI cutover dropped. PERF-01 already deduped
the loadTable in this cluster; this fixes the remaining scan.
Add a connector-side IcebergPartitionCache keyed by (TableIdentifier,
snapshotId), value = the raw partition list. The three consumers --
buildMvccPartitionView (MVCC/MTMV view), listPartitions (selectedPartitionNum),
listPartitionNames (SHOW PARTITIONS) -- all funnel through loadRawPartitions, so
they share one scan per (table, snapshot). The snapshot id resolved inside those
methods is held stable across queries (and across one refresh's enumeration
points) by IcebergLatestSnapshotCache, so the key is stable and the MTMV 4~6x
re-enumeration collapses onto one scan too.
A snapshot is immutable, so the cached value is a pure function of the key
(always correct; a new commit -> new snapshot id -> new key -> a live scan). The
value is pure metadata with NO FileIO/credential, so unlike the PERF-01 table
cache it needs no credential gate and is built for every catalog (only the
meta.cache.iceberg.table.ttl-second knob disables it). The cached list is
unmodifiable; the loader's exception propagates verbatim so listPartitions keeps
its dropped-partition-source-column degradation, and a failed scan is not cached.
REFRESH TABLE/DB/CATALOG invalidate it alongside the other caches.
The MTMV refresh-level MvccSnapshot pin the audit also suggested is intentionally
NOT done: IcebergLatestSnapshotCache already stabilizes the snapshot across a
refresh (ttl>0), so this cache collapses the repeats without a fe-core change
(the only residual is the ttl=0 no-cache catalog's pre-existing enumeration skew,
which is out of scope for a connector-side perf fix).
Metric gate: a real partitioned table + a real cache; repeated
buildMvccPartitionView plus listPartitions at one snapshot run the PARTITIONS
scan exactly once (cache.loadCountForTest()==1, was per-call). Full iceberg module
suite: 943 pass / 0 fail / 1 skip; checkstyle clean. Behavior parity: cached
enumeration equals a live one; snapshot/schema visibility unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG31 parent 65cc6d4 commit 518d059
7 files changed
Lines changed: 493 additions & 11 deletions
File tree
- fe/fe-connector/fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
164 | 164 | | |
165 | 165 | | |
166 | 166 | | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
167 | 171 | | |
168 | 172 | | |
169 | 173 | | |
| |||
205 | 209 | | |
206 | 210 | | |
207 | 211 | | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
208 | 215 | | |
209 | 216 | | |
210 | 217 | | |
| |||
229 | 236 | | |
230 | 237 | | |
231 | 238 | | |
232 | | - | |
| 239 | + | |
233 | 240 | | |
234 | 241 | | |
235 | 242 | | |
| |||
543 | 550 | | |
544 | 551 | | |
545 | 552 | | |
| 553 | + | |
546 | 554 | | |
547 | 555 | | |
548 | 556 | | |
| |||
562 | 570 | | |
563 | 571 | | |
564 | 572 | | |
| 573 | + | |
565 | 574 | | |
566 | 575 | | |
567 | 576 | | |
| |||
577 | 586 | | |
578 | 587 | | |
579 | 588 | | |
| 589 | + | |
580 | 590 | | |
581 | 591 | | |
582 | 592 | | |
| |||
611 | 621 | | |
612 | 622 | | |
613 | 623 | | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
614 | 629 | | |
615 | 630 | | |
616 | 631 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
148 | 148 | | |
149 | 149 | | |
150 | 150 | | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
151 | 154 | | |
152 | 155 | | |
153 | 156 | | |
154 | | - | |
| 157 | + | |
155 | 158 | | |
156 | 159 | | |
157 | 160 | | |
158 | 161 | | |
159 | 162 | | |
160 | | - | |
| 163 | + | |
161 | 164 | | |
162 | 165 | | |
| 166 | + | |
163 | 167 | | |
164 | 168 | | |
165 | 169 | | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
166 | 176 | | |
167 | 177 | | |
168 | 178 | | |
169 | 179 | | |
170 | 180 | | |
| 181 | + | |
171 | 182 | | |
172 | 183 | | |
173 | 184 | | |
| |||
1490 | 1501 | | |
1491 | 1502 | | |
1492 | 1503 | | |
1493 | | - | |
| 1504 | + | |
| 1505 | + | |
1494 | 1506 | | |
1495 | 1507 | | |
1496 | 1508 | | |
| |||
1518 | 1530 | | |
1519 | 1531 | | |
1520 | 1532 | | |
1521 | | - | |
| 1533 | + | |
| 1534 | + | |
1522 | 1535 | | |
1523 | 1536 | | |
1524 | 1537 | | |
| |||
1549 | 1562 | | |
1550 | 1563 | | |
1551 | 1564 | | |
1552 | | - | |
| 1565 | + | |
| 1566 | + | |
1553 | 1567 | | |
1554 | 1568 | | |
1555 | 1569 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
0 commit comments