Repository navigation
Commit 484f0e0
[perf](catalog) fe-connector-iceberg: memoize the resolved Table within and across queries (PERF-01)
A plain WHERE-filtered iceberg SELECT re-loaded the same table 3~7 times per
planning pass: getColumnHandles, getTableStatistics, the scan provider's
resolveTable (getScanNodeProperties / planScan / streamingSplitEstimate) and the
analysis-phase getMvccPartitionView / listPartitions each issued their own remote
loadTable (a metastore RPC + a metadata.json read). The SPI cutover split the
legacy IcebergExternalMetaCache and kept only the latest-snapshot pin
(IcebergLatestSnapshotCache), dropping the cached-Table half.
Restore it in two connector-side layers (fe-core untouched):
- Query-scoped fat handle: IcebergTableHandle carries a transient raw Table memo
(not serialized, not part of identity), carried forward by withSnapshot /
withRewriteFileScope / withTopnLazyMaterialize. Reads resolve the table once and
reuse the same instance. PluginDrivenScanNode.currentHandle is per-query, so the
memo is GC'd with the plan (zero per-query accumulation) => exactly one remote
loadTable per table per planning pass.
- Cross-query IcebergTableCache (new; mirrors IcebergLatestSnapshotCache, value =
raw Table): 24h TTL, REFRESH-invalidated, backs the fat handle's first miss and
restores legacy cross-query caching. Disabled (null) for iceberg.rest.session=user
and REST vended-credentials, where a cached raw Table's FileIO carries a per-user
/ soon-expiring token (cross-user leak / 403 mid-scan); the fat handle stays on
there (token fresh within one query).
Reads go through resolveTableForRead (fat handle -> cache -> raw loadTable), which
throws NoSuchTableException unwrapped so listPartitions / getMvccPartitionView keep
their concurrent-drop degradation. The scan provider re-applies wrapTableForScan
(Kerberos FileIO) per call, never freezing it into the memo. DDL / write / procedure
/ sys-table paths bypass the helper (fresh remote base). Part B (convertPredicate
narrowing) is intentionally dropped (red-team proved it a no-op).
Metric gate: RecordingIcebergCatalogOps asserts one planning pass
(getColumnHandles + planScan on a single handle) issues exactly one remote
loadTable (was two). Full iceberg module suite: 932 pass / 0 fail / 1 skip;
checkstyle clean. Behavior parity: pure de-duplication -- snapshot semantics,
schema visibility and time travel are unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG31 parent 9a2c39c commit 484f0e0
9 files changed
Lines changed: 633 additions & 26 deletions
File tree
- fe/fe-connector/fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
159 | 159 | | |
160 | 160 | | |
161 | 161 | | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
162 | 167 | | |
163 | 168 | | |
164 | 169 | | |
| |||
187 | 192 | | |
188 | 193 | | |
189 | 194 | | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
190 | 208 | | |
191 | 209 | | |
192 | 210 | | |
| |||
211 | 229 | | |
212 | 230 | | |
213 | 231 | | |
214 | | - | |
| 232 | + | |
215 | 233 | | |
216 | 234 | | |
217 | 235 | | |
| |||
522 | 540 | | |
523 | 541 | | |
524 | 542 | | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
525 | 546 | | |
526 | 547 | | |
527 | 548 | | |
| |||
538 | 559 | | |
539 | 560 | | |
540 | 561 | | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
541 | 565 | | |
542 | 566 | | |
543 | 567 | | |
| |||
550 | 574 | | |
551 | 575 | | |
552 | 576 | | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
553 | 580 | | |
554 | 581 | | |
555 | 582 | | |
| |||
579 | 606 | | |
580 | 607 | | |
581 | 608 | | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
582 | 614 | | |
583 | 615 | | |
584 | 616 | | |
| |||
588 | 620 | | |
589 | 621 | | |
590 | 622 | | |
591 | | - | |
| 623 | + | |
592 | 624 | | |
593 | 625 | | |
594 | 626 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
147 | 151 | | |
148 | 152 | | |
149 | 153 | | |
150 | | - | |
| 154 | + | |
151 | 155 | | |
152 | 156 | | |
| 157 | + | |
153 | 158 | | |
154 | 159 | | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
155 | 166 | | |
156 | 167 | | |
157 | 168 | | |
158 | 169 | | |
| 170 | + | |
159 | 171 | | |
160 | 172 | | |
161 | 173 | | |
| |||
536 | 548 | | |
537 | 549 | | |
538 | 550 | | |
539 | | - | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
540 | 557 | | |
541 | 558 | | |
542 | | - | |
543 | | - | |
| 559 | + | |
544 | 560 | | |
545 | 561 | | |
546 | 562 | | |
547 | 563 | | |
548 | 564 | | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
549 | 588 | | |
550 | 589 | | |
551 | 590 | | |
| |||
1450 | 1489 | | |
1451 | 1490 | | |
1452 | 1491 | | |
1453 | | - | |
| 1492 | + | |
1454 | 1493 | | |
1455 | 1494 | | |
1456 | 1495 | | |
| |||
1473 | 1512 | | |
1474 | 1513 | | |
1475 | 1514 | | |
1476 | | - | |
| 1515 | + | |
1477 | 1516 | | |
1478 | 1517 | | |
1479 | 1518 | | |
| |||
1504 | 1543 | | |
1505 | 1544 | | |
1506 | 1545 | | |
1507 | | - | |
| 1546 | + | |
1508 | 1547 | | |
1509 | 1548 | | |
1510 | 1549 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
65 | 65 | | |
66 | 66 | | |
67 | 67 | | |
| 68 | + | |
68 | 69 | | |
69 | 70 | | |
70 | 71 | | |
| |||
222 | 223 | | |
223 | 224 | | |
224 | 225 | | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
225 | 231 | | |
226 | 232 | | |
227 | 233 | | |
| |||
248 | 254 | | |
249 | 255 | | |
250 | 256 | | |
251 | | - | |
252 | | - | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
253 | 271 | | |
254 | 272 | | |
255 | 273 | | |
| |||
261 | 279 | | |
262 | 280 | | |
263 | 281 | | |
264 | | - | |
| 282 | + | |
265 | 283 | | |
266 | 284 | | |
267 | 285 | | |
268 | 286 | | |
269 | 287 | | |
| 288 | + | |
270 | 289 | | |
271 | 290 | | |
272 | 291 | | |
| |||
1979 | 1998 | | |
1980 | 1999 | | |
1981 | 2000 | | |
| 2001 | + | |
| 2002 | + | |
| 2003 | + | |
| 2004 | + | |
| 2005 | + | |
| 2006 | + | |
| 2007 | + | |
1982 | 2008 | | |
1983 | 2009 | | |
| 2010 | + | |
1984 | 2011 | | |
1985 | | - | |
| 2012 | + | |
| 2013 | + | |
| 2014 | + | |
| 2015 | + | |
| 2016 | + | |
| 2017 | + | |
| 2018 | + | |
1986 | 2019 | | |
1987 | | - | |
1988 | | - | |
1989 | | - | |
1990 | | - | |
1991 | | - | |
| 2020 | + | |
| 2021 | + | |
| 2022 | + | |
| 2023 | + | |
| 2024 | + | |
| 2025 | + | |
| 2026 | + | |
| 2027 | + | |
| 2028 | + | |
| 2029 | + | |
| 2030 | + | |
| 2031 | + | |
| 2032 | + | |
1992 | 2033 | | |
| 2034 | + | |
1993 | 2035 | | |
1994 | 2036 | | |
1995 | 2037 | | |
| |||
0 commit comments