Skip to content

Commit 484f0e0

Browse files
morningmanclaude
andcommitted
[perf](catalog) fe-connector-iceberg: memoize the resolved Table within and across queries (PERF-01)
A plain WHERE-filtered iceberg SELECT re-loaded the same table 3~7 times per planning pass: getColumnHandles, getTableStatistics, the scan provider's resolveTable (getScanNodeProperties / planScan / streamingSplitEstimate) and the analysis-phase getMvccPartitionView / listPartitions each issued their own remote loadTable (a metastore RPC + a metadata.json read). The SPI cutover split the legacy IcebergExternalMetaCache and kept only the latest-snapshot pin (IcebergLatestSnapshotCache), dropping the cached-Table half. Restore it in two connector-side layers (fe-core untouched): - Query-scoped fat handle: IcebergTableHandle carries a transient raw Table memo (not serialized, not part of identity), carried forward by withSnapshot / withRewriteFileScope / withTopnLazyMaterialize. Reads resolve the table once and reuse the same instance. PluginDrivenScanNode.currentHandle is per-query, so the memo is GC'd with the plan (zero per-query accumulation) => exactly one remote loadTable per table per planning pass. - Cross-query IcebergTableCache (new; mirrors IcebergLatestSnapshotCache, value = raw Table): 24h TTL, REFRESH-invalidated, backs the fat handle's first miss and restores legacy cross-query caching. Disabled (null) for iceberg.rest.session=user and REST vended-credentials, where a cached raw Table's FileIO carries a per-user / soon-expiring token (cross-user leak / 403 mid-scan); the fat handle stays on there (token fresh within one query). Reads go through resolveTableForRead (fat handle -> cache -> raw loadTable), which throws NoSuchTableException unwrapped so listPartitions / getMvccPartitionView keep their concurrent-drop degradation. The scan provider re-applies wrapTableForScan (Kerberos FileIO) per call, never freezing it into the memo. DDL / write / procedure / sys-table paths bypass the helper (fresh remote base). Part B (convertPredicate narrowing) is intentionally dropped (red-team proved it a no-op). Metric gate: RecordingIcebergCatalogOps asserts one planning pass (getColumnHandles + planScan on a single handle) issues exactly one remote loadTable (was two). Full iceberg module suite: 932 pass / 0 fail / 1 skip; checkstyle clean. Behavior parity: pure de-duplication -- snapshot semantics, schema visibility and time travel are unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
1 parent 9a2c39c commit 484f0e0

9 files changed

Lines changed: 633 additions & 26 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java‎

Lines changed: 34 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -159,6 +159,11 @@ public class IcebergConnector implements Connector {
159159
// manifest cache is path-keyed, no-TTL, capacity-bounded; it is consumed only when
160160
// meta.cache.iceberg.manifest.enable is set (default off → scan uses the SDK planFiles path).
161161
private final IcebergLatestSnapshotCache latestSnapshotCache;
162+
// PERF-01: cross-query cache of the RAW iceberg Table (restores the legacy IcebergExternalMetaCache table
163+
// cache that the SPI cutover dropped). null when the catalog's credentials are query-dependent
164+
// (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The query-scoped fat handle
165+
// (IcebergTableHandle.resolvedTable) is always on and independent of this field.
166+
private final IcebergTableCache tableCache;
162167
private final IcebergManifestCache manifestCache = new IcebergManifestCache();
163168
// commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply
164169
// across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it
@@ -187,6 +192,19 @@ public IcebergConnector(Map<String, String> properties, ConnectorContext context
187192
this::pluginAuthenticator);
188193
this.latestSnapshotCache = new IcebergLatestSnapshotCache(
189194
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
195+
// PERF-01 cross-query RAW-table cache. Disabled (null) when the catalog's credentials are
196+
// query-dependent, because a cached raw Table carries its FileIO's credentials:
197+
// - iceberg.rest.session=user: per-user delegated FileIO -> sharing across users leaks credentials.
198+
// - REST vended-credentials: the FileIO carries a server-vended token that expires within ~an hour;
199+
// iceberg keeps it fresh by reloading the table each query, so a 24h-TTL hit would hand BE an
200+
// expired token (403 mid-scan). Both gates are independent; either one disables this layer.
201+
// The query-scoped fat handle stays on in all cases (its token is fresh within the one query). Same
202+
// TTL/capacity as the snapshot cache (the single meta.cache.iceberg.table.ttl-second knob).
203+
this.tableCache = (isUserSessionEnabled()
204+
|| IcebergScanPlanProvider.restVendedCredentialsEnabled(this.properties))
205+
? null
206+
: new IcebergTableCache(
207+
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
190208
}
191209

192210
/**
@@ -211,7 +229,7 @@ static long resolveTableCacheTtlSecond(Map<String, String> properties) {
211229
@Override
212230
public ConnectorMetadata getMetadata(ConnectorSession session) {
213231
return new IcebergConnectorMetadata(
214-
newCatalogBackedOps(session), properties, context, latestSnapshotCache);
232+
newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache);
215233
}
216234

217235
/**
@@ -522,6 +540,9 @@ private IcebergCatalogOps newCatalogBackedOps(ConnectorSession session) {
522540
@Override
523541
public void invalidateTable(String dbName, String tableName) {
524542
latestSnapshotCache.invalidate(TableIdentifier.of(dbName, tableName));
543+
if (tableCache != null) {
544+
tableCache.invalidate(TableIdentifier.of(dbName, tableName));
545+
}
525546
}
526547

527548
/**
@@ -538,6 +559,9 @@ public void invalidateTable(String dbName, String tableName) {
538559
@Override
539560
public void invalidateDb(String dbName) {
540561
latestSnapshotCache.invalidateDb(dbName);
562+
if (tableCache != null) {
563+
tableCache.invalidateDb(dbName);
564+
}
541565
}
542566

543567
/**
@@ -550,6 +574,9 @@ public void invalidateDb(String dbName) {
550574
@Override
551575
public void invalidateAll() {
552576
latestSnapshotCache.invalidateAll();
577+
if (tableCache != null) {
578+
tableCache.invalidateAll();
579+
}
553580
manifestCache.invalidateAll();
554581
}
555582

@@ -579,6 +606,11 @@ IcebergManifestCache manifestCacheForTest() {
579606
return manifestCache;
580607
}
581608

609+
/** Test-only: the cross-query table cache, or {@code null} when disabled by the credential gate (PERF-01). */
610+
IcebergTableCache tableCacheForTest() {
611+
return tableCache;
612+
}
613+
582614
@Override
583615
public ConnectorScanPlanProvider getScanPlanProvider() {
584616
// Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built
@@ -588,7 +620,7 @@ public ConnectorScanPlanProvider getScanPlanProvider() {
588620
// threaded for parity with the legacy single per-catalog IcebergMetadataOps.
589621
return new IcebergScanPlanProvider(properties,
590622
this::newCatalogBackedOps, context, manifestCache,
591-
rewritableDeleteStash);
623+
rewritableDeleteStash, tableCache);
592624
}
593625

594626
@Override

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java‎

Lines changed: 46 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -144,18 +144,30 @@ public class IcebergConnectorMetadata implements ConnectorMetadata {
144144
// IcebergExternalMetaCache parity, mirrors paimon). The 3-arg ctor (direct-construction tests) passes a
145145
// DISABLED cache so those reads stay always-live.
146146
private final IcebergLatestSnapshotCache latestSnapshotCache;
147+
// PERF-01: cross-query RAW-table cache (null = no cross-query layer). The 3-arg direct-construction tests
148+
// and the credential-gated catalogs pass null; the query-scoped fat handle (IcebergTableHandle) works
149+
// regardless. Consumed only by resolveTableForRead.
150+
private final IcebergTableCache tableCache;
147151

148152
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
149153
ConnectorContext context) {
150-
this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1));
154+
this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null);
151155
}
152156

157+
/** Convenience ctor without a cross-query table cache (tableCache null); used by MVCC/statistics tests. */
153158
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
154159
ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache) {
160+
this(catalogOps, properties, context, latestSnapshotCache, null);
161+
}
162+
163+
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
164+
ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache,
165+
IcebergTableCache tableCache) {
155166
this.catalogOps = catalogOps;
156167
this.properties = properties;
157168
this.context = context;
158169
this.latestSnapshotCache = latestSnapshotCache;
170+
this.tableCache = tableCache;
159171
}
160172

161173
// ========== ConnectorSchemaOps ==========
@@ -536,16 +548,43 @@ static String buildShowSortClause(Table table) {
536548
return "ORDER BY (" + String.join(", ", sortItems) + ")";
537549
}
538550

539-
/** Loads the iceberg {@link Table} through the seam, wrapped in the FE-injected auth context (Kerberos UGI). */
551+
/**
552+
* Loads the iceberg {@link Table} for {@code handle}, wrapped in the FE-injected auth context (Kerberos
553+
* UGI). Resolution goes through {@link #resolveTableForRead} (fat handle -&gt; cross-query cache -&gt;
554+
* remote), so the many reads sharing one handle in a planning/analysis pass collapse onto a single remote
555+
* {@code loadTable} (PERF-01); a fat-handle hit returns without any remote call.
556+
*/
540557
private Table loadTable(IcebergTableHandle handle) {
541558
try {
542-
return context.executeAuthenticated(
543-
() -> catalogOps.loadTable(handle.getDbName(), handle.getTableName()));
559+
return context.executeAuthenticated(() -> resolveTableForRead(handle));
544560
} catch (Exception e) {
545561
throw new RuntimeException("Failed to load table, error message is:" + e.getMessage(), e);
546562
}
547563
}
548564

565+
/**
566+
* Resolves the RAW iceberg {@link Table} for {@code handle} with PERF-01's two-layer memo, WITHOUT opening
567+
* an auth scope or wrapping exceptions — callers own both. The query-scoped fat handle comes first (same
568+
* handle instance -&gt; same table); then the cross-query {@link IcebergTableCache} when enabled (else a
569+
* direct remote load); finally the result is memoized back onto the handle. The remote loader's exception
570+
* propagates verbatim (the cache re-throws it unwrapped), so a caller's own {@code NoSuchTableException}
571+
* degradation (the partition-view readers) still fires. Callers needing the auth scope wrap the call in
572+
* {@code executeAuthenticated} (see {@link #loadTable}). NOT used by the sys-table path
573+
* ({@link #loadSysTable}) or the DDL/write path (both take a fresh remote base by design).
574+
*/
575+
private Table resolveTableForRead(IcebergTableHandle handle) {
576+
Table cached = handle.getResolvedTable();
577+
if (cached != null) {
578+
return cached;
579+
}
580+
Table table = tableCache != null
581+
? tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()),
582+
() -> catalogOps.loadTable(handle.getDbName(), handle.getTableName()))
583+
: catalogOps.loadTable(handle.getDbName(), handle.getTableName());
584+
handle.setResolvedTable(table);
585+
return table;
586+
}
587+
549588
/**
550589
* Loads the iceberg metadata (system) table for {@code handle} through the seam, wrapped in the
551590
* FE-injected auth context (Kerberos UGI). Mirrors legacy
@@ -1450,7 +1489,7 @@ public Optional<ConnectorMvccPartitionView> getMvccPartitionView(
14501489
IcebergTableHandle iceHandle = (IcebergTableHandle) handle;
14511490
try {
14521491
return context.executeAuthenticated(() -> {
1453-
Table table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName());
1492+
Table table = resolveTableForRead(iceHandle);
14541493
return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId()));
14551494
});
14561495
} catch (Exception e) {
@@ -1473,7 +1512,7 @@ public List<String> listPartitionNames(ConnectorSession session, ConnectorTableH
14731512
return context.executeAuthenticated(() -> {
14741513
Table table;
14751514
try {
1476-
table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName());
1515+
table = resolveTableForRead(iceHandle);
14771516
} catch (NoSuchTableException e) {
14781517
LOG.warn("Iceberg table not found while listing partitions: {}.{}",
14791518
iceHandle.getDbName(), iceHandle.getTableName(), e);
@@ -1504,7 +1543,7 @@ public List<ConnectorPartitionInfo> listPartitions(ConnectorSession session,
15041543
return context.executeAuthenticated(() -> {
15051544
Table table;
15061545
try {
1507-
table = catalogOps.loadTable(iceHandle.getDbName(), iceHandle.getTableName());
1546+
table = resolveTableForRead(iceHandle);
15081547
} catch (NoSuchTableException e) {
15091548
LOG.warn("Iceberg table not found while listing partitions: {}.{}",
15101549
iceHandle.getDbName(), iceHandle.getTableName(), e);

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java‎

Lines changed: 51 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@
6565
import org.apache.iceberg.TableOperations;
6666
import org.apache.iceberg.TableProperties;
6767
import org.apache.iceberg.TableScan;
68+
import org.apache.iceberg.catalog.TableIdentifier;
6869
import org.apache.iceberg.exceptions.ValidationException;
6970
import org.apache.iceberg.expressions.Expression;
7071
import org.apache.iceberg.expressions.Expressions;
@@ -222,6 +223,11 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider {
222223
// (offline tests), in which case stashing is skipped (the supply is exercised only on the post-cutover write
223224
// path; pre-flip the provider never runs at all).
224225
private final IcebergRewritableDeleteStash rewritableDeleteStash;
226+
// PERF-01: cross-query RAW-table cache shared with the metadata layer, owned by the long-lived
227+
// IcebergConnector and injected via getScanPlanProvider. Nullable — null via the offline-test ctors and
228+
// when the connector's credential gate disables the cross-query layer; when null resolveTable still uses the
229+
// query-scoped fat handle and falls back to a direct remote load.
230+
private final IcebergTableCache tableCache;
225231

226232
// FIX-SCAN-METRICS: per-query stash of the iceberg SDK scan diagnostics captured by the attached
227233
// IcebergScanProfileReporter during planScan, keyed by session queryId. fe-core drains it
@@ -248,8 +254,20 @@ public IcebergScanPlanProvider(Map<String, String> properties, IcebergCatalogOps
248254
ConnectorContext context, IcebergManifestCache manifestCache,
249255
IcebergRewritableDeleteStash rewritableDeleteStash) {
250256
// Constant resolver: these ctors (offline tests + the pre-session connector paths) bind a single ops that
251-
// ignores the session, so existing behaviour/tests are byte-identical.
252-
this(properties, session -> catalogOps, context, manifestCache, rewritableDeleteStash);
257+
// ignores the session, so existing behaviour/tests are byte-identical. No cross-query cache (tableCache
258+
// null) — the fat handle still dedups within a planning pass.
259+
this(properties, session -> catalogOps, context, manifestCache, rewritableDeleteStash, null);
260+
}
261+
262+
/**
263+
* Session-aware convenience ctor without a cross-query table cache (tableCache null); used by the offline
264+
* session-routing tests. The query-scoped fat handle still dedups within a planning pass.
265+
*/
266+
public IcebergScanPlanProvider(Map<String, String> properties,
267+
Function<ConnectorSession, IcebergCatalogOps> catalogOpsResolver,
268+
ConnectorContext context, IcebergManifestCache manifestCache,
269+
IcebergRewritableDeleteStash rewritableDeleteStash) {
270+
this(properties, catalogOpsResolver, context, manifestCache, rewritableDeleteStash, null);
253271
}
254272

255273
/**
@@ -261,12 +279,13 @@ public IcebergScanPlanProvider(Map<String, String> properties, IcebergCatalogOps
261279
public IcebergScanPlanProvider(Map<String, String> properties,
262280
Function<ConnectorSession, IcebergCatalogOps> catalogOpsResolver,
263281
ConnectorContext context, IcebergManifestCache manifestCache,
264-
IcebergRewritableDeleteStash rewritableDeleteStash) {
282+
IcebergRewritableDeleteStash rewritableDeleteStash, IcebergTableCache tableCache) {
265283
this.properties = properties;
266284
this.catalogOpsResolver = catalogOpsResolver;
267285
this.context = context;
268286
this.manifestCache = manifestCache;
269287
this.rewritableDeleteStash = rewritableDeleteStash;
288+
this.tableCache = tableCache;
270289
}
271290

272291
/**
@@ -1979,17 +1998,40 @@ static ZoneId resolveSessionZone(ConnectorSession session) {
19791998
* {@code null} context (offline unit tests / simple-auth) resolves directly.
19801999
*/
19812000
private Table resolveTable(ConnectorSession session, IcebergTableHandle handle) {
2001+
// Fat handle first (PERF-01): one handle threaded through getColumnHandles + planScan resolves the table
2002+
// once. The memo holds the RAW table; wrapTableForScan (the Kerberos doAs FileIO) is re-applied per call
2003+
// below so no per-request authenticator is ever frozen into the shared memo/cache.
2004+
Table cached = handle.getResolvedTable();
2005+
if (cached != null) {
2006+
return wrapTableForScan(cached);
2007+
}
19822008
// Resolve the per-request ops before the auth scope so a session=user fail-closed surfaces verbatim.
19832009
IcebergCatalogOps ops = catalogOpsResolver.apply(session);
2010+
Table raw;
19842011
if (context == null) {
1985-
return ops.loadTable(handle.getDbName(), handle.getTableName());
2012+
raw = loadRawTable(ops, handle);
2013+
} else {
2014+
try {
2015+
raw = context.executeAuthenticated(() -> loadRawTable(ops, handle));
2016+
} catch (Exception e) {
2017+
throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e);
2018+
}
19862019
}
1987-
try {
1988-
return wrapTableForScan(context.executeAuthenticated(
1989-
() -> ops.loadTable(handle.getDbName(), handle.getTableName())));
1990-
} catch (Exception e) {
1991-
throw new RuntimeException("Failed to load table for scan, error message is:" + e.getMessage(), e);
2020+
handle.setResolvedTable(raw);
2021+
return wrapTableForScan(raw);
2022+
}
2023+
2024+
/**
2025+
* Loads the RAW iceberg table for {@code handle} through the cross-query {@link IcebergTableCache} when
2026+
* enabled (the connector disables it for credential-dependent catalogs), else a direct remote
2027+
* {@code loadTable}. No wrap and no auth scope here — {@link #resolveTable} owns both.
2028+
*/
2029+
private Table loadRawTable(IcebergCatalogOps ops, IcebergTableHandle handle) {
2030+
if (tableCache != null) {
2031+
return tableCache.getOrLoad(TableIdentifier.of(handle.getDbName(), handle.getTableName()),
2032+
() -> ops.loadTable(handle.getDbName(), handle.getTableName()));
19922033
}
2034+
return ops.loadTable(handle.getDbName(), handle.getTableName());
19932035
}
19942036

19952037
/**

0 commit comments

Comments
 (0)