Skip to content

Commit 518d059

Browse files
morningmanclaude
andcommitted
[perf](catalog) fe-connector-iceberg: cache the partition-view scan across queries (PERF-02)
For a partitioned table, every query pays an analysis-phase PARTITIONS metadata-table scan (IcebergPartitionUtils.loadRawPartitions -> partitionsTable.newScan().useSnapshot(id).planFiles()+rows()), which the iceberg SDK materializes by reading EVERY data+delete manifest of the snapshot (O(#manifests) remote IO, independent of query selectivity). It ran once per query (memoized only within the statement) and 4~6 times per MTMV refresh, with no cross-query cache -- the partition-info half of the legacy IcebergExternalMetaCache that the SPI cutover dropped. PERF-01 already deduped the loadTable in this cluster; this fixes the remaining scan. Add a connector-side IcebergPartitionCache keyed by (TableIdentifier, snapshotId), value = the raw partition list. The three consumers -- buildMvccPartitionView (MVCC/MTMV view), listPartitions (selectedPartitionNum), listPartitionNames (SHOW PARTITIONS) -- all funnel through loadRawPartitions, so they share one scan per (table, snapshot). The snapshot id resolved inside those methods is held stable across queries (and across one refresh's enumeration points) by IcebergLatestSnapshotCache, so the key is stable and the MTMV 4~6x re-enumeration collapses onto one scan too. A snapshot is immutable, so the cached value is a pure function of the key (always correct; a new commit -> new snapshot id -> new key -> a live scan). The value is pure metadata with NO FileIO/credential, so unlike the PERF-01 table cache it needs no credential gate and is built for every catalog (only the meta.cache.iceberg.table.ttl-second knob disables it). The cached list is unmodifiable; the loader's exception propagates verbatim so listPartitions keeps its dropped-partition-source-column degradation, and a failed scan is not cached. REFRESH TABLE/DB/CATALOG invalidate it alongside the other caches. The MTMV refresh-level MvccSnapshot pin the audit also suggested is intentionally NOT done: IcebergLatestSnapshotCache already stabilizes the snapshot across a refresh (ttl>0), so this cache collapses the repeats without a fe-core change (the only residual is the ttl=0 no-cache catalog's pre-existing enumeration skew, which is out of scope for a connector-side perf fix). Metric gate: a real partitioned table + a real cache; repeated buildMvccPartitionView plus listPartitions at one snapshot run the PARTITIONS scan exactly once (cache.loadCountForTest()==1, was per-call). Full iceberg module suite: 943 pass / 0 fail / 1 skip; checkstyle clean. Behavior parity: cached enumeration equals a live one; snapshot/schema visibility unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SMtYwYyyubZZiC1odLZTG3
1 parent 65cc6d4 commit 518d059

7 files changed

Lines changed: 493 additions & 11 deletions

File tree

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -164,6 +164,10 @@ public class IcebergConnector implements Connector {
164164
// (iceberg.rest.session=user / REST vended-credentials) — see the constructor. The query-scoped fat handle
165165
// (IcebergTableHandle.resolvedTable) is always on and independent of this field.
166166
private final IcebergTableCache tableCache;
167+
// PERF-02: cross-query partition-view cache (the raw PARTITIONS-scan result, keyed by (table, snapshotId)).
168+
// Built unconditionally — the cached value is pure metadata with no FileIO/credential, so no credential gate
169+
// (unlike tableCache); only the TTL knob disables it.
170+
private final IcebergPartitionCache partitionCache;
167171
private final IcebergManifestCache manifestCache = new IcebergManifestCache();
168172
// commit-bridge supply (S4 part 2): per-catalog stash carrying a row-level DML's non-equality delete supply
169173
// across the scan->write seam — the scan provider fills it (keyed by queryId), the write provider drains it
@@ -205,6 +209,9 @@ public IcebergConnector(Map<String, String> properties, ConnectorContext context
205209
? null
206210
: new IcebergTableCache(
207211
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
212+
// PERF-02: partition-view cache. Pure metadata (no credentials) -> no gate; same TTL/capacity as above.
213+
this.partitionCache = new IcebergPartitionCache(
214+
resolveTableCacheTtlSecond(this.properties), DEFAULT_TABLE_CACHE_CAPACITY);
208215
}
209216

210217
/**
@@ -229,7 +236,7 @@ static long resolveTableCacheTtlSecond(Map<String, String> properties) {
229236
@Override
230237
public ConnectorMetadata getMetadata(ConnectorSession session) {
231238
return new IcebergConnectorMetadata(
232-
newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache);
239+
newCatalogBackedOps(session), properties, context, latestSnapshotCache, tableCache, partitionCache);
233240
}
234241

235242
/**
@@ -543,6 +550,7 @@ public void invalidateTable(String dbName, String tableName) {
543550
if (tableCache != null) {
544551
tableCache.invalidate(TableIdentifier.of(dbName, tableName));
545552
}
553+
partitionCache.invalidate(TableIdentifier.of(dbName, tableName));
546554
}
547555

548556
/**
@@ -562,6 +570,7 @@ public void invalidateDb(String dbName) {
562570
if (tableCache != null) {
563571
tableCache.invalidateDb(dbName);
564572
}
573+
partitionCache.invalidateDb(dbName);
565574
}
566575

567576
/**
@@ -577,6 +586,7 @@ public void invalidateAll() {
577586
if (tableCache != null) {
578587
tableCache.invalidateAll();
579588
}
589+
partitionCache.invalidateAll();
580590
manifestCache.invalidateAll();
581591
}
582592

@@ -611,6 +621,11 @@ IcebergTableCache tableCacheForTest() {
611621
return tableCache;
612622
}
613623

624+
/** Test-only: the cross-query partition-view cache (PERF-02; always built, no credential gate). */
625+
IcebergPartitionCache partitionCacheForTest() {
626+
return partitionCache;
627+
}
628+
614629
@Override
615630
public ConnectorScanPlanProvider getScanPlanProvider() {
616631
// Mirrors PaimonConnector.getScanPlanProvider: build a fresh provider per call over the lazily-built

‎fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java‎

Lines changed: 19 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -148,26 +148,37 @@ public class IcebergConnectorMetadata implements ConnectorMetadata {
148148
// and the credential-gated catalogs pass null; the query-scoped fat handle (IcebergTableHandle) works
149149
// regardless. Consumed only by resolveTableForRead.
150150
private final IcebergTableCache tableCache;
151+
// PERF-02: cross-query partition-view cache (null = no cross-query layer; the convenience ctors used by
152+
// direct-construction tests pass null). Consumed by getMvccPartitionView / listPartitions / listPartitionNames.
153+
private final IcebergPartitionCache partitionCache;
151154

152155
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
153156
ConnectorContext context) {
154-
this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null);
157+
this(catalogOps, properties, context, new IcebergLatestSnapshotCache(0L, 1), null, null);
155158
}
156159

157160
/** Convenience ctor without a cross-query table cache (tableCache null); used by MVCC/statistics tests. */
158161
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
159162
ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache) {
160-
this(catalogOps, properties, context, latestSnapshotCache, null);
163+
this(catalogOps, properties, context, latestSnapshotCache, null, null);
161164
}
162165

166+
/** Convenience ctor without a partition-view cache (partitionCache null). */
163167
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
164168
ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache,
165169
IcebergTableCache tableCache) {
170+
this(catalogOps, properties, context, latestSnapshotCache, tableCache, null);
171+
}
172+
173+
public IcebergConnectorMetadata(IcebergCatalogOps catalogOps, Map<String, String> properties,
174+
ConnectorContext context, IcebergLatestSnapshotCache latestSnapshotCache,
175+
IcebergTableCache tableCache, IcebergPartitionCache partitionCache) {
166176
this.catalogOps = catalogOps;
167177
this.properties = properties;
168178
this.context = context;
169179
this.latestSnapshotCache = latestSnapshotCache;
170180
this.tableCache = tableCache;
181+
this.partitionCache = partitionCache;
171182
}
172183

173184
// ========== ConnectorSchemaOps ==========
@@ -1490,7 +1501,8 @@ public Optional<ConnectorMvccPartitionView> getMvccPartitionView(
14901501
try {
14911502
return context.executeAuthenticated(() -> {
14921503
Table table = resolveTableForRead(iceHandle);
1493-
return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId()));
1504+
return Optional.of(IcebergPartitionUtils.buildMvccPartitionView(table, iceHandle.getSnapshotId(),
1505+
TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache));
14941506
});
14951507
} catch (Exception e) {
14961508
throw new RuntimeException("Failed to build iceberg MVCC partition view, error message is:"
@@ -1518,7 +1530,8 @@ public List<String> listPartitionNames(ConnectorSession session, ConnectorTableH
15181530
iceHandle.getDbName(), iceHandle.getTableName(), e);
15191531
return Collections.<String>emptyList();
15201532
}
1521-
return IcebergPartitionUtils.listPartitionNames(table);
1533+
return IcebergPartitionUtils.listPartitionNames(table,
1534+
TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache);
15221535
});
15231536
} catch (Exception e) {
15241537
throw new RuntimeException("Failed to list iceberg partition names, error message is:"
@@ -1549,7 +1562,8 @@ public List<ConnectorPartitionInfo> listPartitions(ConnectorSession session,
15491562
iceHandle.getDbName(), iceHandle.getTableName(), e);
15501563
return Collections.<ConnectorPartitionInfo>emptyList();
15511564
}
1552-
return IcebergPartitionUtils.listPartitions(table);
1565+
return IcebergPartitionUtils.listPartitions(table,
1566+
TableIdentifier.of(iceHandle.getDbName(), iceHandle.getTableName()), partitionCache);
15531567
});
15541568
} catch (Exception e) {
15551569
throw new RuntimeException("Failed to list iceberg partitions, error message is:"
Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.iceberg;
19+
20+
import org.apache.doris.connector.cache.CacheSpec;
21+
import org.apache.doris.connector.cache.MetaCacheEntry;
22+
import org.apache.doris.connector.iceberg.IcebergPartitionUtils.IcebergRawPartition;
23+
24+
import org.apache.iceberg.catalog.Namespace;
25+
import org.apache.iceberg.catalog.TableIdentifier;
26+
27+
import java.util.List;
28+
import java.util.Objects;
29+
import java.util.concurrent.ForkJoinPool;
30+
import java.util.function.Supplier;
31+
32+
/**
33+
* Per-catalog cache of an iceberg table's raw partition list (PERF-02), keyed by {@code (TableIdentifier,
34+
* snapshotId)}. Restores the partition-info half of the legacy {@code IcebergExternalMetaCache} that the SPI
35+
* cutover dropped: the analysis-phase PARTITIONS metadata-table scan
36+
* ({@link IcebergPartitionUtils#loadRawPartitionsUncached}, which the iceberg SDK materializes by reading EVERY
37+
* data+delete manifest of the snapshot) was re-run per query and re-run 4~6 times per MTMV refresh, with no
38+
* cross-query reuse. The three consumers ({@code buildMvccPartitionView} for the MVCC/MTMV partition view,
39+
* {@code listPartitions} for {@code selectedPartitionNum}, {@code listPartitionNames} for SHOW PARTITIONS) all
40+
* funnel through it, so they share a single scan per {@code (table, snapshot)}.
41+
*
42+
* <p><b>Snapshot-keyed, so always correct.</b> A snapshot is immutable, so the derived partitions are a pure
43+
* function of the key; a new commit yields a new snapshot id (a new key -&gt; a live scan). Within the TTL the
44+
* snapshot id itself is held stable by {@link IcebergLatestSnapshotCache}, which is what makes the key stable
45+
* across queries and across the enumeration points of one MTMV refresh.
46+
*
47+
* <p><b>No credential gate</b> (unlike {@link IcebergTableCache}): the cached value is pure metadata (partition
48+
* names, values, transforms, timestamps, snapshot ids) and carries no {@code FileIO} / credential, so it is
49+
* safe to share across users and is built unconditionally (only the TTL knob disables it).
50+
*
51+
* <p>Backed identically to {@link IcebergLatestSnapshotCache}: a contextual, access-TTL {@link MetaCacheEntry}
52+
* with manual miss-load, so the scan runs OUTSIDE Caffeine's compute lock and its exception (e.g. the
53+
* dropped-partition-source-column {@link org.apache.iceberg.exceptions.ValidationException} that
54+
* {@code listPartitions} degrades on) propagates verbatim and a failed scan is not cached. TTL is
55+
* {@code meta.cache.iceberg.table.ttl-second}; {@code <= 0} disables (read live). Lives on the long-lived
56+
* per-catalog {@link IcebergConnector}; a REFRESH CATALOG rebuilds the connector and thus the cache.
57+
*/
58+
final class IcebergPartitionCache {
59+
60+
/** Immutable composite key: a table's partition list is distinct per pinned snapshot id. */
61+
static final class Key {
62+
final TableIdentifier id;
63+
final long snapshotId;
64+
65+
Key(TableIdentifier id, long snapshotId) {
66+
this.id = id;
67+
this.snapshotId = snapshotId;
68+
}
69+
70+
@Override
71+
public boolean equals(Object o) {
72+
if (this == o) {
73+
return true;
74+
}
75+
if (!(o instanceof Key)) {
76+
return false;
77+
}
78+
Key that = (Key) o;
79+
return snapshotId == that.snapshotId && Objects.equals(id, that.id);
80+
}
81+
82+
@Override
83+
public int hashCode() {
84+
return Objects.hash(id, snapshotId);
85+
}
86+
}
87+
88+
private final MetaCacheEntry<Key, List<IcebergRawPartition>> entry;
89+
90+
IcebergPartitionCache(long ttlSeconds, int maxSize) {
91+
// Mirror IcebergLatestSnapshotCache: translate the connector's "<= 0 disables" contract to CacheSpec's
92+
// ttl == 0 (disabled) rather than passing a negative value through (which CacheSpec reads as "no
93+
// expiration").
94+
CacheSpec spec = ttlSeconds > 0
95+
? CacheSpec.of(true, ttlSeconds, maxSize)
96+
: CacheSpec.of(true, CacheSpec.CACHE_TTL_DISABLE_CACHE, maxSize);
97+
this.entry = new MetaCacheEntry<>("iceberg-partition", null, spec,
98+
ForkJoinPool.commonPool(), false, true, 0L, true);
99+
}
100+
101+
/** Caching is on only when the TTL is positive; ttl-second &lt;= 0 means "always scan live". */
102+
boolean isEnabled() {
103+
return entry.stats().isEffectiveEnabled();
104+
}
105+
106+
/**
107+
* Returns the cached raw partition list for {@code key} if present and unexpired, else runs {@code loader}
108+
* (the live PARTITIONS scan), caches and returns it. Disabled cache -&gt; {@code loader} every call. The
109+
* loader runs OUTSIDE Caffeine's compute lock (single-flight per key) and its exception propagates unwrapped.
110+
*/
111+
List<IcebergRawPartition> getOrLoad(Key key, Supplier<List<IcebergRawPartition>> loader) {
112+
return entry.get(key, ignored -> loader.get());
113+
}
114+
115+
/** Drops every cached snapshot entry for one table so the next read scans live (REFRESH TABLE). */
116+
void invalidate(TableIdentifier id) {
117+
entry.invalidateIf(key -> key.id.equals(id));
118+
}
119+
120+
/** Drops every cached entry for one database (REFRESH DATABASE / DROP DATABASE); db match = namespace equality. */
121+
void invalidateDb(String dbName) {
122+
Namespace ns = Namespace.of(dbName);
123+
entry.invalidateIf(key -> key.id.namespace().equals(ns));
124+
}
125+
126+
/** Drops all cached entries (REFRESH CATALOG). */
127+
void invalidateAll() {
128+
entry.invalidateAll();
129+
}
130+
131+
/** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */
132+
int size() {
133+
int[] count = {0};
134+
entry.forEach((key, value) -> count[0]++);
135+
return count[0];
136+
}
137+
138+
/** Test-only: how many times the live loader (the PARTITIONS scan) actually ran — the metric gate. */
139+
long loadCountForTest() {
140+
return entry.stats().getLoadSuccessCount();
141+
}
142+
}

0 commit comments

Comments
 (0)