Skip to content

Commit e3be4ad

Browse files
morningmanclaude
andcommitted
[perf](catalog) wire cross-query derived partition-view cache A into the paimon connector (S4)
Layer the generic ConnectorPartitionViewCache<V> (S2 framework) above paimon's raw catalogOps.listPartitions round-trip so a repeated query on a partitioned paimon table skips the derived-view BUILD (display-name rendering + null-sentinel normalization, PaimonConnectorMetadata#collectPartitions), not just avoids re-deriving from an already-cached remote result -- there was no such cache before. ONE typed field (unlike iceberg's two): paimon does not override getMvccPartitionView (the generic MTMV model falls back to its default listPartitions/LIST/timestamp path for paimon), so listPartitions is the only enumeration hook to wrap. - Key = (db, table, snapshotId, schemaId). snapshotId reads the SAME per-catalog PaimonLatestSnapshotCache that beginQuerySnapshot pins queries to -- collectPartitions' remote call is base-identifier-only (it does not apply the handle's pinned scanOptions, unlike the scan path) so it always reflects the CURRENT catalog state; keying off "current" (not whatever pin happens to be on the handle) keeps the cache key an honest descriptor of what was actually computed. schemaId is pinned -1 ("unversioned" for that axis): PaimonTableHandle carries no schemaId field, applySnapshot never threads one onto it, and beginQuerySnapshot's own ConnectorMvccSnapshot leaves it at the builder default (-1) for the common latest-pin path -- and collectPartitions' output does not depend on schema version in practice (paimon partition columns are immutable post-creation). - listPartitions: bypass cache when filter.isPresent() (mirrors the iceberg pattern; not currently load-bearing since paimon ignores the filter value, but keeps the two paths independent) and when the handle is unpartitioned (preserves the pre-existing "no seam call for unpartitioned tables" contract). - Paimon has NO iceberg.rest.session=user-style per-user cache-disabling convention (a paimon catalog authenticates at catalog-creation time -- Kerberos UGI / HMS principal -- not per-query session identity), so the cache is constructed unconditionally on PaimonConnector (never null there); PaimonConnectorMetadata's convenience/test ctors still pass null (no cross-query layer) for the ~15 existing direct-construction tests. - Invalidation routed from invalidateTable/invalidateDb/invalidateAll to the new cache alongside the existing latestSnapshotCache/schemaAtMemo. Tests: new PaimonConnectorMetadataPartitionViewCacheTest (7) proves same-key-hit enumerates once, distinct-latest-snapshot re-enumerates, invalidateTable/All re-enumerate, filter bypass, null-cache re-enumeration, and the unpartitioned fast path touches no remote seam; PaimonConnectorCacheTest (+2) proves the cache is always built (no session=user gate) and the 3-hook invalidation routing on the real field. Full module suite 370/370 green (1 pre-existing skip, PaimonLiveConnectivityTest). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_014ZngdXGBAXNDRe7jA42mAS
1 parent 96d5611 commit e3be4ad

4 files changed

Lines changed: 431 additions & 2 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java‎

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,9 +20,11 @@
2020
import org.apache.doris.connector.api.Connector;
2121
import org.apache.doris.connector.api.ConnectorCapability;
2222
import org.apache.doris.connector.api.ConnectorMetadata;
23+
import org.apache.doris.connector.api.ConnectorPartitionInfo;
2324
import org.apache.doris.connector.api.ConnectorSession;
2425
import org.apache.doris.connector.api.ConnectorValidationContext;
2526
import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider;
27+
import org.apache.doris.connector.cache.ConnectorPartitionViewCache;
2628
import org.apache.doris.connector.metastore.HmsMetaStoreProperties;
2729
import org.apache.doris.connector.metastore.spi.JdbcDriverSupport;
2830
import org.apache.doris.connector.metastore.spi.MetaStoreProviders;
@@ -50,6 +52,7 @@
5052
import java.util.Collections;
5153
import java.util.EnumSet;
5254
import java.util.HashMap;
55+
import java.util.List;
5356
import java.util.Map;
5457
import java.util.Optional;
5558
import java.util.OptionalLong;
@@ -126,6 +129,19 @@ public class PaimonConnector implements Connector {
126129
// Cleared wholesale on REFRESH CATALOG (the connector is rebuilt). See PaimonSchemaAtMemo.
127130
private final PaimonSchemaAtMemo schemaAtMemo = new PaimonSchemaAtMemo(PaimonSchemaAtMemo.DEFAULT_MAX_SIZE);
128131

132+
// PERF-06: cross-query DERIVED partition-view cache ("cache A", the generic ConnectorPartitionViewCache from
133+
// fe-connector-cache), layered ABOVE the raw remote catalog.listPartitions call (PaimonCatalogOps#listPartitions):
134+
// it memoizes the BUILT List<ConnectorPartitionInfo> (display-name rendering + null-sentinel normalization,
135+
// see PaimonConnectorMetadata#collectPartitions) keyed by (db, table, snapshotId, schemaId), so a repeated
136+
// query on a partitioned table skips the derived rebuild AND the remote catalog round-trip. ONE typed field
137+
// (unlike iceberg's two): paimon does not override getMvccPartitionView, so the generic MTMV model falls
138+
// back to its default listPartitions/LIST/timestamp path for paimon -- listPartitions is the only
139+
// enumeration hook to wrap. Unlike iceberg, paimon has NO session=user / per-user credential-isolation
140+
// cache-disabling convention (a paimon catalog authenticates at catalog-creation time -- Kerberos UGI /
141+
// HMS principal -- not per-query session identity), so this is constructed unconditionally: never null on
142+
// a live connector (only PaimonConnectorMetadata's convenience/test constructors pass null).
143+
private final ConnectorPartitionViewCache<List<ConnectorPartitionInfo>> partitionViewCache;
144+
129145
public PaimonConnector(Map<String, String> properties, ConnectorContext context) {
130146
this.properties = properties;
131147
// Wrap the FE-injected context so every executeAuthenticated pins the TCCL to the plugin loader (the
@@ -137,6 +153,9 @@ public PaimonConnector(Map<String, String> properties, ConnectorContext context)
137153
this::pluginAuthenticator);
138154
this.latestSnapshotCache =
139155
new PaimonLatestSnapshotCache(resolveTableCacheTtlSecond(properties), DEFAULT_TABLE_CACHE_CAPACITY);
156+
// Reads its own meta.cache.paimon.partition_view.(enable|ttl-second|capacity) from the catalog
157+
// properties via the framework's CacheSpec (default ON / 24h / 1000).
158+
this.partitionViewCache = new ConnectorPartitionViewCache<>("paimon", properties);
140159
}
141160

142161
/**
@@ -225,7 +244,7 @@ private static long resolveTableCacheTtlSecond(Map<String, String> properties) {
225244
public ConnectorMetadata getMetadata(ConnectorSession session) {
226245
return new PaimonConnectorMetadata(
227246
new PaimonCatalogOps.CatalogBackedPaimonCatalogOps(ensureCatalog()), properties, context,
228-
schemaAtMemo, latestSnapshotCache);
247+
schemaAtMemo, latestSnapshotCache, partitionViewCache);
229248
}
230249

231250
@Override
@@ -239,6 +258,9 @@ public void invalidateTable(String dbName, String tableName) {
239258
// (db,table,sysTable,branch,schemaId) and would otherwise serve a stale schema-at-snapshot after a
240259
// drop+recreate that reuses a schemaId (the memo's narrow write-once-per-schemaId assumption breaks).
241260
schemaAtMemo.invalidate(dbName, tableName);
261+
// PERF-06: also drop this table's cached derived partition-view entries (every snapshotId cached for
262+
// it), so the next listPartitions re-enumerates live.
263+
partitionViewCache.invalidateTable(dbName, tableName);
242264
}
243265

244266
/**
@@ -255,12 +277,14 @@ public void invalidateTable(String dbName, String tableName) {
255277
public void invalidateDb(String dbName) {
256278
latestSnapshotCache.invalidateDb(dbName);
257279
schemaAtMemo.invalidateDb(dbName);
280+
partitionViewCache.invalidateDb(dbName);
258281
}
259282

260283
@Override
261284
public void invalidateAll() {
262285
latestSnapshotCache.invalidateAll();
263286
schemaAtMemo.invalidateAll();
287+
partitionViewCache.invalidateAll();
264288
}
265289

266290
@Override
@@ -316,6 +340,11 @@ public Set<ConnectorCapability> getCapabilities() {
316340
ConnectorCapability.SUPPORTS_SHOW_CREATE_DDL);
317341
}
318342

343+
/** Test-only: the derived listPartitions view cache (PERF-06). Never null (paimon has no session=user gate). */
344+
ConnectorPartitionViewCache<List<ConnectorPartitionInfo>> partitionViewCacheForTest() {
345+
return partitionViewCache;
346+
}
347+
319348
private Catalog ensureCatalog() {
320349
if (catalog == null) {
321350
synchronized (this) {

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,8 @@
3232
import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec;
3333
import org.apache.doris.connector.api.pushdown.ConnectorExpression;
3434
import org.apache.doris.connector.api.scan.ConnectorPartitionValues;
35+
import org.apache.doris.connector.cache.ConnectorPartitionViewCache;
36+
import org.apache.doris.connector.cache.PartitionViewCacheKey;
3537
import org.apache.doris.connector.spi.ConnectorContext;
3638
import org.apache.doris.thrift.THiveTable;
3739
import org.apache.doris.thrift.TTableDescriptor;
@@ -95,6 +97,15 @@ public class PaimonConnectorMetadata implements ConnectorMetadata {
9597
// existing direct-construction tests compile unchanged; production goes through the 5-arg ctor.
9698
private final PaimonLatestSnapshotCache latestSnapshotCache;
9799

100+
// PERF-06: cross-query DERIVED partition-view cache A (generic ConnectorPartitionViewCache), injected by the
101+
// owning PaimonConnector; null = no cross-query derived layer (the convenience/test ctors used by ~15
102+
// existing direct-construction tests pass null). Layered ABOVE the raw remote catalogOps.listPartitions
103+
// call: a hit skips both the derived-view BUILD (collectPartitions) and the remote round-trip, keyed by
104+
// (db, table, snapshotId, schemaId). Consumed only by listPartitions -- paimon does not override
105+
// getMvccPartitionView (see ConnectorMetadata's default), so the generic MTMV model already uses
106+
// listPartitions for its LIST/timestamp partition view; there is no second hook to wrap.
107+
private final ConnectorPartitionViewCache<List<ConnectorPartitionInfo>> partitionViewCache;
108+
98109
public PaimonConnectorMetadata(PaimonCatalogOps catalogOps, Map<String, String> properties,
99110
ConnectorContext context) {
100111
this(catalogOps, properties, context, new PaimonSchemaAtMemo(PaimonSchemaAtMemo.DEFAULT_MAX_SIZE));
@@ -105,15 +116,30 @@ public PaimonConnectorMetadata(PaimonCatalogOps catalogOps, Map<String, String>
105116
this(catalogOps, properties, context, schemaAtMemo, new PaimonLatestSnapshotCache(0L, 1));
106117
}
107118

119+
/** Convenience ctor without the PERF-06 derived partition-view cache (null -> listPartitions always live). */
108120
PaimonConnectorMetadata(PaimonCatalogOps catalogOps, Map<String, String> properties,
109121
ConnectorContext context, PaimonSchemaAtMemo schemaAtMemo,
110122
PaimonLatestSnapshotCache latestSnapshotCache) {
123+
this(catalogOps, properties, context, schemaAtMemo, latestSnapshotCache, null);
124+
}
125+
126+
/**
127+
* Full ctor used by {@link PaimonConnector#getMetadata}, adding the PERF-06 derived partition-view cache
128+
* (cache A): {@code partitionViewCache} memoizes {@link #listPartitions}'s built
129+
* {@code List<ConnectorPartitionInfo>}, keyed by {@code (db, table, snapshotId, schemaId)}. {@code null}
130+
* for the convenience/test ctors (no cross-query derived layer -&gt; compute directly every call).
131+
*/
132+
PaimonConnectorMetadata(PaimonCatalogOps catalogOps, Map<String, String> properties,
133+
ConnectorContext context, PaimonSchemaAtMemo schemaAtMemo,
134+
PaimonLatestSnapshotCache latestSnapshotCache,
135+
ConnectorPartitionViewCache<List<ConnectorPartitionInfo>> partitionViewCache) {
111136
this.catalogOps = catalogOps;
112137
this.typeMappingOptions = buildTypeMappingOptions(properties);
113138
this.context = context;
114139
this.catalogProperties = properties;
115140
this.schemaAtMemo = schemaAtMemo;
116141
this.latestSnapshotCache = latestSnapshotCache;
142+
this.partitionViewCache = partitionViewCache;
117143
}
118144

119145
@Override
@@ -955,11 +981,57 @@ public List<String> listPartitionNames(ConnectorSession session, ConnectorTableH
955981
* {@code PaimonExternalCatalog.getPaimonPartitions} returns the full partition set without
956982
* pushing predicates into the Paimon catalog, and this preserves that behavior (mirrors
957983
* {@code MaxComputeConnectorMetadata}).
984+
*
985+
* <p>PERF-06 cache A: the BUILT {@code List<ConnectorPartitionInfo>} is memoized across queries in
986+
* {@link #partitionViewCache}, keyed by {@code (db, table, snapshotId, schemaId)} (see
987+
* {@link #partitionViewCacheKey}) — a hit skips both {@link #collectPartitions} and the remote
988+
* {@code catalogOps.listPartitions} round-trip. The cache is BYPASSED (compute directly, never populated)
989+
* when: {@code partitionViewCache} is {@code null} (the convenience/test ctors); {@code filter} is present
990+
* (not the pruning path, and not keyed by filter); or the handle is unpartitioned (mirrors
991+
* {@link #collectPartitions}'s own empty-partitionKeys short-circuit, so an unpartitioned table never
992+
* touches {@link #latestSnapshotCache} either — preserving the "no seam call" contract the unpartitioned
993+
* path already guarantees).
958994
*/
959995
@Override
960996
public List<ConnectorPartitionInfo> listPartitions(ConnectorSession session,
961997
ConnectorTableHandle handle, Optional<ConnectorExpression> filter) {
962-
return collectPartitions((PaimonTableHandle) handle);
998+
PaimonTableHandle paimonHandle = (PaimonTableHandle) handle;
999+
List<String> partitionKeys = paimonHandle.getPartitionKeys();
1000+
if (partitionViewCache == null || filter.isPresent()
1001+
|| partitionKeys == null || partitionKeys.isEmpty()) {
1002+
return collectPartitions(paimonHandle);
1003+
}
1004+
PartitionViewCacheKey key = partitionViewCacheKey(paimonHandle);
1005+
return partitionViewCache.get(key, () -> collectPartitions(paimonHandle));
1006+
}
1007+
1008+
/**
1009+
* Builds cache A's key for {@code paimonHandle}: {@code (db, table, snapshotId, schemaId)}.
1010+
*
1011+
* <p><b>snapshotId</b>: {@link #collectPartitions}'s remote call ({@code catalogOps.listPartitions(Identifier)})
1012+
* is BASE-identifier-only — it does not apply the handle's pinned {@code scanOptions} (unlike the scan path),
1013+
* so it always reflects the CURRENT catalog state, never a time-travel pin (branch / time-travel reads never
1014+
* reach this path at all — see {@link #collectPartitions}). The key must therefore track "current", not
1015+
* whatever snapshot happens to be threaded on the handle: it reads the SAME per-catalog
1016+
* {@link #latestSnapshotCache} that {@link #beginQuerySnapshot} pins queries to (a cheap in-memory hit within
1017+
* the query — {@code beginQuerySnapshot} already warmed it), so a repeat query within the TTL hits this cache,
1018+
* and a new snapshot (data change, once the entry expires or REFRESH invalidates it) naturally mints a new key.
1019+
*
1020+
* <p><b>schemaId</b>: pinned {@code -1} ("unversioned" for that axis, matching
1021+
* {@link PartitionViewCacheKey}'s documented convention). Unlike iceberg, paimon's {@link PaimonTableHandle}
1022+
* carries no schemaId — {@code applySnapshot} threads only {@code scanOptions} (an opaque properties map;
1023+
* see its javadoc) onto the handle, and {@link #beginQuerySnapshot} (the common latest-pin path) never
1024+
* resolves a schemaId either (its {@code ConnectorMvccSnapshot} keeps the builder default {@code -1}). This
1025+
* is not a loss for THIS view: {@link #collectPartitions} derives its output from {@code partitionKeys}
1026+
* (fixed at handle-build time) and paimon's raw partition specs, and paimon partition columns are immutable
1027+
* post-creation, so schema evolution (e.g. ADD COLUMN) does not change what this method computes.
1028+
*/
1029+
private PartitionViewCacheKey partitionViewCacheKey(PaimonTableHandle paimonHandle) {
1030+
Identifier identifier = Identifier.create(paimonHandle.getDatabaseName(), paimonHandle.getTableName());
1031+
long snapshotId = latestSnapshotCache.getOrLoad(identifier,
1032+
() -> catalogOps.latestSnapshotId(resolveTable(paimonHandle)).orElse(-1L));
1033+
return new PartitionViewCacheKey(
1034+
paimonHandle.getDatabaseName(), paimonHandle.getTableName(), snapshotId, -1L);
9631035
}
9641036

9651037
@Override

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorCacheTest.java‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,13 +17,19 @@
1717

1818
package org.apache.doris.connector.paimon;
1919

20+
import org.apache.doris.connector.api.ConnectorPartitionInfo;
21+
import org.apache.doris.connector.cache.ConnectorPartitionViewCache;
22+
import org.apache.doris.connector.cache.PartitionViewCacheKey;
23+
2024
import org.junit.jupiter.api.Assertions;
2125
import org.junit.jupiter.api.Test;
2226

2327
import java.util.Collections;
2428
import java.util.HashMap;
29+
import java.util.List;
2530
import java.util.Map;
2631
import java.util.OptionalLong;
32+
import java.util.function.Supplier;
2733

2834
/**
2935
* Tests PaimonConnector's FIX-4 cache knobs (CI 973411): the {@code meta.cache.paimon.table.ttl-second}
@@ -83,4 +89,64 @@ public void invalidateHooksAreNoThrowOnFreshConnector() {
8389
Assertions.assertDoesNotThrow(() -> connector.invalidateDb("db1"));
8490
Assertions.assertDoesNotThrow(connector::invalidateAll);
8591
}
92+
93+
// ============ PERF-06: derived partition-view cache A (no session=user gate) + invalidation ============
94+
95+
@Test
96+
public void partitionViewCacheAlwaysBuilt() {
97+
// Unlike iceberg, paimon has NO session=user / per-user credential-isolation cache-disabling
98+
// convention (a paimon catalog authenticates at catalog-creation time, not per-query session
99+
// identity), so the connector must construct cache A unconditionally on every flavor. MUTATION: a
100+
// stray session-like gate leaving the field null on some property combination -> red.
101+
Assertions.assertNotNull(connector(Collections.emptyMap()).partitionViewCacheForTest(),
102+
"a fresh paimon connector must always build the partition-view cache");
103+
Map<String, String> withTtl = props("3600");
104+
Assertions.assertNotNull(connector(withTtl).partitionViewCacheForTest(),
105+
"the partition-view cache is independent of meta.cache.paimon.table.ttl-second");
106+
}
107+
108+
@Test
109+
public void refreshHooksInvalidatePartitionViewCache() {
110+
// The REFRESH hooks must clear cache A too (else external DDL/writes stay invisible beyond the TTL):
111+
// REFRESH TABLE drops that table's snapshot entries, REFRESH DATABASE that db's, REFRESH CATALOG
112+
// everything. Asserted via a counting loader (the framework's size() is package-private): after
113+
// invalidation the loader must run again. MUTATION: an invalidate* hook not routed to the view cache
114+
// -> the entry survives -> loader not re-run -> a loads assert below red.
115+
PaimonConnector connector = connector(Collections.emptyMap());
116+
ConnectorPartitionViewCache<List<ConnectorPartitionInfo>> cache = connector.partitionViewCacheForTest();
117+
Assertions.assertNotNull(cache);
118+
int[] loads = {0};
119+
Supplier<List<ConnectorPartitionInfo>> loader = () -> {
120+
loads[0]++;
121+
return Collections.emptyList();
122+
};
123+
PartitionViewCacheKey db1t1 = new PartitionViewCacheKey("db1", "t1", 1L, -1L);
124+
PartitionViewCacheKey db1t2 = new PartitionViewCacheKey("db1", "t2", 1L, -1L);
125+
PartitionViewCacheKey db2t1 = new PartitionViewCacheKey("db2", "t1", 1L, -1L);
126+
127+
// REFRESH TABLE db1.t1 -> only db1.t1 re-loads.
128+
cache.get(db1t1, loader);
129+
cache.get(db1t1, loader);
130+
Assertions.assertEquals(1, loads[0], "second get is a hit");
131+
connector.invalidateTable("db1", "t1");
132+
cache.get(db1t1, loader);
133+
Assertions.assertEquals(2, loads[0], "REFRESH TABLE forces a reload of db1.t1");
134+
135+
// REFRESH DATABASE db1 -> db1.t2 re-loads; db2.t1 unaffected.
136+
cache.get(db1t2, loader); // loads=3 (miss)
137+
cache.get(db2t1, loader); // loads=4 (miss)
138+
cache.get(db1t2, loader); // hit
139+
cache.get(db2t1, loader); // hit
140+
Assertions.assertEquals(4, loads[0]);
141+
connector.invalidateDb("db1");
142+
cache.get(db2t1, loader); // db2 untouched -> hit
143+
Assertions.assertEquals(4, loads[0], "REFRESH DATABASE db1 must NOT drop db2's entries");
144+
cache.get(db1t2, loader); // db1.t2 dropped -> miss
145+
Assertions.assertEquals(5, loads[0], "REFRESH DATABASE db1 drops db1's entries");
146+
147+
// REFRESH CATALOG -> everything re-loads.
148+
connector.invalidateAll();
149+
cache.get(db2t1, loader);
150+
Assertions.assertEquals(6, loads[0], "REFRESH CATALOG drops everything");
151+
}
86152
}

0 commit comments

Comments
 (0)