Skip to content

Commit 0944b63

Browse files
committed
[perf](catalog) two-level cross-query cache for external partition derived views (#65829)
### What & why The catalog-SPI cutover routes every external partition lookup through the connector SPI per query, dropping the cross-query partition caching the legacy fe-core path (CACHE-P1) provided: a repeated query on a partitioned external table re-enumerated the remote partition set **and** rebuilt the derived partition view / `SortedPartitionRanges` from scratch every time. This PR restores that caching for the iceberg, paimon and hive connectors without breaking the one-way SPI isolation (`fe-connector → fe-core`) and without re-introducing the #65659 per-statement partition TOCTOU. It uses two independent layers, both consulted only at the frozen per-statement prune, so neither can serve data inconsistent with the pinned snapshot: **Cache A — connector derived-partition-view cache (below the SPI line).** A generic, engine-agnostic `ConnectorPartitionViewCache<V>` keyed by `(db, table, snapshotId, schemaId)`, modeled on the existing `IcebergPartitionCache` pattern (contextual-only, manual-miss-load, `CacheSpec`-driven config). It caches the *derived* view (display-name rendering, null-sentinel normalization, `ConnectorPartitionInfo` construction) — so a repeat query skips the BUILD, not only the round-trip. Wired into iceberg (S3), paimon (S4) and hive (S6). Iceberg `session=user` catalogs null the cache to keep the per-user authorization isolation from #65785; paimon and hive have no per-query session identity and build it unconditionally. **Cache B — fe-core `SortedPartitionRanges` reuse.** External MVCC tables now implement `SupportBinarySearchFilteringPartitions`, routing through the same `NereidsSortedPartitionsCacheManager` native tables use. The version token is the **frozen partition name-set** read from the same pinned snapshot the scan reads, so a cache hit can only return ranges built from the exact partition set the current statement prunes against — uniform across snapshot-bearing (iceberg/paimon) and snapshot-less (hive) connectors, with no generation counter and no SPI change. `PruneFileScanPartition` consults it strictly as a fallback between the frozen `scan.getSelectedPartitions()` field and the build-fresh path, leaving the #65659 TOCTOU guard intact. Invalidation is routed from `ExternalMetaCacheMgr.invalidateTable/Db/Catalog/removeCatalog` so REFRESH and metastore events drop both layers. ### How verified Per-connector cache tests (iceberg / paimon / hive) assert a same-key repeat enumerates the remote seam exactly once, a distinct snapshot / name-set re-enumerates, and REFRESH invalidation re-enumerates. fe-core `NereidsSortedPartitionsCacheManagerExternalTest` and `BinarySearchPartitionInconsistencyTest` prove a cache hit returns the same ranges, rebuilds on a partition-set change, and never breaks the #65659 within-query consistency invariant. The iceberg / paimon / hive connector suites and the fe-core partition-cache suites are green. Part of the catalog-SPI migration tracked in #65185.
1 parent 7ac3ca7 commit 0944b63

26 files changed

Lines changed: 3205 additions & 28 deletions

File tree

‎.gitleaks.toml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,12 @@ useDefault = true
66
[[rules]]
77
id = "generic-api-key"
88

9+
[[rules.allowlists]]
10+
description = "Ignore prose 'version token: iceberg/paimon' in code comments/plan-doc — a cache version-token discussion, not a credential (generic-api-key false positive: 'iceberg/paimon' entropy 3.52 barely clears the 3.5 threshold)"
11+
condition = "AND"
12+
regexTarget = "match"
13+
regexes = ['''token:\s*iceberg/paimon''']
14+
915
[[rules.allowlists]]
1016
description = "Ignore the fake OIDC token fixture in AuthenticatorManagerTest"
1117
condition = "AND"
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.cache;
19+
20+
import java.util.Collections;
21+
import java.util.Map;
22+
import java.util.Objects;
23+
import java.util.concurrent.ForkJoinPool;
24+
import java.util.function.Supplier;
25+
26+
/**
27+
* GENERIC (engine-agnostic) cross-query cache of a connector's derived partition view — "cache A" of the
28+
* external-partition-derived-cache design (design doc {@code 2026-07-20-external-partition-derived-cache-design.md}
29+
* §5). It is the generic version of {@code org.apache.doris.connector.iceberg.IcebergPartitionCache}: same
30+
* construction pattern (a contextual-only, manual-miss-load {@link MetaCacheEntry}), same {@link CacheSpec} wiring,
31+
* same invalidation style — but keyed by the engine-agnostic {@link PartitionViewCacheKey} and holding an opaque
32+
* value {@code V} instead of an iceberg-specific raw-partition list, so it has no engine-specific imports and can be
33+
* shared by every connector (iceberg/paimon/hive/maxcompute, wired in later subsystems — this class has NO
34+
* consumers yet).
35+
*
36+
* <p><b>Config</b>: {@code meta.cache.<engine>.partition_view.(enable|ttl-second|capacity)}, default ON / 86400s /
37+
* 1000 entries (matching {@code IcebergPartitionCache}'s {@code DEFAULT_TABLE_CACHE_CAPACITY}). {@code enable=false}
38+
* / {@code ttl-second=0} / {@code capacity=0} each disable the cache (see {@link CacheSpec#isCacheEnabled}): {@link
39+
* #get} then calls the loader on every call, matching {@code IcebergPartitionCache}'s disabled-cache bypass.
40+
*
41+
* <p><b>Concurrency</b>: mirrors {@code IcebergPartitionCache} / {@code MaxComputePartitionCache} exactly — the
42+
* entry is contextual-only (no built-in loader; the caller supplies one per {@link #get} call) with manual miss
43+
* load, so a slow remote enumeration runs OUTSIDE Caffeine's compute lock (deduplicated per key by a striped lock)
44+
* on the calling thread, and its exception propagates to the caller unwrapped without poisoning the cache.
45+
*/
46+
public final class ConnectorPartitionViewCache<V> {
47+
48+
/** {@code meta.cache.<engine>.partition_view.*} — the property-namespace entry name for this cache. */
49+
static final String ENTRY_PARTITION_VIEW = "partition_view";
50+
/** Default TTL: 24h, matching the design doc's stated default and sibling caches' 24h TTL. */
51+
static final long DEFAULT_TTL_SECOND = 86400L;
52+
/** Default capacity, matching {@code IcebergPartitionCache}'s {@code DEFAULT_TABLE_CACHE_CAPACITY}. */
53+
static final long DEFAULT_CAPACITY = 1000L;
54+
55+
private final MetaCacheEntry<PartitionViewCacheKey, V> entry;
56+
57+
/**
58+
* @param engine engine token for the {@code meta.cache.<engine>.partition_view.*} property namespace, e.g.
59+
* {@code "iceberg"}/{@code "paimon"}/{@code "hive"}/{@code "max_compute"}.
60+
* @param props the catalog properties; drives the {@link CacheSpec} (enable/ttl-second/capacity). May be
61+
* {@code null}, treated as empty (defaults apply).
62+
*/
63+
public ConnectorPartitionViewCache(String engine, Map<String, String> props) {
64+
Objects.requireNonNull(engine, "engine can not be null");
65+
Map<String, String> properties = props == null ? Collections.emptyMap() : props;
66+
CacheSpec spec = CacheSpec.fromProperties(properties, engine, ENTRY_PARTITION_VIEW,
67+
CacheSpec.of(true, DEFAULT_TTL_SECOND, DEFAULT_CAPACITY));
68+
// contextual-only (loader == null, supplied per-call by get()) + manual-miss-load, no auto-refresh --
69+
// identical shape to IcebergPartitionCache / MaxComputePartitionCache's entry construction.
70+
this.entry = new MetaCacheEntry<>(engine + "." + ENTRY_PARTITION_VIEW, null, spec,
71+
ForkJoinPool.commonPool(), false, true, 0L, true);
72+
}
73+
74+
/** Caching is on only when the resolved {@link CacheSpec} is effectively enabled (see {@link #entry}'s spec). */
75+
public boolean isEnabled() {
76+
return entry.stats().isEffectiveEnabled();
77+
}
78+
79+
/**
80+
* Returns the cached value for {@code key} if present, else runs {@code loader}, caches and returns it.
81+
* Disabled cache -&gt; {@code loader} runs on every call. The loader runs OUTSIDE Caffeine's compute lock
82+
* (single-flight per key) and its exception propagates unwrapped; a failed load is never cached.
83+
*/
84+
public V get(PartitionViewCacheKey key, Supplier<V> loader) {
85+
Objects.requireNonNull(loader, "loader can not be null");
86+
return entry.get(key, ignored -> loader.get());
87+
}
88+
89+
/** Evicts every cached snapshot/schema entry of one (db, table). Backs REFRESH TABLE / table-level invalidation. */
90+
public void invalidateTable(String db, String table) {
91+
entry.invalidateIf(key -> key.matches(db, table));
92+
}
93+
94+
/** Evicts every cached entry of one db (all its tables). Backs REFRESH DATABASE / db-level invalidation. */
95+
public void invalidateDb(String db) {
96+
entry.invalidateIf(key -> key.matchesDb(db));
97+
}
98+
99+
/** Evicts every cached entry. Backs REFRESH CATALOG. */
100+
public void invalidateAll() {
101+
entry.invalidateAll();
102+
}
103+
104+
/** Test-only: current number of cached entries (accurate map membership, not Caffeine's estimate). */
105+
long size() {
106+
int[] count = {0};
107+
entry.forEach((key, value) -> count[0]++);
108+
return count[0];
109+
}
110+
}
Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.cache;
19+
20+
import java.util.Objects;
21+
22+
/**
23+
* Immutable cache key for {@link ConnectorPartitionViewCache}: {@code (db, table, snapshotId, schemaId)}.
24+
*
25+
* <p>Engine-agnostic (external-partition-derived-cache design doc §5, "cache A"): a table's derived partition
26+
* view is a pure function of its identity plus the MVCC coordinate it was read at, so pinning that coordinate
27+
* into the key is what makes the cache "always correct" — a new snapshot/schema yields a new key, never a stale
28+
* hit. Non-MVCC engines (hive) or engines without a separate schema version pass {@code snapshotId = -1} /
29+
* {@code schemaId = -1}; the key still holds them, it just means "unversioned" for that axis.
30+
*
31+
* <p>{@link #matches} / {@link #matchesDb} back {@link ConnectorPartitionViewCache#invalidateTable} /
32+
* {@link ConnectorPartitionViewCache#invalidateDb}, which must drop every snapshot/schema of a (db, table) or
33+
* every table of a db — mirrors the {@code matches}/{@code matchesDb} helpers on the sibling connector caches
34+
* ({@code MaxComputePartitionCache.PartitionKey}, {@code HiveFileListingCache.FileListingKey}).
35+
*/
36+
public final class PartitionViewCacheKey {
37+
private final String db;
38+
private final String table;
39+
private final long snapshotId;
40+
private final long schemaId;
41+
42+
public PartitionViewCacheKey(String db, String table, long snapshotId, long schemaId) {
43+
this.db = db;
44+
this.table = table;
45+
this.snapshotId = snapshotId;
46+
this.schemaId = schemaId;
47+
}
48+
49+
public String getDb() {
50+
return db;
51+
}
52+
53+
public String getTable() {
54+
return table;
55+
}
56+
57+
public long getSnapshotId() {
58+
return snapshotId;
59+
}
60+
61+
public long getSchemaId() {
62+
return schemaId;
63+
}
64+
65+
/** Whether this key belongs to the given (db, table), regardless of snapshotId/schemaId. */
66+
public boolean matches(String db, String table) {
67+
return Objects.equals(this.db, db) && Objects.equals(this.table, table);
68+
}
69+
70+
/** Whether this key belongs to the given db, regardless of table/snapshotId/schemaId. */
71+
public boolean matchesDb(String db) {
72+
return Objects.equals(this.db, db);
73+
}
74+
75+
@Override
76+
public boolean equals(Object o) {
77+
if (this == o) {
78+
return true;
79+
}
80+
if (!(o instanceof PartitionViewCacheKey)) {
81+
return false;
82+
}
83+
PartitionViewCacheKey that = (PartitionViewCacheKey) o;
84+
return snapshotId == that.snapshotId
85+
&& schemaId == that.schemaId
86+
&& Objects.equals(db, that.db)
87+
&& Objects.equals(table, that.table);
88+
}
89+
90+
@Override
91+
public int hashCode() {
92+
return Objects.hash(db, table, snapshotId, schemaId);
93+
}
94+
95+
@Override
96+
public String toString() {
97+
return "PartitionViewCacheKey{db=" + db + ", table=" + table
98+
+ ", snapshotId=" + snapshotId + ", schemaId=" + schemaId + '}';
99+
}
100+
}

0 commit comments

Comments
 (0)