Skip to content

Commit c954090

Browse files
morningmanclaude
andcommitted
[opt](catalog) fe-connector: unify the connector cache foundation, memoize per-statement hot paths, fix iceberg case-insensitive column DDL (#66029)
### What problem does this PR solve? Issue Number: #65185 Problem Summary: Part of the catalog-SPI migration (#65185). Review round 19, targeting `branch-catalog-spi`. Three themes plus one bug fix. **1. Connector cache foundation unification (`fe-connector-cache` / `fe-connector-api`)** Several connectors had each grown their own copy of the same two caching shapes. This round factors them out instead of copying them a fourth time: - `ConnectorPartitionViewCache` is generalized into a reusable `ConnectorMetadataCache`, so a connector can cache a derived metadata view without re-implementing the loader / invalidation plumbing. - The per-statement table memo that iceberg, paimon and hudi each open-coded is extracted into a shared `ConnectorStatementScopes` helper in `fe-connector-api`. - The "`ttl <= 0` disables the cache" contract is folded into `CacheSpec.ofConnectorTtl` rather than being re-derived at every call site. - Statement-scope namespace constants are pushed DOWN into each connector (jdbc's lands on `JdbcConnectorMetadata`); `fe-connector-api` keeps only the neutral `resolveInStatement` plus the key convention, so the API stays source-agnostic. Namespaces are prefixed with `getType()`, which makes cross-connector collisions impossible by construction, and each connector reflectively walks its own `*_NAMESPACE` constants to assert the prefix, so the check extends itself. - A fe-core test guards that the connector per-statement scope is reset across prepared executions of `ExecuteCommand`. **2. Round-1 hot-path memoization (perf)** Removes repeated remote round-trips and repeated local CPU within a single statement: | connector | change | | --- | --- | | hudi | cache the HMS client (fresh listing + REFRESH flush); memoize latest schema + instant per statement | | maxcompute | memoize the resolved table handle per statement | | es | hoist per-scan metadata state, memoize schema, and share ONE index-mapping fetch per statement across the schema and scan paths | | paimon | route `SHOW PARTITIONS` / `partition_values()` through the partition-view cache | | jdbc | dedup the remote column fetch per statement across the read and write paths | Some of these save remote IO and some save only local CPU (repeated rendering / parsing / object construction behind an already-warm lower-level cache); both are counted as real wins here. **3. Build-gate cleanup** `tools/check-authz-cache-sharding.sh` (and its self-test) are removed. A shell/regex gate cannot decide Java boolean semantics — two adversarial review rounds produced 10 and 11 false positives/negatives respectively, and a false positive is the worse failure mode because it blocks legitimate builds. The invariant it guarded is already proven by runtime behavior tests (`IcebergConnectorCacheTest`) and is now stated as an ATTN comment at the call sites. Static gates stay for existence/prefix/marker invariants only (e.g. `check-connector-imports`). **4. Fix: iceberg now rejects case-insensitive column-name collisions on the top-level ALTER ops** A port gap from #65329. Upstream `IcebergMetadataOps` called `validateNoCaseInsensitiveSiblingCollision` from five sites — flat ADD, nested ADD, the ADD COLUMNS batch, flat RENAME and nested RENAME — but the connector port carried only the two NESTED sites. Top-level column DDL never reaches that class: the fe-core bridge short-circuits every non-nested `ColumnPath` back to the flat SPI, and the connector short-circuits it again, so the flat ops went straight to `UpdateSchema` with no validation. Iceberg could not catch it either: `SchemaUpdate` sets `caseSensitive = true`, so `findField("id")` does not see an existing `Id`, while Doris column names are case-insensitive. Against a Spark-created table holding `Id INT, Label STRING, Info STRUCT<Metric:INT, Label:STRING>`: - `ALTER TABLE t ADD COLUMN id STRING` silently committed a second top-level column differing from `Id` only in case; - `ALTER TABLE t RENAME COLUMN label TO ...` failed with iceberg's raw `Cannot rename missing column: label`, because the real field is `Label`. The three top-level ops now use the validators the nested arm already had: - `addColumn` rejects a name colliding case-insensitively with an existing field. - `addColumns` does the same per column and additionally rejects two names of one request that collide with each other; the whole batch is validated before anything is staged, so a partly-valid request commits nothing. - `renameColumn` resolves the SOURCE name case-insensitively, lets a pure re-casing rename through via the field-identity escape, and restates identifier-field paths through the existing `applyRenameColumn`. `validateNoCaseInsensitiveSiblingCollision` becomes package-private; its `parentPath.isEmpty()` top-level arm had already been written but was unreachable, which is what flagged the missing call sites. Known remaining gaps in the same flat arm, deliberately left out of this PR and not exercised by any current test: flat `dropColumn` and `modifyColumn` still resolve names case-sensitively, the `AFTER` position reference is not resolved case-insensitively, and `reorderColumns` neither canonicalizes names nor rejects duplicates. ### Release note Fixed an Iceberg external-catalog issue where `ALTER TABLE ... ADD COLUMN` and `RENAME COLUMN` ignored case-insensitive collisions with existing column names, so a column differing from an existing one only in case could be added; renaming a column whose name was typed in a different case than the underlying Iceberg field also failed. ### Check List (For Author) - Test - [x] Regression test - [x] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason The iceberg fix is covered by the existing suite `external_table_p0/iceberg/test_iceberg_nested_schema_evolution_spark_doris_interop.groovy` (lines 273-297), which was failing on this branch; four of its assertions after line 273 had never been reached before. New unit tests: 7 cases in `CatalogBackedIcebergCatalogOpsColumnEvolutionTest` against a real `InMemoryCatalog` with a mixed-case fixture, verified to fail on the pre-fix code with the exact CI symptoms. The cache and perf work adds unit tests per connector (`HudiConnectorHmsCacheTest`, `HudiStatementMemoTest`, `MaxComputeConnectorMetadataHandleMemoTest`, `JdbcConnectorMetadataTest`, `JdbcWritePlanProviderTest`, `IcebergStatementScopeTest`, `PaimonConnectorMetadataPartitionViewCacheTest`, `ConnectorStatementScopeTest`, ...). - Behavior changed: - [x] Yes. Iceberg `ALTER TABLE ... ADD COLUMN` / `ADD COLUMN (...)` / `RENAME COLUMN` now fail with an explicit "conflicts with existing Iceberg field '<name>' (case-insensitive)" message where they previously succeeded (or failed with iceberg's raw message). A top-level `RENAME COLUMN` whose source name differs only in case from the underlying Iceberg field now succeeds instead of failing. - Does this need documentation? - [x] No. --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 565d31e commit c954090

102 files changed

Lines changed: 7223 additions & 614 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
Lines changed: 76 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,76 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.api;
19+
20+
import java.util.function.Supplier;
21+
22+
/**
23+
* Statement-scoped resolution helper over the neutral {@link ConnectorStatementScope} (reached via
24+
* {@link ConnectorSession#getStatementScope()}). A connector routes its "resolve {@code db.table} once per
25+
* statement" memo through {@link #resolveInStatement}, so every read / scan / write resolver of one statement
26+
* shares the single loaded value and the loader runs at most once — while a reused prepared-statement scope,
27+
* reset per execution, still isolates each execution (its {@code queryId} is part of the key).
28+
*
29+
* <p>This standardizes the security-critical key convention once instead of letting each connector re-derive it:
30+
* dropping {@code queryId} would leak a table across executions of a reused prepared statement; dropping
31+
* {@code catalogId} would collide across a cross-catalog {@code MERGE}.
32+
*
33+
* <p><b>{@code keyNamespace} is connector-owned; this module stays source-neutral.</b> {@code keyNamespace}
34+
* namespaces the value <em>type</em> stored under the shared {@code (catalogId, db, table, queryId)} coordinate,
35+
* so a heterogeneous gateway statement touching two connectors cannot collide on {@code (db, table)} and hand one
36+
* connector another's value (a {@link ClassCastException}). Each connector declares its own namespace constant in
37+
* its own module (e.g. {@code IcebergStatementScope}, {@code HudiStatementScope}, {@code EsStatementScope},
38+
* {@code JdbcConnectorMetadata}) — this API deliberately holds no source names. The convention every
39+
* {@code keyNamespace} passed to {@link #resolveInStatement} MUST follow: it is a compile-time constant prefixed
40+
* with the connector's connector-type name (its {@code ConnectorProvider.getType()}, e.g. {@code "iceberg"},
41+
* {@code "hudi"}). Because {@code getType()} is a connector's unique identity, source-prefixing makes these
42+
* namespaces distinct across connectors <em>by construction</em>; each connector's {@code *StatementScope} guards
43+
* its own prefix with a unit test.
44+
*
45+
* <p>This convention governs only the {@code keyNamespace} passed to {@link #resolveInStatement}. Other
46+
* per-statement memos that use {@link ConnectorStatementScope}'s {@code computeIfAbsent} /
47+
* {@code getOrCreateMetadata} directly own their own key discipline and are out of scope here — e.g. an
48+
* engine-reserved family that stores a single value type per key, discriminated by catalog id, stays collision-safe
49+
* without a source prefix. A connector that instead memoizes on its own per-statement metadata instance needs no
50+
* namespace at all.
51+
*/
52+
public final class ConnectorStatementScopes {
53+
54+
private ConnectorStatementScopes() {
55+
}
56+
57+
/**
58+
* Resolves {@code db.table} once per statement and shares the single value across every resolver of the
59+
* statement. The key is {@code keyNamespace + ":" + catalogId + ":" + db + ":" + table + ":" + queryId}: the
60+
* catalog id isolates a cross-catalog {@code MERGE}, the {@code queryId} isolates each execution of a reused
61+
* prepared statement, and {@code keyNamespace} isolates value types across a heterogeneous gateway.
62+
* {@code keyNamespace} MUST be a connector-owned, source-prefixed constant (see the class javadoc).
63+
* {@code loader} runs at most once per statement; under a {@code null} session or
64+
* {@link ConnectorStatementScope#NONE} (offline / no live statement) it runs on every call — byte-identical
65+
* to loading every time.
66+
*/
67+
public static <T> T resolveInStatement(ConnectorSession session, String keyNamespace,
68+
String db, String table, Supplier<T> loader) {
69+
if (session == null) {
70+
return loader.get();
71+
}
72+
String key = keyNamespace + ":" + session.getCatalogId() + ":" + db + ":" + table
73+
+ ":" + session.getQueryId();
74+
return session.getStatementScope().computeIfAbsent(key, loader);
75+
}
76+
}
Lines changed: 225 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,225 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.api;
19+
20+
import org.junit.jupiter.api.Assertions;
21+
import org.junit.jupiter.api.Test;
22+
23+
import java.util.Collections;
24+
import java.util.Map;
25+
import java.util.concurrent.ConcurrentHashMap;
26+
import java.util.concurrent.atomic.AtomicInteger;
27+
import java.util.function.Supplier;
28+
29+
/**
30+
* Tests for {@link ConnectorStatementScopes#resolveInStatement}: the shared per-statement resolver whose key is
31+
* {@code keyNamespace + ":" + catalogId + ":" + db + ":" + table + ":" + queryId}. It proves the memo collapses
32+
* repeat resolves within a statement, that each key axis (namespace / catalog id / db / table / queryId) isolates,
33+
* that a null session or {@link ConnectorStatementScope#NONE} degrades to load-every-time, and that the emitted
34+
* key is exactly the documented string (the byte contract every connector consumer depends on).
35+
*/
36+
public class ConnectorStatementScopesTest {
37+
38+
@Test
39+
public void sameCoordinateResolvesOncePerStatement() {
40+
// Same (namespace, catalog, db, table, queryId) within one scope -> loader runs once, one shared instance.
41+
// MUTATION: not memoizing -> two loads / two instances -> red.
42+
RecordingMemoScope scope = new RecordingMemoScope();
43+
TestSession session = new TestSession(7L, "q1", scope);
44+
AtomicInteger loads = new AtomicInteger();
45+
Object a = ConnectorStatementScopes.resolveInStatement(session, "x.table", "db1", "t", () -> {
46+
loads.incrementAndGet();
47+
return new Object();
48+
});
49+
Object b = ConnectorStatementScopes.resolveInStatement(session, "x.table", "db1", "t", () -> {
50+
loads.incrementAndGet();
51+
return new Object();
52+
});
53+
Assertions.assertSame(a, b, "same coordinate -> one shared instance");
54+
Assertions.assertEquals(1, loads.get(), "loaded once per statement");
55+
}
56+
57+
@Test
58+
public void differentQueryIdIsolatesTheLoad() {
59+
// A reused prepared statement runs each execution under its own queryId; the memo never crosses executions.
60+
RecordingMemoScope scope = new RecordingMemoScope();
61+
Object a = ConnectorStatementScopes.resolveInStatement(
62+
new TestSession(7L, "q1", scope), "x.table", "db1", "t", Object::new);
63+
Object b = ConnectorStatementScopes.resolveInStatement(
64+
new TestSession(7L, "q2", scope), "x.table", "db1", "t", Object::new);
65+
Assertions.assertNotSame(a, b, "different queryId -> isolated load");
66+
}
67+
68+
@Test
69+
public void differentCatalogIdIsolatesTheLoad() {
70+
// A cross-catalog MERGE resolves the two catalogs' tables independently (the key carries the catalog id).
71+
RecordingMemoScope scope = new RecordingMemoScope();
72+
Object a = ConnectorStatementScopes.resolveInStatement(
73+
new TestSession(1L, "q1", scope), "x.table", "db1", "t", Object::new);
74+
Object b = ConnectorStatementScopes.resolveInStatement(
75+
new TestSession(2L, "q1", scope), "x.table", "db1", "t", Object::new);
76+
Assertions.assertNotSame(a, b, "different catalog id -> isolated load");
77+
}
78+
79+
@Test
80+
public void differentDbOrTableIsolatesTheLoad() {
81+
RecordingMemoScope scope = new RecordingMemoScope();
82+
TestSession session = new TestSession(7L, "q1", scope);
83+
Object t1 = ConnectorStatementScopes.resolveInStatement(session, "x.table", "db1", "t", Object::new);
84+
Object t2 = ConnectorStatementScopes.resolveInStatement(session, "x.table", "db1", "u", Object::new);
85+
Object t3 = ConnectorStatementScopes.resolveInStatement(session, "x.table", "db2", "t", Object::new);
86+
Assertions.assertNotSame(t1, t2, "different table -> isolated load");
87+
Assertions.assertNotSame(t1, t3, "different db -> isolated load");
88+
}
89+
90+
@Test
91+
public void differentNamespaceIsolatesValueTypes() {
92+
// The namespace guards a heterogeneous-gateway statement: two connectors sharing (db, table, catalog,
93+
// queryId) but storing different value types must not collide. Without namespacing, the second resolve
94+
// would return the first's memoized value cast to the wrong type -> ClassCastException.
95+
// MUTATION: dropping keyNamespace from the key -> ClassCastException here -> red.
96+
RecordingMemoScope scope = new RecordingMemoScope();
97+
TestSession session = new TestSession(7L, "q1", scope);
98+
String asString = ConnectorStatementScopes.resolveInStatement(
99+
session, "a.value", "db1", "t", () -> "stringValue");
100+
Integer asInt = ConnectorStatementScopes.resolveInStatement(
101+
session, "b.value", "db1", "t", () -> 42);
102+
Assertions.assertEquals("stringValue", asString, "namespace a keeps its String value");
103+
Assertions.assertEquals(Integer.valueOf(42), asInt, "namespace b keeps its Integer value (no collision)");
104+
}
105+
106+
@Test
107+
public void nullSessionLoadsEveryTime() {
108+
// No session (offline / direct-construction tests): each call loads, byte-identical to loading every time.
109+
AtomicInteger loads = new AtomicInteger();
110+
ConnectorStatementScopes.resolveInStatement(null, "x.table", "db1", "t", () -> {
111+
loads.incrementAndGet();
112+
return new Object();
113+
});
114+
ConnectorStatementScopes.resolveInStatement(null, "x.table", "db1", "t", () -> {
115+
loads.incrementAndGet();
116+
return new Object();
117+
});
118+
Assertions.assertEquals(2, loads.get(), "null session -> load every time");
119+
}
120+
121+
@Test
122+
public void noneScopeLoadsEveryTime() {
123+
// A live session whose scope is NONE (no per-statement context) also loads every time.
124+
TestSession session = new TestSession(7L, "q1", ConnectorStatementScope.NONE);
125+
AtomicInteger loads = new AtomicInteger();
126+
ConnectorStatementScopes.resolveInStatement(session, "x.table", "db1", "t", () -> {
127+
loads.incrementAndGet();
128+
return new Object();
129+
});
130+
ConnectorStatementScopes.resolveInStatement(session, "x.table", "db1", "t", () -> {
131+
loads.incrementAndGet();
132+
return new Object();
133+
});
134+
Assertions.assertEquals(2, loads.get(), "NONE scope -> load every time");
135+
}
136+
137+
@Test
138+
public void keyIsNamespaceCatalogDbTableQueryId() {
139+
// The exact byte contract: keyNamespace + ":" + catalogId + ":" + db + ":" + table + ":" + queryId.
140+
// Every connector consumer's cross-statement isolation depends on this string; assert it verbatim.
141+
RecordingMemoScope scope = new RecordingMemoScope();
142+
ConnectorStatementScopes.resolveInStatement(
143+
new TestSession(7L, "q1", scope), "test.ns", "db1", "t", Object::new);
144+
Assertions.assertEquals("test.ns:7:db1:t:q1", scope.lastKey, "emitted key must match the documented format");
145+
}
146+
147+
/** A statement scope that memoizes like the engine's real one and records the last key, for the assertions. */
148+
private static final class RecordingMemoScope implements ConnectorStatementScope {
149+
private final ConcurrentHashMap<String, Object> cache = new ConcurrentHashMap<>();
150+
private String lastKey;
151+
152+
@Override
153+
@SuppressWarnings("unchecked")
154+
public <T> T computeIfAbsent(String key, Supplier<T> loader) {
155+
lastKey = key;
156+
return (T) cache.computeIfAbsent(key, k -> loader.get());
157+
}
158+
}
159+
160+
/** Minimal {@link ConnectorSession} carrying a catalog id, queryId and scope for the key + memo assertions. */
161+
private static final class TestSession implements ConnectorSession {
162+
private final long catalogId;
163+
private final String queryId;
164+
private final ConnectorStatementScope scope;
165+
166+
TestSession(long catalogId, String queryId, ConnectorStatementScope scope) {
167+
this.catalogId = catalogId;
168+
this.queryId = queryId;
169+
this.scope = scope;
170+
}
171+
172+
@Override
173+
public long getCatalogId() {
174+
return catalogId;
175+
}
176+
177+
@Override
178+
public String getQueryId() {
179+
return queryId;
180+
}
181+
182+
@Override
183+
public String getSessionId() {
184+
// Deliberately != queryId so keyIsNamespaceCatalogDbTableQueryId pins the per-execution queryId
185+
// (cross-query isolation), not the stable per-connection sessionId; a queryId->sessionId swap in the
186+
// helper's key would share a table across queries of one connection and MUST turn that assertion red.
187+
return "session-" + queryId;
188+
}
189+
190+
@Override
191+
public ConnectorStatementScope getStatementScope() {
192+
return scope;
193+
}
194+
195+
@Override
196+
public String getUser() {
197+
return "u";
198+
}
199+
200+
@Override
201+
public String getTimeZone() {
202+
return "UTC";
203+
}
204+
205+
@Override
206+
public String getLocale() {
207+
return "en_US";
208+
}
209+
210+
@Override
211+
public String getCatalogName() {
212+
return "c";
213+
}
214+
215+
@Override
216+
public <T> T getProperty(String name, Class<T> type) {
217+
return null;
218+
}
219+
220+
@Override
221+
public Map<String, String> getCatalogProperties() {
222+
return Collections.emptyMap();
223+
}
224+
}
225+
}

‎fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheSpec.java‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -67,6 +67,23 @@ public static CacheSpec of(boolean enable, long ttlSecond, long capacity) {
6767
return new CacheSpec(enable, ttlSecond, capacity);
6868
}
6969

70+
/**
71+
* Build an ENABLED spec from a connector-resolved TTL under the "{@code <= 0} disables" contract.
72+
*
73+
* <p>A connector that resolves its own single {@code ttl-second} knob (iceberg's shared
74+
* {@code meta.cache.iceberg.table.ttl-second}, paimon's snapshot cache) treats any non-positive TTL as
75+
* "disable caching, always read live". That is NOT the raw {@link CacheSpec} contract, which reads
76+
* {@code ttlSecond == -1} as {@link #CACHE_NO_TTL} ("no expiration", still ENABLED) and only
77+
* {@code ttlSecond == 0} as {@link #CACHE_TTL_DISABLE_CACHE} ("disabled"). This factory folds any
78+
* non-positive TTL to the disable sentinel so a negative operator value disables the cache rather than
79+
* silently becoming a never-expiring one. It is exactly the
80+
* {@code ttlSecond > 0 ? of(true, ttlSecond, capacity) : of(true, CACHE_TTL_DISABLE_CACHE, capacity)}
81+
* expression each per-catalog cache used to inline.
82+
*/
83+
public static CacheSpec ofConnectorTtl(long ttlSecond, long capacity) {
84+
return of(true, ttlSecond > 0 ? ttlSecond : CACHE_TTL_DISABLE_CACHE, capacity);
85+
}
86+
7087
public static PropertySpec.Builder propertySpecBuilder() {
7188
return new PropertySpec.Builder();
7289
}

0 commit comments

Comments
 (0)