Skip to content

Commit 67a9b9d

Browse files
morningmanclaude
andcommitted
fix: FIX-CREATE-TABLE-LOCAL-CONFLICT — restore legacy local-conflict rejection in PluginDrivenExternalCatalog.createTable
Root cause: the generic fe-core bridge PluginDrivenExternalCatalog.createTable collapsed legacy PaimonMetadataOps.performCreateTable's ordered remote-then-local existence probe into a single `exists` OR that was consumed ONLY by the IF NOT EXISTS branch. The !IF NOT EXISTS path ignored it and unconditionally called metadata.createTable. So a table present only in the local FE cache (a case-variant folded onto an existing name under lower_case_meta_names, absent on a case-sensitive remote) was CREATED remotely instead of rejected with ERR_TABLE_EXISTS_ERROR -- silent metadata corruption. Found by the P3 plugin-vs-legacy parity audit (adversarially verified); narrow, backend-dependent trigger (filesystem/jdbc paimon; HMS lowercases so both sides reject). Generic bridge -> also affects MaxCompute / future iceberg/hudi. Solution (fe-core bridge only; zero SPI/connector/BE): split the `exists` OR into remoteExists/localExists; under !IF NOT EXISTS, when localExists is true throw ERR_TABLE_EXISTS_ERROR (legacy local-arm parity). A remote-only conflict still falls through to connector.createTable (case A unchanged). Option-2 surgical (D-056); the residual case-A / all-DDL-op generic-error-code collapse is pre-existing and out of scope (DV-034). Tests: new PluginDrivenExternalCatalogDdlRoutingTest .testCreateTableLocalConflictWithoutIfNotExistsRejects (local-hit + remote-miss + !IF NOT EXISTS -> asserts DdlException thrown + metadata.createTable never called + no edit log). fail-before: exactly 1 new test red ("Expected DdlException...nothing was thrown"); pass-after: 26/0/0. fe-core checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 673f45b commit 67a9b9d

6 files changed

Lines changed: 272 additions & 16 deletions

File tree

‎fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java‎

Lines changed: 21 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,8 @@
1919

2020
import org.apache.doris.catalog.Env;
2121
import org.apache.doris.common.DdlException;
22+
import org.apache.doris.common.ErrorCode;
23+
import org.apache.doris.common.ErrorReport;
2224
import org.apache.doris.common.UserException;
2325
import org.apache.doris.connector.ConnectorFactory;
2426
import org.apache.doris.connector.ConnectorSessionBuilder;
@@ -290,16 +292,26 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException
290292
// short-circuit (Env.createTable contract: return true when the table already exists), so a
291293
// "CREATE TABLE IF NOT EXISTS ... AS SELECT" does NOT fall through to an INSERT into the
292294
// pre-existing table. The table name is intentionally NOT remote-resolved (legacy parity).
293-
boolean exists = metadata.getTableHandle(session, db.getRemoteName(),
294-
createTableInfo.getTableName()).isPresent()
295-
|| db.getTableNullable(createTableInfo.getTableName()) != null;
296-
if (exists && createTableInfo.isIfNotExists()) {
297-
LOG.info("create table[{}.{}.{}] which already exists; skipping (IF NOT EXISTS)",
298-
getName(), createTableInfo.getDbName(), createTableInfo.getTableName());
299-
return true;
295+
boolean remoteExists = metadata.getTableHandle(session, db.getRemoteName(),
296+
createTableInfo.getTableName()).isPresent();
297+
boolean localExists = db.getTableNullable(createTableInfo.getTableName()) != null;
298+
if (remoteExists || localExists) {
299+
if (createTableInfo.isIfNotExists()) {
300+
LOG.info("create table[{}.{}.{}] which already exists; skipping (IF NOT EXISTS)",
301+
getName(), createTableInfo.getDbName(), createTableInfo.getTableName());
302+
return true;
303+
}
304+
// !IF NOT EXISTS: a table present ONLY in the local FE cache (folded onto an existing name
305+
// under lower_case_meta_names while the case-sensitive remote has no such table) must be
306+
// rejected HERE -- connector.createTable would otherwise CREATE it remotely instead of
307+
// failing. Mirrors legacy PaimonMetadataOps.performCreateTable:206-214 (local arm). A
308+
// remote-only conflict still falls through to connector.createTable, which throws
309+
// "already exists" -> DdlException (unchanged).
310+
if (localExists) {
311+
ErrorReport.reportDdlException(ErrorCode.ERR_TABLE_EXISTS_ERROR,
312+
createTableInfo.getTableName());
313+
}
300314
}
301-
// existing + !IF NOT EXISTS falls through to connector.createTable, which throws
302-
// "already exists" -> DdlException (unchanged); only the IF NOT EXISTS hit short-circuits.
303315
ConnectorCreateTableRequest request = CreateTableInfoToConnectorRequestConverter
304316
.convert(createTableInfo, db.getRemoteName());
305317
try {

‎fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -551,6 +551,42 @@ public void testCreateTableExistingTableWithoutIfNotExistsStillErrors() throws E
551551
}
552552
}
553553

554+
@Test
555+
public void testCreateTableLocalConflictWithoutIfNotExistsRejects() throws Exception {
556+
// Remote says ABSENT (getTableHandle empty) but the FE cache HAS the table -- the local arm of the
557+
// legacy remote-then-local probe (PaimonMetadataOps.performCreateTable:206-214). Under
558+
// lower_case_meta_names a case-variant name folds onto an existing local table while the
559+
// case-sensitive remote has no such table. Legacy throws ERR_TABLE_EXISTS_ERROR here; the bridge
560+
// must NOT fall through to metadata.createTable, which would CREATE a duplicate remote table
561+
// (silent metadata corruption).
562+
ExternalDatabase<? extends ExternalTable> db = mockExternalDatabase();
563+
Mockito.when(db.getRemoteName()).thenReturn("DB1");
564+
Mockito.doReturn(Mockito.mock(ExternalTable.class)).when(db).getTableNullable("t1");
565+
catalog.dbNullableResult = db;
566+
Mockito.when(metadata.getTableHandle(session, "DB1", "t1")).thenReturn(Optional.empty());
567+
568+
try (MockedStatic<CreateTableInfoToConnectorRequestConverter> conv =
569+
Mockito.mockStatic(CreateTableInfoToConnectorRequestConverter.class)) {
570+
ConnectorCreateTableRequest req = Mockito.mock(ConnectorCreateTableRequest.class);
571+
conv.when(() -> CreateTableInfoToConnectorRequestConverter.convert(Mockito.any(), Mockito.any()))
572+
.thenReturn(req);
573+
CreateTableInfo info = Mockito.mock(CreateTableInfo.class);
574+
Mockito.when(info.getDbName()).thenReturn("db1");
575+
Mockito.when(info.getTableName()).thenReturn("t1");
576+
Mockito.when(info.isIfNotExists()).thenReturn(false);
577+
578+
// WHY (Rule 9 / Rule 12): a local-ONLY conflict without IF NOT EXISTS must be REJECTED at the FE
579+
// level (ERR_TABLE_EXISTS_ERROR), never handed to connector.createTable. The pre-fix code
580+
// consumed the existence probe only in the IF NOT EXISTS branch and fell through here, calling
581+
// metadata.createTable -> created a duplicate remote table. Mutation (drop the localExists
582+
// guard) -> no throw + createTable called -> both assertions go red.
583+
DdlException ex = Assertions.assertThrows(DdlException.class, () -> catalog.createTable(info));
584+
Assertions.assertTrue(ex.getMessage().contains("already exists"));
585+
Mockito.verify(metadata, Mockito.never()).createTable(Mockito.any(), Mockito.any());
586+
Mockito.verify(mockEditLog, Mockito.never()).logCreateTable(Mockito.any());
587+
}
588+
}
589+
554590
// ==================== helpers ====================
555591

556592
@SuppressWarnings("unchecked")

‎plan-doc/decisions-log.md‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
1616
| 编号 | 别名 | 简述 | 日期 | 状态 |
1717
|---|---|---|---|---|
18+
| D-056 | P3-fix | **FIX-CREATE-TABLE-LOCAL-CONFLICT(P3 覆盖缺口核查揪出的真分歧,MAJOR correctness)= fix-now(用户签字,2026-06-12)+ Option-2 外科最小修(仅补 local-conflict 闸,不动 remote-hit 路)**:P3「去查」对抗 review(`wf_25450c36-b7a`,4 项 plugin-vs-legacy paimon parity;tracer→对抗 verifier→completeness critic)3 项 PARITY_HOLDS(HMS-CONFRES:key 拼写恰 `hive.conf.resources` 无 `config.resources` 别名、round-1 wiring 在、BE-downflow 两侧同——legacy HMS hive-site.xml 本就不入 BE scan props;ANALYZE/列统计:`getColumnStatistic` 两侧 `Optional.empty()`、`createAnalysisTask` byte-同;split-count:post-sub-split 数经共享父 `FileQueryScanNode.selectedSplitNum` 喂 `SqlBlockRuleMgr` 两侧同,2 项 cosmetic/NIT 且 pre-date #9),唯 DDL 写揪出真分歧:通用 fe-core 桥 `PluginDrivenExternalCatalog.createTable` 把 legacy `PaimonMetadataOps.performCreateTable:182-214` 的**有序双探**(先 remote `tableExist:190` 后 local `getTableNullable:206`,任一命中+`!IF NOT EXISTS`→`ERR_TABLE_EXISTS_ERROR` 1050)**合并成单 `exists` OR**(`:293-295`),且 `exists` **只被 IF NOT EXISTS 臂消费**(`:296`)→`!IF NOT EXISTS` 臂(`:303-309`)忽略它无条件调 `metadata.createTable`。后果:**local-cache 命中但 remote 缺**(`lower_case_meta_names` 下 case-variant 折叠到既有本地表、case-sensitive remote 无此表)+`!IF NOT EXISTS` 时,legacy 报 1050 拒绝、plugin **静默在 remote 建重复表**(元数据腐败)。触发窄+backend-dependent(filesystem/jdbc case-sensitive 才中;HMS 小写化两侧都拒)但 silent correctness。**通用桥**(paimon+MaxCompute+未来 iceberg/hudi 共用)→修一处跨连接器收口。**修=纯 fe-core 桥、零 SPI/连接器/BE**:单 OR 拆回 `remoteExists`/`localExists` 两臂,`!IF NOT EXISTS` 下 `localExists`→`ErrorReport.reportDdlException(ERR_TABLE_EXISTS_ERROR,name)`(legacy local 臂逐字);remote-only 仍 fall-through 连接器抛(case-A 不动、既有 intentional 测绿)。**否决 Option 1 full-parity**(对整 `exists&&!ifNotExists` retype 1050)——改非分歧 case-A+破既有测+越界;case-A error-code-generic 是 pre-existing 跨全 DDL op cosmetic 残留=[DV-034]。守门:fe-core `PluginDrivenExternalCatalogDdlRoutingTest` **fail-before 恰 1 新测红**("Expected DdlException…nothing was thrown")→**pass-after 26/0/0**、checkstyle 0。真值闸=live-e2e(`lower_case_meta_names=1`+case-variant CREATE 无 IF NOT EXISTS 于 case-sensitive paimon catalog;既有 legacy paimon DDL regression 覆盖契约、本 fix 无 BE 改)。设计 [`P5-fix-CREATE-TABLE-LOCAL-CONFLICT-design.md`](./tasks/designs/P5-fix-CREATE-TABLE-LOCAL-CONFLICT-design.md) | 2026-06-12 | ✅ |
1819
| D-055 | P5-fix#9 | **FIX-NATIVE-SUBSPLIT(M-3,round-2 MAJOR/round-1 MINOR,perf-parity)= fix-now(用户签字,2026-06-12)+ 纯连接器零 SPI/零 fe-core**:翻闸后大 native ORC/Parquet paimon 文件得**一个** scanner(无文件内并行)——连接器 native 臂每 RawFile 发**一个** `PaimonScanRange`(`.start(0).length(file.length())`),legacy `PaimonScanNode:434-465` 经 `determineTargetFileSplitSize`+`fileSplitter.splitFile` 把大文件切成多 split。结果正确仅并行度回归。recon(5-scout + 对抗 synthesizer `wf_ad764bf6-1c9`)三大结论:① **真 gap 非 no-op**——ORC/Parquet `compressType=PLAIN`(`FileSplitter:115`),`(!splittable||!=PLAIN)` 闸不触发→真切分跑;② **DV×sub-split 安全无须 guard**——paimon DV rowid 是文件**全局**行位置,BE native reader 在部分 byte range 内仍报全局行位(ORC `getRowNumber()` 由 stripe 起播种、Parquet `first_row` 跨 row-group 累计),`_kv_cache` 按 `path+offset` 跨 sub-split 共享 DV 位图,iceberg 用同机制于常规切分文件→**规则=同一 per-RawFile DeletionFile 原样附到每个 sub-range、不 re-base offset**(legacy `:459-460` parity);③ **纯连接器**——切分 math 是对 5 个 session var(`VariableMgr.toMap` 通道,同 `isCppReaderEnabled`)的 long 算术,连接器禁 import fe-core `FileSplitter`/`SessionVariable` 故逐字重述;`start/length/fileSize` 经既有 `PaimonScanRange.Builder`→`PluginDrivenSplit` FileSplit ctor→`FileQueryScanNode.createFileRangeDesc` 已序列化到 BE。**仅 specified-size 分支可达**(连接器传 blockLocations=null + target 恒>0 因 paimon 非 batch;block-based 分支死)。**修=纯连接器**:2 纯静态(`computeFileSplitOffsets` 逐字移植含 **`>1.1D` 尾吸收 guard**、`determineTargetSplitSize` 移植 determineTargetFileSplitSize+applyMaxFileSplitNumLimit 略去 isBatchMode→0)+ `sessionLong`/`resolveTargetSplitSize`(lazy once)+ native 臂改 buildNativeRange 加 (start,length) 内层 loop。守门:连接器 256/0/0(1 CI-gated skip)、checkstyle 0、import-gate 净、**fail-before 恰 3 splitting 测红**(neuter `computeFileSplitOffsets`→单 range)其余绿、end-to-end append-only 真表小 file_split_size→≥2 contig sub-range。split-weight 调度 nicety 不移植(pre-existing native 路已缺)= [DV-033]。真值闸=live-e2e 大文件并行 + DV-file 多 range 读(既有 legacy paimon regression 覆盖 BE 契约、本 fix 无 BE 改)。设计 [`P5-fix-NATIVE-SUBSPLIT-design.md`](./tasks/designs/P5-fix-NATIVE-SUBSPLIT-design.md) | 2026-06-12 | ✅ |
1920
| D-054 | P5-fix#8 | **FIX-COUNT-PUSHDOWN(M-2,round-2 MAJOR/round-1 MINOR,perf-parity)= fix-now + 新增 default `planScan` 7-arg overload 携 `boolean countPushdown` + 连接器 collapse-to-one(用户签字,2026-06-12)**:翻闸后 plugin-driven paimon `COUNT(*)` **结果正确但慢**——COUNT 枚举已达 BE(`FileScanNode.toThrift:90` 发 `pushDownAggNoGroupingOp`、`PhysicalPlanTranslator:873` 在 plugin 节点设 COUNT、未排除)且 per-range emit 缝**已建全**(`PaimonScanRange.Builder.rowCount`→`paimon.row_count`→`setTableLevelRowCount`,与 legacy `PaimonScanNode:303-308` byte-一致),唯独**信号+计算**缺:merged count `DataSplit.mergedRowCount()` 是 paimon-SDK-only 须连接器算,而 COUNT 信号 `getPushDownAggNoGroupingOp()==COUNT` 只在 fe-core 节点、`PluginDrivenScanNode.getSplits` 从不读(grep 0)也不在任何 `planScan`/`ConnectorSession`/`ConnectorContext`/handle → 连接器每 split 发 `table_level_row_count=-1` → BE 物化全 post-merge 行去 count(`file_scanner.cpp:1298-1326`),PK/MOR merge 表尤贵。**故非纯连接器(更正动手前 framing)**:信号须过 SPI 边界。**否决经 `ConnectorSession` 穿**(FIX-FORCE-JNI 先例)——agg-op 是 per-query planner 输出非 SET-var,会成静默无类型通道(本项目反复踩的 bug 类)。**用户定(vs defer)= fix-now**,且 **count-split 形状 = 连接器 collapse-to-one**(vs full-parity fe-core trim / vs per-split)。**修=3 文件**:① SPI `ConnectorScanPlanProvider` +1 **default** 7-arg `planScan(...,boolean countPushdown)` 委托 6-arg(镜像 limit/requiredPartitions 扩展链,其余连接器零改 no-op)[E15];② fe-core `PluginDrivenScanNode.getSplits` 读 `getPushDownAggNoGroupingOp()==TPushAggOp.COUNT` 传入(**无 post-loop 数学**);③ 连接器抽 `planScanInternal(...,countPushdown)`(4-arg 委托 false、7-arg 委托 flag)+ count 短路(**第一 routing 臂**,count-eligible split 不再发数据 range,否则 BE 双计 vs DV/PK-merge):累加全 eligible split 的 `mergedRowCount` 入 `countSum`、留首个为代表、循环后发**一** JNI count range 携 `countSum`(=legacy `<=10000` singletonList+assignCountToSplits 收一 split case);无 merged count 的 split 走常规 native/JNI 路 BE 自计(footer/物化)。两新成员=纯静态 `isCountPushdownSplit(boolean,DataSplit)`(mutation-test 路由闸)+ `buildCountRange`。**参数形状 `boolean`**(BE 只需 COUNT-vs-not、`TPushAggOp` 过度泛化)+ **paimon-only**=工程判断(未被否)。legacy `>10000` 并行 split trim **有意丢**(连接器无 numBackends,fe-core-only)= perf-only 偏差 [DV-032]。守门:连接器 252/0/0(1 CI-gated skip)、fe-core compile+checkstyle 0、import-gate 净、**fail-before 恰 2 新测红**(neuter `isCountPushdownSplit`→false)其余 33 绿、end-to-end 真 local PK 表测断言 collapse-to-one 携 merged total(2)。真值闸=live-e2e BE CountReader 选择/EXPLAIN(既有 legacy paimon count regression 覆盖 BE 契约、本 fix 无 BE 改)。设计 [`P5-fix-COUNT-PUSHDOWN-design.md`](./tasks/designs/P5-fix-COUNT-PUSHDOWN-design.md) | 2026-06-12 | ✅ |
2021
| D-053 | P5-fix#6 | **FIX-KERBEROS-DOAS / M-8(MAJOR,Kerberos-only,fe-core,filesystem+jdbc)= fix-now(用户签字,2026-06-11)**:翻闸后 filesystem/jdbc flavor 在 Kerberized HDFS 上丢 UGI `doAs`——连接器 `PaimonConnector.createCatalog` 已把建 catalog 包进 `context.executeAuthenticated`(:194),但其背后 authenticator 对这两 flavor 是**基类 no-op**:HDFS `HadoopExecutionAuthenticator` 仅在 `initializeCatalog()` 内构建(`PaimonFileSystemMetaStoreProperties:46`/`PaimonJdbcMetaStoreProperties:120`),而 `initializeCatalog` 在翻闸路径**死代码**(唯一 live 调用方=legacy `PaimonExternalCatalog:147`;plugin 路径经 `PaimonCatalogFactory` 自建 catalog)→ `PluginDrivenExternalCatalog.initPreExecutionAuthenticator:130` 读到 `AbstractPaimonProperties:45` 的 no-op → `executeAuthenticated` 不 doAs。HMS 不受影响(authenticator 在 `initNormalizeAndCheckProps:70` 即建、必跑)。**作用域=filesystem+jdbc only**(用户签):DLF/REST 排除——`PaimonAliyunDLFMetaStoreProperties` 从不设 authenticator、用 Aliyun AK/SK/STS 入 HiveConf 非 Kerberos UGI(无 doAs 可丢),故 review「DLF」从句 **overstated**;HMS 已对。**修=fe-core,零连接器改/零连接器-SPI**:新 fe-core hook `MetastoreProperties.initExecutionAuthenticator(List<StorageProperties>)`(default no-op)由 `PluginDrivenExternalCatalog.initPreExecutionAuthenticator` 用**已安全建好**的 `catalogProperty.getOrderedStoragePropertiesList()` 调(catalog-init 时机、与 legacy 同、避免每次 `MetastoreProperties.create` eager 重复 kerberos login);filesystem/jdbc override 之经 `AbstractPaimonProperties.initHdfsExecutionAuthenticator` 共享 helper 从 HDFS `StorageProperties` 建 authenticator(镜像 HMS)。**FE-unit 可测 wiring**(断言 `getExecutionAuthenticator()` 返 `HadoopExecutionAuthenticator`、不调 initializeCatalog),**真 doAs 端到端=live-Kerberos-e2e only**(无 paimon-kerberos regression 套件,[DV-031](./deviations-log.md))。守门:fe-core `Paimon{FileSystem,Jdbc}MetaStorePropertiesTest` 14/0/0、fail-before 双 red(no-op `AbstractPaimonProperties$1`)、checkstyle 0。设计 [`P5-fix-KERBEROS-DOAS-design.md`](./tasks/designs/P5-fix-KERBEROS-DOAS-design.md) | 2026-06-11 | ✅ |
@@ -67,6 +68,16 @@
6768

6869
## 详细记录(时间倒序)
6970

71+
### D-056 — `FIX-CREATE-TABLE-LOCAL-CONFLICT`(P3 揪出,MAJOR correctness)= fix-now + Option-2 外科最小修
72+
73+
- **日期**:2026-06-12
74+
- **状态**:✅ 生效
75+
- **关联**:[task-list §P3](./task-list-P5-rereview2-fixes.md)、[设计](./tasks/designs/P5-fix-CREATE-TABLE-LOCAL-CONFLICT-design.md)、[DV-034](./deviations-log.md)、P3 对抗 review `wf_25450c36-b7a`
76+
- **背景**:P3 覆盖缺口核查(「去查」非「去改」)的 4 项 plugin-vs-legacy paimon parity 中,3 项 PARITY_HOLDS(HMS-CONFRES:key 拼写恰 `hive.conf.resources`、无 `config.resources` 别名、round-1 wiring 在、BE-downflow 两侧同——legacy HMS hive-site.xml 本就不入 BE scan props;ANALYZE/列统计:`getColumnStatistic` 两侧 `Optional.empty()`、`createAnalysisTask` byte-同、generic 于桥非 paimon regression;split-count:post-sub-split 数经共享父 `FileQueryScanNode.selectedSplitNum` 喂 `SqlBlockRuleMgr` 两侧同、2 项 cosmetic/NIT 且 pre-date #9),唯 DDL 写揪出真分歧:对抗 verifier 把 tracer 的 createTable PARITY 推翻为 DIVERGENCE——通用桥 `PluginDrivenExternalCatalog.createTable` 丢了 legacy `PaimonMetadataOps:206-214` 的 local-arm 拒绝(详见 §索引 D-056 正文)。
77+
- **决策**:用户签字 **convert-to-FIX now**(vs log-as-deviation / investigate-more)。实现取 **Option 2 外科最小修**:仅补 local-conflict 闸,case-A(remote-hit)行为不动。
78+
- **替代方案**:Option 1 full-parity(对 `exists&&!ifNotExists` 全 retype 1050)——否决:改非分歧 case-A + 破既有 intentional 测(`testCreateTableExistingTableWithoutIfNotExistsStillErrors` 钉 remote-hit→连接器抛 generic)+ 越 finding 界。
79+
- **影响**:fe-core `PluginDrivenExternalCatalog.java`(拆 OR + 加 guard、+2 import `ErrorCode`/`ErrorReport`)+ test(+1)。零 SPI/连接器/BE/RFC。**通用桥修跨连接器收口**(MaxCompute/未来 iceberg/hudi 同受益,呼应 P3 item-5 跨连接器 follow-up)。残留 case-A error-code-generic = [DV-034] 留 P4 cleanup。
80+
7081
### D-055 — `FIX-NATIVE-SUBSPLIT`(#9 M-3)= 连接器侧移植 native 文件切分(纯连接器,零 SPI/零 fe-core)
7182

7283
- **日期**:2026-06-12

0 commit comments

Comments
 (0)