Skip to content

Commit 9dcf6d1

Browse files
morningmanclaude
andcommitted
fix: FIX-MAPPING-FLAG-KEYS — read canonical dotted catalog keys for paimon type-mapping toggles
Root cause: after the SPI cutover the paimon connector reads the type-mapping toggles from UNDERSCORE keys (enable_mapping_binary_as_varbinary / enable_mapping_timestamp_tz; PaimonConnectorProperties:39,42 -> PaimonConnectorMetadata.buildTypeMappingOptions), but fe-core only ever writes the canonical DOTTED catalog keys (enable.mapping.varbinary / enable.mapping.timestamp_tz; CatalogProperty:50,52, written/defaulted by ExternalCatalog.setDefaultPropsIfMissing and hidden via HIDDEN_PROPERTIES). PluginDrivenExternalCatalog.createConnectorFromProperties hands the connector the raw catalog property map verbatim, so getOrDefault(underscore,"false") is always false. Even when the user enables the mapping at CREATE CATALOG, Paimon BINARY stays STRING and TIMESTAMP_WITH_LOCAL_TIME_ZONE stays DATETIMEV2 — a silent cutover regression (legacy PaimonExternalTable:350 reads the dotted key and honors it). The binary key is doubly drifted (separator . -> _ AND token varbinary -> binary_as_varbinary), so a generic dot->underscore normalizer would not fix it. Latent until the flag is enabled. Re-confirmation: M-crit was critic-surfaced (not 3-lens-gated), so the finding was independently re-verified by a 5-agent scout + adversarial synthesizer (REAL_BUG, high confidence; false-positive steelman rejected — dotted is canonical per the original feature PRs, every regression CREATE CATALOG, legacy parity, and the JDBC connector which kept dotted in the same SPI PR). Solution (connector-only, zero new SPI, no BE): re-point the two PaimonConnectorProperties constants to the canonical dotted keys (ENABLE_MAPPING_VARBINARY = "enable.mapping.varbinary", renamed from ENABLE_MAPPING_BINARY_AS_VARBINARY to match the CatalogProperty/JDBC/iceberg convention and fix both separator and token; ENABLE_MAPPING_TIMESTAMP_TZ = "enable.mapping.timestamp_tz") and update the one reference in PaimonConnectorMetadata. No logic change — the Options(mapBinaryToVarbinary, mapTimestampTz) arg order is already correct. BE-side consistency verified: PluginDrivenScanNode extends FileQueryScanNode and inherits the dotted-key read for the BE scan param (FileQueryScanNode:192-193,635-678), so FE column type and BE scan param now agree (they diverged before this fix). Scope: paimon-only (user-signed D-051). NEW hive + iceberg connectors share the identical root cause; logged as a cross-connector follow-up (DV-030), not fixed here. Rejected an fe-core dot->underscore normalizer (broader blast, breaks JDBC which already reads dotted, and insufficient for paimon's renamed token). Tests (PaimonConnectorMetadataTest): +2 UT. getTableSchemaHonorsDottedMappingKeys (bug-catcher) sets the dotted keys true and asserts BINARY->VARBINARY / LTZ->TIMESTAMPTZ; getTableSchemaDefaultsMappingFlagsOff (guard) asserts the default-off STRING/DATETIMEV2. Module 234/0/0 (1 CI-gated skip), checkstyle 0, import-gate clean. Fail-before verified: the bug-catcher reddens on the underscore key (expected <VARBINARY> but was <STRING>) while the guard stays green. E2E test_paimon_catalog_{varbinary,timestamp_tz}.groovy are CI-gated (enablePaimonTest=false + external fixture) — not run. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 65609bf commit 9dcf6d1

7 files changed

Lines changed: 167 additions & 7 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1017,7 +1017,7 @@ private List<ConnectorColumn> mapFields(List<DataField> fields, List<String> pri
10171017
private static PaimonTypeMapping.Options buildTypeMappingOptions(Map<String, String> props) {
10181018
boolean binaryAsVarbinary = Boolean.parseBoolean(
10191019
props.getOrDefault(
1020-
PaimonConnectorProperties.ENABLE_MAPPING_BINARY_AS_VARBINARY,
1020+
PaimonConnectorProperties.ENABLE_MAPPING_VARBINARY,
10211021
"false"));
10221022
boolean timestampTz = Boolean.parseBoolean(
10231023
props.getOrDefault(

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorProperties.java‎

Lines changed: 17 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,11 +35,24 @@ public final class PaimonConnectorProperties {
3535
/** Warehouse location for the Paimon catalog. */
3636
public static final String WAREHOUSE = "warehouse";
3737

38-
/** Whether to map Paimon BINARY/VARBINARY to Doris VARBINARY instead of STRING. */
39-
public static final String ENABLE_MAPPING_BINARY_AS_VARBINARY = "enable_mapping_binary_as_varbinary";
38+
/**
39+
* Whether to map Paimon BINARY/VARBINARY to Doris VARBINARY instead of STRING.
40+
*
41+
* <p>Canonical (dotted) CREATE-CATALOG key, mirroring fe-core
42+
* {@code CatalogProperty.ENABLE_MAPPING_VARBINARY} and the legacy paimon path. The connector
43+
* receives the raw catalog property map ({@code catalogProperty.getProperties()}), which only
44+
* ever carries this dotted key (fe-core {@code setDefaultPropsIfMissing} writes only it), so the
45+
* read MUST use the dotted spelling — an underscore variant is never present and would read false.
46+
*/
47+
public static final String ENABLE_MAPPING_VARBINARY = "enable.mapping.varbinary";
4048

41-
/** Whether to map Paimon TIMESTAMP_WITH_LOCAL_TIME_ZONE to TIMESTAMPTZ. */
42-
public static final String ENABLE_MAPPING_TIMESTAMP_TZ = "enable_mapping_timestamp_tz";
49+
/**
50+
* Whether to map Paimon TIMESTAMP_WITH_LOCAL_TIME_ZONE to TIMESTAMPTZ.
51+
*
52+
* <p>Canonical (dotted) CREATE-CATALOG key, mirroring fe-core
53+
* {@code CatalogProperty.ENABLE_MAPPING_TIMESTAMP_TZ} and the legacy paimon path.
54+
*/
55+
public static final String ENABLE_MAPPING_TIMESTAMP_TZ = "enable.mapping.timestamp_tz";
4356

4457
/** Default catalog type when not specified. */
4558
public static final String DEFAULT_CATALOG_TYPE = "filesystem";

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTest.java‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,4 +287,71 @@ public void getTableSchemaAtSnapshotAlsoForcesNullable() {
287287
Assertions.assertTrue(schema.getColumns().get(0).isNullable(),
288288
"the at-snapshot read path must also force columns nullable (legacy parity)");
289289
}
290+
291+
// ---------------------------------------------------------------------
292+
// FIX-MAPPING-FLAG-KEYS — type-mapping toggles read the canonical dotted
293+
// CREATE-CATALOG keys (enable.mapping.varbinary / enable.mapping.timestamp_tz)
294+
// ---------------------------------------------------------------------
295+
296+
private static RowType binaryAndLtzRowType() {
297+
return RowType.builder()
298+
.field("b", DataTypes.BINARY(16))
299+
.field("ts_ltz", DataTypes.TIMESTAMP_WITH_LOCAL_TIME_ZONE(3))
300+
.build();
301+
}
302+
303+
@Test
304+
public void getTableSchemaHonorsDottedMappingKeys() {
305+
RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps();
306+
ops.table = new FakePaimonTable(
307+
"t1", binaryAndLtzRowType(), Collections.emptyList(), Collections.emptyList());
308+
309+
// The user enables both mappings via the canonical DOTTED CREATE-CATALOG keys — the only
310+
// spelling fe-core ever writes into the catalog property map (CatalogProperty.java:50,52;
311+
// ExternalCatalog.setDefaultPropsIfMissing). The connector receives that raw map verbatim.
312+
Map<String, String> props = new java.util.HashMap<>();
313+
props.put("enable.mapping.varbinary", "true");
314+
props.put("enable.mapping.timestamp_tz", "true");
315+
PaimonConnectorMetadata metadata =
316+
new PaimonConnectorMetadata(ops, props, new RecordingConnectorContext());
317+
318+
ConnectorTableHandle handle = metadata.getTableHandle(null, "db1", "t1").get();
319+
ConnectorTableSchema schema = metadata.getTableSchema(null, handle);
320+
321+
// WHY: when the user enables the mapping at CREATE CATALOG, a Paimon BINARY column must
322+
// surface as VARBINARY and a TIMESTAMP_WITH_LOCAL_TIME_ZONE column as TIMESTAMPTZ — legacy
323+
// parity (PaimonExternalTable.java:350 reads the same dotted key and honors it). MUTATION:
324+
// reverting the connector constants to the underscore spelling (the cutover bug:
325+
// enable_mapping_binary_as_varbinary / enable_mapping_timestamp_tz) makes getOrDefault miss
326+
// the dotted keys the map actually carries -> both flags read false -> the column types fall
327+
// back to STRING / DATETIMEV2 -> red. This closes critic coverage-gap #2.
328+
Assertions.assertEquals("VARBINARY", schema.getColumns().get(0).getType().getTypeName(),
329+
"enable.mapping.varbinary=true must map Paimon BINARY to Doris VARBINARY");
330+
Assertions.assertEquals("TIMESTAMPTZ", schema.getColumns().get(1).getType().getTypeName(),
331+
"enable.mapping.timestamp_tz=true must map Paimon LTZ to Doris TIMESTAMPTZ");
332+
}
333+
334+
@Test
335+
public void getTableSchemaDefaultsMappingFlagsOff() {
336+
RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps();
337+
ops.table = new FakePaimonTable(
338+
"t1", binaryAndLtzRowType(), Collections.emptyList(), Collections.emptyList());
339+
340+
// No mapping keys set — the default (legacy-compatible) behavior.
341+
PaimonConnectorMetadata metadata =
342+
new PaimonConnectorMetadata(ops, Collections.emptyMap(), new RecordingConnectorContext());
343+
344+
ConnectorTableHandle handle = metadata.getTableHandle(null, "db1", "t1").get();
345+
ConnectorTableSchema schema = metadata.getTableSchema(null, handle);
346+
347+
// WHY: with the toggles absent, BINARY must map to STRING and LTZ to DATETIMEV2 (default
348+
// false), matching legacy. This guards against a fix that accidentally flips the defaults on
349+
// (e.g. reading the wrong default or inverting the boolean). MUTATION: defaulting either flag
350+
// to true -> VARBINARY / TIMESTAMPTZ -> red. Green in both the buggy and fixed states (it
351+
// pins the default, not the key spelling), so it is a regression guard, not the bug-catcher.
352+
Assertions.assertEquals("STRING", schema.getColumns().get(0).getType().getTypeName(),
353+
"absent enable.mapping.varbinary must leave Paimon BINARY as STRING (default off)");
354+
Assertions.assertEquals("DATETIMEV2", schema.getColumns().get(1).getType().getTypeName(),
355+
"absent enable.mapping.timestamp_tz must leave Paimon LTZ as DATETIMEV2 (default off)");
356+
}
290357
}

‎plan-doc/decisions-log.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
1616
| 编号 | 别名 | 简述 | 日期 | 状态 |
1717
|---|---|---|---|---|
18+
| D-051 | P5-fix#5 | **FIX-MAPPING-FLAG-KEYS(M-crit MAJOR,纯连接器/FE-wiring,无 BE/无 SPI)作用域 = paimon-only 修 + hive/iceberg 跨连接器 follow-up(用户签字,2026-06-11)**:翻闸后 paimon 连接器类型映射两开关**静默失效**——连接器读**下划线**键 `enable_mapping_binary_as_varbinary`/`enable_mapping_timestamp_tz`(`PaimonConnectorProperties:39,42`→`PaimonConnectorMetadata.buildTypeMappingOptions`),但 fe-core 只写**点分**键 `enable.mapping.varbinary`/`enable.mapping.timestamp_tz`(`CatalogProperty:50,52`;`ExternalCatalog.setDefaultPropsIfMissing:302-306` 仅写点分键;`HIDDEN_PROPERTIES` 仅藏点分键)→ `PluginDrivenExternalCatalog.createConnectorFromProperties` 把**原始** catalog map 原样喂连接器 → `getOrDefault(下划线,"false")` 恒 false → 即便用户在 CREATE CATALOG 开启,BINARY 仍→STRING、LTZ 仍→DATETIMEV2(legacy `PaimonExternalTable:350` 读点分键并 honor → cutover 回归,flag 启用前 latent)。binary 键**双重漂移**(分隔符 `.`→`_` 且 token `varbinary`→`binary_as_varbinary`)→ 通用归一化器修不了。**M-crit 是 critic-surfaced 未过 3-lens → 先独立复核**(5-agent scout + 对抗 synthesizer workflow `wf_a3626c54-0db` → REAL_BUG high-conf,false-positive steelman 被否:原始 feature PR #57821/#59720、全 regression CREATE CATALOG(paimon/iceberg/hive/jdbc 皆点分)、legacy parity、同 SPI PR 迁移的 JDBC 连接器正确保点分 `JdbcConnectorProperties:66-67` 均证点分为 canonical)。**修(纯连接器、零 SPI/BE)**:`PaimonConnectorProperties` 两常量重指 canonical 点分键(binary 常量并改名 `ENABLE_MAPPING_VARBINARY` 对齐 CatalogProperty/JDBC/iceberg 约定,同修分隔符+token)+ 更新 `PaimonConnectorMetadata` 一处引用;`Options(mapBinaryToVarbinary,mapTimestampTz)` 顺序本就对、无逻辑改。**BE 一致性已核**:`PluginDrivenScanNode extends FileQueryScanNode` 不 override mapping getter → BE scan param 经继承的 `getEnableMappingVarbinary/Tz` 本就读点分键(`FileQueryScanNode:192-193,635-678`),故修连接器 FE 侧读后 FE 列型与 BE scan param 一致(修前两侧分歧)。**用户定 = paimon-only**(vs 一次修全 3 连接器)→ hive/iceberg 同根因登 [DV-030](./deviations-log.md) 跨连接器 follow-up(hive `enable_mapping_binary_as_string` 是误名非语义反转)。否决 fe-core 归一化器(blast 大/破 JDBC 已正确读点分/对 paimon 双重漂移不足)。守门:模块 234/0/0(1 CI-gated skip)、checkstyle 0、import-gate 净、fail-before bug-catcher 向红(期望 VARBINARY 实得 STRING)+guard 两态绿。真值闸=`test_paimon_catalog_{varbinary,timestamp_tz}.groovy`(CI-gated,enablePaimonTest=false+外部 fixture)。设计 [`P5-fix-MAPPING-FLAG-KEYS-design.md`](./tasks/designs/P5-fix-MAPPING-FLAG-KEYS-design.md) | 2026-06-11 | ✅ |
1819
| D-050 | P5-fix#4 | **FIX-JDBC-DRIVER-URL(B-8a BLOCKER + B-8b 安全)作用域 = CREATE-time 校验 parity(用户签字,2026-06-11)**:JDBC flavor 连接器(a)`PaimonScanPlanProvider.getBackendPaimonOptions` 把 `driver_url` **裸**转发给 BE 且 `startsWith("jdbc.")` filter 丢 `paimon.jdbc.*` 别名 → BE `JdbcDriverUtils.registerDriver` 的 `new URL("mysql.jar")` 抛 `MalformedURLException`(B-8a 功能 BLOCKER);(b)driver_url 无 format/allow-list/secure-path 校验 + stale「paimon 不在 SPI_READY_TYPES」注释(B7 后已假,`CatalogFactory:51` 含 paimon)(B-8b 安全)。**修 = 纯连接器、零新 SPI**(复用既有 `Connector.preCreateValidation` + `ConnectorValidationContext.validateAndResolveDriverPath`):B-8a 在 `getBackendPaimonOptions` 用 `firstNonBlank(JDBC_DRIVER_URL)` 认两别名 + 抽出共享 `PaimonCatalogFactory.resolveDriverUrl`(FE 注册与 BE 选项同解析)发 canonical `jdbc.driver_url`(resolved)+`jdbc.driver_class`(镜像 legacy `PaimonJdbcMetaStoreProperties.getBackendPaimonOptions`);B-8b override `PaimonConnector.preCreateValidation` 对 jdbc flavor 调 `validateAndResolveDriverPath`(镜像 `JdbcDorisConnector`)+ 删 stale 注释。**4-lens clean-room review 确认 B-8a + CREATE-time B-8b 正确**,揪出**校验仅 CREATE-time**(FE-restart reload/ALTER CATALOG/scan-time 不复校)= **pre-existing fe-core 缝、所有 plugin 连接器共有(含 JDBC 参考连接器)、默认配置 permissive 无可绕**,legacy 更强(每 scan 经 getFullDriverUrl 复校)。**用户定 = 接受 CREATE-time parity**(vs 扩到 fe-core ALTER hook + scan-time 校验 SPI——触 fe-core+全连接器+ALTER 可能触发 BE 连通测,非 surgical)→ 登 [DV-028](./deviations-log.md)(CREATE-time-only 校验 gap + 跨连接器 follow-up)+ [DV-029](./deviations-log.md)(简化 resolver + BE-side user/password/uri 别名 out-of-scope)。守门:模块 232/0/0、checkstyle 0、import-gate 净、fail-before 5/9 向红。真值闸=`test_paimon_jdbc_catalog`(CI-gated)。设计 [`P5-fix-JDBC-DRIVER-URL-design.md`](./tasks/designs/P5-fix-JDBC-DRIVER-URL-design.md) | 2026-06-11 | ✅ |
1920
| D-049 | P5-fix#3 | **FIX-SCHEMA-EVOLUTION(B-1a BLOCKER + M-10 MAJOR)= Design C「连接器直建 thrift schema 字典」(用户签字,2026-06-11;M-10 deferred)**:翻闸后 native(ORC/Parquet) 读丢 paimon schema-evolution——连接器只发 per-file `TPaimonFileDesc.schema_id`、从不设 scan 级 `TFileScanRangeParams.current_schema_id`/`history_schema_info` → BE `table_schema_change_helper.h:219-237` 走 `!__isset` 分支退化**名匹配** → schema-evolved(改名/重排)表旧 schema 文件**静默错行/读 NULL**(JNI 路不受影响、native 是默认)。**关键事实**(5 层 trace + BE `table_schema_change_helper.cpp:312-430`):field-id 匹配路 BE 只读 `TField.{id,name,type.type 当 nested-vs-scalar tag}`、**从不读 Doris Type 也不读 tuple descriptor** → 连接器可**直建** `TSchema`(`org.apache.doris.thrift.*` import-legal)、**无需 Doris Type/无新 SPI**;`Column.uniqueId`(M-10) 仅当 FE 从 Doris 列建 history 才有关、直接从 paimon `DataField.id()` 建则 B-1a 独立于 M-10。**用户定 = Design C(vs Design B「穿 ConnectorColumn/ConnectorType field-id + fe-core ExternalUtil 建」)**:连接器在 `getScanNodeProperties` 从 live(snapshot-pinned)表建 `current_schema_id=-1`+`history_schema_info`(-1 entry=pinned schema、外加每个 `SchemaManager.listAllIds()` 提交 schema)→ base64 thrift carrier prop 经既有 `populateScanLevelParams` SPI hook(复用 DV-006 同缝)落 params。**零新 SPI surface**(task-list 原标「SPI?=yes」修正为 no)、连接器局部、最小 blast radius;M-10(`Column.uniqueId=-1`)**deferred**(rereview2 §4 已证伪 standalone repro、无消费者,[DV-026](./deviations-log.md))。**3-lens clean-room review 揪出 2 真 BLOCKER(均在 -1 entry,已修+复验 clean)**:① 列名 casing(BE verbatim key vs lowercase slot name + `current_schema_id=-1` 永不走 ConstNode 快路 → 大小写混合列即崩、连未演化表都回归)→ 修 = -1 entry **只** lowercase 顶层名(default-locale,byte-match slot 产出方+legacy `parseSchema:507`;嵌套 struct 名保 paimon-case=legacy `PaimonUtil:302` 非对称);② 时间旅行(-1 entry 取 `schemaManager.latest()` 绝对最新、但 tuple 用 pinned schema → 改名列崩/错行)→ 修 = -1 entry 取 `((FileStoreTable)table).schema()`(pinned)、guard `DataTable`→`FileStoreTable`。MINOR(eager 读全 schema 无 cache)= 接受的 fail-loud 偏差 [DV-027](./deviations-log.md)。守门:模块 222/0/0(+5 schema-evo UT)、checkstyle 0、import-gate 净。真值闸=`test_paimon_full_schema_change.groovy`(CI-gated)。设计 [`P5-fix-SCHEMA-EVOLUTION-design.md`](./tasks/designs/P5-fix-SCHEMA-EVOLUTION-design.md) | 2026-06-11 | ✅ |
2021
| D-048 | P5-fix#2 | **FIX-STATIC-CREDS-BE(B-9 BLOCKER)作用域 = full legacy-parity 替换(用户签字,2026-06-11)**:翻闸后 paimon 连接器 `PaimonScanPlanProvider.getScanNodeProperties:372-381` 把静态 catalog 凭据/配置(`s3.`/`oss.`/`cos.`/`obs.`/`hadoop.`/`fs.`/`dfs.`/`hive.` 前缀)裸拷进 `location.<rawkey>`,fe-core bridge `PluginDrivenScanNode.getLocationProperties` 只剥前缀不归一化 → BE native(FILE_S3) reader 只认 `AWS_ACCESS_KEY`/`AWS_SECRET_KEY`/`AWS_ENDPOINT`/`AWS_REGION`/`AWS_TOKEN`(`s3_util.cpp`)→ 私有桶 native 读拿不到凭据 403。是 review §9.3 凭据**第三道缝**(static→BE-scan,FIX-STORAGE-CREDS 修 catalog FileIO 缝、FIX-REST-VENDED 修 vended 缝,本缝两轮均漏)。legacy `PaimonScanNode.getLocationProperties:650-652` 仅返回 `CredentialUtils.getBackendPropertiesFromStorageMap(storagePropertiesMap)`(canonical map)。**用户定 = 方案 A(full legacy-parity,非窄 object-store-only)**:新 SPI `ConnectorContext.getBackendStorageProperties()`(default 空,仅 paimon 调)= 引擎用 #1 已接线的 `storagePropertiesSupplier`(`catalogProperty.getStoragePropertiesMap()`)跑同一 `getBackendPropertiesFromStorageMap` → 连接器**整段**替换裸拷循环为该 overlay(vended overlay 仍后置、collision 胜,legacy 优先序)。object-store→`AWS_*`;HDFS→canonical(保留用户 `hadoop.`/`dfs.`/`fs.`/`juicefs.` override + 补 legacy 默认 `ipc.client.fallback-to-simple-auth-allowed` 等,**顺修 review §211 MINOR**);丢非-parity `hive.*` 裸键(legacy 本不发 scan location)。一处 SPI 调用替掉前缀循环、单一真相源、无漂移。**ANONYMOUS-leak 边角经 BE 证伪无回归**(`s3_util.cpp` v1:383/v2:448 显式 ak/sk 优先于 `cred_provider_type`,vended keys 在则永不走 Anonymous 支)。无 ctor 改、无连接器新 import(import-gate 净)。SPI RFC §22(E14)。测:fe-core `DefaultConnectorContextBackendStoragePropsTest`(2)+连接器 `PaimonScanPlanProviderTest`(3 改/增,red-check 2/3 向红);模块 217/0/0。设计 [`P5-fix-STATIC-CREDS-BE-design.md`](./tasks/designs/P5-fix-STATIC-CREDS-BE-design.md) | 2026-06-11 | ✅ |

0 commit comments

Comments
 (0)