You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 9dcf6d1
Browse filesBrowse the repository at this point in the historyBrowse files
Root cause: after the SPI cutover the paimon connector reads the type-mapping
toggles from UNDERSCORE keys (enable_mapping_binary_as_varbinary /
enable_mapping_timestamp_tz; PaimonConnectorProperties:39,42 ->
PaimonConnectorMetadata.buildTypeMappingOptions), but fe-core only ever writes
the canonical DOTTED catalog keys (enable.mapping.varbinary /
enable.mapping.timestamp_tz; CatalogProperty:50,52, written/defaulted by
ExternalCatalog.setDefaultPropsIfMissing and hidden via HIDDEN_PROPERTIES).
PluginDrivenExternalCatalog.createConnectorFromProperties hands the connector
the raw catalog property map verbatim, so getOrDefault(underscore,"false") is
always false. Even when the user enables the mapping at CREATE CATALOG, Paimon
BINARY stays STRING and TIMESTAMP_WITH_LOCAL_TIME_ZONE stays DATETIMEV2 — a
silent cutover regression (legacy PaimonExternalTable:350 reads the dotted key
and honors it). The binary key is doubly drifted (separator . -> _ AND token
varbinary -> binary_as_varbinary), so a generic dot->underscore normalizer
would not fix it. Latent until the flag is enabled.
Re-confirmation: M-crit was critic-surfaced (not 3-lens-gated), so the finding
was independently re-verified by a 5-agent scout + adversarial synthesizer
(REAL_BUG, high confidence; false-positive steelman rejected — dotted is
canonical per the original feature PRs, every regression CREATE CATALOG, legacy
parity, and the JDBC connector which kept dotted in the same SPI PR).
Solution (connector-only, zero new SPI, no BE): re-point the two
PaimonConnectorProperties constants to the canonical dotted keys
(ENABLE_MAPPING_VARBINARY = "enable.mapping.varbinary", renamed from
ENABLE_MAPPING_BINARY_AS_VARBINARY to match the CatalogProperty/JDBC/iceberg
convention and fix both separator and token; ENABLE_MAPPING_TIMESTAMP_TZ =
"enable.mapping.timestamp_tz") and update the one reference in
PaimonConnectorMetadata. No logic change — the Options(mapBinaryToVarbinary,
mapTimestampTz) arg order is already correct. BE-side consistency verified:
PluginDrivenScanNode extends FileQueryScanNode and inherits the dotted-key read
for the BE scan param (FileQueryScanNode:192-193,635-678), so FE column type
and BE scan param now agree (they diverged before this fix).
Scope: paimon-only (user-signed D-051). NEW hive + iceberg connectors share the
identical root cause; logged as a cross-connector follow-up (DV-030), not fixed
here. Rejected an fe-core dot->underscore normalizer (broader blast, breaks
JDBC which already reads dotted, and insufficient for paimon's renamed token).
Tests (PaimonConnectorMetadataTest): +2 UT. getTableSchemaHonorsDottedMappingKeys
(bug-catcher) sets the dotted keys true and asserts BINARY->VARBINARY /
LTZ->TIMESTAMPTZ; getTableSchemaDefaultsMappingFlagsOff (guard) asserts the
default-off STRING/DATETIMEV2. Module 234/0/0 (1 CI-gated skip), checkstyle 0,
import-gate clean. Fail-before verified: the bug-catcher reddens on the
underscore key (expected <VARBINARY> but was <STRING>) while the guard stays
green. E2E test_paimon_catalog_{varbinary,timestamp_tz}.groovy are CI-gated
(enablePaimonTest=false + external fixture) — not run.
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java
+1-1Lines changed: 1 addition & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -1017,7 +1017,7 @@ private List<ConnectorColumn> mapFields(List<DataField> fields, List<String> pri
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorProperties.java
+17-4Lines changed: 17 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -35,11 +35,24 @@ public final class PaimonConnectorProperties {
35
35
/** Warehouse location for the Paimon catalog. */
36
36
publicstaticfinalStringWAREHOUSE = "warehouse";
37
37
38
-
/** Whether to map Paimon BINARY/VARBINARY to Doris VARBINARY instead of STRING. */
Copy file name to clipboardExpand all lines: fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTest.java
+67Lines changed: 67 additions & 0 deletions
Original file line number
Diff line number
Diff line change
@@ -287,4 +287,71 @@ public void getTableSchemaAtSnapshotAlsoForcesNullable() {
0 commit comments