Skip to content

Commit fbe6a9d

Browse files
morningmansuxiaogang223claude
committed
[fix](paimon) Cast static overwrite partition values to their column types
### What problem does this PR solve? Issue Number: None Related PR: #65868 Problem Summary: An INSERT OVERWRITE with a static PARTITION clause named the Paimon partition by the literal as written: PluginDrivenInsertCommandContext keeps Literal.getStringValue() of the user's literal, and PaimonWriteBinding handed it to Paimon's withOverwrite unchanged, while the written rows carry the literal cast to the column type. On a local filesystem catalog, PARTITION (p = true) on an INT column failed with NumberFormatException, PARTITION (dt = 20240101) on a DATE column failed with "the changes in dt=19723/ does not belong to this partition", and an overwrite of a TIMESTAMP WITH LOCAL TIME ZONE partition failed the same way, because Paimon parses that value as local time in the FE JVM zone while the rows were written in the session zone. An overwrite without rows passes Paimon's check and replaced the wrong partition silently: an empty LTZ overwrite deleted the partition that lies the session's offset from the JVM zone away and kept the named one, and an empty overwrite of a value equal to partition.default-name deleted the NULL partition, which is what Paimon reads that string as. branch-4.1 #65868 casts each static partition literal to its column type, moves an LTZ value from the session zone to the FE JVM zone in Paimon's timestamp format, and rejects a value equal to partition.default-name. Master builds the static partition spec in fe-core and hands it to connectors through ConnectorWriteHandle, so the cast values are offered by a new SPI getter, getCastStaticPartitionSpec, beside the spec as written. fe-core casts each literal to the type of its column in the bound target schema, the cast that constant folding applies to the value materialized into the rows, and only when a connector asks: Iceberg, MaxCompute and Hive keep reading the spec as written and see no change. Paimon's write binding resolves its static partition from the cast values; an LTZ column mapped to TIMESTAMPTZ yields a UTC instant with its offset, which moves to the JVM zone as well. A literal that does not fit its column type now fails the statement, as on branch-4.x, where master wrote NULL into the partition column. ### Release note Fix INSERT OVERWRITE ... PARTITION on Paimon tables failing, or replacing the wrong partition, when a static partition value is not spelled in its column type (p = true on an INT column, dt = 20240101 on a DATE column) or the column is TIMESTAMP WITH LOCAL TIME ZONE. A static partition value equal to the table's partition.default-name, or one that is not a valid value of its column type, is now rejected. ### Check List (For Author) - Test: Unit Test, Manual test; regression test added but not run locally - PaimonWriteBindingTest ports the branch-4.x cases for the cast value, the partition.default-name rejection and the LTZ zone move, and adds the TIMESTAMPTZ mapping and SQL NULL. PluginDrivenTableSinkBindingTest checks the cast values on the write handle, the session-zone forms of DATETIMEV2 and TIMESTAMPTZ values, and that an invalid value fails only when cast. ConnectorWriteHandleTest pins the SPI default. Reading the spec as written, dropping the default-name check, skipping either zone move, or skipping the cast in fe-core makes them fail. - test_paimon_write_transaction gains the INT, DATE, LTZ and default-name cases. It needs the Paimon docker environment and was not run locally; the same statements on a local filesystem Paimon catalog give the asserted rows with this change and fail or delete the wrong partition on master. - Behavior changed: Yes (see Release note) - Does this need documentation: No Co-authored-by: Socrates <[email protected]> Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent e62ed74 commit fbe6a9d

13 files changed

Lines changed: 569 additions & 35 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -946,7 +946,7 @@ private long pinnedSnapshotId(Table table, Map<String, String> resolved) {
946946
* overrides — PST and EST resolve via {@code SHORT_IDS}, so a 4-entry-only map would still
947947
* reject them (verified by JDK harness).
948948
*/
949-
private static final Map<String, String> SESSION_TIME_ZONE_ALIASES;
949+
static final Map<String, String> SESSION_TIME_ZONE_ALIASES;
950950

951951
static {
952952
Map<String, String> m = new java.util.TreeMap<>(String.CASE_INSENSITIVE_ORDER);

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWriteBinding.java‎

Lines changed: 69 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -17,18 +17,26 @@
1717

1818
package org.apache.doris.connector.paimon;
1919

20+
import org.apache.doris.connector.spi.ConnectorColumn;
2021
import org.apache.doris.connector.spi.DorisConnectorException;
22+
import org.apache.doris.connector.spi.handle.ConnectorWriteHandle;
2123

2224
import org.apache.paimon.CoreOptions;
2325
import org.apache.paimon.table.FileStoreTable;
26+
import org.apache.paimon.types.DataTypeRoot;
2427
import org.apache.paimon.utils.InstantiationUtil;
2528

2629
import java.io.IOException;
30+
import java.time.LocalDateTime;
31+
import java.time.OffsetDateTime;
32+
import java.time.ZoneId;
33+
import java.time.ZonedDateTime;
34+
import java.time.format.DateTimeFormatter;
2735
import java.util.Base64;
2836
import java.util.Collections;
2937
import java.util.LinkedHashMap;
3038
import java.util.Map;
31-
import java.util.Set;
39+
import java.util.Objects;
3240
import java.util.TreeMap;
3341

3442
/** Statement-scoped Paimon write target shared by sink planning and transaction commit. */
@@ -55,13 +63,11 @@ private PaimonWriteBinding(String tableName, FileStoreTable table,
5563
}
5664

5765
static PaimonWriteBinding create(PaimonTableHandle handle, FileStoreTable table,
58-
Map<String, String> hadoopConfig, boolean overwrite,
59-
Map<String, String> requestedStaticPartition, Set<String> staticPartitionNullKeys) {
60-
Map<String, String> staticPartition = resolveStaticPartition(
61-
table, requestedStaticPartition, staticPartitionNullKeys);
62-
FileStoreTable writeTable = configureTableForWrite(table, overwrite, staticPartition);
66+
Map<String, String> hadoopConfig, ConnectorWriteHandle writeHandle, String sessionTimeZone) {
67+
Map<String, String> staticPartition = resolveStaticPartition(table, writeHandle, sessionTimeZone);
68+
FileStoreTable writeTable = configureTableForWrite(table, writeHandle.isOverwrite(), staticPartition);
6369
return new PaimonWriteBinding(handle.getDatabaseName() + "." + handle.getTableName(),
64-
writeTable, hadoopConfig, overwrite, staticPartition);
70+
writeTable, hadoopConfig, writeHandle.isOverwrite(), staticPartition);
6571
}
6672

6773
static FileStoreTable configureTableForWrite(FileStoreTable table, boolean overwrite,
@@ -78,26 +84,75 @@ static FileStoreTable configureTableForWrite(FileStoreTable table, boolean overw
7884
return table.copy(Collections.singletonMap(dynamicOverwriteKey, Boolean.FALSE.toString()));
7985
}
8086

81-
private static Map<String, String> resolveStaticPartition(FileStoreTable table,
82-
Map<String, String> requested, Set<String> nullKeys) {
87+
/**
88+
* The static partition as Paimon's static overwrite parses it: SQL NULL becomes the table's
89+
* {@code partition.default-name}, and every other value is the one the written rows carry, cast to the
90+
* column type.
91+
*/
92+
static Map<String, String> resolveStaticPartition(FileStoreTable table, ConnectorWriteHandle writeHandle,
93+
String sessionTimeZone) {
8394
Map<String, String> canonicalNames = new TreeMap<>(String.CASE_INSENSITIVE_ORDER);
8495
for (String partitionKey : table.partitionKeys()) {
8596
canonicalNames.put(partitionKey, partitionKey);
8697
}
8798
String defaultPartitionName = CoreOptions.fromMap(table.options()).partitionDefaultName();
99+
Map<String, String> castValues = writeHandle.getCastStaticPartitionSpec();
88100
Map<String, String> result = new LinkedHashMap<>();
89-
for (Map.Entry<String, String> entry : requested.entrySet()) {
90-
String canonicalName = canonicalNames.get(entry.getKey());
101+
for (String key : writeHandle.getStaticPartitionSpec().keySet()) {
102+
String canonicalName = canonicalNames.get(key);
91103
if (canonicalName == null) {
92-
throw new DorisConnectorException("Column '" + entry.getKey()
104+
throw new DorisConnectorException("Column '" + key
93105
+ "' is not a partition column of Paimon table");
94106
}
95-
String value = entry.getValue();
96-
result.put(canonicalName, nullKeys.contains(entry.getKey()) ? defaultPartitionName : value);
107+
if (writeHandle.getStaticPartitionNullKeys().contains(key)) {
108+
result.put(canonicalName, defaultPartitionName);
109+
continue;
110+
}
111+
String value = Objects.requireNonNull(castValues.get(key),
112+
() -> "missing the cast value of static partition column " + key);
113+
if (table.rowType().getField(canonicalName).type().getTypeRoot()
114+
== DataTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZONE) {
115+
value = toSdkLocalTime(value, isTimestampTz(writeHandle, canonicalName), sessionTimeZone);
116+
}
117+
if (writeHandle.isOverwrite() && defaultPartitionName.equals(value)) {
118+
// Paimon's static overwrite reads this string as the NULL partition of any partition type, so
119+
// a value equal to it would overwrite the NULL partition instead.
120+
throw new DorisConnectorException("Static partition value for column '" + canonicalName
121+
+ "' equals Paimon partition.default-name '" + defaultPartitionName
122+
+ "' and cannot be represented in a static overwrite");
123+
}
124+
result.put(canonicalName, value);
97125
}
98126
return result;
99127
}
100128

129+
private static boolean isTimestampTz(ConnectorWriteHandle writeHandle, String columnName) {
130+
for (ConnectorColumn column : writeHandle.getBoundTargetColumns()) {
131+
if (column.getName().equalsIgnoreCase(columnName)) {
132+
return "TIMESTAMPTZ".equalsIgnoreCase(column.getType().getTypeName());
133+
}
134+
}
135+
throw new DorisConnectorException("Paimon partition column is missing from the write schema: "
136+
+ columnName);
137+
}
138+
139+
/**
140+
* Paimon parses a static overwrite value of a TIMESTAMP WITH LOCAL TIME ZONE column as local time in the FE
141+
* JVM's default zone, so the value is moved there from the zone it was written in: the session zone for a
142+
* DATETIMEV2 value, UTC for a TIMESTAMPTZ value, which carries its offset. Paimon's parser takes a space,
143+
* not ISO's 'T', between the date and the time.
144+
*/
145+
private static String toSdkLocalTime(String value, boolean timestampTz, String sessionTimeZone) {
146+
String isoValue = value.replace(' ', 'T');
147+
ZonedDateTime instant = timestampTz
148+
? OffsetDateTime.parse(isoValue, DateTimeFormatter.ISO_OFFSET_DATE_TIME).toZonedDateTime()
149+
: LocalDateTime.parse(isoValue, DateTimeFormatter.ISO_LOCAL_DATE_TIME)
150+
.atZone(ZoneId.of(sessionTimeZone, PaimonConnectorMetadata.SESSION_TIME_ZONE_ALIASES));
151+
return instant.withZoneSameInstant(ZoneId.systemDefault()).toLocalDateTime()
152+
.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME)
153+
.replace('T', ' ');
154+
}
155+
101156
private static String serialize(FileStoreTable table) {
102157
try {
103158
return Base64.getEncoder().encodeToString(InstantiationUtil.serializeObject(table));

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWritePlanProvider.java‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -152,8 +152,7 @@ public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandl
152152
}
153153
PaimonConnectorTransaction transaction = currentTransaction(session);
154154
PaimonWriteBinding binding = PaimonWriteBinding.create(
155-
tableHandle, table, buildHadoopConfig(), handle.isOverwrite(),
156-
handle.getStaticPartitionSpec(), handle.getStaticPartitionNullKeys());
155+
tableHandle, table, buildHadoopConfig(), handle, session.getTimeZone());
157156
transaction.bind(binding);
158157

159158
TPaimonTableSink sink = new TPaimonTableSink();

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java‎

Lines changed: 39 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@
1717

1818
package org.apache.doris.connector.paimon;
1919

20+
import org.apache.doris.connector.spi.ConnectorColumn;
21+
import org.apache.doris.connector.spi.handle.ConnectorTableHandle;
22+
import org.apache.doris.connector.spi.handle.ConnectorWriteHandle;
23+
2024
import org.apache.paimon.CoreOptions;
2125
import org.apache.paimon.Snapshot;
2226
import org.apache.paimon.catalog.FileSystemCatalog;
@@ -60,6 +64,7 @@
6064
import java.util.List;
6165
import java.util.Map;
6266
import java.util.Optional;
67+
import java.util.Set;
6368

6469
/**
6570
* The table {@link PaimonScanPlanProvider#getScanNodeProperties} serializes to the BE must carry no
@@ -113,8 +118,8 @@ public void writeBindingKeepsCatalogOnFeButRemovesItFromBePayload(@TempDir Path
113118
FileStoreTable table = newTable(warehouse, catalogEnvironment(catalog), C1);
114119

115120
PaimonWriteBinding binding = PaimonWriteBinding.create(
116-
dataHandle(), table, Collections.emptyMap(), false,
117-
Collections.emptyMap(), Collections.emptySet());
121+
dataHandle(), table, Collections.emptyMap(),
122+
writeHandle(Collections.emptyMap(), Collections.emptySet()), "UTC");
118123
FileStoreTable backendTable = deserializeTable(binding.getSerializedTable());
119124

120125
Assertions.assertNotNull(binding.getTable().catalogEnvironment().catalogLoader());
@@ -135,9 +140,9 @@ public void staticPartitionDistinguishesSqlNullFromNullString(@TempDir Path ware
135140
Map<String, String> requested = Collections.singletonMap("pt", "NULL");
136141

137142
PaimonWriteBinding literal = PaimonWriteBinding.create(dataHandle(), table,
138-
Collections.emptyMap(), false, requested, Collections.emptySet());
143+
Collections.emptyMap(), writeHandle(requested, Collections.emptySet()), "UTC");
139144
PaimonWriteBinding sqlNull = PaimonWriteBinding.create(dataHandle(), table,
140-
Collections.emptyMap(), false, requested, Collections.singleton("pt"));
145+
Collections.emptyMap(), writeHandle(requested, Collections.singleton("pt")), "UTC");
141146

142147
Assertions.assertEquals("NULL", literal.getStaticPartition().get("pt"));
143148
Assertions.assertEquals(table.coreOptions().partitionDefaultName(),
@@ -769,6 +774,36 @@ private static PaimonTableHandle dataHandle() {
769774
return new PaimonTableHandle("db", "tbl", Collections.emptyList(), Collections.emptyList());
770775
}
771776

777+
/** An INSERT whose static partition values need no cast, so the handle's default cast spec applies. */
778+
private static ConnectorWriteHandle writeHandle(Map<String, String> staticPartition, Set<String> nullKeys) {
779+
return new ConnectorWriteHandle() {
780+
@Override
781+
public ConnectorTableHandle getTableHandle() {
782+
return null;
783+
}
784+
785+
@Override
786+
public List<ConnectorColumn> getColumns() {
787+
return Collections.emptyList();
788+
}
789+
790+
@Override
791+
public boolean isOverwrite() {
792+
return false;
793+
}
794+
795+
@Override
796+
public Map<String, String> getStaticPartitionSpec() {
797+
return staticPartition;
798+
}
799+
800+
@Override
801+
public Set<String> getStaticPartitionNullKeys() {
802+
return nullKeys;
803+
}
804+
};
805+
}
806+
772807
private static PaimonTableHandle sysHandle(String sysTableType, FileStoreTable base) {
773808
PaimonTableHandle handle = PaimonTableHandle.forSystemTable("db", "tbl", sysTableType,
774809
PaimonScanParams.requiresPaimonReader(sysTableType));

0 commit comments

Comments
 (0)