Skip to content

Commit 454af09

Browse files
morningmanclaude
andcommitted
[fix](paimon) Resolve the static partition of a Paimon overwrite only
### What problem does this PR solve? Issue Number: None Related PR: #65868 Problem Summary: Only a static overwrite reads the static partition: the commit passes it to withOverwrite to name the partition to replace. An INSERT writes each row to the partition its own values name. The write binding still resolved the static partition for an INSERT, and so asked fe-core for the values cast to their column types. A value that cast rejects failed an INSERT whose rows accept it: PARTITION (dt = '2024-13-45') on a DATE column, which the non-strict row cast writes as NULL, and PARTITION (p = X'4142') on a VARBINARY(2) column, whose constant-folding cast from a VARBINARY literal is not supported. Resolve the static partition for an overwrite only, so an INSERT is planned as before this change. branch-4.x also casts for an INSERT, but nothing depends on it. ### Release note None ### Check List (For Author) - Test: Unit Test - PaimonWriteBindingTest.insertLeavesTheStaticPartitionValuesAlone: an INSERT binding never asks for the cast values and carries no static partition; resolving it for an INSERT makes the case fail. - PaimonBackendBoundTableTest checks SQL NULL against the string 'NULL' on an overwrite, the only write that resolves the static partition. - Behavior changed: No. An INSERT with a static partition is planned as on master. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent b9874e4 commit 454af09

3 files changed

Lines changed: 50 additions & 28 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWriteBinding.java‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -62,10 +62,14 @@ private PaimonWriteBinding(String tableName, FileStoreTable table,
6262

6363
static PaimonWriteBinding create(PaimonTableHandle handle, FileStoreTable table,
6464
Map<String, String> hadoopConfig, ConnectorWriteHandle writeHandle) {
65-
Map<String, String> staticPartition = resolveStaticPartition(table, writeHandle);
66-
FileStoreTable writeTable = configureTableForWrite(table, writeHandle.isOverwrite(), staticPartition);
65+
boolean overwrite = writeHandle.isOverwrite();
66+
// Only a static overwrite reads the partition values. An INSERT writes each row to the partition its own
67+
// values name, so it leaves them alone: casting them could only fail a write the rows would accept.
68+
Map<String, String> staticPartition = overwrite
69+
? resolveStaticPartition(table, writeHandle) : Collections.emptyMap();
70+
FileStoreTable writeTable = configureTableForWrite(table, overwrite, staticPartition);
6771
return new PaimonWriteBinding(handle.getDatabaseName() + "." + handle.getTableName(),
68-
writeTable, hadoopConfig, writeHandle.isOverwrite(), staticPartition);
72+
writeTable, hadoopConfig, overwrite, staticPartition);
6973
}
7074

7175
static FileStoreTable configureTableForWrite(FileStoreTable table, boolean overwrite,
@@ -83,7 +87,7 @@ static FileStoreTable configureTableForWrite(FileStoreTable table, boolean overw
8387
}
8488

8589
/**
86-
* The static partition as Paimon's static overwrite parses it: SQL NULL becomes the table's
90+
* The static partition of an overwrite, as Paimon's static overwrite parses it: SQL NULL becomes the table's
8791
* {@code partition.default-name}, and every other value is the one the written rows carry, cast to the
8892
* column type.
8993
*/
@@ -109,9 +113,9 @@ static Map<String, String> resolveStaticPartition(FileStoreTable table, Connecto
109113
() -> "missing the cast value of static partition column " + key);
110114
if (table.rowType().getField(canonicalName).type().getTypeRoot()
111115
== DataTypeRoot.TIMESTAMP_WITH_LOCAL_TIME_ZONE) {
112-
value = toSdkLocalTime(canonicalName, value, writeHandle.isOverwrite());
116+
value = toSdkLocalTime(canonicalName, value);
113117
}
114-
if (writeHandle.isOverwrite() && defaultPartitionName.equals(value)) {
118+
if (defaultPartitionName.equals(value)) {
115119
// Paimon's static overwrite reads this string as the NULL partition of any partition type, so
116120
// a value equal to it would overwrite the NULL partition instead.
117121
throw new DorisConnectorException("Static partition value for column '" + canonicalName
@@ -127,14 +131,14 @@ static Map<String, String> resolveStaticPartition(FileStoreTable table, Connecto
127131
* Paimon parses a static overwrite value of a TIMESTAMP WITH LOCAL TIME ZONE column as local time in the FE
128132
* JVM's default zone. Doris binds such a column as TIMESTAMPTZ, so the cast value is the instant in UTC with
129133
* its offset; it is moved to the JVM zone, with a space between the date and the time as Paimon's parser
130-
* needs. An overwrite rejects an instant whose local time a DST change repeats there: both instants of the
131-
* overlap format to the same value, which Paimon reads as the earlier one.
134+
* needs. An instant whose local time a DST change repeats there is rejected: both instants of the overlap
135+
* format to the same value, which Paimon reads as the earlier one.
132136
*/
133-
private static String toSdkLocalTime(String column, String value, boolean overwrite) {
137+
private static String toSdkLocalTime(String column, String value) {
134138
ZoneId sdkZone = ZoneId.systemDefault();
135139
LocalDateTime sdkLocal = OffsetDateTime.parse(value.replace(' ', 'T'), DateTimeFormatter.ISO_OFFSET_DATE_TIME)
136140
.atZoneSameInstant(sdkZone).toLocalDateTime();
137-
if (overwrite && sdkZone.getRules().getValidOffsets(sdkLocal).size() > 1) {
141+
if (sdkZone.getRules().getValidOffsets(sdkLocal).size() > 1) {
138142
throw new DorisConnectorException("Static LTZ partition value for column '" + column
139143
+ "' is ambiguous in FE JVM time zone " + sdkZone
140144
+ " and cannot be represented in a static overwrite");

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonBackendBoundTableTest.java‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -119,7 +119,7 @@ public void writeBindingKeepsCatalogOnFeButRemovesItFromBePayload(@TempDir Path
119119

120120
PaimonWriteBinding binding = PaimonWriteBinding.create(
121121
dataHandle(), table, Collections.emptyMap(),
122-
writeHandle(Collections.emptyMap(), Collections.emptySet()));
122+
writeHandle(false, Collections.emptyMap(), Collections.emptySet()));
123123
FileStoreTable backendTable = deserializeTable(binding.getSerializedTable());
124124

125125
Assertions.assertNotNull(binding.getTable().catalogEnvironment().catalogLoader());
@@ -140,9 +140,9 @@ public void staticPartitionDistinguishesSqlNullFromNullString(@TempDir Path ware
140140
Map<String, String> requested = Collections.singletonMap("pt", "NULL");
141141

142142
PaimonWriteBinding literal = PaimonWriteBinding.create(dataHandle(), table,
143-
Collections.emptyMap(), writeHandle(requested, Collections.emptySet()));
143+
Collections.emptyMap(), writeHandle(true, requested, Collections.emptySet()));
144144
PaimonWriteBinding sqlNull = PaimonWriteBinding.create(dataHandle(), table,
145-
Collections.emptyMap(), writeHandle(requested, Collections.singleton("pt")));
145+
Collections.emptyMap(), writeHandle(true, requested, Collections.singleton("pt")));
146146

147147
Assertions.assertEquals("NULL", literal.getStaticPartition().get("pt"));
148148
Assertions.assertEquals(table.coreOptions().partitionDefaultName(),
@@ -774,8 +774,12 @@ private static PaimonTableHandle dataHandle() {
774774
return new PaimonTableHandle("db", "tbl", Collections.emptyList(), Collections.emptyList());
775775
}
776776

777-
/** An INSERT whose static partition values need no cast, so the handle's default cast spec applies. */
778-
private static ConnectorWriteHandle writeHandle(Map<String, String> staticPartition, Set<String> nullKeys) {
777+
/**
778+
* A write whose static partition values need no cast, so the handle's default cast spec applies. Only an
779+
* overwrite resolves the static partition.
780+
*/
781+
private static ConnectorWriteHandle writeHandle(boolean overwrite, Map<String, String> staticPartition,
782+
Set<String> nullKeys) {
779783
return new ConnectorWriteHandle() {
780784
@Override
781785
public ConnectorTableHandle getTableHandle() {
@@ -789,7 +793,7 @@ public List<ConnectorColumn> getColumns() {
789793

790794
@Override
791795
public boolean isOverwrite() {
792-
return false;
796+
return overwrite;
793797
}
794798

795799
@Override

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWriteBindingTest.java‎

Lines changed: 26 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@
4545
import java.util.Map;
4646
import java.util.Set;
4747
import java.util.TimeZone;
48+
import java.util.function.Supplier;
4849

4950
/**
5051
* How a static partition reaches Paimon's static overwrite. The written rows carry each PARTITION literal cast
@@ -99,10 +100,6 @@ public void staticOverwriteRejectsTheDefaultPartitionName(@TempDir Path warehous
99100
() -> PaimonWriteBinding.resolveStaticPartition(table,
100101
write(true, "part", DEFAULT_NAME, DEFAULT_NAME)));
101102
Assertions.assertTrue(error.getMessage().contains("cannot be represented"), error.getMessage());
102-
103-
// An INSERT does not overwrite anything, and its rows keep the value as written.
104-
Assertions.assertEquals(DEFAULT_NAME, PaimonWriteBinding.resolveStaticPartition(table,
105-
write(false, "part", DEFAULT_NAME, DEFAULT_NAME)).get("part"));
106103
}
107104

108105
@Test
@@ -132,10 +129,6 @@ public void staticLtzOverwriteRejectsAnInstantTheSdkZoneRepeats(@TempDir Path wa
132129
() -> PaimonWriteBinding.resolveStaticPartition(table, write(true, "part", instant, instant)));
133130
Assertions.assertTrue(error.getMessage().contains("ambiguous"), error.getMessage());
134131
Assertions.assertTrue(error.getMessage().contains("America/Los_Angeles"), error.getMessage());
135-
136-
// An INSERT does not overwrite anything, so it keeps the value.
137-
Assertions.assertEquals("2023-11-05 01:30:00.123456", PaimonWriteBinding.resolveStaticPartition(table,
138-
write(false, "part", instant, instant)).get("part"));
139132
}
140133
}
141134

@@ -160,12 +153,33 @@ public void staticPartitionNullUsesTheDefaultPartitionName(@TempDir Path warehou
160153
Collections.singletonMap("partition.default-name", DEFAULT_NAME),
161154
new DataField(1, "part", DataTypes.INT()));
162155
ConnectorWriteHandle write = handle(true, Collections.singletonMap("part", "NULL"),
163-
Collections.emptyMap(), Collections.singleton("part"));
156+
Collections::emptyMap, Collections.singleton("part"));
164157

165158
Assertions.assertEquals(DEFAULT_NAME,
166159
PaimonWriteBinding.resolveStaticPartition(table, write).get("part"));
167160
}
168161

162+
@Test
163+
public void insertLeavesTheStaticPartitionValuesAlone(@TempDir Path warehouse) {
164+
// Only a static overwrite reads the static partition. An INSERT writes each row to the partition its own
165+
// values name, so a value the cast rejects (a row keeps it as NULL, or as a fitting binary) or an
166+
// overwrite cannot represent must not fail it. MUTATION: resolving the static partition of an INSERT
167+
// -> red.
168+
FileStoreTable table = partitionedTable(warehouse,
169+
Collections.singletonMap("partition.default-name", DEFAULT_NAME),
170+
new DataField(1, "part", DataTypes.STRING()));
171+
ConnectorWriteHandle insert = handle(false, Collections.singletonMap("part", DEFAULT_NAME), () -> {
172+
throw new AssertionError("an INSERT must not cast its static partition values");
173+
}, Collections.emptySet());
174+
175+
PaimonWriteBinding binding = PaimonWriteBinding.create(
176+
new PaimonTableHandle("db", "tbl", Collections.emptyList(), Collections.emptyList()),
177+
table, Collections.emptyMap(), insert);
178+
179+
Assertions.assertFalse(binding.isOverwrite());
180+
Assertions.assertTrue(binding.getStaticPartition().isEmpty(), binding.getStaticPartition().toString());
181+
}
182+
169183
private static void assertStaticLtzRoundTrip(FileStoreTable table, String instant) {
170184
String value = PaimonWriteBinding.resolveStaticPartition(table, write(true, "part", instant, instant))
171185
.get("part");
@@ -189,11 +203,11 @@ private static FileStoreTable partitionedTable(Path warehouse, Map<String, Strin
189203
private static ConnectorWriteHandle write(boolean overwrite, String name, String writtenValue,
190204
String castValue) {
191205
return handle(overwrite, Collections.singletonMap(name, writtenValue),
192-
Collections.singletonMap(name, castValue), Collections.emptySet());
206+
() -> Collections.singletonMap(name, castValue), Collections.emptySet());
193207
}
194208

195209
private static ConnectorWriteHandle handle(boolean overwrite, Map<String, String> spec,
196-
Map<String, String> castSpec, Set<String> nullKeys) {
210+
Supplier<Map<String, String>> castSpec, Set<String> nullKeys) {
197211
return new ConnectorWriteHandle() {
198212
@Override
199213
public ConnectorTableHandle getTableHandle() {
@@ -222,7 +236,7 @@ public Set<String> getStaticPartitionNullKeys() {
222236

223237
@Override
224238
public Map<String, String> getCastStaticPartitionSpec() {
225-
return castSpec;
239+
return castSpec.get();
226240
}
227241
};
228242
}

0 commit comments

Comments
 (0)