Skip to content

Commit 2a790ff

Browse files
morningmanclaude
andcommitted
[fix](paimon) Name binary partitions of Paimon writes by their bytes
### What problem does this PR solve? Issue Number: None Related PR: #68874 Problem Summary: Paimon names a partition after the string of each value. With partition.legacy-name = true, the default, that string is toString() of the value, which for a BINARY / VARBINARY value is the Java array object ([B@1b6d3586) rather than its bytes, and every resolution of the name differs. With Paimon 2.1.0, Paimon's own writer and reader cannot round-trip such a partition (the read fails with FileNotFoundException), and a Doris INSERT into such a table had the BE write its files under one [B@... directory, then failed the commit with "Cannot recover from this checkpoint because some files in the snapshot that need to be resubmitted have been deleted", leaving the files behind. A write to a table with a binary partition column and legacy partition names now fails while it is planned, naming the table option 'partition.legacy-name' = 'false' that makes such a table work. With partition.legacy-name = false, Paimon names a binary partition after the text of its bytes, and Doris INSERT already writes and reads it correctly. #68874 rejected a static overwrite of a binary partition column with a non-NULL value: Paimon's static overwrite takes the UTF-8 bytes of the value's string, and the cast value reaches the connector as 0x and its hex digits, which names another partition (an overwrite without rows then reported success and left the given partition in place). The connector now decodes the hex digits and passes the text the bytes decode to as UTF-8, which Paimon encodes back to the same bytes. Bytes that are not UTF-8 text, such as 0xEF, are still rejected, since no string names them. ### Release note Fix: a write to a Paimon table whose binary partition column uses legacy partition names fails at planning with a hint to set 'partition.legacy-name' = 'false', instead of failing its commit and leaving files behind. INSERT OVERWRITE ... PARTITION (p = X'...') on a binary partition column of a Paimon table is supported when the bytes are UTF-8 text. ### Check List (For Author) - Test: Unit Test, Manual test - PaimonWriteBindingTest: binary static values become the text of their bytes, bytes that are not UTF-8 are rejected, a write of a legacy-named binary partition fails while one with new names passes - fe-connector-paimon module tests - Manual: local file:// Paimon catalog with tables created by the Paimon API (see the PR description) - Behavior changed: Yes. A write to a Paimon table with a binary partition column and legacy partition names fails at planning instead of at commit; a static overwrite of a binary partition column with a binary literal whose bytes are UTF-8 text is accepted instead of rejected. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 1bb3537 commit 2a790ff

2 files changed

Lines changed: 115 additions & 25 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWriteBinding.java‎

Lines changed: 61 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,18 @@
2929
import org.apache.paimon.utils.InstantiationUtil;
3030

3131
import java.io.IOException;
32+
import java.nio.ByteBuffer;
33+
import java.nio.charset.CharacterCodingException;
34+
import java.nio.charset.CodingErrorAction;
35+
import java.nio.charset.StandardCharsets;
3236
import java.time.LocalDateTime;
3337
import java.time.OffsetDateTime;
3438
import java.time.ZoneId;
3539
import java.time.format.DateTimeFormatter;
3640
import java.util.Base64;
3741
import java.util.Collections;
3842
import java.util.HashMap;
43+
import java.util.HexFormat;
3944
import java.util.LinkedHashMap;
4045
import java.util.List;
4146
import java.util.Map;
@@ -80,14 +85,39 @@ private PaimonWriteBinding(String tableName, FileStoreTable table,
8085
*/
8186
static PaimonWriteBinding create(PaimonTableHandle handle, FileStoreTable table,
8287
Map<String, String> hadoopConfig, ConnectorWriteHandle writeHandle, boolean charPadding) {
88+
String tableName = handle.getDatabaseName() + "." + handle.getTableName();
89+
checkBinaryPartitionNames(tableName, table);
8390
boolean overwrite = writeHandle.isOverwrite();
8491
// Only a static overwrite reads the partition values. An INSERT writes each row to the partition its own
8592
// values name, so it leaves them alone: casting them could only fail a write the rows would accept.
8693
Map<String, String> staticPartition = overwrite
8794
? resolveStaticPartition(table, writeHandle, charPadding) : Collections.emptyMap();
8895
FileStoreTable writeTable = configureTableForWrite(table, overwrite, staticPartition);
89-
return new PaimonWriteBinding(handle.getDatabaseName() + "." + handle.getTableName(),
90-
writeTable, hadoopConfig, overwrite, staticPartition, charPadding);
96+
return new PaimonWriteBinding(tableName, writeTable, hadoopConfig, overwrite, staticPartition, charPadding);
97+
}
98+
99+
/**
100+
* With {@code partition.legacy-name} = true, the default, Paimon names a partition after the string of each
101+
* value, which for a binary value is the Java array object ({@code [B@1b6d3586}) rather than its bytes. Every
102+
* resolution of the name differs, so a write fails its commit after the BE wrote its files under one of them,
103+
* and Paimon itself cannot read such a partition back.
104+
*/
105+
static void checkBinaryPartitionNames(String tableName, FileStoreTable table) {
106+
if (!CoreOptions.fromMap(table.options()).legacyPartitionName()) {
107+
return;
108+
}
109+
for (String partitionKey : table.partitionKeys()) {
110+
if (isBinary(table.rowType().getField(partitionKey).type())) {
111+
throw new DorisConnectorException("Cannot write Paimon table " + tableName + ": its binary"
112+
+ " partition column '" + partitionKey + "' needs the table option 'partition.legacy-name'"
113+
+ " = 'false'. With legacy partition names Paimon names a binary partition after a Java"
114+
+ " array object rather than its bytes and cannot read the partition back");
115+
}
116+
}
117+
}
118+
119+
private static boolean isBinary(DataType type) {
120+
return type.getTypeRoot() == DataTypeRoot.BINARY || type.getTypeRoot() == DataTypeRoot.VARBINARY;
91121
}
92122

93123
/**
@@ -118,7 +148,8 @@ static FileStoreTable configureTableForWrite(FileStoreTable table, boolean overw
118148
* The static partition of an overwrite, as Paimon's static overwrite parses it: SQL NULL becomes the table's
119149
* {@code partition.default-name}, and every other value is the one the written rows carry, cast to the
120150
* column type, and padded as the BE writer pads the CHAR values of the rows ({@link #charPaddedFields}): Paimon
121-
* parses a partition value without padding it. A binary column can only be statically overwritten with NULL.
151+
* parses a partition value without padding it. A binary value is the text of its bytes ({@link
152+
* #binaryStaticValue}).
122153
*/
123154
static Map<String, String> resolveStaticPartition(FileStoreTable table, ConnectorWriteHandle writeHandle,
124155
boolean charPadding) {
@@ -133,15 +164,6 @@ static Map<String, String> resolveStaticPartition(FileStoreTable table, Connecto
133164
throw new DorisConnectorException("Column '" + key
134165
+ "' is not a partition column of Paimon table");
135166
}
136-
DataTypeRoot typeRoot = table.rowType().getField(canonicalName).type().getTypeRoot();
137-
if ((typeRoot == DataTypeRoot.BINARY || typeRoot == DataTypeRoot.VARBINARY)
138-
&& !writeHandle.getStaticPartitionNullKeys().contains(key)) {
139-
// Paimon's static overwrite takes the UTF-8 bytes of a value's string, which cannot name every
140-
// binary partition, and a wrong name overwrites another partition without an error. Checked
141-
// before the values are cast.
142-
throw new DorisConnectorException("Static overwrite of binary partition column '"
143-
+ canonicalName + "' is not supported");
144-
}
145167
requestedColumns.put(key, canonicalName);
146168
}
147169
CoreOptions options = CoreOptions.fromMap(table.options());
@@ -165,6 +187,9 @@ static Map<String, String> resolveStaticPartition(FileStoreTable table, Connecto
165187
if (type.getTypeRoot() == DataTypeRoot.CHAR && charPaddedFields.test(canonicalName)) {
166188
value = padChar(value, ((CharType) type).getLength());
167189
}
190+
if (isBinary(type)) {
191+
value = binaryStaticValue(canonicalName, value);
192+
}
168193
if (defaultPartitionName.equals(value)) {
169194
// Paimon's static overwrite reads this string as the NULL partition of any partition type, so
170195
// a value equal to it would overwrite the NULL partition instead.
@@ -219,6 +244,30 @@ private static String toSdkLocalTime(String column, String value) {
219244
* Pads a CHAR value with spaces to the column length in code points, as PaimonWriteSchema pads the rows' CHAR
220245
* values; a longer value is kept, as it is there.
221246
*/
247+
/**
248+
* Paimon's static overwrite takes the UTF-8 bytes of a partition value's string, so the bytes the rows carry,
249+
* which reach it as {@code 0x} and their hex digits, name their partition only as the text they decode to. A
250+
* wrong name would overwrite another partition without an error, so bytes that are not UTF-8 are rejected;
251+
* strict decoding also rejects overlong and surrogate sequences, so the text encodes back to the same bytes.
252+
*/
253+
private static String binaryStaticValue(String column, String value) {
254+
if (!value.startsWith("0x")) {
255+
throw new IllegalStateException("Binary static partition value of column " + column
256+
+ " lacks the 0x prefix: " + value);
257+
}
258+
byte[] bytes = HexFormat.of().parseHex(value.substring(2));
259+
try {
260+
return StandardCharsets.UTF_8.newDecoder()
261+
.onMalformedInput(CodingErrorAction.REPORT)
262+
.onUnmappableCharacter(CodingErrorAction.REPORT)
263+
.decode(ByteBuffer.wrap(bytes))
264+
.toString();
265+
} catch (CharacterCodingException e) {
266+
throw new DorisConnectorException("Static partition value " + value + " of binary column '" + column
267+
+ "' is not UTF-8 text, which a static overwrite of a Paimon table needs to name its partition");
268+
}
269+
}
270+
222271
private static String padChar(String value, int length) {
223272
int padLength = length - value.codePointCount(0, value.length());
224273
return padLength > 0 ? value + " ".repeat(padLength) : value;

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWriteBindingTest.java‎

Lines changed: 54 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,8 @@ public class PaimonWriteBindingTest {
6161

6262
private static final String DEFAULT_NAME = "__CUSTOM_DEFAULT__";
6363
private static final String DYNAMIC_OVERWRITE = CoreOptions.DYNAMIC_PARTITION_OVERWRITE.key();
64+
private static final Map<String, String> NEW_PARTITION_NAMES =
65+
Collections.singletonMap("partition.legacy-name", "false");
6466

6567
private TimeZone jvmZone;
6668

@@ -197,27 +199,66 @@ public void staticLtzPartitionOutsideAnOverlapRoundTripsThroughPaimon(@TempDir P
197199
}
198200

199201
@Test
200-
public void staticOverwriteRejectsABinaryPartitionValue(@TempDir Path warehouse) {
201-
// Paimon's static overwrite takes the UTF-8 bytes of a value's string: "0x4142" names the partition of
202-
// those six bytes, not of the two bytes 41 42 the rows carry, so an overwrite without rows would report
203-
// success and leave the partition it was given in place. The check comes before the values are cast,
204-
// which a binary value may not survive. MUTATION: dropping the check -> red.
202+
public void staticBinaryPartitionIsTheTextOfTheBytesTheRowsCarry(@TempDir Path warehouse) {
203+
// Paimon's static overwrite takes the UTF-8 bytes of a value's string: "0x4142", as the cast value of a
204+
// VARBINARY column reaches the connector, names the partition of those six bytes, not of the two bytes
205+
// 41 42 the rows carry, so an overwrite without rows would report success and leave the partition it was
206+
// given in place. MUTATION: passing the cast value through -> red.
205207
Assertions.assertArrayEquals("0x4142".getBytes(StandardCharsets.UTF_8),
206208
(byte[]) TypeUtils.castFromString("0x4142", DataTypes.BYTES()));
207209
for (DataType type : Arrays.asList(DataTypes.BYTES(), DataTypes.VARBINARY(2), DataTypes.BINARY(2))) {
208-
FileStoreTable table = partitionedTable(warehouse, Collections.emptyMap(),
209-
new DataField(1, "part", type));
210-
ConnectorWriteHandle overwrite = handle(true, Collections.singletonMap("part", "0x4142"), () -> {
211-
throw new AssertionError("a binary static partition value must be rejected before it is cast");
212-
}, Collections.emptySet());
210+
FileStoreTable table = partitionedTable(warehouse, NEW_PARTITION_NAMES, new DataField(1, "part", type));
211+
212+
String value = PaimonWriteBinding.resolveStaticPartition(table, write(true, "part", "AB", "0x4142"),
213+
true).get("part");
213214

215+
Assertions.assertArrayEquals(new byte[] {0x41, 0x42}, (byte[]) TypeUtils.castFromString(value, type),
216+
type.toString());
217+
}
218+
}
219+
220+
@Test
221+
public void staticBinaryPartitionRejectsBytesThatAreNotUtf8(@TempDir Path warehouse) {
222+
// No string's UTF-8 bytes are a lone 0xEF, an overlong '/' or an encoded surrogate, so Paimon's static
223+
// overwrite cannot name such a partition; a lenient decoder would name another one through U+FFFD.
224+
// MUTATION: decoding with replacement -> red.
225+
FileStoreTable table = partitionedTable(warehouse, NEW_PARTITION_NAMES,
226+
new DataField(1, "part", DataTypes.BYTES()));
227+
for (String castValue : Arrays.asList("0xEF", "0xC0AF", "0xEDA080")) {
214228
DorisConnectorException error = Assertions.assertThrows(DorisConnectorException.class,
215-
() -> PaimonWriteBinding.resolveStaticPartition(table, overwrite, true));
216-
Assertions.assertTrue(error.getMessage().contains("binary partition column 'part'"),
217-
error.getMessage());
229+
() -> PaimonWriteBinding.resolveStaticPartition(table, write(true, "part", castValue, castValue),
230+
true));
231+
Assertions.assertTrue(error.getMessage().contains("is not UTF-8 text"), error.getMessage());
218232
}
219233
}
220234

235+
@Test
236+
public void writeOfABinaryPartitionNeedsNewPartitionNames(@TempDir Path warehouse) {
237+
// With partition.legacy-name = true, the default, Paimon names a binary partition after the Java array
238+
// object, so the commit of the written files fails and Paimon cannot read the partition back. The write
239+
// fails while it is planned; a table with new partition names, or without a binary partition, is written.
240+
// MUTATION: dropping the check -> red.
241+
PaimonTableHandle handle = new PaimonTableHandle("db", "tbl", Collections.emptyList(),
242+
Collections.emptyList());
243+
for (DataType type : Arrays.asList(DataTypes.BYTES(), DataTypes.VARBINARY(2), DataTypes.BINARY(2))) {
244+
FileStoreTable legacy = partitionedTable(warehouse, Collections.emptyMap(), new DataField(1, "part", type));
245+
for (boolean overwrite : new boolean[] {false, true}) {
246+
DorisConnectorException error = Assertions.assertThrows(DorisConnectorException.class,
247+
() -> PaimonWriteBinding.create(handle, legacy, Collections.emptyMap(),
248+
write(overwrite, "part", "AB", "0x4142"), true));
249+
Assertions.assertTrue(error.getMessage().contains("'partition.legacy-name' = 'false'"),
250+
error.getMessage());
251+
}
252+
FileStoreTable renamed = partitionedTable(warehouse, NEW_PARTITION_NAMES, new DataField(1, "part", type));
253+
Assertions.assertEquals("AB", PaimonWriteBinding.create(handle, renamed, Collections.emptyMap(),
254+
write(true, "part", "AB", "0x4142"), true).getStaticPartition().get("part"));
255+
}
256+
FileStoreTable legacyString = partitionedTable(warehouse, Collections.emptyMap(),
257+
new DataField(1, "part", DataTypes.STRING()));
258+
Assertions.assertFalse(PaimonWriteBinding.create(handle, legacyString, Collections.emptyMap(),
259+
write(false, "part", "AB", "AB"), true).isOverwrite());
260+
}
261+
221262
@Test
222263
public void staticPartitionNullUsesTheDefaultPartitionName(@TempDir Path warehouse) {
223264
// A binary column too: SQL NULL names its partition by partition.default-name, not by bytes.

0 commit comments

Comments
 (0)