Skip to content

Commit 2c95fb1

Browse files
morningmanclaude
andcommitted
[fix](paimon) Cast binary static partition values to columns of any length
### What problem does this PR solve? Issue Number: None Related PR: #68874 Problem Summary: Paimon BINARY(n) and VARBINARY(n) columns map to Doris VARBINARY(n), while a binary literal such as X'4142' is VARBINARY of the maximum length. PluginDrivenInsertCommandContext.castStaticPartitionSpec cast a static partition value with Literal.checkedCastTo, which casts a VARBINARY to no other length, so INSERT OVERWRITE ... PARTITION (p = X'4142') on such a column failed while it was planned with "Cast from 4142 to VARBINARY(2) not supported". Only BYTES columns, whose type is VARBINARY of the maximum length, reached the decoding of the previous commit. The rows of the same statement are written with their bytes: BindSink leaves their cast to BE, which keeps them, since a VARBINARY length is metadata only. The cast value of a binary literal now keeps its bytes and takes the column's VARBINARY type, as the rows do. Paimon parses the static value back to those bytes for a BINARY(n) column whatever their length, as the rows carry them; for a VARBINARY(n) column it fails the commit, after the BE wrote its files, when the bytes are longer than n, so such a value is now rejected while the write is planned, naming the column, the value and its length. The Javadoc of padChar, which the previous commit had separated from its method, is moved back. ### Release note None ### Check List (For Author) - Test: Unit Test - PluginDrivenTableSinkBindingTest: X'41', X'4142' and X'414243' keep their bytes when cast for a VARBINARY(2) column through the sink binding - PaimonWriteBindingTest: a BINARY(2) static value keeps one or three bytes, and a VARBINARY(2) value longer than two bytes is rejected - IcebergStaticPartitionLiteralTest, PluginDrivenTableSinkTest, fe-connector-paimon module tests - Behavior changed: Yes. INSERT OVERWRITE ... PARTITION (p = X'...') on a Paimon BINARY(n) / VARBINARY(n) partition column is planned instead of failing on the cast; a value longer than a VARBINARY(n) column fails while planned instead of at commit. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 2a790ff commit 2c95fb1

4 files changed

Lines changed: 65 additions & 7 deletions

File tree

‎fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonWriteBinding.java‎

Lines changed: 15 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
import org.apache.paimon.types.CharType;
2727
import org.apache.paimon.types.DataType;
2828
import org.apache.paimon.types.DataTypeRoot;
29+
import org.apache.paimon.types.VarBinaryType;
2930
import org.apache.paimon.utils.InstantiationUtil;
3031

3132
import java.io.IOException;
@@ -188,7 +189,7 @@ static Map<String, String> resolveStaticPartition(FileStoreTable table, Connecto
188189
value = padChar(value, ((CharType) type).getLength());
189190
}
190191
if (isBinary(type)) {
191-
value = binaryStaticValue(canonicalName, value);
192+
value = binaryStaticValue(canonicalName, value, type);
192193
}
193194
if (defaultPartitionName.equals(value)) {
194195
// Paimon's static overwrite reads this string as the NULL partition of any partition type, so
@@ -240,22 +241,26 @@ private static String toSdkLocalTime(String column, String value) {
240241
return sdkLocal.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME).replace('T', ' ');
241242
}
242243

243-
/**
244-
* Pads a CHAR value with spaces to the column length in code points, as PaimonWriteSchema pads the rows' CHAR
245-
* values; a longer value is kept, as it is there.
246-
*/
247244
/**
248245
* Paimon's static overwrite takes the UTF-8 bytes of a partition value's string, so the bytes the rows carry,
249246
* which reach it as {@code 0x} and their hex digits, name their partition only as the text they decode to. A
250247
* wrong name would overwrite another partition without an error, so bytes that are not UTF-8 are rejected;
251248
* strict decoding also rejects overlong and surrogate sequences, so the text encodes back to the same bytes.
249+
* Paimon keeps those bytes for a BINARY column whatever their length, as the rows keep them, and fails the
250+
* commit for a VARBINARY value longer than its column, so such a value is rejected here, before any file is
251+
* written.
252252
*/
253-
private static String binaryStaticValue(String column, String value) {
253+
private static String binaryStaticValue(String column, String value, DataType type) {
254254
if (!value.startsWith("0x")) {
255255
throw new IllegalStateException("Binary static partition value of column " + column
256256
+ " lacks the 0x prefix: " + value);
257257
}
258258
byte[] bytes = HexFormat.of().parseHex(value.substring(2));
259+
if (type.getTypeRoot() == DataTypeRoot.VARBINARY && bytes.length > ((VarBinaryType) type).getLength()) {
260+
throw new DorisConnectorException("Static partition value " + value + " of binary column '" + column
261+
+ "' has " + bytes.length + " bytes, more than its type " + type + " holds, which a static"
262+
+ " overwrite of a Paimon table rejects");
263+
}
259264
try {
260265
return StandardCharsets.UTF_8.newDecoder()
261266
.onMalformedInput(CodingErrorAction.REPORT)
@@ -268,6 +273,10 @@ private static String binaryStaticValue(String column, String value) {
268273
}
269274
}
270275

276+
/**
277+
* Pads a CHAR value with spaces to the column length in code points, as PaimonWriteSchema pads the rows' CHAR
278+
* values; a longer value is kept, as it is there.
279+
*/
271280
private static String padChar(String value, int length) {
272281
int padLength = length - value.codePointCount(0, value.length());
273282
return padLength > 0 ? value + " ".repeat(padLength) : value;

‎fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonWriteBindingTest.java‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
import java.time.OffsetDateTime;
4545
import java.util.Arrays;
4646
import java.util.Collections;
47+
import java.util.HexFormat;
4748
import java.util.LinkedHashMap;
4849
import java.util.List;
4950
import java.util.Map;
@@ -232,6 +233,33 @@ public void staticBinaryPartitionRejectsBytesThatAreNotUtf8(@TempDir Path wareho
232233
}
233234
}
234235

236+
@Test
237+
public void staticBinaryPartitionKeepsTheLengthOfTheBytes(@TempDir Path warehouse) {
238+
// The rows keep their bytes whatever the column length, and Paimon's static overwrite keeps them for a
239+
// BINARY(n) column too. For a VARBINARY(n) column it fails the commit, after the files are written, when the
240+
// bytes are longer, so such a value fails while the write is planned. MUTATION: no length check -> red.
241+
FileStoreTable binary = partitionedTable(warehouse, NEW_PARTITION_NAMES,
242+
new DataField(1, "part", DataTypes.BINARY(2)));
243+
for (String castValue : Arrays.asList("0x41", "0x414243")) {
244+
String value = PaimonWriteBinding.resolveStaticPartition(binary,
245+
write(true, "part", castValue, castValue), true).get("part");
246+
Assertions.assertArrayEquals(HexFormat.of().parseHex(castValue.substring(2)),
247+
(byte[]) TypeUtils.castFromString(value, DataTypes.BINARY(2)), castValue);
248+
}
249+
250+
FileStoreTable varbinary = partitionedTable(warehouse, NEW_PARTITION_NAMES,
251+
new DataField(1, "part", DataTypes.VARBINARY(2)));
252+
Assertions.assertEquals("A", PaimonWriteBinding.resolveStaticPartition(varbinary,
253+
write(true, "part", "0x41", "0x41"), true).get("part"));
254+
Assertions.assertThrows(IllegalArgumentException.class,
255+
() -> TypeUtils.castFromString("ABC", DataTypes.VARBINARY(2)));
256+
DorisConnectorException error = Assertions.assertThrows(DorisConnectorException.class,
257+
() -> PaimonWriteBinding.resolveStaticPartition(varbinary,
258+
write(true, "part", "0x414243", "0x414243"), true));
259+
Assertions.assertTrue(error.getMessage().contains("has 3 bytes, more than its type VARBINARY(2) holds"),
260+
error.getMessage());
261+
}
262+
235263
@Test
236264
public void writeOfABinaryPartitionNeedsNewPartitionNames(@TempDir Path warehouse) {
237265
// With partition.legacy-name = true, the default, Paimon names a binary partition after the Java array

‎fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertCommandContext.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828
import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
2929
import org.apache.doris.nereids.trees.expressions.literal.VarBinaryLiteral;
3030
import org.apache.doris.nereids.types.DataType;
31+
import org.apache.doris.nereids.types.VarBinaryType;
3132
import org.apache.doris.nereids.types.coercion.CharacterType;
3233

3334
import com.google.common.base.Preconditions;
@@ -139,6 +140,11 @@ private static String specValue(Literal literal) {
139140
}
140141

141142
private static Literal writtenValue(Literal value, DataType columnType) {
143+
if (value instanceof VarBinaryLiteral && columnType instanceof VarBinaryType) {
144+
// A binary literal is VARBINARY of the maximum length, which Literal casts to no other length. BindSink
145+
// leaves that cast of the rows to BE, which keeps the bytes: a VARBINARY length is metadata only.
146+
return new VarBinaryLiteral((VarBinaryType) columnType, (byte[]) value.getValue());
147+
}
142148
if (!value.getDataType().isStringLikeType() || !columnType.isStringLikeType()) {
143149
Expression cast = value.checkedCastTo(columnType);
144150
Preconditions.checkState(cast instanceof Literal, "cast of literal %s is not a literal", value);

‎fe/fe-core/src/test/java/org/apache/doris/planner/PluginDrivenTableSinkBindingTest.java‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -208,13 +208,28 @@ public void staticPartitionBinaryValueKeepsItsBytesWhenCast() throws AnalysisExc
208208
// byte array ([B@...), which names no partition. MUTATION: getStringValue() for a cast binary value -> red.
209209
Map<String, Expression> partition = Collections.singletonMap("b",
210210
new VarBinaryLiteral(new byte[] {0x41, 0x42}));
211-
// The type Paimon's VARBINARY maps to.
211+
// The type Paimon's BYTES maps to.
212212
List<Column> schema = Collections.singletonList(
213213
new Column("b", ScalarType.createVarbinaryType(ScalarType.MAX_VARBINARY_LENGTH)));
214214

215215
Assertions.assertEquals(Collections.singletonMap("b", "0x4142"), castStaticPartitionSpec(partition, schema));
216216
}
217217

218+
@Test
219+
public void staticPartitionBinaryValueKeepsItsBytesForAColumnOfAnyLength() throws AnalysisException {
220+
// Paimon's BINARY(n) and VARBINARY(n) map to VARBINARY(n). A binary literal is VARBINARY of the maximum
221+
// length, which Literal casts to no other length; the rows keep their bytes in such a column, shorter or
222+
// longer than it, and so does the cast value. MUTATION: no binary case in writtenValue -> red ("Cast from
223+
// 4142 to VARBINARY(2) not supported").
224+
List<Column> schema = Collections.singletonList(new Column("b", ScalarType.createVarbinaryType(2)));
225+
for (String hex : Arrays.asList("41", "4142", "414243")) {
226+
Map<String, Expression> partition = Collections.singletonMap("b", new VarBinaryLiteral(hex));
227+
228+
Assertions.assertEquals(Collections.singletonMap("b", "0x" + hex),
229+
castStaticPartitionSpec(partition, schema));
230+
}
231+
}
232+
218233
@Test
219234
public void staticPartitionValueThatDoesNotFitItsTypeFailsOnlyWhenCast() throws AnalysisException {
220235
// Only a connector that reads the cast values may fail on them; the others keep reading the literal.

0 commit comments

Comments
 (0)