Skip to content

Commit f6b658b

Browse files
morningmanclaude
andcommitted
[fix](paimon) Pad CHAR values written to Paimon tables as Spark and Flink do
### What problem does this PR solve? Issue Number: None Related PR: #69045 Problem Summary: Doris wrote a CHAR(n) value into a Paimon table as it was, without the trailing spaces that pad it to n characters. Spark (with the paimon-spark extensions) and Flink pad CHAR values when they write a Paimon table, nested values too, and Paimon compares primary keys and hashes buckets by bytes, so a key that Doris wrote as 'a' and Spark as 'a ' made two rows of a primary-key table. On a local cluster with a file:// Paimon catalog, a CHAR(3) primary-key table into which Doris inserted 'k' and the Paimon API then wrote 'k ', as Spark and Flink write 'k', had two rows; it now has one. The Paimon JNI writer (PaimonWriteSchema) now pads the CHAR values of the rows it inserts, top-level and nested in ARRAY, MAP and ROW, with spaces to the column length in code points, as Paimon's own cast to CHAR does. A longer value is kept: Doris has already cut it or kept it, as the session asks. The static partition value of a CHAR partition column is padded the same way, because Paimon parses it without padding it, and an unpadded value would name another partition than the rows reach. UPDATE and DELETE rows are not padded: they carry the key Doris read back from the table, which has to stay byte-equal to the stored key. The new session variable enable_paimon_write_char_padding (default true) turns this off, for a table whose CHAR keys an earlier Doris wrote unpadded. FE passes it to the BE writer as the option doris.write.char-padding on the table it serializes for the writer; the FE committer's table goes without it. A writer that finds no such option, as from an older FE, does not pad. ### Release note Doris now pads CHAR(n) values written into Paimon tables with spaces to n characters, as Spark and Flink do, including values nested in ARRAY, MAP and ROW and static partition values; UPDATE and DELETE keep keys as stored. Doris reads a padded value back with its trailing spaces, so WHERE c = 'a' does not match it, as with data that Spark and Flink write. A key that an earlier Doris wrote unpadded is another key than its padded form; set enable_paimon_write_char_padding to false to keep writing CHAR values unpadded. ### Check List (For Author) - Test: Regression test, Unit Test, Manual test - test_paimon_write_char_padding (external_table_p0/paimon/write): a key that Doris and Spark both write is one row, nested and partition values are padded, a second static overwrite replaces the padded partition, the switch off, and DELETE and UPDATE of unpadded keys; it fails without this change at its first check - PaimonWriteSchemaTest: top-level and nested CHAR padding, padding off, UPDATE and DELETE rows left alone, reading the table option - PaimonWriteBindingTest: static CHAR partition value padding - PaimonBackendBoundTableTest: only the BE table carries the option - PaimonWritePlanProviderTest: the session variable, on by default - Manual test on a local cluster with a file:// Paimon catalog: the primary-key table above; CHAR values nested in ARRAY, MAP and ROW and the partition values of a static overwrite and of an INSERT are padded; with the switch off values are written unpadded, and with it on DELETE and UPDATE still match those unpadded keys - Behavior changed: Yes. CHAR values written into Paimon tables are padded to the column length unless enable_paimon_write_char_padding is false. - Does this need documentation: Yes (the new session variable) Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>
1 parent 2137067 commit f6b658b

12 files changed

Lines changed: 626 additions & 31 deletions

File tree

‎fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniWriter.java‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -183,8 +183,8 @@ public void open(String serializedTable, Map<String, String> hadoopConfig,
183183
this.bucketMode = table.bucketMode();
184184

185185
CoreOptions coreOptions = CoreOptions.fromMap(table.options());
186-
this.writeSchema = PaimonWriteSchema.create(
187-
table.rowType(), columnNames, changelogWrite);
186+
this.writeSchema = PaimonWriteSchema.create(table.rowType(), columnNames, changelogWrite,
187+
PaimonWriteSchema.charPadding(table.options()));
188188
this.arrowAdapter = new PaimonArrowBatchAdapter(
189189
writeSchema.inputType(), ZoneId.of(timeZone), allocator);
190190
validateWriteColumnsForMergeEngine(

‎fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonWriteSchema.java‎

Lines changed: 162 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -17,17 +17,27 @@
1717

1818
package org.apache.doris.paimon;
1919

20+
import org.apache.paimon.data.BinaryString;
21+
import org.apache.paimon.data.GenericArray;
2022
import org.apache.paimon.data.GenericRow;
23+
import org.apache.paimon.data.InternalArray;
24+
import org.apache.paimon.data.InternalMap;
2125
import org.apache.paimon.data.InternalRow;
26+
import org.apache.paimon.types.ArrayType;
27+
import org.apache.paimon.types.CharType;
2228
import org.apache.paimon.types.DataField;
29+
import org.apache.paimon.types.DataType;
30+
import org.apache.paimon.types.MapType;
2331
import org.apache.paimon.types.RowKind;
2432
import org.apache.paimon.types.RowType;
2533
import org.apache.paimon.types.TinyIntType;
34+
import org.apache.paimon.utils.BinaryStringUtils;
2635
import org.apache.paimon.utils.DefaultValueUtils;
2736

2837
import java.util.ArrayList;
2938
import java.util.Arrays;
3039
import java.util.List;
40+
import java.util.Map;
3141

3242
/**
3343
* Immutable mapping from Doris input columns to a Paimon table row.
@@ -41,8 +51,16 @@ final class PaimonWriteSchema {
4151
static final byte INSERT_OPERATION = 0;
4252
static final byte UPDATE_OPERATION = 1;
4353
static final byte DELETE_OPERATION = 2;
54+
/**
55+
* Table option through which FE asks the writer to pad CHAR values, set from the session variable
56+
* enable_paimon_write_char_padding. Keep in sync with PaimonWriteBinding.DORIS_WRITE_CHAR_PADDING in
57+
* fe-connector-paimon.
58+
*/
59+
static final String DORIS_WRITE_CHAR_PADDING = "doris.write.char-padding";
4460

4561
private final InternalRow.FieldGetter[] fieldGetters;
62+
/** Per input column, pads the CHAR values it holds; null when it holds none or padding is off. */
63+
private final CharPadder[] charPadders;
4664
private final RowType inputType;
4765
/** Maps Doris input-column position → Paimon table-schema position. */
4866
private final int[] tableFieldIndexes;
@@ -52,13 +70,15 @@ final class PaimonWriteSchema {
5270
private final int tableFieldCount;
5371

5472
private PaimonWriteSchema(int[] tableFieldIndexes, int[] omittedDefaultFieldIndexes,
55-
Object[] omittedDefaultValues, int tableFieldCount, RowType inputType) {
73+
Object[] omittedDefaultValues, int tableFieldCount, RowType inputType, boolean charPadding) {
5674
this.fieldGetters = new InternalRow.FieldGetter[inputType.getFieldCount()];
75+
this.charPadders = new CharPadder[inputType.getFieldCount()];
5776
for (int i = 0; i < fieldGetters.length; i++) {
5877
// Getters describe the transport row, not the target table. Transport fields are
5978
// nullable so an invalid Doris NULL survives projection and reaches Paimon's
6079
// authoritative table-schema validation.
6180
this.fieldGetters[i] = InternalRow.createFieldGetter(inputType.getTypeAt(i), i);
81+
this.charPadders[i] = charPadding ? charPadder(inputType.getTypeAt(i)) : null;
6282
}
6383
this.inputType = inputType;
6484
this.tableFieldIndexes = tableFieldIndexes;
@@ -82,6 +102,15 @@ static PaimonWriteSchema create(RowType tableType, String[] columnNames) {
82102

83103
static PaimonWriteSchema create(
84104
RowType tableType, String[] columnNames, boolean changelogWrite) {
105+
return create(tableType, columnNames, changelogWrite, false);
106+
}
107+
108+
/**
109+
* @param charPadding whether to pad the CHAR values of inserted rows as Paimon's cast to CHAR does; see
110+
* {@link #charPadder}
111+
*/
112+
static PaimonWriteSchema create(
113+
RowType tableType, String[] columnNames, boolean changelogWrite, boolean charPadding) {
85114
if (columnNames == null || columnNames.length == 0) {
86115
throw new IllegalArgumentException(
87116
"PaimonJniWriter requires explicit column names");
@@ -139,7 +168,16 @@ static PaimonWriteSchema create(
139168
Arrays.copyOf(omittedDefaultFieldIndexes, omittedDefaultCount),
140169
Arrays.copyOf(omittedDefaultValues, omittedDefaultCount),
141170
tableType.getFieldCount(),
142-
new RowType(inputFields));
171+
new RowType(inputFields),
172+
charPadding);
173+
}
174+
175+
/**
176+
* Whether FE asked for CHAR padding through {@link #DORIS_WRITE_CHAR_PADDING}. An FE that predates the option
177+
* does not set it, and does not pad its static partition values either.
178+
*/
179+
static boolean charPadding(Map<String, String> tableOptions) {
180+
return Boolean.parseBoolean(tableOptions.get(DORIS_WRITE_CHAR_PADDING));
143181
}
144182

145183
/** Paimon input fields in the exact order transported by Arrow C Data. */
@@ -163,13 +201,135 @@ GenericRow tableRow(InternalRow columnValues) {
163201
row.setRowKind(toRowKind(value));
164202
continue;
165203
}
204+
// Only an inserted row is padded; its row kind, if any, is the first input column. An UPDATE or
205+
// DELETE row carries the key Doris read back from the table, which has to stay byte-equal to the
206+
// stored key for Paimon to match it, padded or not.
207+
if (charPadders[i] != null && value != null && row.getRowKind() == RowKind.INSERT) {
208+
value = charPadders[i].pad(value);
209+
}
166210
// Actual Doris input is applied last so an explicit NULL remains distinct
167211
// from an omitted field and retains Paimon's writer-side semantics.
168212
row.setField(tableFieldIndexes[i], value);
169213
}
170214
return row;
171215
}
172216

217+
/**
218+
* Returns the padder of a value of {@code type}, or null when the type holds no CHAR. A CHAR value is padded
219+
* with spaces to the type length in code points, as Paimon's cast to CHAR pads it
220+
* ({@code BinaryStringUtils.toCharacterString}), and so are the CHAR values nested in an ARRAY, a MAP or a
221+
* ROW. Unlike that cast, a longer value is kept: Doris has already cut it to the length or kept it, as the
222+
* session asks.
223+
*/
224+
private static CharPadder charPadder(DataType type) {
225+
switch (type.getTypeRoot()) {
226+
case CHAR: {
227+
int length = ((CharType) type).getLength();
228+
return value -> padChar((BinaryString) value, length);
229+
}
230+
case ARRAY: {
231+
DataType elementType = ((ArrayType) type).getElementType();
232+
CharPadder elementPadder = charPadder(elementType);
233+
if (elementPadder == null) {
234+
return null;
235+
}
236+
InternalArray.ElementGetter elementGetter = InternalArray.createElementGetter(elementType.nullable());
237+
return value -> padArray((InternalArray) value, elementGetter, elementPadder);
238+
}
239+
case MAP: {
240+
MapType mapType = (MapType) type;
241+
CharPadder keyPadder = charPadder(mapType.getKeyType());
242+
CharPadder valuePadder = charPadder(mapType.getValueType());
243+
if (keyPadder == null && valuePadder == null) {
244+
return null;
245+
}
246+
InternalArray.ElementGetter keyGetter =
247+
InternalArray.createElementGetter(mapType.getKeyType().nullable());
248+
InternalArray.ElementGetter valueGetter =
249+
InternalArray.createElementGetter(mapType.getValueType().nullable());
250+
return value -> {
251+
InternalMap map = (InternalMap) value;
252+
return new PaddedMap(
253+
keyPadder == null ? map.keyArray() : padArray(map.keyArray(), keyGetter, keyPadder),
254+
valuePadder == null
255+
? map.valueArray() : padArray(map.valueArray(), valueGetter, valuePadder));
256+
};
257+
}
258+
case ROW: {
259+
List<DataField> fields = ((RowType) type).getFields();
260+
CharPadder[] fieldPadders = new CharPadder[fields.size()];
261+
InternalRow.FieldGetter[] fieldGetters = new InternalRow.FieldGetter[fields.size()];
262+
boolean holdsChar = false;
263+
for (int i = 0; i < fields.size(); i++) {
264+
fieldPadders[i] = charPadder(fields.get(i).type());
265+
fieldGetters[i] = InternalRow.createFieldGetter(fields.get(i).type().nullable(), i);
266+
holdsChar |= fieldPadders[i] != null;
267+
}
268+
if (!holdsChar) {
269+
return null;
270+
}
271+
return value -> {
272+
InternalRow nested = (InternalRow) value;
273+
GenericRow padded = new GenericRow(fieldGetters.length);
274+
for (int i = 0; i < fieldGetters.length; i++) {
275+
Object field = fieldGetters[i].getFieldOrNull(nested);
276+
padded.setField(i, field == null || fieldPadders[i] == null
277+
? field : fieldPadders[i].pad(field));
278+
}
279+
return padded;
280+
};
281+
}
282+
default:
283+
return null;
284+
}
285+
}
286+
287+
private static BinaryString padChar(BinaryString value, int length) {
288+
int padLength = length - value.numChars();
289+
return padLength > 0 ? BinaryStringUtils.concat(value, BinaryString.blankString(padLength)) : value;
290+
}
291+
292+
private static InternalArray padArray(InternalArray array, InternalArray.ElementGetter elementGetter,
293+
CharPadder elementPadder) {
294+
Object[] elements = new Object[array.size()];
295+
for (int i = 0; i < elements.length; i++) {
296+
Object element = elementGetter.getElementOrNull(array, i);
297+
elements[i] = element == null ? null : elementPadder.pad(element);
298+
}
299+
return new GenericArray(elements);
300+
}
301+
302+
/** Pads the CHAR values held by one non-null value of an input column. */
303+
private interface CharPadder {
304+
Object pad(Object value);
305+
}
306+
307+
/** A map over padded key and value arrays. Unlike GenericMap it keeps every entry of the input map. */
308+
private static final class PaddedMap implements InternalMap {
309+
private final InternalArray keys;
310+
private final InternalArray values;
311+
312+
private PaddedMap(InternalArray keys, InternalArray values) {
313+
this.keys = keys;
314+
this.values = values;
315+
}
316+
317+
@Override
318+
public int size() {
319+
return keys.size();
320+
}
321+
322+
@Override
323+
public InternalArray keyArray() {
324+
return keys;
325+
}
326+
327+
@Override
328+
public InternalArray valueArray() {
329+
return values;
330+
}
331+
}
332+
173333
private static RowKind toRowKind(Object operation) {
174334
if (!(operation instanceof Byte)) {
175335
throw new IllegalArgumentException("Paimon row change operation must be a TINYINT");

‎fe/be-java-extensions/paimon-scanner/src/test/java/org/apache/doris/paimon/PaimonWriteSchemaTest.java‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,10 +19,13 @@
1919

2020
import org.apache.paimon.casting.DefaultValueRow;
2121
import org.apache.paimon.data.BinaryString;
22+
import org.apache.paimon.data.GenericArray;
23+
import org.apache.paimon.data.GenericMap;
2224
import org.apache.paimon.data.GenericRow;
2325
import org.apache.paimon.data.InternalArray;
2426
import org.apache.paimon.data.InternalMap;
2527
import org.apache.paimon.data.InternalRow;
28+
import org.apache.paimon.types.CharType;
2629
import org.apache.paimon.types.DataField;
2730
import org.apache.paimon.types.DataTypes;
2831
import org.apache.paimon.types.DoubleType;
@@ -34,7 +37,9 @@
3437
import org.junit.jupiter.api.Test;
3538

3639
import java.util.Arrays;
40+
import java.util.Collections;
3741
import java.util.HashMap;
42+
import java.util.LinkedHashMap;
3843
import java.util.Map;
3944

4045
public class PaimonWriteSchemaTest {
@@ -243,6 +248,107 @@ public void testOmittedComplexDefaultsUsePaimonInternalValues() {
243248
Assertions.assertEquals("default-value", nested.getString(1).toString());
244249
}
245250

251+
@Test
252+
public void testCharPaddingPadsCharValuesToTheColumnLengthInCodePoints() {
253+
// Spark and Flink pad a CHAR(n) value with spaces to n characters when they write it, and Paimon compares
254+
// primary keys and hashes buckets by bytes, so an unpadded 'a' would be another key than their 'a '. A
255+
// longer value is kept: Doris has already decided whether to cut it, and VARCHAR is never padded.
256+
// MUTATION: not padding, or padding to the length in bytes -> red.
257+
PaimonWriteSchema schema = PaimonWriteSchema.create(charTableType(),
258+
new String[] {"id", "code", "note"}, false, true);
259+
260+
Assertions.assertEquals("a ", tableRow(schema, 1, string("a"), string("a")).getString(1).toString());
261+
Assertions.assertEquals("中 ", tableRow(schema, 1, string("中"), null).getString(1).toString());
262+
Assertions.assertEquals("abc", tableRow(schema, 1, string("abc"), null).getString(1).toString());
263+
Assertions.assertEquals("abcd", tableRow(schema, 1, string("abcd"), null).getString(1).toString());
264+
Assertions.assertTrue(tableRow(schema, 1, null, null).isNullAt(1));
265+
Assertions.assertEquals("a", tableRow(schema, 1, string("a"), string("a")).getString(2).toString());
266+
}
267+
268+
@Test
269+
public void testCharPaddingPadsCharValuesNestedInArraysMapsAndRows() {
270+
// Spark and Flink pad nested CHAR values too. A map keeps every entry, in order, as it did before.
271+
// MUTATION: padding only top-level values -> red.
272+
RowType tableType = new RowType(Arrays.asList(
273+
new DataField(0, "codes", DataTypes.ARRAY(DataTypes.CHAR(3))),
274+
new DataField(1, "labels", DataTypes.MAP(DataTypes.CHAR(2), DataTypes.CHAR(3))),
275+
new DataField(2, "item", DataTypes.ROW(DataTypes.FIELD(3, "code", DataTypes.CHAR(3)),
276+
DataTypes.FIELD(4, "count", DataTypes.INT()))),
277+
new DataField(5, "rows", DataTypes.ARRAY(DataTypes.ROW(
278+
DataTypes.FIELD(6, "code", DataTypes.CHAR(2))))),
279+
new DataField(7, "numbers", DataTypes.ARRAY(DataTypes.INT()))));
280+
PaimonWriteSchema schema = PaimonWriteSchema.create(tableType,
281+
new String[] {"codes", "labels", "item", "rows", "numbers"}, false, true);
282+
GenericArray numbers = new GenericArray(new Object[] {1, 2});
283+
284+
InternalRow tableRow = tableRow(schema,
285+
new GenericArray(new Object[] {string("a"), null, string("abc")}),
286+
new GenericMap(Collections.singletonMap(string("k"), string("v"))),
287+
GenericRow.of(string("a"), 7),
288+
new GenericArray(new Object[] {GenericRow.of(string("x"))}),
289+
numbers);
290+
291+
InternalArray codes = tableRow.getArray(0);
292+
Assertions.assertEquals("a ", codes.getString(0).toString());
293+
Assertions.assertTrue(codes.isNullAt(1));
294+
Assertions.assertEquals("abc", codes.getString(2).toString());
295+
InternalMap labels = tableRow.getMap(1);
296+
Assertions.assertEquals(1, labels.size());
297+
Assertions.assertEquals("k ", labels.keyArray().getString(0).toString());
298+
Assertions.assertEquals("v ", labels.valueArray().getString(0).toString());
299+
InternalRow item = tableRow.getRow(2, 2);
300+
Assertions.assertEquals("a ", item.getString(0).toString());
301+
Assertions.assertEquals(7, item.getInt(1));
302+
Assertions.assertEquals("x ", tableRow.getArray(3).getRow(0, 1).getString(0).toString());
303+
Assertions.assertSame(numbers, tableRow.getArray(4));
304+
305+
// Two keys that only padding makes equal both stay; the writer does not deduplicate map keys.
306+
Map<BinaryString, BinaryString> keys = new LinkedHashMap<>();
307+
keys.put(string("k"), string("1"));
308+
keys.put(string("k "), string("2"));
309+
InternalMap padded = tableRow(schema, null, new GenericMap(keys), null, null, null).getMap(1);
310+
Assertions.assertEquals(2, padded.size());
311+
Assertions.assertEquals("k ", padded.keyArray().getString(0).toString());
312+
Assertions.assertEquals("k ", padded.keyArray().getString(1).toString());
313+
Assertions.assertEquals("1 ", padded.valueArray().getString(0).toString());
314+
Assertions.assertEquals("2 ", padded.valueArray().getString(1).toString());
315+
}
316+
317+
@Test
318+
public void testCharValuesAreWrittenAsTheyAreWhenPaddingIsOff() {
319+
PaimonWriteSchema schema = PaimonWriteSchema.create(charTableType(),
320+
new String[] {"id", "code", "note"}, false, false);
321+
322+
Assertions.assertEquals("a", tableRow(schema, 1, string("a"), null).getString(1).toString());
323+
}
324+
325+
@Test
326+
public void testCharPaddingLeavesUpdateAndDeleteRowsAlone() {
327+
// An UPDATE or DELETE row carries the key that Doris read back from the table, which must stay byte-equal
328+
// to the stored key for Paimon to match it: a key an earlier Doris wrote without padding would otherwise
329+
// never be updated or deleted. MUTATION: padding every row kind -> red.
330+
PaimonWriteSchema schema = PaimonWriteSchema.create(charTableType(),
331+
new String[] {PaimonWriteSchema.ROW_KIND_COLUMN, "id", "code"}, true, true);
332+
333+
Assertions.assertEquals("a ", tableRow(schema, PaimonWriteSchema.INSERT_OPERATION, 1, string("a"))
334+
.getString(1).toString());
335+
Assertions.assertEquals("a", tableRow(schema, PaimonWriteSchema.UPDATE_OPERATION, 1, string("a"))
336+
.getString(1).toString());
337+
Assertions.assertEquals("a", tableRow(schema, PaimonWriteSchema.DELETE_OPERATION, 1, string("a"))
338+
.getString(1).toString());
339+
}
340+
341+
@Test
342+
public void testCharPaddingIsReadFromTheTableOptionsTheFeSets() {
343+
// FE puts doris.write.char-padding on the table it serializes for the writer (PaimonWriteBinding). An FE
344+
// that predates the option leaves it out and does not pad its static partition values, so neither do we.
345+
Assertions.assertTrue(PaimonWriteSchema.charPadding(
346+
Collections.singletonMap("doris.write.char-padding", "true")));
347+
Assertions.assertFalse(PaimonWriteSchema.charPadding(
348+
Collections.singletonMap("doris.write.char-padding", "false")));
349+
Assertions.assertFalse(PaimonWriteSchema.charPadding(Collections.emptyMap()));
350+
}
351+
246352
@Test
247353
public void testDuplicateColumnRejectedDuringInitialization() {
248354
IllegalArgumentException exception = Assertions.assertThrows(
@@ -260,6 +366,17 @@ private static RowType tableType() {
260366
new DataField(3, "region", new VarCharType())));
261367
}
262368

369+
private static RowType charTableType() {
370+
return new RowType(Arrays.asList(
371+
new DataField(0, "id", new IntType()),
372+
new DataField(1, "code", new CharType(3)),
373+
new DataField(2, "note", new VarCharType(3))));
374+
}
375+
376+
private static BinaryString string(String value) {
377+
return BinaryString.fromString(value);
378+
}
379+
263380
private static InternalRow changelogRow(PaimonWriteSchema schema, byte operation) {
264381
return tableRow(schema,
265382
operation,

0 commit comments

Comments
 (0)