Repository navigation
Commit f6b658b
[fix](paimon) Pad CHAR values written to Paimon tables as Spark and Flink do
### What problem does this PR solve?
Issue Number: None
Related PR: #69045
Problem Summary:
Doris wrote a CHAR(n) value into a Paimon table as it was, without the
trailing spaces that pad it to n characters. Spark (with the
paimon-spark extensions) and Flink pad CHAR values when they write a
Paimon table, nested values too, and Paimon compares primary keys and
hashes buckets by bytes, so a key that Doris wrote as 'a' and Spark as
'a ' made two rows of a primary-key table. On a local cluster with a
file:// Paimon catalog, a CHAR(3) primary-key table into which Doris
inserted 'k' and the Paimon API then wrote 'k ', as Spark and Flink
write 'k', had two rows; it now has one.
The Paimon JNI writer (PaimonWriteSchema) now pads the CHAR values of
the rows it inserts, top-level and nested in ARRAY, MAP and ROW, with
spaces to the column length in code points, as Paimon's own cast to
CHAR does. A longer value is kept: Doris has already cut it or kept it,
as the session asks. The static partition value of a CHAR partition
column is padded the same way, because Paimon parses it without padding
it, and an unpadded value would name another partition than the rows
reach. UPDATE and DELETE rows are not padded: they carry the key Doris
read back from the table, which has to stay byte-equal to the stored
key.
The new session variable enable_paimon_write_char_padding (default
true) turns this off, for a table whose CHAR keys an earlier Doris
wrote unpadded. FE passes it to the BE writer as the option
doris.write.char-padding on the table it serializes for the writer; the
FE committer's table goes without it. A writer that finds no such
option, as from an older FE, does not pad.
### Release note
Doris now pads CHAR(n) values written into Paimon tables with spaces to
n characters, as Spark and Flink do, including values nested in ARRAY,
MAP and ROW and static partition values; UPDATE and DELETE keep keys as
stored. Doris reads a padded value back with its trailing spaces, so
WHERE c = 'a' does not match it, as with data that Spark and Flink
write. A key that an earlier Doris wrote unpadded is another key than
its padded form; set enable_paimon_write_char_padding to false to keep
writing CHAR values unpadded.
### Check List (For Author)
- Test: Regression test, Unit Test, Manual test
- test_paimon_write_char_padding (external_table_p0/paimon/write): a
key that Doris and Spark both write is one row, nested and partition
values are padded, a second static overwrite replaces the padded
partition, the switch off, and DELETE and UPDATE of unpadded keys;
it fails without this change at its first check
- PaimonWriteSchemaTest: top-level and nested CHAR padding, padding
off, UPDATE and DELETE rows left alone, reading the table option
- PaimonWriteBindingTest: static CHAR partition value padding
- PaimonBackendBoundTableTest: only the BE table carries the option
- PaimonWritePlanProviderTest: the session variable, on by default
- Manual test on a local cluster with a file:// Paimon catalog: the
primary-key table above; CHAR values nested in ARRAY, MAP and ROW
and the partition values of a static overwrite and of an INSERT
are padded; with the switch off values are written unpadded, and
with it on DELETE and UPDATE still match those unpadded keys
- Behavior changed: Yes. CHAR values written into Paimon tables are
padded to the column length unless enable_paimon_write_char_padding
is false.
- Does this need documentation: Yes (the new session variable)
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 2137067 commit f6b658b
12 files changed
Lines changed: 626 additions & 31 deletions
File tree
- fe
- be-java-extensions/paimon-scanner/src
- main/java/org/apache/doris/paimon
- test/java/org/apache/doris/paimon
- fe-connector/fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-core/src/main/java/org/apache/doris/qe
- regression-test
- data/external_table_p0/paimon/write
- suites/external_table_p0/paimon/write
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
183 | 183 | | |
184 | 184 | | |
185 | 185 | | |
186 | | - | |
187 | | - | |
| 186 | + | |
| 187 | + | |
188 | 188 | | |
189 | 189 | | |
190 | 190 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
| 23 | + | |
| 24 | + | |
21 | 25 | | |
| 26 | + | |
| 27 | + | |
22 | 28 | | |
| 29 | + | |
| 30 | + | |
23 | 31 | | |
24 | 32 | | |
25 | 33 | | |
| 34 | + | |
26 | 35 | | |
27 | 36 | | |
28 | 37 | | |
29 | 38 | | |
30 | 39 | | |
| 40 | + | |
31 | 41 | | |
32 | 42 | | |
33 | 43 | | |
| |||
41 | 51 | | |
42 | 52 | | |
43 | 53 | | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
44 | 60 | | |
45 | 61 | | |
| 62 | + | |
| 63 | + | |
46 | 64 | | |
47 | 65 | | |
48 | 66 | | |
| |||
52 | 70 | | |
53 | 71 | | |
54 | 72 | | |
55 | | - | |
| 73 | + | |
56 | 74 | | |
| 75 | + | |
57 | 76 | | |
58 | 77 | | |
59 | 78 | | |
60 | 79 | | |
61 | 80 | | |
| 81 | + | |
62 | 82 | | |
63 | 83 | | |
64 | 84 | | |
| |||
82 | 102 | | |
83 | 103 | | |
84 | 104 | | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
85 | 114 | | |
86 | 115 | | |
87 | 116 | | |
| |||
139 | 168 | | |
140 | 169 | | |
141 | 170 | | |
142 | | - | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
143 | 181 | | |
144 | 182 | | |
145 | 183 | | |
| |||
163 | 201 | | |
164 | 202 | | |
165 | 203 | | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
166 | 210 | | |
167 | 211 | | |
168 | 212 | | |
169 | 213 | | |
170 | 214 | | |
171 | 215 | | |
172 | 216 | | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
173 | 333 | | |
174 | 334 | | |
175 | 335 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| 28 | + | |
26 | 29 | | |
27 | 30 | | |
28 | 31 | | |
| |||
34 | 37 | | |
35 | 38 | | |
36 | 39 | | |
| 40 | + | |
37 | 41 | | |
| 42 | + | |
38 | 43 | | |
39 | 44 | | |
40 | 45 | | |
| |||
243 | 248 | | |
244 | 249 | | |
245 | 250 | | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
246 | 352 | | |
247 | 353 | | |
248 | 354 | | |
| |||
260 | 366 | | |
261 | 367 | | |
262 | 368 | | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
263 | 380 | | |
264 | 381 | | |
265 | 382 | | |
| |||
0 commit comments