Repository navigation
Commit 2a790ff
[fix](paimon) Name binary partitions of Paimon writes by their bytes
### What problem does this PR solve?
Issue Number: None
Related PR: #68874
Problem Summary: Paimon names a partition after the string of each value.
With partition.legacy-name = true, the default, that string is toString()
of the value, which for a BINARY / VARBINARY value is the Java array object
([B@1b6d3586) rather than its bytes, and every resolution of the name
differs. With Paimon 2.1.0, Paimon's own writer and reader cannot
round-trip such a partition (the read fails with FileNotFoundException),
and a Doris INSERT into such a table had the BE write its files under one
[B@... directory, then failed the commit with "Cannot recover from this
checkpoint because some files in the snapshot that need to be resubmitted
have been deleted", leaving the files behind. A write to a table with a
binary partition column and legacy partition names now fails while it is
planned, naming the table option 'partition.legacy-name' = 'false' that
makes such a table work.
With partition.legacy-name = false, Paimon names a binary partition after
the text of its bytes, and Doris INSERT already writes and reads it
correctly. #68874 rejected a static overwrite of a binary partition column
with a non-NULL value: Paimon's static overwrite takes the UTF-8 bytes of
the value's string, and the cast value reaches the connector as 0x and its
hex digits, which names another partition (an overwrite without rows then
reported success and left the given partition in place). The connector now
decodes the hex digits and passes the text the bytes decode to as UTF-8,
which Paimon encodes back to the same bytes. Bytes that are not UTF-8 text,
such as 0xEF, are still rejected, since no string names them.
### Release note
Fix: a write to a Paimon table whose binary partition column uses legacy
partition names fails at planning with a hint to set
'partition.legacy-name' = 'false', instead of failing its commit and
leaving files behind. INSERT OVERWRITE ... PARTITION (p = X'...') on a
binary partition column of a Paimon table is supported when the bytes are
UTF-8 text.
### Check List (For Author)
- Test: Unit Test, Manual test
- PaimonWriteBindingTest: binary static values become the text of their bytes, bytes that are not UTF-8 are rejected, a write of a legacy-named binary partition fails while one with new names passes
- fe-connector-paimon module tests
- Manual: local file:// Paimon catalog with tables created by the Paimon API (see the PR description)
- Behavior changed: Yes. A write to a Paimon table with a binary partition column and legacy partition names fails at planning instead of at commit; a static overwrite of a binary partition column with a binary literal whose bytes are UTF-8 text is accepted instead of rejected.
- Does this need documentation: No
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>1 parent 1bb3537 commit 2a790ff
2 files changed
Lines changed: 115 additions & 25 deletions
File tree
- fe/fe-connector/fe-connector-paimon/src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
32 | 36 | | |
33 | 37 | | |
34 | 38 | | |
35 | 39 | | |
36 | 40 | | |
37 | 41 | | |
38 | 42 | | |
| 43 | + | |
39 | 44 | | |
40 | 45 | | |
41 | 46 | | |
| |||
80 | 85 | | |
81 | 86 | | |
82 | 87 | | |
| 88 | + | |
| 89 | + | |
83 | 90 | | |
84 | 91 | | |
85 | 92 | | |
86 | 93 | | |
87 | 94 | | |
88 | 95 | | |
89 | | - | |
90 | | - | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
91 | 121 | | |
92 | 122 | | |
93 | 123 | | |
| |||
118 | 148 | | |
119 | 149 | | |
120 | 150 | | |
121 | | - | |
| 151 | + | |
| 152 | + | |
122 | 153 | | |
123 | 154 | | |
124 | 155 | | |
| |||
133 | 164 | | |
134 | 165 | | |
135 | 166 | | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | 167 | | |
146 | 168 | | |
147 | 169 | | |
| |||
165 | 187 | | |
166 | 188 | | |
167 | 189 | | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
168 | 193 | | |
169 | 194 | | |
170 | 195 | | |
| |||
219 | 244 | | |
220 | 245 | | |
221 | 246 | | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
222 | 271 | | |
223 | 272 | | |
224 | 273 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
64 | 66 | | |
65 | 67 | | |
66 | 68 | | |
| |||
197 | 199 | | |
198 | 200 | | |
199 | 201 | | |
200 | | - | |
201 | | - | |
202 | | - | |
203 | | - | |
204 | | - | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
205 | 207 | | |
206 | 208 | | |
207 | 209 | | |
208 | | - | |
209 | | - | |
210 | | - | |
211 | | - | |
212 | | - | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
213 | 214 | | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
214 | 228 | | |
215 | | - | |
216 | | - | |
217 | | - | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
218 | 232 | | |
219 | 233 | | |
220 | 234 | | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
221 | 262 | | |
222 | 263 | | |
223 | 264 | | |
| |||
0 commit comments