Repository navigation
Commit 0a4a71a
authored
[feature](external) Add generic connector write framework prerequisites (#68320)
### What problem does this PR solve?
Issue Number: #65086
Related PR: #67395
Problem Summary:
Forward-porting Paimon table writes exposes four gaps in the generic
connector write framework. Implementing them as Paimon-specific branches
in FE Core would duplicate the existing write pipeline and would also
cause Paimon row-level changes to be planned with Iceberg
position-delete semantics.
This PR adds the missing connector-neutral framework capabilities:
1. **Write distribution.** A connector can describe routing columns, an
opaque partition-function name and options, and writer-assignment
semantics. FE Core only resolves columns and transports the description;
BE performs the routing. Unsupported functions and unsupported BE
execution versions fail explicitly. The existing reserved BE execution
version 13 is reused, with no Paimon-specific execution version.
2. **Row-level DML.** Connectors declare whether changes use
file-position deletes or changelog rows. Iceberg retains its
position-delete plan. The generic changelog plan converts DELETE,
UPDATE, MERGE, and DELETE USING into an operation marker plus the output
row, while the connector supplies the marker encoding, primary keys,
validation, and label prefix. Unsupported representations fail instead
of falling through to an unrelated write path.
3. **Commit reports.** BE can return opaque connector commit fragments
through the existing external-write report lifecycle. FE acknowledges
only accepted reports, retries remain safe after a lost response, and
duplicate reports do not add commit data twice. FE Core does not
interpret connector-owned bytes.
4. **Variant writes.** FE type checking/construction and BE Arrow
serialization support Binary Variant V2 as a generic capability required
by connector writes.
The implementation intentionally does not add Paimon-specific planner
classes, hidden-column names, Thrift structures, or a new write system.
Paimon-specific table rules, fixed-bucket hash implementation, writer,
and transaction interpretation remain in the Paimon integration that
will follow this prerequisite PR.
The connector SPI surface changes once, so
`connector.plugin.api.version` is increased from 10 to 11 once for the
whole PR.
### Release note
None
### Check List (For Author)
- Test:
- Targeted FE unit tests for connector SPI, row-level DML planning,
commit report acknowledgement/serialization, Variant type handling, and
connector write distribution
- Relevant FE Maven reactors compiled successfully while running the
targeted unit tests
- BE formatting with clang-format 16 and `build-support/check-format.sh`
- `build-support/check-build-hygiene.sh`
- BE compilation and BE unit tests were not run for the latest
distribution change
- Behavior changed: Yes. Connectors can declare row-change
representation and write distribution, return opaque acknowledged commit
data, and write Binary Variant V2 through the shared framework.
- Does this need documentation: No1 parent 31c5836 commit 0a4a71a
118 files changed
Lines changed: 4635 additions & 491 deletions
File tree
- be
- src
- core/data_type_serde
- exec
- operator
- partitioner
- external
- pipeline
- sink/writer
- exprs/function/cast/variant_v2
- format/transformer
- runtime
- test
- core/data_type_serde
- exec
- operator
- partitioner
- exprs/function/cast
- runtime
- fe
- fe-catalog/src/main/java/org/apache/doris
- analysis
- persist/gson
- fe-connector
- fe-connector-iceberg/src
- main/java/org/apache/doris/connector/iceberg
- test/java/org/apache/doris/connector/iceberg
- fe-connector-spi/src
- main/java/org/apache/doris/connector/spi/write
- test
- java/org/apache/doris/connector/spi
- write
- resources
- fe-core/src
- main/java/org/apache/doris
- analysis
- datasource/plugin
- nereids
- analyzer
- glue/translator
- parser
- processor/post
- properties
- rules
- analysis
- expression
- check
- rules
- implementation
- trees
- expressions/functions
- scalar
- plans
- commands
- info
- insert
- merge
- logical
- physical
- types
- persist/gson
- planner
- qe
- runtime
- transaction
- test/java/org/apache/doris
- analysis
- datasource/plugin
- nereids
- glue/translator
- rules
- analysis
- expression
- check
- rules
- trees/plans
- commands
- merge
- physical
- qe
- transaction
- gensrc/thrift
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| 21 | + | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| |||
182 | 183 | | |
183 | 184 | | |
184 | 185 | | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
185 | 313 | | |
186 | 314 | | |
187 | 315 | | |
| |||
553 | 681 | | |
554 | 682 | | |
555 | 683 | | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
556 | 689 | | |
557 | 690 | | |
558 | 691 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| 31 | + | |
31 | 32 | | |
32 | 33 | | |
33 | 34 | | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
37 | 38 | | |
| 39 | + | |
38 | 40 | | |
39 | 41 | | |
40 | 42 | | |
| |||
98 | 100 | | |
99 | 101 | | |
100 | 102 | | |
101 | | - | |
| 103 | + | |
| 104 | + | |
102 | 105 | | |
103 | 106 | | |
104 | 107 | | |
| |||
175 | 178 | | |
176 | 179 | | |
177 | 180 | | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
178 | 202 | | |
179 | 203 | | |
180 | 204 | | |
| |||
272 | 296 | | |
273 | 297 | | |
274 | 298 | | |
| 299 | + | |
275 | 300 | | |
276 | 301 | | |
277 | 302 | | |
| |||
322 | 347 | | |
323 | 348 | | |
324 | 349 | | |
| 350 | + | |
| 351 | + | |
325 | 352 | | |
326 | 353 | | |
327 | 354 | | |
| |||
333 | 360 | | |
334 | 361 | | |
335 | 362 | | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
336 | 368 | | |
337 | 369 | | |
338 | 370 | | |
| |||
535 | 567 | | |
536 | 568 | | |
537 | 569 | | |
| 570 | + | |
538 | 571 | | |
539 | 572 | | |
540 | | - | |
| 573 | + | |
| 574 | + | |
541 | 575 | | |
| 576 | + | |
542 | 577 | | |
543 | | - | |
544 | | - | |
545 | | - | |
546 | | - | |
547 | | - | |
548 | | - | |
549 | | - | |
550 | | - | |
551 | | - | |
| 578 | + | |
| 579 | + | |
552 | 580 | | |
553 | 581 | | |
554 | 582 | | |
| |||
587 | 615 | | |
588 | 616 | | |
589 | 617 | | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
| 621 | + | |
| 622 | + | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
590 | 630 | | |
591 | 631 | | |
592 | 632 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
200 | 200 | | |
201 | 201 | | |
202 | 202 | | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
203 | 214 | | |
204 | 215 | | |
205 | 216 | | |
| |||
239 | 250 | | |
240 | 251 | | |
241 | 252 | | |
| 253 | + | |
242 | 254 | | |
243 | 255 | | |
244 | 256 | | |
245 | 257 | | |
| 258 | + | |
| 259 | + | |
246 | 260 | | |
247 | 261 | | |
248 | 262 | | |
| |||
278 | 292 | | |
279 | 293 | | |
280 | 294 | | |
281 | | - | |
282 | | - | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
283 | 298 | | |
284 | 299 | | |
285 | 300 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
0 commit comments