Repository navigation
Commit eeae58b
[feat](catalog) P7.3 hive: plugin-side write transaction + committer + write plan provider + ACID read producer (dormant)
Port the Hive DML/write path and the ACID read path into the self-contained
fe-connector-hive / fe-connector-hms plugins, mirroring the iceberg connector
template, so fe-core no longer needs native Hive write logic. Dormant: HIVE is
not in SPI_READY_TYPES yet, so this compiles and is unit-tested but not routed
at runtime; cutover (flip the flag, retype the fe-core write chain, swap the
read-txn manager, remove legacy datasource/hive) lands in a later atomic batch.
fe-connector-hms (write-side HMS client):
- HmsClient/ThriftHmsClient: open/commit/abort transaction, acquire locks, get
valid write ids, add/alter partitions, table & partition statistics;
convertTable now carries bucket columns / bucket count.
- HmsWriteConverter: partition + statistics conversion for writes.
- New HmsAcidConstants, HmsCommonStatistics, HmsPartitionStatistics,
HmsPartitionWithStatistics.
fe-connector-hive:
- Write chain: HiveConnectorTransaction (HMS commit protocol + rollback),
HiveWriteContext, HiveSinkHelper, HiveWritePlanProvider, HiveWriteUtils,
NameMapping; HiveConnectorMetadata.beginTransaction override; full-ACID writes
are hard-rejected.
- Read chain: HiveAcidUtil (pure directory-name parsing + snapshot selection),
HiveReadTransaction / HiveReadTransactionManager (per-table read-txn
lifecycle); HiveScanPlanProvider.planAcidScan emits acidInfo only for
full-ACID tables, insert-only tables read plain files by snapshot.
- HiveTableHandle.isTransactional/isFullAcid derived from table parameters.
Connector-agnostic write-plan seam (fe-connector-api + fe-core), default-off:
- ConnectorWritePlanProvider on Connector + ConnectorContractValidator;
PluginDrivenExternalTable / PhysicalConnectorTableSink consume it with no
source-specific code in fe-core.
Verified fresh: fe-connector-hive 104 tests, fe-connector-hms 49, fe-core
write-seam 16 (all 0 failures/skips); checkstyle 0 violations across the touched
modules; connector import-gate clean.
Refs: apache#65185
Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT1 parent 79fd45f commit eeae58b
40 files changed
Lines changed: 7456 additions & 34 deletions
File tree
- fe
- fe-connector
- fe-connector-api/src/main/java/org/apache/doris/connector/api
- write
- fe-connector-hive
- src
- main/java/org/apache/doris/connector/hive
- test/java/org/apache/doris/connector/hive
- fe-connector-hms
- src
- main/java/org/apache/doris/connector/hms
- test/java/org/apache/doris/connector/hms
- fe-core/src
- main/java/org/apache/doris
- datasource
- nereids/trees/plans/physical
- test/java/org/apache/doris
- connector
- nereids/trees/plans/physical
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
89 | 89 | | |
90 | 90 | | |
91 | 91 | | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
92 | 98 | | |
93 | 99 | | |
94 | 100 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
52 | 52 | | |
53 | 53 | | |
54 | 54 | | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
55 | 70 | | |
56 | 71 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
195 | 195 | | |
196 | 196 | | |
197 | 197 | | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
198 | 214 | | |
199 | 215 | | |
200 | 216 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
62 | 62 | | |
63 | 63 | | |
64 | 64 | | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
65 | 76 | | |
66 | 77 | | |
67 | 78 | | |
| |||
79 | 90 | | |
80 | 91 | | |
81 | 92 | | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
82 | 104 | | |
83 | 105 | | |
84 | 106 | | |
| |||
0 commit comments