Skip to content

Commit eeae58b

Browse files
morningmanclaude
andcommitted
[feat](catalog) P7.3 hive: plugin-side write transaction + committer + write plan provider + ACID read producer (dormant)
Port the Hive DML/write path and the ACID read path into the self-contained fe-connector-hive / fe-connector-hms plugins, mirroring the iceberg connector template, so fe-core no longer needs native Hive write logic. Dormant: HIVE is not in SPI_READY_TYPES yet, so this compiles and is unit-tested but not routed at runtime; cutover (flip the flag, retype the fe-core write chain, swap the read-txn manager, remove legacy datasource/hive) lands in a later atomic batch. fe-connector-hms (write-side HMS client): - HmsClient/ThriftHmsClient: open/commit/abort transaction, acquire locks, get valid write ids, add/alter partitions, table & partition statistics; convertTable now carries bucket columns / bucket count. - HmsWriteConverter: partition + statistics conversion for writes. - New HmsAcidConstants, HmsCommonStatistics, HmsPartitionStatistics, HmsPartitionWithStatistics. fe-connector-hive: - Write chain: HiveConnectorTransaction (HMS commit protocol + rollback), HiveWriteContext, HiveSinkHelper, HiveWritePlanProvider, HiveWriteUtils, NameMapping; HiveConnectorMetadata.beginTransaction override; full-ACID writes are hard-rejected. - Read chain: HiveAcidUtil (pure directory-name parsing + snapshot selection), HiveReadTransaction / HiveReadTransactionManager (per-table read-txn lifecycle); HiveScanPlanProvider.planAcidScan emits acidInfo only for full-ACID tables, insert-only tables read plain files by snapshot. - HiveTableHandle.isTransactional/isFullAcid derived from table parameters. Connector-agnostic write-plan seam (fe-connector-api + fe-core), default-off: - ConnectorWritePlanProvider on Connector + ConnectorContractValidator; PluginDrivenExternalTable / PhysicalConnectorTableSink consume it with no source-specific code in fe-core. Verified fresh: fe-connector-hive 104 tests, fe-connector-hms 49, fe-core write-seam 16 (all 0 failures/skips); checkstyle 0 violations across the touched modules; connector import-gate clean. Refs: apache#65185 Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT
1 parent 79fd45f commit eeae58b

40 files changed

Lines changed: 7456 additions & 34 deletions

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,12 @@ default boolean requiresPartitionLocalSort() {
8989
return p != null && p.requiresPartitionLocalSort();
9090
}
9191

92+
/** Null-safe view of {@link ConnectorWritePlanProvider#requiresPartitionHashWrite()}. No provider ⇒ false. */
93+
default boolean requiresPartitionHashWrite() {
94+
ConnectorWritePlanProvider p = getWritePlanProvider();
95+
return p != null && p.requiresPartitionHashWrite();
96+
}
97+
9298
/**
9399
* Null-safe view of {@link ConnectorWritePlanProvider#requiresMaterializeStaticPartitionValues()}. No
94100
* provider ⇒ false.

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,5 +52,20 @@ public static void validate(Connector connector, String catalogType) {
5252
+ "' declares requiresPartitionLocalSort without requiresParallelWrite"
5353
+ " AND requiresFullSchemaWriteOrder");
5454
}
55+
// #4 partition-hash-write (hash without sort) likewise implies parallel write AND full-schema write
56+
// order (the sink indexes partition columns by full-schema position and distributes in parallel).
57+
if (connector.requiresPartitionHashWrite()
58+
&& !(connector.requiresParallelWrite() && connector.requiresFullSchemaWriteOrder())) {
59+
throw new IllegalStateException("Connector '" + catalogType
60+
+ "' declares requiresPartitionHashWrite without requiresParallelWrite"
61+
+ " AND requiresFullSchemaWriteOrder");
62+
}
63+
// #5 the two hash arms are mutually exclusive: the engine checks local-sort first, so declaring both
64+
// would silently ignore the hash-without-sort request. Fail loud instead.
65+
if (connector.requiresPartitionLocalSort() && connector.requiresPartitionHashWrite()) {
66+
throw new IllegalStateException("Connector '" + catalogType
67+
+ "' declares both requiresPartitionLocalSort and requiresPartitionHashWrite;"
68+
+ " a connector must pick at most one partition-distribution arm");
69+
}
5570
}
5671
}

‎fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -195,6 +195,22 @@ default boolean requiresPartitionLocalSort() {
195195
return false;
196196
}
197197

198+
/**
199+
* Whether dynamic-partition writes must be hash-distributed by partition columns but <b>not</b> locally
200+
* sorted by them. A hive-style file writer buffers a separate partition writer per partition value, so —
201+
* unlike {@link #requiresPartitionLocalSort()} (MaxCompute's streaming Storage-API writer, which closes a
202+
* partition writer as soon as a different partition value appears and therefore needs the rows grouped by a
203+
* local sort) — the hash distribution alone keeps each partition's rows on one instance and the output file
204+
* count at ~one-per-partition, with no sort cost. The engine ({@code PhysicalConnectorTableSink}) picks the
205+
* matching distribution: {@code requiresPartitionLocalSort} ⇒ hash + {@code MustLocalSortOrderSpec};
206+
* {@code requiresPartitionHashWrite} ⇒ hash only. A connector sets at most one of the two hash arms; a
207+
* connector declaring this should also declare {@link #requiresParallelWrite()} +
208+
* {@link #requiresFullSchemaWriteOrder()}. Default: no.
209+
*/
210+
default boolean requiresPartitionHashWrite() {
211+
return false;
212+
}
213+
198214
/**
199215
* Whether the connector's data files physically retain partition columns, so a static-partition write
200216
* must materialize the PARTITION-clause literal into the data column instead of NULL-filling it (e.g.

‎fe/fe-connector/fe-connector-hive/pom.xml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,17 @@ under the License.
6262
<scope>provided</scope>
6363
</dependency>
6464

65+
<!-- Object-store FileSystem SPI (ObjFileSystem / ObjStorage / FileSystemProvider) for the write
66+
transaction's multipart-upload commit/abort. MUST be provided (never bundled): these types come
67+
from the shared/parent classpath at runtime, and a second bundled copy would make the plugin's
68+
ObjFileSystem.class != the runtime instance.class, so the instanceof/downcast throws a CCE. -->
69+
<dependency>
70+
<groupId>${project.groupId}</groupId>
71+
<artifactId>fe-filesystem-spi</artifactId>
72+
<version>${project.version}</version>
73+
<scope>provided</scope>
74+
</dependency>
75+
6576
<!-- Shared HMS client library -->
6677
<dependency>
6778
<groupId>${project.groupId}</groupId>
@@ -79,6 +90,17 @@ under the License.
7990
<artifactId>junit-jupiter</artifactId>
8091
<scope>test</scope>
8192
</dependency>
93+
94+
<!-- TEST ONLY: HiveAcidUtilTest exercises the ACID descent against a real Hadoop LocalFileSystem;
95+
Hadoop's Configuration/LocalFileSystem reference commons-lang 2.x
96+
(org.apache.commons.lang.StringUtils), which is present only transitively in the runtime
97+
plugin closure, not on the unit-test classpath. Test-scope only — NOT bundled. Mirrors
98+
fe-connector-hms. Version from dependencyManagement. -->
99+
<dependency>
100+
<groupId>commons-lang</groupId>
101+
<artifactId>commons-lang</artifactId>
102+
<scope>test</scope>
103+
</dependency>
82104
</dependencies>
83105

84106
<build>

0 commit comments

Comments
 (0)