Skip to content

Commit a7b45c6

Browse files
morningmanclaude
andcommitted
[refactor](catalog) Catalog spi 11 hive (#65473)
Refs #65185. Part of **P7 — hive (+HMS)** in the Catalog SPI migration tracking issue. This PR starts the hive/HMS connector migration on `branch-catalog-spi`, covering the HMS metadata path, metastore-event pipeline, ACID transaction write path, and the hudi live cutover dependency tracked in P7. --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent caca4c9 commit a7b45c6

790 files changed

Lines changed: 61591 additions & 59454 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/wf-d2-cache-recon.js‎

Lines changed: 266 additions & 0 deletions
Large diffs are not rendered by default.

‎be/src/common/config.cpp‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1550,7 +1550,10 @@ DEFINE_mInt64(s3_put_token_limit, "0");
15501550
DEFINE_mInt64(s3_rate_limiter_log_interval, "1000");
15511551
DEFINE_Validator(s3_rate_limiter_log_interval, [](int64_t config) -> bool { return config >= 0; });
15521552

1553-
DEFINE_String(trino_connector_plugin_dir, "${DORIS_HOME}/plugins/connectors");
1553+
// The dir TrinoConnectorPluginLoader loads Trino's own plugins from, used verbatim. Keep the default
1554+
// in sync with FE Config.trino_connector_plugin_dir: FE and BE load the same plugins and an operator
1555+
// who leaves both untouched expects both to find them.
1556+
DEFINE_String(trino_connector_plugin_dir, "${DORIS_HOME}/plugins/trino_plugins");
15541557

15551558
// ca_cert_file is in this path by default, Normally no modification is required
15561559
// ca cert default path is different from different OS

‎be/src/io/fs/connectivity/s3_connectivity_tester.cpp‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,10 @@ namespace doris::io {
2424

2525
Status S3ConnectivityTester::test(const std::map<std::string, std::string>& properties) {
2626
auto it = properties.find(TEST_LOCATION);
27+
if (it == properties.end()) {
28+
return Status::InvalidArgument("Missing '{}' property in S3 connectivity test request",
29+
TEST_LOCATION);
30+
}
2731
S3URI s3_uri(it->second);
2832
RETURN_IF_ERROR(s3_uri.parse());
2933

‎build.sh‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1041,7 +1041,11 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
10411041
mkdir -p "${DORIS_OUTPUT}/fe/conf/ssl"
10421042
mkdir -p "${DORIS_OUTPUT}/fe/plugins/jdbc_drivers/"
10431043
mkdir -p "${DORIS_OUTPUT}/fe/plugins/java_udf/"
1044-
mkdir -p "${DORIS_OUTPUT}/fe/plugins/connectors/"
1044+
# Drop point for the trino-connector's own Trino plugins. Deliberately NOT the legacy
1045+
# plugins/connectors/: that name is still read as a fallback for deployments upgrading from
1046+
# <= 2.1.8, so a fresh install must not create it (an empty dir would be harmless, but the
1047+
# one-letter gap to the plugins/connector/ tree above is not).
1048+
mkdir -p "${DORIS_OUTPUT}/fe/plugins/trino_plugins/"
10451049
mkdir -p "${DORIS_OUTPUT}/fe/plugins/hadoop_conf/"
10461050
mkdir -p "${DORIS_OUTPUT}/fe/plugins/java_extensions/"
10471051

@@ -1266,7 +1270,8 @@ EOF
12661270
mkdir -p "${DORIS_OUTPUT}/be/plugins/jdbc_drivers/"
12671271
mkdir -p "${DORIS_OUTPUT}/be/plugins/java_udf/"
12681272
mkdir -p "${DORIS_OUTPUT}/be/plugins/python_udf/"
1269-
mkdir -p "${DORIS_OUTPUT}/be/plugins/connectors/"
1273+
# Mirrors the FE drop point above; the BE JNI scanner loads the same Trino plugins independently.
1274+
mkdir -p "${DORIS_OUTPUT}/be/plugins/trino_plugins/"
12701275
mkdir -p "${DORIS_OUTPUT}/be/plugins/hadoop_conf/"
12711276
mkdir -p "${DORIS_OUTPUT}/be/plugins/java_extensions/"
12721277
cp -r -p "${DORIS_HOME}/be/src/udf/python/python_server.py" "${DORIS_OUTPUT}/be/plugins/python_udf/"
Lines changed: 162 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,162 @@
1+
# Hive Regression Local Environment
2+
3+
This document records the local Docker and regression configuration needed to run:
4+
5+
```bash
6+
regression-test/suites/external_table_p0/hive/
7+
```
8+
9+
## Required Thirdparty Components
10+
11+
Run Hive2 and Hive3 through the thirdparty launcher:
12+
13+
```bash
14+
./docker/thirdparties/run-thirdparties-docker.sh -c hive2,hive3 --hive-mode refresh
15+
```
16+
17+
The Hive startup script may also start `mysql` automatically. Both
18+
`docker-compose/hive/hive-2x_settings.env` and
19+
`docker-compose/hive/hive-3x_settings.env` default `JFS_CLUSTER_META` to:
20+
21+
```bash
22+
mysql://root:123456@(127.0.0.1:3316)/juicefs_meta
23+
```
24+
25+
Because of that setting, `run-thirdparties-docker.sh` treats local MySQL 5.7 as
26+
an implicit dependency for Hive.
27+
28+
Each Hive version starts its own compose services:
29+
30+
- Hive2: `hive2-server`, `hive2-metastore`, `hadoop2-namenode`,
31+
`hadoop2-datanode`, `hive2-metastore-postgresql`
32+
- Hive3: `hive3-server`, `hive3-metastore`, `hadoop3-namenode`,
33+
`hadoop3-datanode`, `hive3-metastore-postgresql`
34+
- MySQL 5.7: required by the default JuiceFS metadata URI when using the local
35+
`127.0.0.1:3316` endpoint
36+
37+
The Hive p0 directory does not require these components by default:
38+
39+
- `pg`
40+
- `iceberg`
41+
- `minio`
42+
- `hudi`
43+
- `kerberos`
44+
- `ranger`
45+
46+
`test_external_catalog_hive.groovy` has a Ranger-specific branch guarded by
47+
`enableRangerTest`; if this option is not set to `true`, that branch is skipped.
48+
49+
## Current Running Containers Observed
50+
51+
The following relevant containers were running when checked with
52+
`sudo docker ps`:
53+
54+
- Hive3 stack: `hive3-server`, `hive3-metastore`, `hadoop3-datanode`,
55+
`hadoop3-namenode`, `hive3-metastore-postgresql`; all healthy
56+
- Hive2 stack: `hive2-server`, `hive2-metastore`, `hadoop2-datanode`,
57+
`hadoop2-namenode`, `hive2-metastore-postgresql`; all healthy
58+
- MySQL 5.7: `mysql-doris--syt--mysql_57-1`, healthy, published as
59+
`127.0.0.1:3316 -> 3306`
60+
61+
Other Doris thirdparty containers were also running, but they are not required
62+
for `external_table_p0/hive/`:
63+
64+
- PostgreSQL 14 on `5442`
65+
- Iceberg REST, Spark, Postgres, and MinIO
66+
- Oracle on `1521`
67+
- ClickHouse on `8123`
68+
- SQLServer on `1433`
69+
- OceanBase on `2881`
70+
71+
The observed Hive2/Hive3 and MySQL compose projects came from another checkout
72+
under `/mnt/disk2/suyiteng/doris`, not from this worktree. PostgreSQL and
73+
Iceberg compose projects came from this worktree.
74+
75+
## Regression Configuration
76+
77+
The Hive p0 suite directly reads these keys from
78+
`regression-test/conf/regression-conf.groovy`:
79+
80+
```groovy
81+
enableHiveTest
82+
externalEnvIp
83+
hive2HmsPort
84+
hive2HdfsPort
85+
hive3HmsPort
86+
hive3HdfsPort
87+
hdfsUser
88+
```
89+
90+
It also uses framework helpers backed by:
91+
92+
```groovy
93+
brokerName
94+
hdfsPasswd
95+
hdfsFs
96+
```
97+
98+
For local Hive2/Hive3 Docker with default ports, the relevant values are:
99+
100+
```groovy
101+
enableHiveTest = true
102+
103+
hive2HmsPort = 9083
104+
hive2HdfsPort = 8020
105+
hive2ServerPort = 10000
106+
hive2PgPort = 5432
107+
108+
hive3HmsPort = 9383
109+
hive3HdfsPort = 8320
110+
hive3ServerPort = 13000
111+
hive3PgPort = 5732
112+
113+
hdfsUser = "doris-test"
114+
brokerName = "broker_name"
115+
hdfsPasswd = ""
116+
```
117+
118+
`externalEnvIp` must be the address that Doris BE can use to reach the Hive
119+
Docker services. For a fully local setup this is usually:
120+
121+
```groovy
122+
externalEnvIp = "127.0.0.1"
123+
```
124+
125+
If BE runs in a context where `127.0.0.1` is not the Docker host, use the host
126+
IP selected by `run-thirdparties-docker.sh` through `IP_HOST`.
127+
128+
## Config Gaps Noted In regression-conf.groovy.bak
129+
130+
`regression-test/conf/regression-conf.groovy.bak` already contains the core Hive
131+
keys listed above. The notable follow-ups are:
132+
133+
- `externalEnvIp` was set to `172.20.32.136`; verify this is reachable from BE,
134+
or replace it with `127.0.0.1` for local host-mode Docker.
135+
- FE/JDBC addresses in the file used `9033`, `9022`, and `8033`; align these
136+
with the current Doris worktree cluster ports before running regression.
137+
- `enableRangerTest` is not present. This is acceptable for the Hive p0 suite
138+
because the Ranger-specific branch is guarded and skipped unless explicitly
139+
enabled.
140+
141+
## Thirdparty Script Settings
142+
143+
Before starting Hive through `run-thirdparties-docker.sh`, set a unique
144+
`CONTAINER_UID` in:
145+
146+
```bash
147+
docker/thirdparties/custom_settings.env
148+
```
149+
150+
The default `CONTAINER_UID="doris--"` is rejected by the startup script.
151+
152+
Hive baseline restore also depends on these settings:
153+
154+
```bash
155+
s3Endpoint
156+
s3BucketName
157+
HIVE_BASELINE_VERSION
158+
HIVE_BASELINE_TARBALL_CACHE
159+
```
160+
161+
If baseline download is unavailable, place the matching tarball manually under
162+
`HIVE_BASELINE_TARBALL_CACHE`.

‎fe/be-java-extensions/trino-connector-scanner/src/main/java/org/apache/doris/trinoconnector/TrinoConnectorPluginLoader.java‎

Lines changed: 7 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,10 @@
4040
public class TrinoConnectorPluginLoader {
4141
private static final Logger LOG = LogManager.getLogger(TrinoConnectorPluginLoader.class);
4242

43-
private static String pluginsDir = EnvUtils.getDorisHome() + "/plugins/connectors";
43+
// Overwritten via setPluginsDir() with BE config trino_connector_plugin_dir before the plugins are
44+
// loaded (see be/src/format*/**/trino_connector_jni_reader.cpp); this initializer only matters if
45+
// that call is ever missed. Mirrors that config's default.
46+
private static String pluginsDir = EnvUtils.getDorisHome() + "/plugins/trino_plugins";
4447

4548
// Suppress default constructor for noninstantiability
4649
private TrinoConnectorPluginLoader() {
@@ -87,17 +90,17 @@ private static class TrinoConnectorPluginLoad {
8790
TypeRegistry typeRegistry = new TypeRegistry(typeOperators, featuresConfig);
8891

8992
ServerPluginsProviderConfig serverPluginsProviderConfig = new ServerPluginsProviderConfig()
90-
.setInstalledPluginsDir(new File(checkAndReturnPluginDir()));
93+
.setInstalledPluginsDir(new File(pluginsDir));
9194
ServerPluginsProvider serverPluginsProvider = new ServerPluginsProvider(serverPluginsProviderConfig,
9295
MoreExecutors.directExecutor());
9396
HandleResolver handleResolver = new HandleResolver();
9497
trinoConnectorPluginManager = new TrinoConnectorPluginManager(serverPluginsProvider,
9598
typeRegistry, handleResolver);
9699
trinoConnectorPluginManager.loadPlugins();
97100

98-
LOG.info("TrinoConnectorPluginLoader successfully loaded plugins from: " + checkAndReturnPluginDir());
101+
LOG.info("TrinoConnectorPluginLoader successfully loaded plugins from: " + pluginsDir);
99102
} catch (Exception e) {
100-
LOG.warn("Failed load trino-connector plugins from " + checkAndReturnPluginDir()
103+
LOG.warn("Failed load trino-connector plugins from " + pluginsDir
101104
+ ", Exception:" + e.getMessage(), e);
102105
}
103106
}
@@ -118,29 +121,6 @@ public static void setPluginsDir(String pluginsDir) {
118121
TrinoConnectorPluginLoader.pluginsDir = pluginsDir;
119122
}
120123

121-
private static String checkAndReturnPluginDir() {
122-
final String defaultDir = System.getenv("DORIS_HOME") + "/plugins/connectors";
123-
final String defaultOldDir = System.getenv("DORIS_HOME") + "/connectors";
124-
if (TrinoConnectorPluginLoader.pluginsDir.equals(defaultDir)) {
125-
// If true, which means user does not set `trino_connector_plugin_dir` and use the default one.
126-
// Because in 2.1.8, we change the default value of `trino_connector_plugin_dir`
127-
// from `DORIS_HOME/connectors` to `DORIS_HOME/plugins/connectors`,
128-
// so we need to check the old default dir for compatibility.
129-
File oldDir = new File(defaultOldDir);
130-
if (oldDir.exists() && oldDir.isDirectory()) {
131-
String[] contents = oldDir.list();
132-
if (contents != null && contents.length > 0) {
133-
// there are contents in old dir, use old one
134-
return defaultOldDir;
135-
}
136-
}
137-
return defaultDir;
138-
} else {
139-
// Return user specified dir directly.
140-
return TrinoConnectorPluginLoader.pluginsDir;
141-
}
142-
}
143-
144124
public static TrinoConnectorPluginManager getTrinoConnectorPluginManager() {
145125
return TrinoConnectorPluginLoad.trinoConnectorPluginManager;
146126
}

‎fe/fe-catalog/src/main/java/org/apache/doris/catalog/Column.java‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,15 @@ public static Column generateBeforeValueColumn(Column column) {
186186

187187
private boolean isCompoundKey = false;
188188

189+
// Marks a connector-reserved passthrough column (e.g. iceberg v3 row-lineage _row_id /
190+
// _last_updated_sequence_number). Set by ConnectorColumnConverter from the connector-declared
191+
// ConnectorColumn.reservedPassthrough(); read by engine MERGE/UPDATE and sink binding so they recognize the
192+
// synthetic passthrough column generically instead of string-matching source column names. NOT persisted
193+
// (no @SerializedName on purpose): it is an external-table-only marker rebuilt each load from connector
194+
// metadata and must never enter an internal-table schema image; on replay it stays at its default false
195+
// (mirrors the runtime-only isCompoundKey / defineExpr fields).
196+
private boolean reservedPassthrough = false;
197+
189198
@SerializedName(value = "hasOnUpdateDefaultValue")
190199
private boolean hasOnUpdateDefaultValue = false;
191200

@@ -382,6 +391,7 @@ public Column(Column column) {
382391
this.visible = column.visible;
383392
this.children = column.getChildren();
384393
this.uniqueId = column.getUniqueId();
394+
this.reservedPassthrough = column.reservedPassthrough;
385395
this.defineExpr = column.getDefineExpr();
386396
this.defineName = column.getRealDefineName();
387397
this.hasOnUpdateDefaultValue = column.hasOnUpdateDefaultValue;
@@ -1009,6 +1019,14 @@ public int getUniqueId() {
10091019
return this.uniqueId;
10101020
}
10111021

1022+
public boolean isReservedPassthrough() {
1023+
return reservedPassthrough;
1024+
}
1025+
1026+
public void setReservedPassthrough(boolean reservedPassthrough) {
1027+
this.reservedPassthrough = reservedPassthrough;
1028+
}
1029+
10121030
public long getAutoIncInitValue() {
10131031
return this.autoIncInitValue;
10141032
}

‎fe/fe-common/pom.xml‎

Lines changed: 0 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -84,11 +84,6 @@ under the License.
8484
<groupId>org.apache.commons</groupId>
8585
<artifactId>commons-lang3</artifactId>
8686
</dependency>
87-
<dependency>
88-
<groupId>org.apache.doris</groupId>
89-
<artifactId>hive-catalog-shade</artifactId>
90-
<scope>provided</scope>
91-
</dependency>
9287
<dependency>
9388
<groupId>org.roaringbitmap</groupId>
9489
<artifactId>RoaringBitmap</artifactId>

‎fe/fe-common/src/main/java/org/apache/doris/common/CatalogConfigFileUtils.java‎

Lines changed: 3 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,6 @@
2020
import org.apache.commons.lang3.StringUtils;
2121
import org.apache.hadoop.conf.Configuration;
2222
import org.apache.hadoop.fs.Path;
23-
import org.apache.hadoop.hive.conf.HiveConf;
2423

2524
import java.io.File;
2625
import java.util.function.BiConsumer;
@@ -29,14 +28,13 @@
2928
public class CatalogConfigFileUtils {
3029

3130
/**
32-
* Generic method to load configuration files (e.g., Hadoop or Hive) from a directory.
31+
* Generic method to load Hadoop-style configuration files from a directory.
3332
*
3433
* @param resourcesPath Comma-separated list of resource file names to be loaded.
3534
* @param configDir Directory prefix where the configuration files reside.
36-
* @param configSupplier Supplier that creates a new configuration object
37-
* (e.g., new Configuration or new HiveConf).
35+
* @param configSupplier Supplier that creates a new configuration object.
3836
* @param addResourceMethod Method to add a resource file to the configuration object.
39-
* @param <T> Type of the configuration (e.g., Configuration or HiveConf).
37+
* @param <T> Type of the configuration.
4038
* @return A configuration object loaded with the given resource files.
4139
* @throws IllegalArgumentException if the resourcesPath is empty or if any file does not exist.
4240
*/
@@ -84,20 +82,4 @@ public static Configuration loadConfigurationFromHadoopConfDir(String resourcesP
8482
Configuration::addResource
8583
);
8684
}
87-
88-
/**
89-
* Loads a HiveConf object from a list of files under the specified config directory.
90-
*
91-
* @param resourcesPath Comma-separated list of file names to be loaded.
92-
* @return A HiveConf object.
93-
* @throws IllegalArgumentException if the input is invalid or files are missing.
94-
*/
95-
public static HiveConf loadHiveConfFromHiveConfDir(String resourcesPath) {
96-
return loadConfigFromDir(
97-
resourcesPath,
98-
Config.hadoop_config_dir,
99-
HiveConf::new,
100-
HiveConf::addResource
101-
);
102-
}
10385
}

0 commit comments

Comments
 (0)