Skip to content

Commit 5c32565

Browse files
morningmanCopilot
andauthored
[feature](fe) Introduce fe-connector SPI framework and migrate JDBC/ES catalogs to plugin-driven architecture (apache#62183)
### What problem does this PR solve? Issue Number: close #xxx Problem Summary: The existing external catalog implementation in Doris has type-specific classes scattered throughout fe-core (JdbcExternalCatalog, EsExternalCatalog, JdbcExternalTable, EsExternalTable, JdbcScanNode, PhysicalJdbcTableSink, etc.), making it difficult to add new connector types and leading to duplicated logic across catalog implementations. This PR introduces a Service Provider Interface (SPI) framework for external connectors and migrates JDBC and Elasticsearch catalogs to the new plugin-driven architecture. The key changes are: 1. **New `fe-connector` module hierarchy**: - `fe-connector-api`: Core interfaces (Connector, ConnectorMetadata, ConnectorCapability, ConnectorPushdownOps, ConnectorWriteOps, ConnectorSession, pushdown expression tree, handle types) - `fe-connector-spi`: Service provider loading (ConnectorProvider, ConnectorContext, CatalogFactory) - `fe-connector-jdbc`: Full JDBC connector implementation migrated from fe-core - `fe-connector-es`: Elasticsearch connector implementation migrated from fe-core 2. **Unified catalog/table/scan classes**: - PluginDrivenExternalCatalog replaces JdbcExternalCatalog + EsExternalCatalog - PluginDrivenExternalTable replaces JdbcExternalTable + EsExternalTable - PluginDrivenScanNode replaces JdbcScanNode (and ES scan path) - PhysicalConnectorTableSink replaces PhysicalJdbcTableSink 3. **Capability-based feature detection**: Connectors declare capabilities (SUPPORTS_INSERT, SUPPORTS_PARALLEL_WRITE, SUPPORTS_PASSTHROUGH_QUERY) so the optimizer can make correct distribution and planning decisions without type-checking specific catalog classes. 4. **GSON backward compatibility**: GsonUtils.registerCompatibleSubtype remaps persisted JdbcExternalCatalog/EsExternalCatalog metadata to PluginDrivenExternalCatalog for seamless upgrade from older Doris versions. 5. **JDBC-specific behavioral preservation**: - GATHER distribution for transactional write safety (non-parallel writes) - query() TVF restricted to connectors with SUPPORTS_PASSTHROUGH_QUERY - Property validation (driver_url, jdbc_url, driver_class, etc.) - Identifier case mapping scoped per database/table - DATETIMEV2 fractional-second precision in predicate pushdown - Database-specific boolean literal rendering (TRUE/FALSE vs 1/0) - Database-specific test queries (DB2: SELECT 1 FROM SYSIBM.SYSDUMMY1) - Legacy engine name compatibility for SHOW TABLES/information_schema - Driver URL resolution with backward-compatible directory fallback ### Release note Introduce a pluggable connector SPI framework (fe-connector) that allows external data source integrations to be developed as independent modules. JDBC and Elasticsearch catalogs are migrated to this new architecture. Existing catalogs created on older versions are automatically upgraded -- no user action is required. All existing JDBC and ES catalog functionality is preserved, including predicate pushdown, identifier mapping, property validation, and transactional write semantics. ### Check List (For Author) - Test: Regression test / Unit Test - 21 JdbcConnectorProvider validation unit tests - 7 PluginDrivenExternalTable engine name unit tests - 7 JdbcQueryBuilder datetime fractional-second tests - 6 JdbcQueryBuilder boolean literal tests - 4 JdbcIdentifierMapper scoped uniqueness tests - Concurrency test for PluginDrivenExternalCatalog - JDBC regression tests (jdbc_catalog_db2, jdbc_catalog_pg, jdbc_catalog_oracle, etc.) - Behavior changed: No (backward compatible -- existing catalogs auto-migrated via GSON) - Does this need documentation: Yes (connector SPI developer guide -- to be added separately) --------- Co-authored-by: Copilot <[email protected]>
1 parent e053225 commit 5c32565

370 files changed

Lines changed: 36881 additions & 9166 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎build.sh‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -615,6 +615,15 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
615615
fi
616616
done
617617
unset _fs_mod
618+
# Connector API, SPI, and plugin modules (loaded at runtime as plugins)
619+
modules+=("fe-connector/fe-connector-api")
620+
modules+=("fe-connector/fe-connector-spi")
621+
for _conn_mod in es jdbc maxcompute trino hms hive paimon hudi iceberg; do
622+
if [[ -d "${DORIS_HOME}/fe/fe-connector/fe-connector-${_conn_mod}" ]]; then
623+
modules+=("fe-connector/fe-connector-${_conn_mod}")
624+
fi
625+
done
626+
unset _conn_mod
618627
if [[ "${WITH_TDE_DIR}" != "" ]]; then
619628
modules+=("fe-${WITH_TDE_DIR}")
620629
fi
@@ -807,7 +816,7 @@ function build_fe_modules() {
807816
local -a user_settings_opts=()
808817
local -a mvn_cmd=(
809818
"${MVN_CMD}"
810-
package
819+
install
811820
-pl
812821
"${FE_MODULES}"
813822
-am
@@ -926,6 +935,24 @@ if [[ "${BUILD_FE}" -eq 1 ]]; then
926935
done
927936
unset FS_PLUGIN_DIR fs_module fs_plugin_target fs_module_dir
928937

938+
# Deploy connector provider plugins as independent plugin directories.
939+
# Each sub-directory is one connector backend loaded at runtime by ConnectorPluginManager.
940+
CONN_PLUGIN_DIR="${DORIS_OUTPUT}/fe/plugins/connector"
941+
for conn_module in es jdbc maxcompute trino hms hive paimon hudi iceberg; do
942+
conn_plugin_target="${CONN_PLUGIN_DIR}/${conn_module}"
943+
conn_module_dir="${DORIS_HOME}/fe/fe-connector/fe-connector-${conn_module}"
944+
if [ ! -d "${conn_module_dir}" ]; then
945+
continue
946+
fi
947+
conn_zip=$(find "${conn_module_dir}/target" -maxdepth 1 -name '*.zip' 2>/dev/null | head -1)
948+
if [ -z "${conn_zip}" ]; then
949+
continue
950+
fi
951+
mkdir -p "${conn_plugin_target}"
952+
unzip -o "${conn_zip}" -d "${conn_plugin_target}/"
953+
done
954+
unset CONN_PLUGIN_DIR conn_module conn_plugin_target conn_module_dir conn_zip
955+
929956
if [ "${TARGET_SYSTEM}" = "Darwin" ] || [ "${TARGET_SYSTEM}" = "Linux" ]; then
930957
mkdir -p "${DORIS_OUTPUT}/fe/arthas"
931958
rm -rf "${DORIS_OUTPUT}/fe/arthas/*"

‎fe/be-java-extensions/jdbc-scanner/src/main/java/org/apache/doris/jdbc/MySQLTypeHandler.java‎

Lines changed: 18 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -63,24 +63,37 @@ public MySQLTypeHandler(String tableType) {
6363
this.tableType = tableType != null ? tableType.toUpperCase() : "MYSQL";
6464
}
6565

66+
/**
67+
* MySQL Connector/J 5.1.x's getObject(col, Type.class) may return the primitive
68+
* default (0, 0.0, false) instead of null for SQL NULL columns. Always follow
69+
* up with wasNull() so we propagate NULLs correctly regardless of driver version.
70+
*/
71+
private static <T> T getTypedObject(ResultSet rs, int col, Class<T> type) throws SQLException {
72+
T val = rs.getObject(col, type);
73+
return rs.wasNull() ? null : val;
74+
}
75+
6676
@Override
6777
public Object getColumnValue(ResultSet rs, int columnIndex, ColumnType type,
6878
ResultSetMetaData metadata) throws SQLException {
6979
switch (type.getType()) {
7080
case BOOLEAN:
71-
return rs.getObject(columnIndex, Boolean.class);
81+
return getTypedObject(rs, columnIndex, Boolean.class);
7282
case TINYINT:
7383
case SMALLINT:
84+
// Must use typed getObject() to avoid MySQL YEAR returning java.sql.Date
85+
// when yearIsDateType=true (the default in Connector/J 8.x)
86+
return getTypedObject(rs, columnIndex, Integer.class);
7487
case LARGEINT:
7588
return rs.getObject(columnIndex);
7689
case INT:
77-
return rs.getObject(columnIndex, Integer.class);
90+
return getTypedObject(rs, columnIndex, Integer.class);
7891
case BIGINT:
79-
return rs.getObject(columnIndex, Long.class);
92+
return getTypedObject(rs, columnIndex, Long.class);
8093
case FLOAT:
81-
return rs.getObject(columnIndex, Float.class);
94+
return getTypedObject(rs, columnIndex, Float.class);
8295
case DOUBLE:
83-
return rs.getObject(columnIndex, Double.class);
96+
return getTypedObject(rs, columnIndex, Double.class);
8497
case DECIMALV2:
8598
case DECIMAL32:
8699
case DECIMAL64:

‎fe/fe-common/src/main/java/org/apache/doris/common/Config.java‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3488,6 +3488,11 @@ public static int metaServiceRpcRetryTimes() {
34883488
+ "If empty, only classpath-based built-in providers are used (test/dev mode)."})
34893489
public static String filesystem_plugin_root = EnvUtils.getDorisHome() + "/plugins/filesystem";
34903490

3491+
@ConfField(description = {"Directory containing connector provider plugin subdirectories. "
3492+
+ "Each subdirectory is one connector (e.g., es/, jdbc/, iceberg/). "
3493+
+ "If empty, only classpath-based built-in providers are used (test/dev mode)."})
3494+
public static String connector_plugin_root = EnvUtils.getDorisHome() + "/plugins/connector";
3495+
34913496
@ConfField(description = {"Authorization plugin configuration file path. Must be under DORIS_HOME. "
34923497
+ "Default is conf/authorization.conf."})
34933498
public static String authorization_config_file_path = "/conf/authorization.conf";
Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<!--
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
-->
20+
<project xmlns="http://maven.apache.org/POM/4.0.0"
21+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
22+
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
23+
<modelVersion>4.0.0</modelVersion>
24+
25+
<parent>
26+
<groupId>org.apache.doris</groupId>
27+
<artifactId>fe-connector</artifactId>
28+
<version>${revision}</version>
29+
<relativePath>../pom.xml</relativePath>
30+
</parent>
31+
32+
<artifactId>fe-connector-api</artifactId>
33+
<packaging>jar</packaging>
34+
<name>Doris FE Connector API</name>
35+
<description>
36+
Consumer-facing API for the Doris FE connector abstraction layer.
37+
Contains the core Connector interface, ConnectorMetadata sub-interfaces,
38+
opaque Handle types, ConnectorSession, value objects, and capability enums.
39+
40+
The fe-thrift dependency (provided scope) allows connectors to construct
41+
typed TTableDescriptor objects for the FE-BE protocol. At runtime the
42+
Thrift classes are supplied by the parent classloader (fe-core).
43+
</description>
44+
45+
<dependencies>
46+
<dependency>
47+
<groupId>${project.groupId}</groupId>
48+
<artifactId>fe-thrift</artifactId>
49+
<version>${project.version}</version>
50+
<scope>provided</scope>
51+
</dependency>
52+
<dependency>
53+
<groupId>org.junit.jupiter</groupId>
54+
<artifactId>junit-jupiter</artifactId>
55+
<scope>test</scope>
56+
</dependency>
57+
</dependencies>
58+
59+
<build>
60+
<finalName>doris-fe-connector-api</finalName>
61+
<plugins>
62+
<plugin>
63+
<groupId>org.apache.maven.plugins</groupId>
64+
<artifactId>maven-dependency-plugin</artifactId>
65+
<executions>
66+
<!-- This module is compiled into fe-core, not deployed as a plugin. -->
67+
<execution>
68+
<id>copy-plugin-deps</id>
69+
<phase>none</phase>
70+
</execution>
71+
</executions>
72+
</plugin>
73+
</plugins>
74+
</build>
75+
</project>
Lines changed: 121 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,121 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.api;
19+
20+
import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider;
21+
22+
import java.io.Closeable;
23+
import java.io.IOException;
24+
import java.util.Collections;
25+
import java.util.List;
26+
import java.util.Set;
27+
28+
/**
29+
* Main entry point for a connector implementation.
30+
*
31+
* <p>A {@code Connector} instance is created once per catalog and provides
32+
* access to metadata, scan planning, and optional write operations.</p>
33+
*/
34+
public interface Connector extends Closeable {
35+
36+
/** Returns the metadata interface for the given session. */
37+
ConnectorMetadata getMetadata(ConnectorSession session);
38+
39+
/** Returns the scan plan provider for split generation. */
40+
default ConnectorScanPlanProvider getScanPlanProvider() {
41+
return null;
42+
}
43+
44+
/** Returns the set of capabilities this connector supports. */
45+
default Set<ConnectorCapability> getCapabilities() {
46+
return Collections.emptySet();
47+
}
48+
49+
/** Returns the table-level property descriptors. */
50+
default List<ConnectorPropertyMetadata<?>> getTableProperties() {
51+
return Collections.emptyList();
52+
}
53+
54+
/** Returns the session-level property descriptors. */
55+
default List<ConnectorPropertyMetadata<?>> getSessionProperties() {
56+
return Collections.emptyList();
57+
}
58+
59+
/**
60+
* Returns whether connectivity testing should be enabled by default when
61+
* the user does not explicitly set the {@code test_connection} property.
62+
*
63+
* <p>Connectors that rely on external drivers or remote connectivity
64+
* (e.g., JDBC) should return {@code true} so that bad configurations
65+
* are caught at CREATE CATALOG time rather than at first query.</p>
66+
*
67+
* @return {@code true} if test_connection defaults to enabled
68+
*/
69+
default boolean defaultTestConnection() {
70+
return false;
71+
}
72+
73+
/**
74+
* Performs connector-specific validation during CREATE CATALOG.
75+
*
76+
* <p>Called before {@link #testConnection(ConnectorSession)}. Connectors
77+
* may override this to validate driver security, compute checksums,
78+
* test BE connectivity, or perform any other pre-creation checks.</p>
79+
*
80+
* <p>The engine provides infrastructure services through the
81+
* {@link ConnectorValidationContext}; each connector calls only the
82+
* services relevant to its validation needs.</p>
83+
*
84+
* @param context engine services for validation
85+
* @throws Exception if validation fails
86+
*/
87+
default void preCreateValidation(ConnectorValidationContext context) throws Exception {
88+
// No-op by default
89+
}
90+
91+
/**
92+
* Tests connectivity to the underlying data source.
93+
*
94+
* <p>Connectors should override this to verify they can reach the
95+
* metastore, storage, and any other required services.</p>
96+
*
97+
* @return the test result; default returns success (no-op test)
98+
*/
99+
default ConnectorTestResult testConnection(ConnectorSession session) {
100+
return ConnectorTestResult.success();
101+
}
102+
103+
@Override
104+
default void close() throws IOException {
105+
}
106+
107+
/**
108+
* Execute a REST passthrough request against the underlying data source.
109+
*
110+
* <p>Connectors that expose HTTP endpoints (e.g., Elasticsearch) can
111+
* override this to proxy REST requests from FE REST APIs.</p>
112+
*
113+
* @param path the relative URL path (e.g., "index_name/_search")
114+
* @param body the request body (may be null for GET-style requests)
115+
* @return the response body as a JSON string
116+
* @throws UnsupportedOperationException if the connector doesn't support REST
117+
*/
118+
default String executeRestRequest(String path, String body) {
119+
throw new UnsupportedOperationException("REST passthrough not supported by this connector");
120+
}
121+
}
Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
package org.apache.doris.connector.api;
19+
20+
/**
21+
* Enumerates the optional capabilities a connector may declare.
22+
* The planner and execution engine use these to decide which
23+
* pushdown and write paths are available.
24+
*/
25+
public enum ConnectorCapability {
26+
SUPPORTS_FILTER_PUSHDOWN,
27+
SUPPORTS_PROJECTION_PUSHDOWN,
28+
SUPPORTS_LIMIT_PUSHDOWN,
29+
SUPPORTS_PARTITION_PRUNING,
30+
SUPPORTS_INSERT,
31+
SUPPORTS_DELETE,
32+
SUPPORTS_UPDATE,
33+
SUPPORTS_MERGE,
34+
SUPPORTS_CREATE_TABLE,
35+
SUPPORTS_MVCC_SNAPSHOT,
36+
SUPPORTS_METASTORE_EVENTS,
37+
SUPPORTS_STATISTICS,
38+
SUPPORTS_VENDED_CREDENTIALS,
39+
SUPPORTS_ACID_TRANSACTIONS,
40+
SUPPORTS_TIME_TRAVEL,
41+
/**
42+
* Indicates the connector supports multiple concurrent writers (sink instances).
43+
*
44+
* <p>Connectors that do NOT declare this capability will use GATHER distribution
45+
* (single writer), which is the safe default for transactional sinks like JDBC
46+
* where each writer commits independently.</p>
47+
*
48+
* <p>File-based connectors (Hive, Iceberg, etc.) that can safely handle
49+
* parallel writers should declare this capability.</p>
50+
*/
51+
SUPPORTS_PARALLEL_WRITE,
52+
/**
53+
* Indicates the connector supports passthrough query via the {@code query()} TVF.
54+
*
55+
* <p>Connectors declaring this capability must implement
56+
* {@link ConnectorTableOps#getColumnsFromQuery} to provide column metadata
57+
* for arbitrary SQL queries passed through to the remote data source.</p>
58+
*/
59+
SUPPORTS_PASSTHROUGH_QUERY
60+
}

0 commit comments

Comments
 (0)