Skip to content

Commit 2a92450

Browse files
authored
[feat](catalog) support ADBC catalog that reads external sources over Arrow (#66331)
### What problem does this PR solve? Related Issue: #65615 Problem Summary: Doris reads an external database through JDBC today: the source's rows become JDBC objects and then Doris columns, one value at a time, across JNI. The source is also read by a single backend, because JDBC has no notion of how a result is partitioned. This PR adds a new catalog type, `adbc`, that reads an external source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. Data arrives as Arrow record batches and is handed to BE's scanner without a per-value conversion, and a scan is split into the driver's own result partitions so that N backends read N partitions in parallel. Neither is reachable through JDBC at all. ```sql CREATE CATALOG remote_source PROPERTIES ( "type" = "adbc", "driver_url" = "libadbc_driver_flightsql.so", "uri" = "grpc://remote-doris:8070", "user" = "root", "password" = "" ); SELECT count(*) FROM remote_source.some_db.some_table; ``` Phase one targets Arrow Flight SQL sources, which makes this the intended replacement for the `remote_doris` catalog. Other sources need only their driver's `.so` and, where their SQL differs from ANSI, a dialect implementation -- no architectural change. ### Supported features What an ADBC catalog gives you from the outside -- what you can do with it and what it does for you without being asked. The boundaries are in *Known limits* further down. | Capability | What it means when you use it | Control (default) | |---|---|---| | ADBC catalog | `CREATE CATALOG ... PROPERTIES("type" = "adbc", ...)`, then query the source's tables like any other external table. Phase one targets Arrow Flight SQL sources, including another Doris cluster. | `uri`, `user`, `password` | | Arrow-native transfer | Rows arrive as Arrow batches and go straight into the scanner, with no per-value conversion on the way in -- which is where a JDBC catalog spends much of a wide or high-volume scan. | Always on | | Parallel read across backends | One scan is split into the source's own result partitions and read by several BEs at once, instead of a single BE on a single connection. Falls back to one statement when the source cannot partition. | `partitioned_read` = `auto` / `disabled` / `required` (`auto`), `max_partitions` (`1024`) | | Column pruning | Only the columns the query actually needs are requested from the source, and `EXPLAIN` prints that same statement. | Always on | | Predicate pushdown | `=`, `!=`, `<`, `<=`, `>`, `>=`, `IS [NOT] NULL`, `[NOT] IN` and `AND` / `OR` / `NOT` over them become the remote `WHERE`. Anything else -- functions, arithmetic, `LIKE`, `BETWEEN` -- stays in Doris. Doris re-applies every predicate regardless, so pushdown changes speed only, never the rows you get. | Always on | | `LIMIT` pushdown | Pushed once the whole `WHERE` was pushed, so the source never truncates ahead of a filter Doris still has to apply. | Always on | | `COUNT(*)` without column values | A count reads no column data from the source at all. | Always on | | Metadata browsing | `SHOW DATABASES`, `SHOW TABLES`, `DESC`, `SHOW CREATE TABLE` and `information_schema` over the source's own databases, tables and columns. | -- | | Automatic type mapping | Source columns arrive as Doris types, including `ARRAY` / `MAP` / `STRUCT`, `DECIMAL`, date, and timestamp with and without a zone. | -- | | Metadata cache and `REFRESH` | Name resolution and table schemas are remembered per catalog, so a query stops paying several remote round trips before it is even planned. Listings are still read live, so a table created on the source is visible with no refresh at all, and `REFRESH CATALOG` / `DATABASE` / `TABLE` drops what is remembered. | `meta.cache.adbc.metadata.enable` / `.ttl-second` (`600`) / `.capacity` (`1000`) | | The usual query surface | Joins against internal tables and against other catalogs, aggregation, `ORDER BY`, `UNION`, subqueries, `SELECT ... INTO OUTFILE`, and an MTMV built on an ADBC table. | -- | | Source SQL you can steer | The generated SQL is conservative ANSI. The connector asks the driver which vendor it is talking to, and you can override that when the answer is unhelpful. | `sql_dialect` (auto-detected, `ansi` fallback; `doris` provided) | | Driver options passed through | Anything the driver itself understands can be set on the catalog, e.g. `"adbc.adbc.snowflake.sql.db" = "..."`. | `adbc.*` properties | | Driver placement and pinning | You place the driver library (Doris ships none); a bare file name resolves under the drivers directory, and a wrong or stale build is reported at `CREATE CATALOG` rather than as a puzzling query failure later. | `driver_url`, `driver_checksum`, `driver_entrypoint` | ### Design decisions worth knowing before reviewing - **FE and BE load the same driver `.so`.** FE goes through the ADBC JNI bridge, which wraps the same C driver manager BE links statically, and BE dlopens the identical file. This is a measured constraint, not a preference: partition descriptors are driver-private bytes, and two official implementations of the same protocol already serialize incompatible messages and mis-parse each other silently rather than erroring. - **Doris ships no ADBC driver.** The library is placed by the operator, under `adbc_drivers_dir` on FE and `be/plugins/adbc_drivers` on every BE. `driver_url` therefore accepts local references only (bare name, `file://`, or an absolute path); remote schemes are rejected because a per-node download cannot promise the nodes agree. "Driver file not found" is written as a first-class error for that reason, and `driver_checksum` can pin the build. - **The SQL sent to a source is conservative ANSI by default**, generated through a dialect interface a source can claim by vendor name or by the `sql_dialect` property. Predicates are pushed all-or-nothing per conjunct from a whitelist; BE re-applies every predicate regardless, so pushdown is pure acceleration. - **Planning has side effects on a Flight SQL source**: asking the driver to partition a statement executes it. That makes this the first connector for which `EXPLAIN` had to be told not to plan for real -- see the SPI addition below. ### Changes outside the connector | Change | Why | |---|---| | `ConnectorScanRequest.isExplainOnly()` (SPI) + `PluginDrivenScanNode` fills it | `EXPLAIN` reaches `planScan` for real (its explain level is `NORMAL`, so `NereidsPlanner.distribute()` does not return early). For this connector that would execute the very query it was asked only to describe. | | `PluginDrivenScanNode` re-asks for the display statement with the current columns | The connector properties are cached in `init()`, before Nereids prunes the scan tuple, so `EXPLAIN`'s `QUERY:` line named more columns than the statement that actually runs. Affects any connector that renders remote SQL from the column list (adbc, jdbc). What goes to BE is unchanged. | | `PluginDrivenScanNode.mapFileFormatType()` learns `"arrow"` | Routes an ADBC range to BE's Arrow reader. | | `TTableFormatFileDesc.adbc_params` | The scan range's parameters; BE's ADBC reader does not read the JDBC field. | | `be/src/vec/exec/format/adbc/*` + `file_scanner_v2` gate | The BE reader. | | `FlightSqlSchemaHelper` (FE, Arrow Flight SQL server) | `GetTables` described a `DATEV2` column as date64 while BE writes date32, and described array/map/struct with placeholder children. A client that types its columns from that schema fails on the first batch. Visible to every Arrow Flight SQL client, not only this connector. | | `DataTypeTimeStampTzSerDe::read_column_from_arrow` (BE) | It never overrode the numeric serde's fixed-width path, which memcpy'd Arrow's int64 epochs into a column that stores packed date/time values -- same width, so every row was silently wrong. | | `thirdparty`: `arrow-adbc` | The C driver manager is linked into `doris_be`; the JNI bridge is built for FE (upstream's prebuilt one needs GLIBC 2.34). The SQLite and Flight SQL drivers are fetched for tests only and are **not shipped**. | | `Config.adbc_drivers_dir` / `adbc_driver_secure_path`, `conf/fe.conf` JVM options, `build.sh` | Driver placement and the FE-side plugin build/deploy wiring. | ### Behaviour to be aware of - A view on the source is not listed as a table. A Doris source ignores the base-table filter ADBC sends, so the filter is applied where the answer is read. - `partitioned_read` is `auto` by default: split the scan when the driver can, read it as one statement when it cannot. `required` forbids that downgrade, which is what keeps a test from going green while quietly exercising the fallback; `disabled` is the escape hatch. - Metadata is cached per catalog for 10 minutes by default (`meta.cache.adbc.metadata.*`), and every `REFRESH` statement drops it. A newly created remote table is reachable without any refresh. - Read only: `INSERT`, `CREATE`/`DROP TABLE` and the other write statements against an ADBC catalog are rejected. An MTMV over an ADBC table does build and refresh. ### Test coverage - `fe-connector-adbc`: 186 unit tests, no skips. About 25 of them drive the real SQLite ADBC driver through the same Java -> JNI -> C driver manager -> driver `.so` path FE takes in production; `fe-connector-api` covers the new SPI field. - BE: unit tests for the ADBC reader, the driver registry, the Arrow variant normalizer, the scanner gate and the TIMESTAMPTZ serde. - End-to-end (`regression-test/suites/external_table_p0/adbc/`): 20 suites against a live catalog -- type mapping and semantics, complex and binary types, scan edges, predicate pushdown, column pruning, query shapes, source table models, metadata operations, nested catalogs, cross-source joins, large data, outfile, MTMV, negative cases, and partitioned vs. single-statement reads compared row for row. Each suite returns early when the driver is absent, so a cluster without one does not fail. ### Known limits - Read only. No writes, DDL, statistics or aggregate pushdown. - Partitions are verified to be read completely and without duplication, but **spreading them over several backends has only been reasoned from the code**, not observed -- the test environment has one backend, and `test_adbc_multi_backend` returns early there. - `IPV4` from a Doris source arrives as a bare `INT` (both sides encode it as int32), and a source `DATETIME` arrives as `TIMESTAMPTZ`. Both are asserted, not worked around. - The FE and BE copies of the driver are not checked against each other; `driver_checksum` verifies FE's copy only. ### Release note Support ADBC catalog, which reads an external source through an Arrow Database Connectivity driver: data is transferred as Arrow record batches and a scan is split across backends by the driver's own result partitions. Phase one targets Arrow Flight SQL sources. The driver library is supplied by the operator; Doris ships none. Also fixes two defects on the Arrow Flight SQL path that are visible to existing clients: `GetTables` reported the wrong Arrow type for `DATEV2` and placeholder children for `ARRAY`/`MAP`/`STRUCT`, and a `TIMESTAMPTZ` column read from Arrow was decoded by copying its bits instead of converting the epoch.
1 parent 6ddb9af commit 2a92450

110 files changed

Lines changed: 18099 additions & 8 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.licenserc.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ header:
6464
# the file an administrator edits in plugins/connector/<dir>/. Matched by name
6565
# rather than by a **/*.conf.template glob, so a new one is a deliberate entry
6666
# here rather than something a wildcard silently absorbs.
67+
- "fe/fe-connector/fe-connector-adbc/src/main/resources/adbc.conf.template"
6768
- "fe/fe-connector/fe-connector-hive/src/main/resources/hms.conf.template"
6869
- "fe/fe-connector/fe-connector-iceberg/src/main/resources/iceberg.conf.template"
6970
- "fe/fe-connector/fe-connector-jdbc/src/main/resources/jdbc.conf.template"

‎be/cmake/thirdparty.cmake‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,7 @@ add_thirdparty(arrow_flight LIB64)
109109
add_thirdparty(arrow_flight_sql LIB64)
110110
add_thirdparty(arrow_dataset LIB64)
111111
add_thirdparty(arrow_acero LIB64)
112+
add_thirdparty(adbc_driver_manager LIB64)
112113
add_thirdparty(parquet LIB64)
113114
# liblance_c.a contains compiler_builtins cbrt symbols. Place libm before it
114115
# so the final linker resolves C math symbols from the system library first.

‎be/src/exec/operator/file_scan_operator.cpp‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -113,6 +113,17 @@ bool FileScanLocalState::TEST_should_use_file_scanner_v2(const TQueryOptions& qu
113113
bool FileScanLocalState::_should_use_file_scanner_v2(const TQueryOptions& query_options,
114114
bool is_load,
115115
const TFileScanRangeParams& scan_params) {
116+
// ADBC only has a FileScannerV2 reader, and enable_file_scanner_v2 is a session variable marked
117+
// fuzzy=true, so the regression harness flips it to false at random. Without letting adbc
118+
// through unconditionally, those queries land in v1, which has no "adbc" branch, and come back
119+
// with an undiagnosable NotSupported -- showing up in CI as a random failure. This is the same
120+
// mechanism as is_transactional_hive below, pointed the other way. Loads are left alone: there
121+
// is no ADBC load path, so widening the rule to cover them would only route them somewhere they
122+
// still cannot run.
123+
if (!is_load && scan_params.__isset.table_format_params &&
124+
scan_params.table_format_params.table_format_type == "adbc") {
125+
return true;
126+
}
116127
const bool is_transactional_hive =
117128
scan_params.__isset.table_format_params &&
118129
scan_params.table_format_params.table_format_type == "transactional_hive";

‎be/src/exec/scan/file_scanner_v2.cpp‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
#include "format_v2/jni/jdbc_reader.h"
5555
#include "format_v2/jni/max_compute_jni_reader.h"
5656
#include "format_v2/jni/trino_connector_jni_reader.h"
57+
#include "format_v2/table/adbc_reader.h"
5758
#include "format_v2/table/hive_reader.h"
5859
#include "format_v2/table/hudi_reader.h"
5960
#include "format_v2/table/iceberg_position_delete_sys_table_reader.h"
@@ -103,7 +104,8 @@ bool is_supported_table_format(const TFileRangeDesc& range) {
103104
}
104105

105106
bool is_supported_arrow_table_format(const TFileRangeDesc& range) {
106-
return table_format_name(range) == "remote_doris";
107+
const auto table_format = table_format_name(range);
108+
return table_format == "remote_doris" || table_format == "adbc";
107109
}
108110

109111
bool is_supported_jni_table_format(const TFileRangeDesc& range) {
@@ -660,6 +662,8 @@ Status FileScannerV2::_create_table_reader_for_format(
660662
*reader = std::make_unique<format::trino_connector::TrinoConnectorJniReader>();
661663
} else if (table_format == "remote_doris") {
662664
*reader = std::make_unique<format::remote_doris::RemoteDorisReader>();
665+
} else if (table_format == "adbc") {
666+
*reader = std::make_unique<format::adbc::AdbcReader>();
663667
} else {
664668
return Status::NotSupported("FileScannerV2 does not support table format {}", table_format);
665669
}
Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#include "format/arrow/arrow_array_normalizer.h"
19+
20+
#include <arrow/array/array_base.h>
21+
#include <arrow/compute/cast.h>
22+
#include <arrow/type.h>
23+
24+
#include <memory>
25+
26+
#include "common/check.h"
27+
#include "common/status.h"
28+
29+
namespace doris {
30+
31+
namespace {
32+
33+
// The accepted counterpart of an encoding-only variant. Null when there is none.
34+
std::shared_ptr<arrow::DataType> target_type_for(const arrow::DataType& type) {
35+
switch (type.id()) {
36+
case arrow::Type::LARGE_STRING:
37+
case arrow::Type::STRING_VIEW:
38+
return arrow::utf8();
39+
case arrow::Type::LARGE_BINARY:
40+
case arrow::Type::BINARY_VIEW:
41+
return arrow::binary();
42+
case arrow::Type::DICTIONARY:
43+
return static_cast<const arrow::DictionaryType&>(type).value_type();
44+
case arrow::Type::RUN_END_ENCODED:
45+
return static_cast<const arrow::RunEndEncodedType&>(type).value_type();
46+
default:
47+
return nullptr;
48+
}
49+
}
50+
51+
} // namespace
52+
53+
bool is_serde_acceptable_arrow_type(const arrow::DataType& type) {
54+
switch (type.id()) {
55+
// Encoding-only variants: convertible to an accepted type.
56+
case arrow::Type::LARGE_STRING:
57+
case arrow::Type::LARGE_BINARY:
58+
case arrow::Type::STRING_VIEW:
59+
case arrow::Type::BINARY_VIEW:
60+
case arrow::Type::DICTIONARY:
61+
case arrow::Type::RUN_END_ENCODED:
62+
return false;
63+
// No Doris column can hold these, so they must not reach a serde either.
64+
case arrow::Type::INTERVAL_MONTHS:
65+
case arrow::Type::INTERVAL_DAY_TIME:
66+
case arrow::Type::INTERVAL_MONTH_DAY_NANO:
67+
case arrow::Type::DURATION:
68+
case arrow::Type::SPARSE_UNION:
69+
case arrow::Type::DENSE_UNION:
70+
return false;
71+
default:
72+
return true;
73+
}
74+
}
75+
76+
Status normalize_arrow_array(const std::shared_ptr<arrow::Array>& arr,
77+
std::shared_ptr<arrow::Array>* out) {
78+
DORIS_CHECK(arr != nullptr);
79+
DORIS_CHECK(out != nullptr);
80+
81+
std::shared_ptr<arrow::Array> current = arr;
82+
// dictionary<int32, large_utf8> decodes to large_utf8, which still needs converting. The bound
83+
// stops a driver from turning a malformed type into an endless loop.
84+
constexpr int kMaxPasses = 4;
85+
for (int pass = 0; pass < kMaxPasses; ++pass) {
86+
const auto& type = *current->type();
87+
if (is_serde_acceptable_arrow_type(type)) {
88+
*out = std::move(current);
89+
return Status::OK();
90+
}
91+
92+
auto target = target_type_for(type);
93+
if (target == nullptr) {
94+
return Status::NotSupported(
95+
"ADBC: arrow type '{}' cannot be materialized into a Doris column",
96+
type.ToString());
97+
}
98+
99+
auto casted = arrow::compute::Cast(*current, target);
100+
if (!casted.ok()) {
101+
return Status::InternalError("ADBC: failed to normalize arrow type '{}' to '{}': {}",
102+
type.ToString(), target->ToString(),
103+
casted.status().ToString());
104+
}
105+
current = casted.MoveValueUnsafe();
106+
}
107+
return Status::InternalError(
108+
"ADBC: arrow type '{}' is still not materializable after {} "
109+
"normalization passes",
110+
arr->type()->ToString(), kMaxPasses);
111+
}
112+
113+
} // namespace doris
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
// Licensed to the Apache Software Foundation (ASF) under one
2+
// or more contributor license agreements. See the NOTICE file
3+
// distributed with this work for additional information
4+
// regarding copyright ownership. The ASF licenses this file
5+
// to you under the Apache License, Version 2.0 (the
6+
// "License"); you may not use this file except in compliance
7+
// with the License. You may obtain a copy of the License at
8+
//
9+
// http://www.apache.org/licenses/LICENSE-2.0
10+
//
11+
// Unless required by applicable law or agreed to in writing,
12+
// software distributed under the License is distributed on an
13+
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
// KIND, either express or implied. See the License for the
15+
// specific language governing permissions and limitations
16+
// under the License.
17+
18+
#pragma once
19+
20+
#include <arrow/type_fwd.h>
21+
22+
#include <memory>
23+
24+
#include "common/status.h"
25+
26+
namespace doris {
27+
28+
/// DataTypeSerDe::read_column_from_arrow was written for the Arrow that Doris itself emits, so it
29+
/// accepts only a subset of the Arrow type variants. Third-party ADBC drivers emit others: DuckDB
30+
/// emits string_view, Go-based drivers may emit large_* and dictionary. This normalizes them into
31+
/// a shape the serdes accept.
32+
///
33+
/// Only top-level types are normalized. A nested type whose child is an unaccepted variant (say
34+
/// list<large_utf8>) passes through and fails inside the serde, loudly rather than silently.
35+
36+
/// Whether the serdes take this Arrow type as-is.
37+
bool is_serde_acceptable_arrow_type(const arrow::DataType& type);
38+
39+
/// Normalizes `arr` into a serde-acceptable shape. Returns it unchanged (no copy) when it already
40+
/// is one, and fails with the offending type named when no accepted shape exists.
41+
Status normalize_arrow_array(const std::shared_ptr<arrow::Array>& arr,
42+
std::shared_ptr<arrow::Array>* out);
43+
44+
} // namespace doris

0 commit comments

Comments
 (0)