Skip to content

Commit 0c01156

Browse files
authored
[feat](thirdparty) add arrow-adbc to the thirdparty build (#66358)
### What problem does this PR solve? Related Issue: #65615 Related PR: #66331 Problem Summary: Split out of #66331, which adds an `adbc` catalog type that reads an external source through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/) driver. This PR carries only that PR's `thirdparty/` half, so the dependency can be reviewed and the build-env image rebuilt before the code that links against it lands. Nothing in the tree consumes these artifacts yet -- this PR adds one package to the thirdparty build and declares its license, and changes nothing else. **What comes out of it** | Artifact | How | Used by | |---|---|---| | `libadbc_driver_manager.a` | built from source | to be statically linked into `doris_be` (#66331) | | `libadbc_driver_jni.so` | built from source | to be loaded by the FE ADBC connector (#66331) | | `libadbc_driver_sqlite.so` | built from source | tests only, **not shipped** | | `libadbc_driver_flightsql.so` | prebuilt, from the official release wheel | tests only, **not shipped** | Doris ships no ADBC driver to users; a deployment supplies its own. The two drivers above exist so the ADBC code paths can be tested at all. **Three things upstream does that do not carry over** - *The SQLite driver needs a system SQLite3 development package*, which Doris does not ship and most build hosts lack. The source tree vendors the amalgamation but never references it from CMake, so it is compiled here into a scratch static library, handed to `FindSQLite3`, and dropped afterwards. It ends up statically inside the driver, leaving no sqlite artifacts in thirdparty. - *The JNI bridge header is generated by shelling out to Maven* (`java/driver/jni/CMakeLists.txt` runs `mvn -Pjni,javah compile`). Doing that would make this the first thirdparty package to require Maven, a Maven Central connection and a JDK 11+, while the build-env image runs this script with `JAVA_HOME` on JDK 8. The `javah` output is checked in as a patch instead and `jni_wrapper.cc` is compiled against it directly, needing nothing but `jni.h`. The patch header records how to regenerate it on a version bump. The prebuilt JNI binary inside upstream's Maven jar is not used either: it requires `GLIBC_2.34` and `GLIBCXX_3.4.31`, which excludes CentOS 7/8, Rocky 8 and Ubuntu 20.04. - *The Flight SQL driver is written in Go and no bare shared library is published*, so it is taken from the official release wheel -- a zip the existing download step already knows how to unpack -- rather than adding a Go toolchain to the thirdparty build. It is skipped on platforms upstream publishes no prebuilt binary for, the same way hyperscan is. **On the version pin** The source tree is tag `apache-arrow-adbc-24`, which is release C/Go 1.12.0 (the tag carries neither number). The prebuilt Flight SQL driver is pinned to that same release, and that is not cosmetic: ADBC partition descriptors are driver-private bytes, so every process that hands one to another must have loaded the very same driver build. `dist/LICENSE-dist.txt` gets the corresponding Apache-2.0 entry. It is the only file outside `thirdparty/` here.
1 parent 82646c3 commit 0c01156

5 files changed

Lines changed: 605 additions & 0 deletions

File tree

‎dist/LICENSE-dist.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1463,6 +1463,9 @@ The Apache Software License, Version 2.0
14631463
* brpc: 1.1.0
14641464
* rocksdb: 5.14.2
14651465
* arrow: 7.0.0
1466+
* arrow-adbc: 1.12.0
1467+
- source: https://github.com/apache/arrow-adbc
1468+
- only the driver manager (C++) is linked into the BE binary
14661469
* S2: 0.10.0
14671470
* croaringbitmap: 0.4.0
14681471
* Abseil LTS 20220623

‎thirdparty/build-thirdparty.sh‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1152,6 +1152,91 @@ build_arrow() {
11521152
strip_lib libarrow_acero.a
11531153
}
11541154

1155+
# arrow-adbc
1156+
# Produces three artifacts from one source tree:
1157+
# libadbc_driver_manager.a -- statically linked into doris_be
1158+
# libadbc_driver_jni.so -- loaded by the FE adbc connector
1159+
# libadbc_driver_sqlite.so -- BE unit tests only, not shipped
1160+
# and installs a fourth that is not built here:
1161+
# libadbc_driver_flightsql.so -- prebuilt, adbc tests only, not shipped
1162+
build_arrow_adbc() {
1163+
check_if_source_exist "${ARROW_ADBC_SOURCE}"
1164+
1165+
local adbc_src="${TP_SOURCE_DIR}/${ARROW_ADBC_SOURCE}"
1166+
1167+
# The SQLite driver needs a SQLite3 development package, which Doris does not
1168+
# ship and most build hosts lack. arrow-adbc vendors the amalgamation, so build
1169+
# it here into a scratch prefix and hand the paths to FindSQLite3. It ends up
1170+
# statically inside libadbc_driver_sqlite.so, so nothing sqlite is installed.
1171+
local sqlite_host="${adbc_src}/c/${BUILD_DIR}-sqlite-host"
1172+
rm -rf "${sqlite_host}"
1173+
mkdir -p "${sqlite_host}/include" "${sqlite_host}/lib"
1174+
"${CC}" -O2 -fPIC -DSQLITE_ENABLE_COLUMN_METADATA=1 \
1175+
-c "${adbc_src}/c/vendor/sqlite3/sqlite3.c" -o "${sqlite_host}/sqlite3.o"
1176+
ar rcs "${sqlite_host}/lib/libsqlite3.a" "${sqlite_host}/sqlite3.o"
1177+
cp -f "${adbc_src}/c/vendor/sqlite3/sqlite3.h" "${sqlite_host}/include/"
1178+
1179+
# (1) driver manager + sqlite driver
1180+
cd "${adbc_src}/c"
1181+
rm -rf "${BUILD_DIR}"
1182+
mkdir -p "${BUILD_DIR}"
1183+
cd "${BUILD_DIR}"
1184+
1185+
"${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
1186+
-DCMAKE_BUILD_TYPE=Release \
1187+
-DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}" \
1188+
-DCMAKE_INSTALL_LIBDIR=lib64 \
1189+
-DCMAKE_POSITION_INDEPENDENT_CODE=ON \
1190+
-DADBC_DRIVER_MANAGER=ON \
1191+
-DADBC_DRIVER_SQLITE=ON \
1192+
-DADBC_BUILD_STATIC=ON \
1193+
-DADBC_BUILD_SHARED=ON \
1194+
-DADBC_BUILD_TESTS=OFF \
1195+
-DADBC_USE_CCACHE=OFF \
1196+
-DSQLite3_INCLUDE_DIR="${sqlite_host}/include" \
1197+
-DSQLite3_LIBRARY="${sqlite_host}/lib/libsqlite3.a" \
1198+
..
1199+
"${BUILD_SYSTEM}" -j "${PARALLEL}"
1200+
"${BUILD_SYSTEM}" install
1201+
1202+
# (2) JNI bridge for the FE. Must come after (1): it links the driver manager.
1203+
# Built directly rather than through upstream's java/CMakeLists.txt, which
1204+
# shells out to Maven just to generate the JNI header; that header is
1205+
# checked in as a patch instead (see thirdparty/patches). Also not taken
1206+
# from the Maven jar: the prebuilt binary there requires GLIBC_2.34 and
1207+
# GLIBCXX_3.4.31, which excludes CentOS 7/8, Rocky 8, Ubuntu 20.04 and more.
1208+
# Only jni.h is needed, so any JDK will do.
1209+
if [[ ! -f "${JAVA_HOME}/include/jni.h" ]]; then
1210+
echo "arrow-adbc: JAVA_HOME must point at a JDK (no ${JAVA_HOME}/include/jni.h)"
1211+
exit 1
1212+
fi
1213+
local jni_md_dir='linux'
1214+
if [[ "${KERNEL}" == 'Darwin' ]]; then
1215+
jni_md_dir='darwin'
1216+
fi
1217+
1218+
"${CXX}" -std="c++${TP_CXX_STANDARD}" -O2 -fPIC -shared \
1219+
-I"${JAVA_HOME}/include" \
1220+
-I"${JAVA_HOME}/include/${jni_md_dir}" \
1221+
-I"${adbc_src}/java/driver/jni/doris_generated" \
1222+
-I"${TP_INCLUDE_DIR}" \
1223+
"${adbc_src}/java/driver/jni/src/main/cpp/jni_wrapper.cc" \
1224+
-o "${TP_INSTALL_DIR}/lib64/libadbc_driver_jni.so" \
1225+
"${TP_INSTALL_DIR}/lib64/libadbc_driver_manager.a"
1226+
1227+
# (3) Flight SQL driver. Not built: upstream implements it in Go, so it comes
1228+
# prebuilt out of the release wheel that download-thirdparty.sh unpacked
1229+
# (see vars.sh). Nothing links against it; the FE and BE dlopen it at run
1230+
# time, and only the adbc tests ask for it.
1231+
if [[ -n "${ARROW_ADBC_FLIGHTSQL_SOURCE}" ]]; then
1232+
check_if_source_exist "${ARROW_ADBC_FLIGHTSQL_SOURCE}"
1233+
cp -f "${TP_SOURCE_DIR}/${ARROW_ADBC_FLIGHTSQL_SOURCE}/libadbc_driver_flightsql.so" \
1234+
"${TP_INSTALL_DIR}/lib64/libadbc_driver_flightsql.so"
1235+
fi
1236+
1237+
rm -rf "${sqlite_host}"
1238+
}
1239+
11551240
# abseil
11561241
build_abseil() {
11571242
check_if_source_exist "${ABSEIL_SOURCE}"
@@ -2228,6 +2313,7 @@ if [[ "${#packages[@]}" -eq 0 ]]; then
22282313
cares
22292314
grpc # after cares, protobuf
22302315
arrow
2316+
arrow_adbc
22312317
lance_c
22322318
s2
22332319
bitshuffle
@@ -2319,6 +2405,15 @@ cleanup_package_source() {
23192405
librdkafka) src_var="LIBRDKAFKA_SOURCE" ;;
23202406
flatbuffers) src_var="FLATBUFFERS_SOURCE" ;;
23212407
arrow) src_var="ARROW_SOURCE" ;;
2408+
arrow_adbc)
2409+
# arrow_adbc also unpacks the prebuilt flightsql driver, clean both
2410+
if [[ -n "${ARROW_ADBC_FLIGHTSQL_SOURCE}" && -d "${TP_SOURCE_DIR}/${ARROW_ADBC_FLIGHTSQL_SOURCE}" ]]; then
2411+
echo "Cleaning up source: ${ARROW_ADBC_FLIGHTSQL_SOURCE}"
2412+
rm -rf "${TP_SOURCE_DIR}/${ARROW_ADBC_FLIGHTSQL_SOURCE}" \
2413+
"${TP_SOURCE_DIR}/${ARROW_ADBC_FLIGHTSQL_SOURCE}"-*.dist-info
2414+
fi
2415+
src_var="ARROW_ADBC_SOURCE"
2416+
;;
23222417
brotli) src_var="BROTLI_SOURCE" ;;
23232418
cares) src_var="CARES_SOURCE" ;;
23242419
grpc) src_var="GRPC_SOURCE" ;;

‎thirdparty/download-thirdparty.sh‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -458,6 +458,18 @@ if [[ " ${TP_ARCHIVES[*]} " =~ " ARROW " ]]; then
458458
echo "Finished patching ${ARROW_SOURCE}"
459459
fi
460460

461+
# arrow-adbc patch adds the pre-generated JNI header, so that building the JNI
462+
# bridge needs no Maven. See the patch header for how to regenerate it.
463+
if [[ " ${TP_ARCHIVES[*]} " =~ " ARROW_ADBC " ]]; then
464+
cd "${TP_SOURCE_DIR}/${ARROW_ADBC_SOURCE}"
465+
if [[ ! -f "${PATCHED_MARK}" ]]; then
466+
patch -p1 <"${TP_PATCH_DIR}/apache-arrow-adbc-24-pregenerated-jni-header.patch"
467+
touch "${PATCHED_MARK}"
468+
fi
469+
cd -
470+
echo "Finished patching ${ARROW_ADBC_SOURCE}"
471+
fi
472+
461473
# patch librdkafka to avoid crash
462474
if [[ " ${TP_ARCHIVES[*]} " =~ " LIBRDKAFKA " ]]; then
463475
if [[ "${LIBRDKAFKA_SOURCE}" == "librdkafka-2.11.0" ]]; then

0 commit comments

Comments
 (0)