Repository navigation
Commit 0c01156
authored
[feat](thirdparty) add arrow-adbc to the thirdparty build (#66358)
### What problem does this PR solve?
Related Issue: #65615
Related PR: #66331
Problem Summary:
Split out of #66331, which adds an `adbc` catalog type that reads an
external source
through an [Arrow Database Connectivity](https://arrow.apache.org/adbc/)
driver. This PR
carries only that PR's `thirdparty/` half, so the dependency can be
reviewed and the
build-env image rebuilt before the code that links against it lands.
Nothing in the tree consumes these artifacts yet -- this PR adds one
package to the
thirdparty build and declares its license, and changes nothing else.
**What comes out of it**
| Artifact | How | Used by |
|---|---|---|
| `libadbc_driver_manager.a` | built from source | to be statically
linked into `doris_be` (#66331) |
| `libadbc_driver_jni.so` | built from source | to be loaded by the FE
ADBC connector (#66331) |
| `libadbc_driver_sqlite.so` | built from source | tests only, **not
shipped** |
| `libadbc_driver_flightsql.so` | prebuilt, from the official release
wheel | tests only, **not shipped** |
Doris ships no ADBC driver to users; a deployment supplies its own. The
two drivers above
exist so the ADBC code paths can be tested at all.
**Three things upstream does that do not carry over**
- *The SQLite driver needs a system SQLite3 development package*, which
Doris does not
ship and most build hosts lack. The source tree vendors the amalgamation
but never
references it from CMake, so it is compiled here into a scratch static
library, handed
to `FindSQLite3`, and dropped afterwards. It ends up statically inside
the driver,
leaving no sqlite artifacts in thirdparty.
- *The JNI bridge header is generated by shelling out to Maven*
(`java/driver/jni/CMakeLists.txt` runs `mvn -Pjni,javah compile`). Doing
that would make
this the first thirdparty package to require Maven, a Maven Central
connection and a
JDK 11+, while the build-env image runs this script with `JAVA_HOME` on
JDK 8. The
`javah` output is checked in as a patch instead and `jni_wrapper.cc` is
compiled against
it directly, needing nothing but `jni.h`. The patch header records how
to regenerate it
on a version bump.
The prebuilt JNI binary inside upstream's Maven jar is not used either:
it requires
`GLIBC_2.34` and `GLIBCXX_3.4.31`, which excludes CentOS 7/8, Rocky 8
and Ubuntu 20.04.
- *The Flight SQL driver is written in Go and no bare shared library is
published*, so it
is taken from the official release wheel -- a zip the existing download
step already
knows how to unpack -- rather than adding a Go toolchain to the
thirdparty build. It is
skipped on platforms upstream publishes no prebuilt binary for, the same
way hyperscan
is.
**On the version pin**
The source tree is tag `apache-arrow-adbc-24`, which is release C/Go
1.12.0 (the tag
carries neither number). The prebuilt Flight SQL driver is pinned to
that same release,
and that is not cosmetic: ADBC partition descriptors are driver-private
bytes, so every
process that hands one to another must have loaded the very same driver
build.
`dist/LICENSE-dist.txt` gets the corresponding Apache-2.0 entry. It is
the only file
outside `thirdparty/` here.1 parent 82646c3 commit 0c01156
5 files changed
Lines changed: 605 additions & 0 deletions
File tree
- dist
- thirdparty
- patches
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1463 | 1463 | | |
1464 | 1464 | | |
1465 | 1465 | | |
| 1466 | + | |
| 1467 | + | |
| 1468 | + | |
1466 | 1469 | | |
1467 | 1470 | | |
1468 | 1471 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1152 | 1152 | | |
1153 | 1153 | | |
1154 | 1154 | | |
| 1155 | + | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
| 1175 | + | |
| 1176 | + | |
| 1177 | + | |
| 1178 | + | |
| 1179 | + | |
| 1180 | + | |
| 1181 | + | |
| 1182 | + | |
| 1183 | + | |
| 1184 | + | |
| 1185 | + | |
| 1186 | + | |
| 1187 | + | |
| 1188 | + | |
| 1189 | + | |
| 1190 | + | |
| 1191 | + | |
| 1192 | + | |
| 1193 | + | |
| 1194 | + | |
| 1195 | + | |
| 1196 | + | |
| 1197 | + | |
| 1198 | + | |
| 1199 | + | |
| 1200 | + | |
| 1201 | + | |
| 1202 | + | |
| 1203 | + | |
| 1204 | + | |
| 1205 | + | |
| 1206 | + | |
| 1207 | + | |
| 1208 | + | |
| 1209 | + | |
| 1210 | + | |
| 1211 | + | |
| 1212 | + | |
| 1213 | + | |
| 1214 | + | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
| 1221 | + | |
| 1222 | + | |
| 1223 | + | |
| 1224 | + | |
| 1225 | + | |
| 1226 | + | |
| 1227 | + | |
| 1228 | + | |
| 1229 | + | |
| 1230 | + | |
| 1231 | + | |
| 1232 | + | |
| 1233 | + | |
| 1234 | + | |
| 1235 | + | |
| 1236 | + | |
| 1237 | + | |
| 1238 | + | |
| 1239 | + | |
1155 | 1240 | | |
1156 | 1241 | | |
1157 | 1242 | | |
| |||
2228 | 2313 | | |
2229 | 2314 | | |
2230 | 2315 | | |
| 2316 | + | |
2231 | 2317 | | |
2232 | 2318 | | |
2233 | 2319 | | |
| |||
2319 | 2405 | | |
2320 | 2406 | | |
2321 | 2407 | | |
| 2408 | + | |
| 2409 | + | |
| 2410 | + | |
| 2411 | + | |
| 2412 | + | |
| 2413 | + | |
| 2414 | + | |
| 2415 | + | |
| 2416 | + | |
2322 | 2417 | | |
2323 | 2418 | | |
2324 | 2419 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
458 | 458 | | |
459 | 459 | | |
460 | 460 | | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
461 | 473 | | |
462 | 474 | | |
463 | 475 | | |
| |||
0 commit comments