Skip to content

Commit 0919857

Browse files
Gabriel39Gabriel39
authored
[improvement](lance) Update lance-c to merged upstream main (#68689)
### What problem does this PR solve? Related PR: #68687 Pin lance-c to merged upstream main commit `cd63420bfbe27f6f0a1edcc873b9191af7d52852`, including the distance-bounded search (#90), independent batch nearest search (#92), and boolean scalar-segment predicates (#93) changes. Update the archive checksum and refresh the existing Foyer patch against that base with an unchanged implementation payload. Foyer PR [lance-format/lance-c#73](lance-format/lance-c#73) is still open. Its patch must remain temporarily because Doris already calls its cache APIs; it can be removed after upstream merge. This PR contains the dependency pin/Foyer refresh and its build integration: `build.sh`, `thirdparty/build-thirdparty.sh`, `thirdparty/lance-install.sh`, and the installation regression harness. The build checks a source/archive/patch fingerprint before reusing Lance, invalidates it during publication, and rejects stale external build output. Before removing an existing installation, it verifies required rebuild inputs and rejects external definitions that differ from the checkout. The third-party CI script job runs the installation harness under the existing workflow path filters. FE predicate changes and SQL fixtures/tests from #68687 are not included. ### Validation - Verified the downloaded archive checksum and exact patch application. - Third-party extraction, cached-source reuse/refresh, and invalid-patch rejection tests passed on master. - The updated dependency plus the unchanged Foyer patch passed 75 Rust unit tests, 376 C API tests, and all 3 native C/C++/static OSS consumer tests. - The same dependency passed 27 FE-generated Substrait integration scenarios as part of #68687 validation. - The installation harness passed on master and branch-4.1 using independent external source definitions: legacy/matching installs, missing helper/vars/patch/downloader/builder, mismatched and synchronized pin/patch updates, incomplete artifacts, stale builder output, interrupted publication, and retry. Rejected preflight cases verify that the entire installation and builder invocation count remain unchanged. The missing-helper regression failed before the fix. Shell syntax, workflow YAML, and CI path-filter checks passed. - Full Doris compilation and BE UT for this revision are pending CI. ### Release note Update the Lance C dependency to merged upstream search and scalar-predicate improvements. ### Check List (For Author) - Test: Dependency unit and integration tests described above. - Behavior changed: Yes; incorporates upstream lance-c fixes and additive APIs. - Does this need documentation: No new Doris user interface in this dependency-only update.
1 parent 75d1f03 commit 0919857

7 files changed

Lines changed: 264 additions & 24 deletions

File tree

‎.github/workflows/build-thirdparty.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,9 @@ jobs:
8282
run: |
8383
thirdparty/test/adbc-jni-config-test.sh
8484
85+
- name: Test Lance installation
86+
run: bash thirdparty/test/lance-install-test.sh
87+
8588
build_linux:
8689
name: Build Third Party Libraries (Linux)
8790
needs: changes

‎build.sh‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -480,6 +480,7 @@ if [[ "${CLEAN}" -eq 1 && "${BUILD_BE}" -eq 0 && "${BUILD_FE}" -eq 0 && ${BUILD_
480480
fi
481481

482482
# build thirdparty libraries if necessary. check last thirdparty lib installation
483+
source "${DORIS_HOME}/thirdparty/lance-install.sh"
483484
if [[ "${TARGET_SYSTEM}" == 'Darwin' ]]; then
484485
LAST_THIRDPARTY_LIB='libbrotlienc.a'
485486
else
@@ -495,10 +496,21 @@ if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
495496
! -f "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
496497
! -s "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
497498
! -s "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
498-
-e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]]; then
499-
# Compilation images may contain only installed artifacts; never erase them without a rebuild source.
500-
if [[ ! -f "${DORIS_THIRDPARTY}/build-thirdparty.sh" ]]; then
501-
echo "Third-party dependencies require a rebuild, but build-thirdparty.sh is missing." >&2
499+
-e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]] ||
500+
! lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${DORIS_THIRDPARTY}/installed"; then
501+
# External trees can be partially updated or pinned to another revision. Preserve
502+
# the existing prefix unless their build inputs can produce the requested Lance version.
503+
for input in build-thirdparty.sh download-thirdparty.sh vars.sh lance-install.sh patches/lance-c-foyer.patch; do
504+
if [[ ! -f "${DORIS_THIRDPARTY}/${input}" || ! -r "${DORIS_THIRDPARTY}/${input}" ]]; then
505+
echo "Third-party dependencies require a rebuild, but ${input} is missing or unreadable." >&2
506+
echo "Refresh the compilation image or set DORIS_THIRDPARTY to a complete third-party source tree." >&2
507+
exit 1
508+
fi
509+
done
510+
if ! expected_lance_fingerprint="$(lance_c_install_fingerprint "${DORIS_HOME}/thirdparty")" ||
511+
! rebuild_lance_fingerprint="$(lance_c_install_fingerprint "${DORIS_THIRDPARTY}")" ||
512+
[[ "${rebuild_lance_fingerprint}" != "${expected_lance_fingerprint}" ]]; then
513+
echo "Lance rebuild sources do not match this checkout; installed dependencies have been preserved." >&2
502514
echo "Refresh the compilation image or set DORIS_THIRDPARTY to a complete third-party source tree." >&2
503515
exit 1
504516
fi
@@ -511,6 +523,11 @@ if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
511523
else
512524
bash "${DORIS_THIRDPARTY}/build-thirdparty.sh" -j "${PARALLEL}" --clean
513525
fi
526+
# An external build script can itself be stale. Never link its old output silently.
527+
if ! lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${DORIS_THIRDPARTY}/installed"; then
528+
echo "Lance dependency revision does not match this checkout. Refresh the third-party build tree." >&2
529+
exit 1
530+
fi
514531
fi
515532

516533
update_submodule() {

‎thirdparty/build-thirdparty.sh‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,8 @@ if [[ ! -f "${TP_DIR}/vars.sh" ]]; then
219219
fi
220220

221221
. "${TP_DIR}/vars.sh"
222+
. "${TP_DIR}/lance-install.sh"
223+
LANCE_C_INSTALL_FINGERPRINT="$(lance_c_install_fingerprint "${TP_DIR}")"
222224

223225
cd "${TP_DIR}"
224226

@@ -2507,9 +2509,14 @@ build_lance_c() {
25072509
env "${cargo_env[@]}" "${cargo_bin}" "${cargo_args[@]}"
25082510

25092511
mkdir -p "${TP_INSTALL_DIR}/include" "${TP_INSTALL_DIR}/lib64"
2512+
# Invalidate before publishing either file so interrupted installs cannot reuse
2513+
# a matching marker with a partial header/archive pair.
2514+
rm -f "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
25102515
rm -rf "${TP_INSTALL_DIR}/include/lance"
25112516
cp -av include/lance "${TP_INSTALL_DIR}/include/"
25122517
install_rust_archive "${BUILD_DIR}/release/liblance_c.a"
2518+
printf '%s\n' "${LANCE_C_INSTALL_FINGERPRINT}" > "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint.tmp"
2519+
mv "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint.tmp" "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
25132520
}
25142521

25152522
build_paimon_rust() {

‎thirdparty/lance-install.sh‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
#!/usr/bin/env bash
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
19+
# Keep the check independent of the installed prefix: an external compilation image
20+
# can carry old vars.sh alongside an ABI-compatible but behaviorally stale archive.
21+
lance_c_install_fingerprint() (
22+
local definitions="$1"
23+
local TP_DIR="${definitions}"
24+
source "${definitions}/vars.sh" || return 1
25+
local patch_checksum
26+
patch_checksum="$(cksum < "${definitions}/patches/lance-c-foyer.patch")" || return 1
27+
printf '%s\n' "${LANCE_C_SOURCE}" "${LANCE_C_MD5SUM}" "${patch_checksum}"
28+
)
29+
30+
lance_c_install_is_current() {
31+
local definitions="$1" installed="$2" expected
32+
[[ -s "${installed}/lib64/liblance_c.a" &&
33+
-s "${installed}/include/lance/lance.h" &&
34+
-s "${installed}/include/lance/lance.hpp" &&
35+
-s "${installed}/lib64/.lance-c-fingerprint" ]] || return 1
36+
expected="$(lance_c_install_fingerprint "${definitions}")" || return 1
37+
[[ "$(cat "${installed}/lib64/.lance-c-fingerprint")" == "${expected}" ]]
38+
}

‎thirdparty/patches/lance-c-foyer.patch‎

Lines changed: 15 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,7 @@
11
# Foyer data-cache integration for lance-format/lance-c#73.
2-
# Base: 9bd730add2ac70316c1d642b8459011e2dd92022
2+
# Base: cd63420bfbe27f6f0a1edcc873b9191af7d52852 (lance-format/lance-c main, including #93)
33
# Source: https://github.com/Gabriel39/lance-c/commit/24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f
4-
# Regenerate the payload in lance-c; do not maintain separate downstream edits:
5-
# git diff --full-index --binary 9bd730add2ac70316c1d642b8459011e2dd92022 24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f -- . | sed 's/^ $//'
4+
# The unchanged Foyer payload is reapplied to the newer upstream base with git apply --3way.
65
diff --git a/Cargo.lock b/Cargo.lock
76
index 78812f1330e146db295a14276f90f9e28654f399..daf9f85daf3e0ea59bb906e8e3c32470519b084b 100644
87
--- a/Cargo.lock
@@ -199,7 +198,7 @@ index 18654b862440a90aa6bf0dd846e8f1ad6036fd7c..fd134a7151e1df61da201d867a642e9b
199198
prost = "0.14"
200199
snafu = "0.9"
201200
diff --git a/README.md b/README.md
202-
index 9ceccdc4f3f6b5d13dc271be3d2659c7be400306..94de4d1c26d8ab869d1726e2b416d21f21661d6d 100644
201+
index 9f0871b17b13ebfd96916947b4bee0dd06f6a468..441719c9f3904058127e55d1cd29ea1afc878935 100644
203202
--- a/README.md
204203
+++ b/README.md
205204
@@ -68,6 +68,7 @@ Based on the [liblance RFC](https://github.com/lance-format/lance/discussions/60
@@ -208,9 +207,9 @@ index 9ceccdc4f3f6b5d13dc271be3d2659c7be400306..94de4d1c26d8ab869d1726e2b416d21f
208207
| [x] | Filter pushdown | `lance_scanner_set_substrait_filter()` accepts a serialized Substrait `ExtendedExpression`; `lance_scanner_additional_sql_filter()` adds SQL predicates with AND before scanning starts |
209208
+| [x] | Data-file cache | Optional Foyer memory/disk cache for immutable `data/*.lance` reads |
210209

211-
## Multi-vector search
210+
## Segment-scoped array label filters
212211

213-
@@ -231,6 +232,35 @@ auto ds = lance::Dataset::open_with_session(session, "data.lance");
212+
@@ -362,6 +363,35 @@ auto ds = lance::Dataset::open_with_session(session, "data.lance");
214213
auto stats = session.cache_stats();
215214
```
216215

@@ -247,7 +246,7 @@ index 9ceccdc4f3f6b5d13dc271be3d2659c7be400306..94de4d1c26d8ab869d1726e2b416d21f
247246

248247
`lance_dataset_open` takes a `version` argument — `0` means the latest, any
249248
diff --git a/include/lance/lance.h b/include/lance/lance.h
250-
index 7630913fd7849d0bff2cb0f31e1ab1199cb9f0fb..1c1d0752efdda5023b29eafe5aa9a87e03dc5303 100644
249+
index 94772e2a14bf9a8a950dfb2d39112136e27197c0..fe48eb0d9afd1c7157a3bd2230cce7e769120bac 100644
251250
--- a/include/lance/lance.h
252251
+++ b/include/lance/lance.h
253252
@@ -214,6 +214,36 @@ typedef struct LanceSessionCacheStats {
@@ -335,7 +334,7 @@ index 7630913fd7849d0bff2cb0f31e1ab1199cb9f0fb..1c1d0752efdda5023b29eafe5aa9a87e
335334
void lance_dataset_close(LanceDataset* dataset);
336335

337336
diff --git a/include/lance/lance.hpp b/include/lance/lance.hpp
338-
index 286724e96736e354785e046f2fcfe5ae7c65147b..5070d423453a5d60d31d03ce758ad7dbe83a6ea2 100644
337+
index edcd39fc3caba387ab490fa84139f24e9376130a..17b71957d0f5c419fe0b3392ce11faa757ddbfe1 100644
339338
--- a/include/lance/lance.hpp
340339
+++ b/include/lance/lance.hpp
341340
@@ -176,6 +176,13 @@ struct SqlColumn {
@@ -2244,7 +2243,7 @@ index 1971510de4a13ee6f04fc39c159b088e78e21d55..ba51c87a8cebb6897c48fb55509f71da
22442243
// SAFETY: `out_dataset` is non-NULL (checked above) and the caller
22452244
// guarantees it points to caller-owned, writable storage of size
22462245
diff --git a/tests/c_api_test.rs b/tests/c_api_test.rs
2247-
index 1a25e34a8ca50623812ffedaa7bb5a8c54f7850e..c341859e9434d4578b2833eca6ec313a28f31f32 100644
2246+
index d3bea44529a842fe3ae28ee3f90cd8d45fe0b394..98e6901a2cfeb2cc98df6606e26b04788d4218b2 100644
22482247
--- a/tests/c_api_test.rs
22492248
+++ b/tests/c_api_test.rs
22502249
@@ -100,10 +100,83 @@ fn create_large_dataset(num_rows: i32) -> (tempfile::TempDir, String) {
@@ -2503,7 +2502,7 @@ index 1a25e34a8ca50623812ffedaa7bb5a8c54f7850e..c341859e9434d4578b2833eca6ec313a
25032502
fn test_dataset_restore_to_current_latest_writes_new_manifest() {
25042503
// Restoring to the current latest still writes a new manifest. The
25052504
diff --git a/tests/cpp/test_c_api.c b/tests/cpp/test_c_api.c
2506-
index 5df9cde4d8fa971823b9e6c974a8a0db6c9ec9ca..1444e55161631ac8ad976f427d867ad91d358195 100644
2505+
index efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10b1d21720 100644
25072506
--- a/tests/cpp/test_c_api.c
25082507
+++ b/tests/cpp/test_c_api.c
25092508
@@ -171,6 +171,37 @@ static void test_shared_session(const char *uri) {
@@ -2544,16 +2543,16 @@ index 5df9cde4d8fa971823b9e6c974a8a0db6c9ec9ca..1444e55161631ac8ad976f427d867ad9
25442543
static void test_scan(const char *uri) {
25452544
printf(" test_scan... ");
25462545

2547-
@@ -1323,6 +1354,7 @@ int main(int argc, char **argv) {
2548-
2546+
@@ -1471,6 +1502,7 @@ int main(int argc, char **argv) {
2547+
test_batch_nearest(uri);
25492548
test_open_and_metadata(uri);
25502549
test_shared_session(uri);
25512550
+ test_data_cache_session(uri, write_uri);
25522551
test_scan(uri);
2552+
test_distance_range(uri);
25532553
test_scan_with_limit(uri);
2554-
test_scanner_blob_handling(blob_uri);
25552554
diff --git a/tests/cpp/test_cpp_api.cpp b/tests/cpp/test_cpp_api.cpp
2556-
index 0332d1a2687354984d0551e10716df08344a4013..ef64ab0d02bed2f1750c3d388792e4aa44265e24 100644
2555+
index e2d541e4430889aba89ab85c01cc7e3eacf97baa..ac1c10bccfcf399bbbfcfe908d3188484aff0add 100644
25572556
--- a/tests/cpp/test_cpp_api.cpp
25582557
+++ b/tests/cpp/test_cpp_api.cpp
25592558
@@ -131,6 +131,28 @@ static void test_shared_session(const std::string& uri) {
@@ -2585,11 +2584,11 @@ index 0332d1a2687354984d0551e10716df08344a4013..ef64ab0d02bed2f1750c3d388792e4aa
25852584
static void test_dataset_schema(const std::string& uri) {
25862585
TEST(test_dataset_schema);
25872586

2588-
@@ -1211,6 +1233,7 @@ int main(int argc, char** argv) {
2587+
@@ -1353,6 +1375,7 @@ int main(int argc, char** argv) {
25892588

25902589
test_dataset_open(uri);
25912590
test_shared_session(uri);
25922591
+ test_data_cache_session(uri, write_uri);
25932592
test_dataset_schema(uri);
25942593
test_scanner_fluent(uri);
2595-
test_scanner_async_stream_ownership(uri);
2594+
test_distance_range(uri);

0 commit comments

Comments
 (0)