Skip to content

Commit dabd51e

Browse files
authored
[improvement](lance) Upgrade prewarm dependency and remove install guard (#68765)
### What problem does this PR solve? Related PR: #68698 Related issue: #68692 (dependency preparation only) Port #68698 to master. Upgrade lance-c from `cd63420bfbe27f6f0a1edcc873b9191af7d52852` to `98468344bc9d56aa7ed4a4192e844afa62b34ba1`, including the upstream synchronous index-prewarm C/C++ APIs. Refresh the Foyer patch context while preserving its implementation payload and verify that patch application retains the prewarm APIs and shared-session regression. Remove the Lance-specific installation fingerprint helper, its test harness, and their build/CI calls. Existing installations are no longer rejected or rebuilt because their Lance revision fingerprint differs; unavailable APIs are reported through normal compilation or linking. Preserve master's unrelated build and CI changes. This only prepares the dependency; it does not add a SQL prewarm interface or complete the parent issue. ### Validation - Verified the archive checksum and unchanged Foyer implementation payload. - Passed `lance-prefilter-patch-test.sh`, covering fresh extraction, cached-source reuse, re-extraction, stale patch markers, invalid-patch rejection, macOS definitions, and prewarm API/regression preservation. - Passed `download-thirdparty-fallback-test.sh`, `juicefs-default-mirror-test.sh`, `download-thirdparty-md5-test.sh`, `azure-vcpkg-retry-test.sh`, and `adbc-jni-config-test.sh`. - Passed affected shell syntax checks, workflow YAML parsing, and `git diff --check`. No Doris FE/BE source files change. Full compilation and runtime suites were not run locally; PR CI will validate the build. ### Release note Prepare the Lance dependency for index prewarm and remove Lance-specific installation fingerprint checks. No new SQL interface is exposed. ### Check List (For Author) - Test: Existing dependency/script regression tests and manual validation described above. - Behavior changed: Yes; upgrade lance-c and remove Lance revision fingerprint checks. - Does this need documentation: No new SQL interface. ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label
1 parent 310e302 commit dabd51e

8 files changed

Lines changed: 27 additions & 261 deletions

File tree

‎.github/workflows/build-thirdparty.yml‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -82,9 +82,6 @@ jobs:
8282
run: |
8383
thirdparty/test/adbc-jni-config-test.sh
8484
85-
- name: Test Lance installation
86-
run: bash thirdparty/test/lance-install-test.sh
87-
8885
build_linux:
8986
name: Build Third Party Libraries (Linux)
9087
needs: changes

‎build.sh‎

Lines changed: 4 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -480,7 +480,6 @@ if [[ "${CLEAN}" -eq 1 && "${BUILD_BE}" -eq 0 && "${BUILD_FE}" -eq 0 && ${BUILD_
480480
fi
481481

482482
# build thirdparty libraries if necessary. check last thirdparty lib installation
483-
source "${DORIS_HOME}/thirdparty/lance-install.sh"
484483
if [[ "${TARGET_SYSTEM}" == 'Darwin' ]]; then
485484
LAST_THIRDPARTY_LIB='libbrotlienc.a'
486485
else
@@ -496,21 +495,10 @@ if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
496495
! -f "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
497496
! -s "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
498497
! -s "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
499-
-e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]] ||
500-
! lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${DORIS_THIRDPARTY}/installed"; then
501-
# External trees can be partially updated or pinned to another revision. Preserve
502-
# the existing prefix unless their build inputs can produce the requested Lance version.
503-
for input in build-thirdparty.sh download-thirdparty.sh vars.sh lance-install.sh patches/lance-c-foyer.patch; do
504-
if [[ ! -f "${DORIS_THIRDPARTY}/${input}" || ! -r "${DORIS_THIRDPARTY}/${input}" ]]; then
505-
echo "Third-party dependencies require a rebuild, but ${input} is missing or unreadable." >&2
506-
echo "Refresh the compilation image or set DORIS_THIRDPARTY to a complete third-party source tree." >&2
507-
exit 1
508-
fi
509-
done
510-
if ! expected_lance_fingerprint="$(lance_c_install_fingerprint "${DORIS_HOME}/thirdparty")" ||
511-
! rebuild_lance_fingerprint="$(lance_c_install_fingerprint "${DORIS_THIRDPARTY}")" ||
512-
[[ "${rebuild_lance_fingerprint}" != "${expected_lance_fingerprint}" ]]; then
513-
echo "Lance rebuild sources do not match this checkout; installed dependencies have been preserved." >&2
498+
-e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]]; then
499+
# Compilation images may contain only installed artifacts; never erase them without a rebuild source.
500+
if [[ ! -f "${DORIS_THIRDPARTY}/build-thirdparty.sh" ]]; then
501+
echo "Third-party dependencies require a rebuild, but build-thirdparty.sh is missing." >&2
514502
echo "Refresh the compilation image or set DORIS_THIRDPARTY to a complete third-party source tree." >&2
515503
exit 1
516504
fi
@@ -523,11 +511,6 @@ if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
523511
else
524512
bash "${DORIS_THIRDPARTY}/build-thirdparty.sh" -j "${PARALLEL}" --clean
525513
fi
526-
# An external build script can itself be stale. Never link its old output silently.
527-
if ! lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${DORIS_THIRDPARTY}/installed"; then
528-
echo "Lance dependency revision does not match this checkout. Refresh the third-party build tree." >&2
529-
exit 1
530-
fi
531514
fi
532515

533516
update_submodule() {

‎thirdparty/build-thirdparty.sh‎

Lines changed: 0 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -219,8 +219,6 @@ if [[ ! -f "${TP_DIR}/vars.sh" ]]; then
219219
fi
220220

221221
. "${TP_DIR}/vars.sh"
222-
. "${TP_DIR}/lance-install.sh"
223-
LANCE_C_INSTALL_FINGERPRINT="$(lance_c_install_fingerprint "${TP_DIR}")"
224222

225223
cd "${TP_DIR}"
226224

@@ -2509,14 +2507,9 @@ build_lance_c() {
25092507
env "${cargo_env[@]}" "${cargo_bin}" "${cargo_args[@]}"
25102508

25112509
mkdir -p "${TP_INSTALL_DIR}/include" "${TP_INSTALL_DIR}/lib64"
2512-
# Invalidate before publishing either file so interrupted installs cannot reuse
2513-
# a matching marker with a partial header/archive pair.
2514-
rm -f "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
25152510
rm -rf "${TP_INSTALL_DIR}/include/lance"
25162511
cp -av include/lance "${TP_INSTALL_DIR}/include/"
25172512
install_rust_archive "${BUILD_DIR}/release/liblance_c.a"
2518-
printf '%s\n' "${LANCE_C_INSTALL_FINGERPRINT}" > "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint.tmp"
2519-
mv "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint.tmp" "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
25202513
}
25212514

25222515
build_paimon_rust() {

‎thirdparty/lance-install.sh‎

Lines changed: 0 additions & 38 deletions
This file was deleted.

‎thirdparty/patches/lance-c-foyer.patch‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Foyer data-cache integration for lance-format/lance-c#73.
2-
# Base: cd63420bfbe27f6f0a1edcc873b9191af7d52852 (lance-format/lance-c main, including #93)
2+
# Base: 98468344bc9d56aa7ed4a4192e844afa62b34ba1 (lance-format/lance-c main, including #94)
33
# Source: https://github.com/Gabriel39/lance-c/commit/24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f
44
# The unchanged Foyer payload is reapplied to the newer upstream base with git apply --3way.
55
diff --git a/Cargo.lock b/Cargo.lock
@@ -198,7 +198,7 @@ index 18654b862440a90aa6bf0dd846e8f1ad6036fd7c..fd134a7151e1df61da201d867a642e9b
198198
prost = "0.14"
199199
snafu = "0.9"
200200
diff --git a/README.md b/README.md
201-
index 9f0871b17b13ebfd96916947b4bee0dd06f6a468..441719c9f3904058127e55d1cd29ea1afc878935 100644
201+
index f13ad015826a8ac5468cfea247707d987cc0864d..56422c22969882cd43edd5e83a4a5c4d0dadd3d6 100644
202202
--- a/README.md
203203
+++ b/README.md
204204
@@ -68,6 +68,7 @@ Based on the [liblance RFC](https://github.com/lance-format/lance/discussions/60
@@ -242,11 +242,11 @@ index 9f0871b17b13ebfd96916947b4bee0dd06f6a468..441719c9f3904058127e55d1cd29ea1a
242242
+auto ds = lance::Dataset::open_with_session(session, "s3://bucket/data.lance");
243243
+```
244244
+
245-
### Open at a specific version
245+
### Prewarm an index synchronously
246246

247-
`lance_dataset_open` takes a `version` argument — `0` means the latest, any
247+
Prewarm a logical index before serving queries to move cold index reads out of
248248
diff --git a/include/lance/lance.h b/include/lance/lance.h
249-
index 94772e2a14bf9a8a950dfb2d39112136e27197c0..fe48eb0d9afd1c7157a3bd2230cce7e769120bac 100644
249+
index b91c290a40d37edbb49323d2d11efd3b4f980bb8..0997023974a5205a145a991d53da0ee1a2babc8c 100644
250250
--- a/include/lance/lance.h
251251
+++ b/include/lance/lance.h
252252
@@ -214,6 +214,36 @@ typedef struct LanceSessionCacheStats {
@@ -334,7 +334,7 @@ index 94772e2a14bf9a8a950dfb2d39112136e27197c0..fe48eb0d9afd1c7157a3bd2230cce7e7
334334
void lance_dataset_close(LanceDataset* dataset);
335335

336336
diff --git a/include/lance/lance.hpp b/include/lance/lance.hpp
337-
index edcd39fc3caba387ab490fa84139f24e9376130a..17b71957d0f5c419fe0b3392ce11faa757ddbfe1 100644
337+
index 17e93486c079d3453547af160b875310328da9e1..b049cc519ee9784d4bcc1a087eac237e880b5e74 100644
338338
--- a/include/lance/lance.hpp
339339
+++ b/include/lance/lance.hpp
340340
@@ -176,6 +176,13 @@ struct SqlColumn {
@@ -2243,7 +2243,7 @@ index 1971510de4a13ee6f04fc39c159b088e78e21d55..ba51c87a8cebb6897c48fb55509f71da
22432243
// SAFETY: `out_dataset` is non-NULL (checked above) and the caller
22442244
// guarantees it points to caller-owned, writable storage of size
22452245
diff --git a/tests/c_api_test.rs b/tests/c_api_test.rs
2246-
index d3bea44529a842fe3ae28ee3f90cd8d45fe0b394..98e6901a2cfeb2cc98df6606e26b04788d4218b2 100644
2246+
index b00072df3f966f58663359ccd370b399650c2023..e44871e92bc7ff9787bf5220b1ea8a4270da9710 100644
22472247
--- a/tests/c_api_test.rs
22482248
+++ b/tests/c_api_test.rs
22492249
@@ -100,10 +100,83 @@ fn create_large_dataset(num_rows: i32) -> (tempfile::TempDir, String) {
@@ -2502,7 +2502,7 @@ index d3bea44529a842fe3ae28ee3f90cd8d45fe0b394..98e6901a2cfeb2cc98df6606e26b0478
25022502
fn test_dataset_restore_to_current_latest_writes_new_manifest() {
25032503
// Restoring to the current latest still writes a new manifest. The
25042504
diff --git a/tests/cpp/test_c_api.c b/tests/cpp/test_c_api.c
2505-
index efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10b1d21720 100644
2505+
index 77af1a69b872bd4152a44e642f30190720daa33f..da15749671cb9448c73d3c61546ecab2adcacb15 100644
25062506
--- a/tests/cpp/test_c_api.c
25072507
+++ b/tests/cpp/test_c_api.c
25082508
@@ -171,6 +171,37 @@ static void test_shared_session(const char *uri) {
@@ -2543,7 +2543,7 @@ index efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10
25432543
static void test_scan(const char *uri) {
25442544
printf(" test_scan... ");
25452545

2546-
@@ -1471,6 +1502,7 @@ int main(int argc, char **argv) {
2546+
@@ -1476,6 +1507,7 @@ int main(int argc, char **argv) {
25472547
test_batch_nearest(uri);
25482548
test_open_and_metadata(uri);
25492549
test_shared_session(uri);
@@ -2552,7 +2552,7 @@ index efad5ed55987976a3f1820220e42fc58bef42638..050543aebc02454cab09f6e0bd38ec10
25522552
test_distance_range(uri);
25532553
test_scan_with_limit(uri);
25542554
diff --git a/tests/cpp/test_cpp_api.cpp b/tests/cpp/test_cpp_api.cpp
2555-
index e2d541e4430889aba89ab85c01cc7e3eacf97baa..ac1c10bccfcf399bbbfcfe908d3188484aff0add 100644
2555+
index 3bc313ebecbb0de06badd9f8d75f531fc95809c9..298228f7dd68d28452e6436db5c589183361f2de 100644
25562556
--- a/tests/cpp/test_cpp_api.cpp
25572557
+++ b/tests/cpp/test_cpp_api.cpp
25582558
@@ -131,6 +131,28 @@ static void test_shared_session(const std::string& uri) {
@@ -2584,7 +2584,7 @@ index e2d541e4430889aba89ab85c01cc7e3eacf97baa..ac1c10bccfcf399bbbfcfe908d318848
25842584
static void test_dataset_schema(const std::string& uri) {
25852585
TEST(test_dataset_schema);
25862586

2587-
@@ -1353,6 +1375,7 @@ int main(int argc, char** argv) {
2587+
@@ -1365,6 +1387,7 @@ int main(int argc, char** argv) {
25882588

25892589
test_dataset_open(uri);
25902590
test_shared_session(uri);

‎thirdparty/test/lance-install-test.sh‎

Lines changed: 0 additions & 176 deletions
This file was deleted.

0 commit comments

Comments
 (0)