Skip to content

Commit 29f3df9

Browse files
committed
[improvement](lance) Adopt upstream segment prefilter optimization
### What problem does this PR solve? Related PR: lance-format/lance#9599, lance-format/lance-c#89 Problem Summary: Complete index-segment vector scans without predicates materialize an unnecessary row-ID allowlist. Pin upstream lance-c with the Lance development-branch optimization and metric-based regressions. Remove superseded local patches, retaining only the Foyer cache integration with object-store compatibility and bounded response-metadata reuse. ### Release note Avoid redundant prefilter row-ID construction for unfiltered vector queries covering complete visible index segments. ### Check List (For Author) - Test: Lance prefilter and segment-contract tests; lance-c and Foyer unit/C API suites; Rust format and Clippy; C/C++ executable tests; dependency downloader patch lifecycle, checksum, and fallback regressions. - Behavior changed: Yes, eligible vector scans skip redundant prefilter work; predicates, partial coverage, deletions, and flat fallback retain semantics. - Does this need documentation: No; no SQL or configuration interface change.
1 parent 4750b1c commit 29f3df9

12 files changed

Lines changed: 362 additions & 10102 deletions

‎thirdparty/download-thirdparty.sh‎

Lines changed: 5 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -718,30 +718,14 @@ if [[ " ${TP_ARCHIVES[*]} " =~ " AZURE " ]]; then
718718
echo "Finished patching ${AZURE_SOURCE}"
719719
fi
720720

721-
# Apply Doris lance-c patches as one chain to the pinned release archive.
721+
# Foyer remains a local patch until its cache interface is accepted upstream.
722+
# All search fixes are supplied by the immutable lance-c dependency revision.
722723
if [[ " ${TP_ARCHIVES[*]} " =~ " LANCE_C " ]]; then
723724
cd "${TP_SOURCE_DIR}/${LANCE_C_SOURCE}"
724-
LANCE_C_PATCHED_MARK="${PATCHED_MARK}_community_pr83_prefilter"
725-
# Older source caches carry a different PR #73 and cannot accept this chain incrementally.
726-
if [[ -f "${PATCHED_MARK}" && ! -f "${LANCE_C_PATCHED_MARK}" ]]; then
727-
echo "The lance-c patch chain changed; remove ${TP_SOURCE_DIR}/${LANCE_C_SOURCE} and rebuild."
728-
exit 1
729-
fi
730-
if [[ ! -f "${LANCE_C_PATCHED_MARK}" ]]; then
731-
# PR #77 provides Lance v11 for the following community patches. PR #83
732-
# retains PR #79's scalar-segment path when adding multi-vector execution.
733-
# The final patch pins the full-snapshot prefilter fix and its execution metrics.
734-
for lance_patch in pr-74 pr-75-pr-78 pr-77 pr-73 pr-79 pr-80 pr-83 prefilter; do
735-
patch --batch --forward --reject-file=- --fuzz=0 --no-backup-if-mismatch -s \
736-
-p1 <"${TP_PATCH_DIR}/${LANCE_C_SOURCE}-${lance_patch}.patch"
737-
done
738-
touch "${PATCHED_MARK}" "${LANCE_C_PATCHED_MARK}"
739-
fi
740-
# Cached sources may carry the earlier prefilter pin; upgrade FTS metrics independently.
741-
if [[ ! -f "${PATCHED_MARK}_prefilter_fts" ]]; then
725+
if [[ ! -f "${PATCHED_MARK}_foyer" ]]; then
742726
patch --batch --forward --reject-file=- --fuzz=0 --no-backup-if-mismatch -s \
743-
-p1 <"${TP_PATCH_DIR}/${LANCE_C_SOURCE}-prefilter-fts.patch"
744-
touch "${PATCHED_MARK}_prefilter_fts"
727+
-p1 <"${TP_PATCH_DIR}/lance-c-foyer.patch"
728+
touch "${PATCHED_MARK}_foyer"
745729
fi
746730
cd -
747731
echo "Finished patching ${LANCE_C_SOURCE}"

0 commit comments

Comments
 (0)