fix: isolate Foyer cache origins and avoid warm size writes - #3

…68613)

Unfiltered ANN searches scoped to complete index segments can spend time
materializing redundant row-ID allowlists. Update lance-c to an
immutable upstream revision containing the merged Lance
development-branch fix lance-format/lance#9599 and ANN diagnostics from
lance-format/lance#9602, through lance-format/lance-c#89.

Keep all master changes in `thirdparty/`. Remove the superseded local
lance-c patch chain and retain the Foyer cache integration while its
interface remains outside upstream main. The upstream revision also
carries the merged lance-format/lance#9537 PQ scoring change and ANN
stage timing diagnostics; no Doris FE or BE code changes are included
here.

The prefilter fast path requires complete visible coverage of every
selected segment and no predicate. Partial coverage and predicates
retain row-ID prefiltering, while deletions and unindexed fallback
remain effective.

Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged
Lance #9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This
includes parallel-path partition-preparation timing. The upstream
callback regression checks timer presence and nanosecond units while
accepting valid zero durations.

The Foyer wrapper caches immutable ObjectMeta and Attributes
independently of HTTP response extensions, avoiding repeated HEAD
requests on warm range reads. It does not replay transport extensions
from cached metadata. The HTTP regression checks HEAD/GET counts,
returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD
bypass.

Generate the retained Foyer patch from source commit
`24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned
lance-c baseline. The header records both revisions and the regeneration
command. The source is submitted as
zhangstar333/lance-c#3 against the branch behind
lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris
consumes the exact reviewed source revision while that supplementary PR
awaits merge.

Scope metadata, size, and block keys to a random namespace for each live
underlying object-store instance. The wrapping API omits a complete
stable backend identity; identical bucket/path names alone cannot
distinguish S3-compatible endpoints. Weak identity records preserve
sharing for the same live store without retaining it, while new stores
and process restarts start cold. A fresh Dataset open that creates a new
store consequently needs to warm its own cache. This deliberately
reduces reuse across instances to prevent returning another origin's
data; it does not claim unchanged cache-hit rates or production latency.

Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid
cache: look up both tiers first and populate only absent or invalid size
records. Regression tests cover real HTTP endpoints sharing
bucket/path/ETag, batched data and NotFound isolation, replaced origins
after disk recovery, weak-reference lifetime, and actual disk-write
bytes. Five warm range reads wrote 40 KB before the fix and zero bytes
after it in the regression.

Fingerprint the patch in the downloader so existing source trees refresh
after updates, including legacy empty markers. Identical patches reuse
cached sources. Check platform definitions under nounset with simulated
Darwin x86_64/arm64, and make the optional ADBC source guard safe when
unset on master. Downloader lifecycle and platform handling remain
downstream in Doris.

Validation: 461 regular Rust tests passed on the final source; Rust
formatting and diff checks passed. GNU patch and git apply checks
passed, and all 87 tracked source files match the recorded source
commit. Downloader tests passed on master, branch-4.1, and the
downstream hotfix branch for fresh extraction, idempotence,
re-extraction, generic markers, legacy/mismatched Foyer markers, and
patch failure. Shell syntax and simulated macOS initialization passed.
All three opt-in native consumer tests passed: C calls, C++ calls, and
static OSS HTTP transport. Full Doris builds and BE integration
execution remain in PR CI.

…nch-4.1) (#68615)

Unfiltered ANN searches scoped to complete index segments can
materialize redundant row-ID allowlists, and the existing scan profile
leaves the remaining search work unexplained. Update the upstream
dependency and expose ANN stage timings on `branch-4.1`.

This includes the dependency integration from #68613: adopt the merged
lance-format/lance#9599 and lance-format/lance#9602 through
lance-format/lance-c#89, retain the merged lance-format/lance#9537 PQ
scoring fix, remove superseded local lance-c patches, and retain only
the Foyer cache integration.

Add BE profile counters for index opening, partition
loading/preparation, prefilter readiness, CPU queue wait, partition
search, query lookup-table preparation, fused distance/TopK work, and
result materialization. Export operator baselines for ANN, sort/merge,
take, and vector-distance work. Extend the existing indexed multivector
regression to check that stage timers reach the Doris profile, alongside
existing result and prefilter assertions.

Timings accumulate across concurrent work and overlap parent stages.
Distance/TopK includes candidate filtering in fused paths; these
counters must not be summed to reconstruct wall time.
`docs/lance-ann-profile.md` documents the boundaries, supported paths,
and relationship to the existing scanner and prefilter timers. The BE
changes add observability; the prefilter optimization itself remains in
the upstream dependency.

Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged
Lance #9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This
includes parallel-path partition-preparation timing. The upstream
callback regression checks timer presence and nanosecond units while
accepting valid zero durations.

The Foyer wrapper caches immutable ObjectMeta and Attributes
independently of HTTP response extensions, avoiding repeated HEAD
requests on warm range reads. It does not replay transport extensions
from cached metadata. The HTTP regression checks HEAD/GET counts,
returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD
bypass.

Generate the retained Foyer patch from source commit
`24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned
lance-c baseline. The header records both revisions and the regeneration
command. The source is submitted as
zhangstar333/lance-c#3 against the branch behind
lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris
consumes the exact reviewed source revision while that supplementary PR
awaits merge.

Scope metadata, size, and block keys to a random namespace for each live
underlying object-store instance. The wrapping API omits a complete
stable backend identity; identical bucket/path names alone cannot
distinguish S3-compatible endpoints. Weak identity records preserve
sharing for the same live store without retaining it, while new stores
and process restarts start cold. A fresh Dataset open that creates a new
store consequently needs to warm its own cache. This deliberately
reduces reuse across instances to prevent returning another origin's
data; it does not claim unchanged cache-hit rates or production latency.

Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid
cache: look up both tiers first and populate only absent or invalid size
records. Regression tests cover real HTTP endpoints sharing
bucket/path/ETag, batched data and NotFound isolation, replaced origins
after disk recovery, weak-reference lifetime, and actual disk-write
bytes. Five warm range reads wrote 40 KB before the fix and zero bytes
after it in the regression.

Fingerprint the patch in the downloader so existing source trees refresh
after updates, including legacy empty markers. Identical patches reuse
cached sources. Check platform definitions under nounset with simulated
Darwin x86_64/arm64, and make the optional ADBC source guard safe when
unset on master. Downloader lifecycle and platform handling remain
downstream in Doris.

Validation: 461 regular Rust tests passed on the final source; Rust
formatting and diff checks passed. GNU patch and git apply checks
passed, and all 87 tracked source files match the recorded source
commit. Downloader tests passed on master, branch-4.1, and the
downstream hotfix branch for fresh extraction, idempotence,
re-extraction, generic markers, legacy/mismatched Foyer markers, and
patch failure. Shell syntax and simulated macOS initialization passed.
All three opt-in native consumer tests passed: C calls, C++ calls, and
static OSS HTTP transport. Full Doris builds and BE integration
execution remain in PR CI.

…nch-4.1) (#68615)

Unfiltered ANN searches scoped to complete index segments can
materialize redundant row-ID allowlists, and the existing scan profile
leaves the remaining search work unexplained. Update the upstream
dependency and expose ANN stage timings on `branch-4.1`.

This includes the dependency integration from #68613: adopt the merged
lance-format/lance#9599 and lance-format/lance#9602 through
lance-format/lance-c#89, retain the merged lance-format/lance#9537 PQ
scoring fix, remove superseded local lance-c patches, and retain only
the Foyer cache integration.

Add BE profile counters for index opening, partition
loading/preparation, prefilter readiness, CPU queue wait, partition
search, query lookup-table preparation, fused distance/TopK work, and
result materialization. Export operator baselines for ANN, sort/merge,
take, and vector-distance work. Extend the existing indexed multivector
regression to check that stage timers reach the Doris profile, alongside
existing result and prefilter assertions.

Timings accumulate across concurrent work and overlap parent stages.
Distance/TopK includes candidate filtering in fused paths; these
counters must not be summed to reconstruct wall time.
`docs/lance-ann-profile.md` documents the boundaries, supported paths,
and relationship to the existing scanner and prefilter timers. The BE
changes add observability; the prefilter optimization itself remains in
the upstream dependency.

Pin lance-c `9bd730add2ac70316c1d642b8459011e2dd92022`, using the merged
Lance #9602 revision `68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. This
includes parallel-path partition-preparation timing. The upstream
callback regression checks timer presence and nanosecond units while
accepting valid zero durations.

The Foyer wrapper caches immutable ObjectMeta and Attributes
independently of HTTP response extensions, avoiding repeated HEAD
requests on warm range reads. It does not replay transport extensions
from cached metadata. The HTTP regression checks HEAD/GET counts,
returned bytes/ranges/metadata, fresh dataset scopes, and explicit HEAD
bypass.

Generate the retained Foyer patch from source commit
`24c7ca4bcb9422c113b0d3e07e4efe1173b0bc9f` relative to the pinned
lance-c baseline. The header records both revisions and the regeneration
command. The source is submitted as
zhangstar333/lance-c#3 against the branch behind
lance-format/lance-c#73, on top of merged source-alignment PR #2. Doris
consumes the exact reviewed source revision while that supplementary PR
awaits merge.

Scope metadata, size, and block keys to a random namespace for each live
underlying object-store instance. The wrapping API omits a complete
stable backend identity; identical bucket/path names alone cannot
distinguish S3-compatible endpoints. Weak identity records preserve
sharing for the same live store without retaining it, while new stores
and process restarts start cold. A fresh Dataset open that creates a new
store consequently needs to warm its own cache. This deliberately
reduces reuse across instances to prevent returning another origin's
data; it does not claim unchanged cache-hit rates or production latency.

Avoid inserting an unchanged size entry into the WriteOnInsertion hybrid
cache: look up both tiers first and populate only absent or invalid size
records. Regression tests cover real HTTP endpoints sharing
bucket/path/ETag, batched data and NotFound isolation, replaced origins
after disk recovery, weak-reference lifetime, and actual disk-write
bytes. Five warm range reads wrote 40 KB before the fix and zero bytes
after it in the regression.

Fingerprint the patch in the downloader so existing source trees refresh
after updates, including legacy empty markers. Identical patches reuse
cached sources. Check platform definitions under nounset with simulated
Darwin x86_64/arm64, and make the optional ADBC source guard safe when
unset on master. Downloader lifecycle and platform handling remain
downstream in Doris.

Validation: 461 regular Rust tests passed on the final source; Rust
formatting and diff checks passed. GNU patch and git apply checks
passed, and all 87 tracked source files match the recorded source
commit. Downloader tests passed on master, branch-4.1, and the
downstream hotfix branch for fresh extraction, idempotence,
re-extraction, generic markers, legacy/mismatched Foyer markers, and
patch failure. Shell syntax and simulated macOS initialization passed.
All three opt-in native consumer tests passed: C calls, C++ calls, and
static OSS HTTP transport. Full Doris builds and BE integration
execution remain in PR CI.