feat: add opt-in runtime BTree segment selection - #98

### What problem does this PR solve?

Lance profiles do not expose Session cache-miss observation windows, scalar-index task grouping cannot compare runtime costs, and Foyer memory capacity is tied to the data-cache read block size.

This PR targets **branch-4.2** and adds:

- `LanceSessionIndexCacheMisses` and `LanceSessionMetadataCacheMisses`, with availability and scope information. They report BE-wide Session deltas observed during a reader's dataset-open windows, including opening. Concurrent readers can overlap; these are not query-exclusive counts and must not be summed as an exact query total.
- The startup BE option `enable_lance_scalar_index_auto_selection` (default `true`). FE keeps coverage-based task grouping and does not estimate selectivity or add a session variable. In BE, lance-c compares current BTree page min/max and null-count metadata for segments covering the entire task. Estimated work is matching pages times rows per page; a broad estimate can select a scoped full scan. Serialized lookup bytes, decoded buffers and probe work are bounded. Unknown estimates retain the existing segment path. The current native API cannot bound variable-width expansion, so string/binary costs remain unknown: retain such a grouping index or skip such an alternative before opening its lookup. Bitmap/LabelList remain supported on their existing path and are not cost-ranked; vector/FTS semantics are unchanged.
- `lance_data_cache_memory_capacity_bytes`. Default `0` preserves the previous one-configured-block memory tier, including nondefault block sizes. Explicit capacities configure Foyer independently of block and disk capacities; negative values are rejected before unsigned conversion.

The native selector and decoded-memory fix are now merged upstream in [lance-format/lance-c#98](lance-format/lance-c#98). This PR pins upstream main commit `a51ab47a9c814a746cb1714c99fad93dd3a188b8`, refreshes the source archive checksum, and removes the local runtime-selection patch together with its application/checksum handling. The pre-existing Foyer patch is unchanged and applies with zero fuzz.

Runtime decisions are exposed through Profile counters. Usage, cost-model limits and cache-counter scope are documented in `docs/lance-cache-and-index-selection.md`.

### Release note

Add Lance Session cache-miss diagnostics, bounded BE runtime BTree selection controlled by a BE configuration, and independently configurable Foyer memory capacity.

### Validation

- On the newly downloaded upstream archive plus the unchanged Foyer patch, `cargo test --offline --locked` passed all 515 tests, including 82 unit tests and 385 C API tests. The normal command ignores three C/C++ executable compilation tests. Coverage includes the compressed long-string allocation regression, decoded-budget limits, competing indices, scan fallback, fragment scope, NULL/deletion/stable-ID behavior and LIMIT/OFFSET.
- C and C++ API sources passed syntax-only compilation against the newly patched headers.
- Verified archive MD5, clean zero-fuzz patch application, cached no-op, stale-checksum source replacement, and equality of the resulting tested source. Selector/scanner/segment sources match merged upstream exactly.
- All nine affected C++ files passed the repository formatting runner with clang-format 16.0.6. Shell syntax and staged whitespace checks passed.
- Existing FE validation passed 74 focused tests and FE Checkstyle; this dependency synchronization changes no FE code.
- Full Doris BE build and unit-suite validation remain for CI and have not been verified locally.

### Check List (For Author)

- Test
    - [x] Unit Test
    - [x] Manual test: native source/patch lifecycle and header compatibility checks above.
    - [ ] Regression test
- Behavior changed:
    - [x] Yes. BE runtime BTree selection is enabled by default and can be disabled. Foyer's default memory behavior is preserved.
- Does this need documentation?
    - [x] Yes. Usage and native memory limits are documented.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

)

### What problem does this PR solve?

Related PR: lance-format/lance-c#98

Problem Summary: Update lance-c from
`4fafcd64525b274b113208390bbcca86bbbc093b` to the merged upstream main
commit `a51ab47a9c814a746cb1714c99fad93dd3a188b8`, and refresh the
source archive name and MD5. This brings in the opt-in runtime BTree
segment selector and its pre-decode memory guards. Existing master
callers retain their selection behavior because the new native option
defaults to false. The existing Foyer patch applies without changes and
with zero fuzz.

### Release note

None. This updates the dependency; it does not enable the new opt-in
native feature in Doris.

### Validation

- Verified the downloaded immutable source archive's MD5 and applied the
existing Foyer patch with zero fuzz. The same patch is used on master
and branch-4.2.
- `cargo test --offline --locked` on that patched source: 515 tests
passed, including 82 unit tests and 385 C API tests. The normal command
ignores three C/C++ executable compilation tests.
- C and C++ API test sources passed syntax-only compilation against the
new patched headers.
- Exercised the existing source-patching block for first application, a
cached no-op, and stale-checksum re-extraction; verified resulting
source equality.
- Shell syntax and staged whitespace checks passed. Full Doris BE
build/integration validation remains for CI.

### Check List (For Author)

- Test
    - [x] Unit Test
- [x] Manual test: source checksum/patch lifecycle and C/C++ header
compatibility checks above.
    - [ ] Regression test
- Behavior changed:
- [x] No. Existing Doris callers do not enable the new native option.
- Does this need documentation?
    - [x] No. No Doris configuration or user interface changes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…tion (#69625)

### What problem does this PR solve?

Lance profiles do not expose Session cache-miss observation windows,
scalar-index task grouping cannot compare runtime costs, and Foyer
memory capacity is tied to the data-cache read block size.

This PR targets **branch-4.2** and adds:

- `LanceSessionIndexCacheMisses` and `LanceSessionMetadataCacheMisses`,
with availability and scope information. They report BE-wide Session
deltas observed during a reader's dataset-open windows, including
opening. Concurrent readers can overlap; these are not query-exclusive
counts and must not be summed as an exact query total.
- The startup BE option `enable_lance_scalar_index_auto_selection`
(default `true`). FE keeps coverage-based task grouping and does not
estimate selectivity or add a session variable. In BE, lance-c compares
current BTree page min/max and null-count metadata for segments covering
the entire task. Estimated work is matching pages times rows per page; a
broad estimate can select a scoped full scan. Serialized lookup bytes,
decoded buffers and probe work are bounded. Unknown estimates retain the
existing segment path. The current native API cannot bound
variable-width expansion, so string/binary costs remain unknown: retain
such a grouping index or skip such an alternative before opening its
lookup. Bitmap/LabelList remain supported on their existing path and are
not cost-ranked; vector/FTS semantics are unchanged.
- `lance_data_cache_memory_capacity_bytes`. Default `0` preserves the
previous one-configured-block memory tier, including nondefault block
sizes. Explicit capacities configure Foyer independently of block and
disk capacities; negative values are rejected before unsigned
conversion.

The native selector and decoded-memory fix are now merged upstream in
[lance-format/lance-c#98](lance-format/lance-c#98).
This PR pins upstream main commit
`a51ab47a9c814a746cb1714c99fad93dd3a188b8`, refreshes the source archive
checksum, and removes the local runtime-selection patch together with
its application/checksum handling. The pre-existing Foyer patch is
unchanged and applies with zero fuzz.

Runtime decisions are exposed through Profile counters. Usage,
cost-model limits and cache-counter scope are documented in
`docs/lance-cache-and-index-selection.md`.

### Release note

Add Lance Session cache-miss diagnostics, bounded BE runtime BTree
selection controlled by a BE configuration, and independently
configurable Foyer memory capacity.

### Validation

- On the newly downloaded upstream archive plus the unchanged Foyer
patch, `cargo test --offline --locked` passed all 515 tests, including
82 unit tests and 385 C API tests. The normal command ignores three
C/C++ executable compilation tests. Coverage includes the compressed
long-string allocation regression, decoded-budget limits, competing
indices, scan fallback, fragment scope, NULL/deletion/stable-ID behavior
and LIMIT/OFFSET.
- C and C++ API sources passed syntax-only compilation against the newly
patched headers.
- Verified archive MD5, clean zero-fuzz patch application, cached no-op,
stale-checksum source replacement, and equality of the resulting tested
source. Selector/scanner/segment sources match merged upstream exactly.
- All nine affected C++ files passed the repository formatting runner
with clang-format 16.0.6. Shell syntax and staged whitespace checks
passed.
- Existing FE validation passed 74 focused tests and FE Checkstyle; this
dependency synchronization changes no FE code.
- Updated the array-predicate regression profile checks to account for
explicit runtime cost-selected scans while retaining request/search/scan
accounting, exact candidate counts and strict LabelList coverage. The
actual assertion block passes six valid profile paths and rejects five
invalid profiles; the previous assertion reproduces the observed
failure. The focused native runtime-selection test and full-suite Groovy
4.0.19 parsing pass. Full external regression execution remains for CI.
- Fixed Session counter publication for early LIMIT: active dataset-open
windows are sampled before Thrift/protobuf profile copies, with
synchronized delta accounting shared with close. All 14 focused C++
tests pass with ASAN/UBSAN and leak detection enabled, including a real
Lance native read held in its statistics callback while another worker
completes LIMIT 1. Disabling the snapshot hook reproduces both
missing-window regressions (native saved misses 9 versus at least 19).
The full reader and test sources pass syntax checks, and all 15 affected
PR C++ files pass clang-format 16.0.6.
- Full Doris BE build and unit-suite validation remain for CI and have
not been verified locally.

### Check List (For Author)

- Test
    - [x] Unit Test
- [x] Manual test: native source/patch lifecycle and header
compatibility checks above.
    - [ ] Regression test
- Behavior changed:
- [x] Yes. BE runtime BTree selection is enabled by default and can be
disabled. Foyer's default memory behavior is preserved.
- Does this need documentation?
    - [x] Yes. Usage and native memory limits are documented.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label