[docs](lance) Explain approximation modes and ACORN selection - #4554

### What problem does this PR solve?

`vector_search` cannot currently select Lance's approximation policy.
This PR adds `"approx_mode"="fast|normal|accurate"`, validates the value
before metadata access, transports it as an optional typed Thrift field,
and forwards it to `lance_scanner_set_approx_mode` on each BE scanner.
`EXPLAIN` shows an explicitly requested mode.

Omitting the property preserves the existing native default (`normal` in
the pinned Lance revision). The setting applies through the shared
single-vector and multi-vector scanner setup. Existing refinement, probe
counts, search width, concurrency, split coverage, and distributed
merging remain unchanged. The existing lance-c dependency already
provides this API, so no third-party update or local patch is needed.

Website documentation is provided in
[apache/doris-website](apache/doris-website#4554),
updating the English and Chinese 4.x Lance Catalog pages with mode
semantics, selection guidance, ACORN activation and fallback rules, SQL
usage, and compatibility notes. The temporary guide in the Doris
repository has been removed. The local experiment remains recorded
below. In particular, `accurate` still selects approximate candidates
and does not guarantee exact Top-K or original-vector distances on a
quantized index. During a smooth upgrade, FE rejects explicit `fast` or
`accurate` at scan initialization if any backend in the query's
candidate pool is marked as an old smooth-upgrade source. Older BEs
would ignore the optional field and silently use NORMAL, so rejecting
prevents a global TopN from merging split candidates generated under
different policies. The error names the backend and requested mode and
advises completing the upgrade or using `normal`. Omission and explicit
`normal` remain compatible. This checks only the selected backend
policy, not unrelated backends, and does not detect arbitrary older
builds without the smooth-upgrade-source marker.

### Mode semantics and recommended selection

Use `normal` as the baseline; choose another mode only after measuring
representative queries against an exact reference. These are
recommendations based on the pinned implementation and the local
experiment, not fixed speed/recall guarantees.

| Search path | `fast` | `normal` | `accurate` |
|---|---|---|---|
| RabitQ raw-query distance estimation | Uses one-bit scores even for a
multi-bit index, skipping extension-bit computation and its lower-bound
gate. | Uses the standard 8-bit quantized distance lookup table and
extension-bit computation when available. | Uses a higher-precision
16-bit distance lookup table and extension-bit computation when
available. |
| HNSW with prefiltering | Enables ACORN traversal when the runtime
conditions below permit it. | Uses regular filtered graph traversal. |
Uses the same traversal policy as `normal`; quantized distance
computation can still differ. |
| IVF_FLAT or exhaustive Flat Search | No mode-specific quantization
change. | Same scoring policy. | Same scoring policy. |

The 8-bit and 16-bit values describe the **query-time distance lookup
table**, not the number of bits stored per dimension in the index. For
example, on a 5-bit RabitQ index, `fast` uses one-bit scores on the
raw-query path; the other modes can use the additional four bits. Other
RabitQ paths, including residual-query scoring, can behave differently.
A mode description does not expand the supported vector-type or metric
combinations.

`accurate` is still approximate search. It does not guarantee exact
Top-K, exhaustive candidate evaluation, or original-vector `_distance`
on a quantized index. `refine_factor` controls reranking with original
vectors; increasing it can improve scores and recall but cannot recover
every candidate missed earlier. Multi-vector queries retain their
mandatory original-vector refinement.

| Workload | Starting choice | What to verify |
|---|---|---|
| General workloads or an unknown bottleneck | `normal` | Establish
recall and latency before changing the policy. |
| HNSW_FLAT with an effective prefilter, where latency matters | Compare
`fast` with `normal` | ACORN can reduce wasted distance calculations.
Check recall, result count, median latency, and p95; a speedup is not
guaranteed. |
| Multi-bit RabitQ where recall matters | `normal` | `fast` can discard
useful encoding information and substantially reduce recall. Use it only
if representative queries still meet the quality target. |
| RabitQ without original-vector reranking, where approximate score
error matters | Compare `accurate` with `normal` | Measure score error
and ranking quality against original-vector distances, as well as extra
latency. Better estimation does not guarantee higher recall. |
| RabitQ already using original-vector reranking | `normal` | The final
score benefit of `accurate` may be smaller; candidate selection can
still differ, so validate recall separately. |
| IVF_FLAT or HNSW_FLAT without an effective prefilter | Keep `normal` |
The current implementation offers no corresponding quantization or ACORN
benefit from changing the mode. IVF_FLAT recall still depends on the
probed partitions. |

For a compatible **HNSW index backed by RabitQ**, `fast` can both enable
ACORN traversal and simplify RabitQ distance estimation on the
applicable raw-query path. These effects are coupled by one option. Do
not infer its recall from an HNSW_FLAT test; validate the combined index
independently. `normal` and `accurate` have the same HNSW traversal
policy, but their RabitQ scoring can differ.

Recommended tuning order:

1. Pin the same dataset snapshot, query set, metric, filters, and index
coverage. Use `use_index=false` to obtain exact Top-K ground truth over
the same eligible rows; put eligibility conditions in the TVF `filter`
with `prefilter=true`.
2. Establish a `normal` baseline. Measure Recall@K, result count, score
error where relevant, and median/p95 latency.
3. If recall is insufficient, tune IVF `nprobes` and HNSW `ef`. If
original-vector scores or more reranked candidates are required,
configure `refine_factor`, accounting for the extra vector reads and
computation.
4. With the other settings fixed, compare `fast` for filtered HNSW or
`accurate` for RabitQ estimation precision. Select a mode only if it
meets the quality target and provides a worthwhile measured benefit.
Measure cold and warm cache behavior separately.

### ACORN activation and fallback

The selected compatible index must contain an HNSW graph. Request
`approx_mode="fast"` and a TVF `filter` with `prefilter="true"`; an
outer Doris `WHERE` does not supply this prefilter. `use_index=true`
alone does not select or force HNSW. The website documentation includes
a complete SQL example.

The current native implementation dispatches separately inside each HNSW
subindex (an IVF partition), using the number of nodes passing the
prefilter:

| Runtime condition | Path |
|---|---|
| No effective prefilter, or all nodes pass | Ordinary HNSW traversal. |
| Fewer than roughly 10% of nodes pass | Flat scoring over the remaining
nodes. |
| A partial filter remains and the sparse-filter rule does not apply |
`fast` uses ACORN; `normal` and `accurate` use regular filtered graph
traversal. |
| ACORN returns too few required candidates and the query has no
distance bounds | Retry with regular filtered graph traversal. |

The threshold applies within each subindex, not to the whole table.
There is no separate force-ACORN property in this interface. In Doris,
distance-range safety rules can also select Flat Search before native
graph dispatch.

`EXPLAIN` showing `lanceApproxMode=FAST` confirms the requested policy,
not that every subindex actually ran ACORN. HNSW_RabitQ behavior is
inferred from the coupled traversal/scoring implementation; the local
HNSW measurements below used HNSW_FLAT and do not validate the combined
index.

### Local validation of behavior and effect

These are **native lance-c scanner measurements, not end-to-end Doris
SQL timings**. The library was freshly built from pinned lance-c
`a51ab47a9c814a746cb1714c99fad93dd3a188b8`, using Lance
`68c12dfd7efe02f90ad2d7f3a239a7eb64884e57`. The unoptimized development
build and Python/Arrow result conversion make the timings illustrative,
not production performance claims.

Procedure:

1. Generate 16,384 Gaussian Float32 vectors of dimension 128, then 128
independent query vectors, using xorshift64/Box-Muller with seed
`20261029`. Assign `row_id=0..16383` and `category=row_id%10`.
2. Build fully covering IVF_RQ (32 partitions, 5 bits), IVF_FLAT (32
partitions), and IVF_HNSW_FLAT (32 partitions, 16 edges, construction
ef=100) indexes. Select each index explicitly by its segment UUIDs.
3. Search squared L2 with k=10, nprobes=32, and query_parallelism=1.
Repeat IVF_RQ with refine_factor=4. For HNSW, set ef=64 and prefilter
`category < 5` (about 50% of rows).
4. Compute exact Top-10 independently with NumPy. Warm up each mode with
eight queries; run three rounds of 128 queries per mode, rotating mode
order per query and round. The table below reports mean Recall@10,
median latency, and p95 latency, from 384 searches per mode.
5. Compare omitted mode with explicit normal over all 128 queries per
case, checking ordered IDs and Float32 distances. Independently
recompute original-vector distances for returned IVF_RQ rows to measure
score error.

| Index / refinement | Mode | Recall@10 | Median (ms) | p95 (ms) |
| --- | --- | ---: | ---: | ---: |
| IVF_RQ / none | fast | 26.80% | 14.57 | 16.03 |
| IVF_RQ / none | normal | 91.72% | 15.90 | 18.12 |
| IVF_RQ / none | accurate | 91.72% | 20.52 | 22.91 |
| IVF_RQ / 4 | fast | 54.30% | 20.50 | 22.07 |
| IVF_RQ / 4 | normal | 99.92% | 23.44 | 25.64 |
| IVF_RQ / 4 | accurate | 99.92% | 28.19 | 30.58 |
| IVF_FLAT / none | fast | 100% | 25.71 | 29.62 |
| IVF_FLAT / none | normal | 100% | 25.78 | 29.82 |
| IVF_FLAT / none | accurate | 100% | 25.71 | 30.74 |
| IVF_HNSW_FLAT / 50% prefilter | fast | 100% | 40.45 | 44.04 |
| IVF_HNSW_FLAT / 50% prefilter | normal | 100% | 44.51 | 49.86 |
| IVF_HNSW_FLAT / 50% prefilter | accurate | 100% | 44.48 | 50.05 |

All searches returned ten rows. Omitted mode matched explicit normal
exactly in every case. IVF_FLAT and filtered HNSW returned identical
ordered IDs and distances across modes. The data seed fixes vectors; ANN
index training/layout can vary across rebuilds.

Without refinement, normal and accurate had equal mean recall but
different ordered IDs on 89/128 queries. Mean absolute squared-L2 score
error was 30.7325 for fast, 0.7986 for normal, and 0.7783 for accurate.
Thus accurate slightly improved distance estimation on this fixture
without improving mean recall. With refinement factor 4, normal and
accurate returned identical results, while fast still missed many
neighbors. The mode is not interchangeable with refinement, and its
effect depends on the index/path and data.

### Release note

Add the optional `approx_mode` property to Lance `vector_search`,
accepting `fast`, `normal`, and `accurate`, with the existing default
preserved when omitted.

### Check List (For Author)

- Test
- [x] Unit tests: focused source compilation and JUnit execution of
`VectorSearchTableValuedFunctionTest` and `LanceScanNodeTest`: **59
tests passed**. Coverage includes accepted/invalid modes, typed Thrift
round trips, default omission, EXPLAIN, and smooth-upgrade scan
initialization.
- [x] Review-fix regression: the new real scan-initialization test
failed on the previous code (the expected upgrade rejection was
missing), then passed with the gate. Coverage includes FAST/ACCURATE
with old-only and mixed candidate pools, single/multi-vector requests,
all modes on current BEs, omitted/explicit NORMAL on mixed BEs, and
unaffected FTS initialization.
- [x] Manual test: native mode comparison described above, plus
independent distance-error checks.
- [x] FE `mvn -B clean checkstyle:check` passed; clang-format 16 checks
passed for both affected C++ files; generated Thrift sources rebuilt
successfully.
- [x] Fresh BE reader and test source objects compiled successfully
against the new headers and pinned C API.
- [ ] Full BE unit-test execution: not run locally because the available
prebuilt BE libraries have an incompatible ABI. New request-validation
coverage is included for CI.
- [ ] SQL regression execution: added mode pass-through, EXPLAIN, and
invalid-value coverage to `test_lance_vector_search`; the full suite
parsed successfully with Groovy 4.0.19, but execution is pending CI.

- Behavior changed:
- [x] Yes. Explicit `approx_mode` now selects the native policy;
omission preserves existing behavior.

- Does this need documentation?
- [x] Yes. English and Chinese 4.x documentation: [website
PR](apache/doris-website#4554).

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label