File scans still maintain the legacy scanner, its readers and a second capability/routing matrix. Route all file scans, file loads and external row-ID fetches through `FileScannerV2`, and remove the unused V1 implementation and tests.

- Keep `enable_file_scanner_v2` as a deprecated compatibility variable. Its stored value no longer affects planning or scanning; FE sends `true` in the existing Thrift field for older backends.
- Remove scanner-level capability checks. Factories and concrete readers return errors for unsupported operations, including native Hudi delta-log reads, query-only connector loads and Remote Doris Variant reads.
- Move standalone Iceberg position-delete reads to V2 `TableReader`, and historical Push ingestion to `LoadTableReader`. Preserve Push conversion expressions across ranges and compatibility with requests that omit the file-column count.
- Retain shared writer, metadata, JDBC/JNI and Elasticsearch HTTP components that surviving readers still use. Remove V1-only schema, partition, count and decoder infrastructure.
- Update connector planning, nullable COUNT pushdown, documentation and compatibility regression expectations.

Related PR: apache#69668

File scans always use FileScannerV2. `enable_file_scanner_v2` is deprecated and has no effect. Unsupported reader operations return an error instead of selecting the legacy scanner.

- Test
    - [x] Unit Test
    - [x] Regression test
    - [x] Manual test

Validation:
- Production BE and FE build, including connector plugin packages.
- 523 focused FE tests and FE Checkstyle.
- 593 distinct BE tests; the Remote Doris Flight suite was rerun successfully with localhost excluded from the development proxy.
- clang-format 16, clang-tidy on changed lines, build hygiene and its full self-test suite.
- Final Push fixture rerun passed after the test-data helper cleanup.
- Six standalone regression suites passed: Arrow Stream Load, HTTP CSV/JSON/compression/header formats and schema inference.
- Manual SQL verified that setting the old option to false remains accepted and SHOW VARIABLES marks it deprecated.

External Paimon regression expectations are updated for the no-op option and will be exercised by the external CI environment.

The exhaustive CSV/JSON split tests retain their existing boundary assertions after dropping the V1 parameterization. Four narrow cognitive-complexity suppressions preserve these existing GTest-expanded cross-product tests; new tests are decomposed instead.

- Behavior changed:
    - [x] Yes. The scanner option is now a deprecated no-op; unsupported operations fail in their reader.
- Does this need documentation?
    - [x] Yes. The scanner design and review guide are updated in this PR.

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

Apply the outstanding changes in master merge order:
apache#69545 7d16729
apache#69551 7224a2d
apache#69612 61d1aef
apache#69615 8d2bd1e
apache#69650 3dc7341
apache#69651 c00ecb1
apache#69646 e493c61
apache#69668 7865646

apache#69614 already exists on branch-4.2 through apache#69644.
Adapt datasource APIs and tests to the release branch, retaining its
existing scanner dispatch and statement-bound metadata contracts.

### What problem does this PR solve?

Backport the requested external-data and FileScannerV2 changes to
`branch-4.2` in their actual master merge order:

| Order | Master PR | Change | Backport |
| --- | --- | --- | --- |
| 1 | #69545 | Preserve Paimon schema options after field renames |
Included |
| 2 | #69551 | Expand Iceberg capability, transaction and snapshot tests
| Included with 4.2 API adaptations |
| 3 | #69614 | Elasticsearch HTTP scans in FileScannerV2 | Already
present through #69644 (`2104b7509cda`); not applied twice |
| 4 | #69612 | Check catalog visibility before SHOW DATABASES accesses
metadata | Included |
| 5 | #69615 | Publish Iceberg Parquet split offsets | Included |
| 6 | #69650 | Preserve ES compound residuals and expand scanner
coverage | Included |
| 7 | #69651 | Accept Elasticsearch 9.x during node discovery | Included
|
| 8 | #69646 | Prioritize Parquet V2 row groups using TopN bounds |
Included |
| 9 | #69668 | Arrow loads and schema inference in FileScannerV2 |
Included |

The eight outstanding changes are combined in one commit. This branch
retains the 4.2 FE datasource APIs instead of importing master's
connector-module refactor.

### Backport adaptations

- Port Paimon schema restoration, catalog-option provenance, fallback
handling and tests to the existing table-copy/serialization lifecycle.
Keep statement-bound fields when reader options are applied after a
schema-only rename.
- Exercise Iceberg's existing metadata and transaction entry points with
real in-memory tables and retained-generation fixtures. Preserve the
nine disabled known-bug reproductions; omit two connector-only lifecycle
tests. Expiration checks verify live data-file sets and exact
delete-file counters; 4.2's separate manifest-name/data-counter issue
remains unchanged. The delete/history SQL suite checks visible rows and
eager/late-materialization parity because 4.2 does not expose master's
`_file`/`_pos` columns. The inventory labels upstream-only evidence
explicitly.
- Port ES residual tracking to the legacy planner and add compound
OR/NOT, docvalue eligibility and Thrift-context tests there. Use Java
8-compatible HTTP APIs for opt-in ES 9 tests and the branch's supported
node-address fixtures.
- Keep the existing full-schema Iceberg metadata-table scan, which
already supplies the file-size dependency for split offsets. Adapt the
external write suite to verify offsets without requiring master-only
distribution plans or golden output.
- Use the existing Parquet statistics interface and physical
row-location representation. Keep the TopN scheduling and pruning
invariants.
- Include the shared Arrow array normalizer required by the new reader,
taken from the same master revision. Preserve 4.2 Lance/Avro dispatch;
omit the schema-inference test for the unavailable Hive OpenCSV parser.

### Validation

- FE production and test sources compile with the branch's Java 8
target. Across the targeted runs and focused reruns, 20 test classes
report 181 discovered, 171 passed, zero failures/errors, and 10 skipped:
nine intentionally disabled known-bug reproductions plus the opt-in live
ES 9 class.
- FE Checkstyle: passed with zero violations.
- Clang compiler syntax checks: all 23 affected C++ translation units
passed against the current sources and freshly generated Thrift, using
cached third-party/generated build dependencies. Focused BE runtime
validation is listed below.
- clang-format 16: all 30 affected C++ source/header files passed.
- Groovy 4.0.19 syntax and AST conversion: all seven changed suites
passed. All 105 local inventory links resolve.
- Complete staged-diff self-review, whitespace/conflict checks, and
sensitive-information checks passed.

A full BE rebuild and SQL/external-service regressions were not run
locally. Live ES 9 tests require their opt-in endpoint. CI is requested
for the combined backport.

### Release note

Fix Paimon reads after field renames, external-catalog SHOW DATABASES
authorization, and ES compound residual filtering. Support ES 9.x node
discovery, publish Iceberg Parquet split offsets, improve Parquet V2
TopN row-group scheduling, and use FileScannerV2 for Arrow loads and
text/JSON schema inference.

### Check List (For Author)

- Test
    - [x] Unit Test
- [x] Regression test additions and syntax checks; SQL execution pending
CI
- Behavior changed:
    - [x] Yes. Backport the changes listed above.
- Does this need documentation?
- [x] No. Existing interfaces and settings are retained; the test
coverage inventory is included.

### Follow-up fixes

- Set the Thrift `enable_file_scanner_v2` default to `true`, matching
the session default. Preserve explicit `false` for legacy query scans
and extend the existing load-selection test.
- Keep the selected Paimon snapshot and schema together during relation
projection. Historical sequence/aggregation options must not be combined
with the latest field names. Read-only principals can resolve historical
selectors without INSERT permission.
- Read legacy append schemas with missing or empty bucket keys without
bucket pruning, preserving every historical file. The regression covers
both unaware and multiple fixed buckets.

Follow-up validation: 169 targeted Paimon FE tests passed with zero
failures/errors/skips. FE production compilation and Checkstyle passed.
Both changed C++ test translation units passed compiler syntax checks
and clang-format 16. Regenerated Java/C++ defaults are enabled; a Java
8-targeted protocol check verifies constructor/clear defaults, an
omitted wire field, and an explicit-false round trip. Complete BE CI and
live external SQL suites remain pending.

The two new reproductions first failed with the same missing-bucket-key
and renamed-sequence-field errors as CI. A separate read-only-principal
reproduction also failed before the permission correction. Complete
external regression execution remains pending CI.

### BE unit-test compatibility

- Complete the native Arrow Variant input dependency for `branch-4.2`:
accept binary storage and the Parquet Variant extension, validate
metadata and payloads before appending, and retain owned data after
Arrow buffers are released. Allow that ingestion path through the load
reader while preserving query-format restrictions and source-default
validation.
- Preserve Variant V2 markers in the Arrow test fixtures, including
nested types. Verify timestamp input against this branch's SerDe for
positive, negative, and second-boundary values. Match the existing DATE
range diagnostic in the TopN pruning test.
- Add tests for sliced and empty reads, invalid ranges and carriers,
malformed metadata/payloads, NULL children, atomic failure, buffer
ownership, and the query capability boundary.

Validation: all 13 previously failing BE tests passed. The expanded run
passed all 309 tests across nine suites, including Arrow/JSON loads,
load validation, Iceberg, table mapping, and Variant input/output. The
focused executable rebuilds the affected readers and SerDe against
cached 4.2 build dependencies; this is not a complete BE rebuild. All
seven changed C++ translation units passed compiler checks with current
headers/generated Thrift, and all 33 affected C/C++ files passed
clang-format 16. The complete staged diff, whitespace, and
sensitive-information checks passed. Full CI is requested with `run
buildall` and `/review`.

File scans still maintain the legacy scanner, its readers and a second capability/routing matrix. Route all file scans, file loads and external row-ID fetches through `FileScannerV2`, and remove the unused V1 implementation and tests.

- Keep `enable_file_scanner_v2` as a deprecated compatibility variable. Its stored value no longer affects planning or scanning; FE sends `true` in the existing Thrift field for older backends.
- Remove scanner-level capability checks. Factories and concrete readers return errors for unsupported operations, including native Hudi delta-log reads, query-only connector loads and Remote Doris Variant reads.
- Move standalone Iceberg position-delete reads to V2 `TableReader`, and historical Push ingestion to `LoadTableReader`. Preserve Push conversion expressions across ranges and compatibility with requests that omit the file-column count.
- Retain shared writer, metadata, JDBC/JNI and Elasticsearch HTTP components that surviving readers still use. Remove V1-only schema, partition, count and decoder infrastructure.
- Update connector planning, nullable COUNT pushdown, documentation and compatibility regression expectations.

Related PR: apache#69668

File scans always use FileScannerV2. `enable_file_scanner_v2` is deprecated and has no effect. Unsupported reader operations return an error instead of selecting the legacy scanner.

- Test
    - [x] Unit Test
    - [x] Regression test
    - [x] Manual test

Validation:
- Production BE and FE build, including connector plugin packages.
- 523 focused FE tests and FE Checkstyle.
- 593 distinct BE tests; the Remote Doris Flight suite was rerun successfully with localhost excluded from the development proxy.
- clang-format 16, clang-tidy on changed lines, build hygiene and its full self-test suite.
- Final Push fixture rerun passed after the test-data helper cleanup.
- Six standalone regression suites passed: Arrow Stream Load, HTTP CSV/JSON/compression/header formats and schema inference.
- Manual SQL verified that setting the old option to false remains accepted and SHOW VARIABLES marks it deprecated.

External Paimon regression expectations are updated for the no-op option and will be exercised by the external CI environment.

The exhaustive CSV/JSON split tests retain their existing boundary assertions after dropping the V1 parameterization. Four narrow cognitive-complexity suppressions preserve these existing GTest-expanded cross-product tests; new tests are decomposed instead.

- Behavior changed:
    - [x] Yes. The scanner option is now a deprecated no-op; unsupported operations fail in their reader.
- Does this need documentation?
    - [x] Yes. The scanner design and review guide are updated in this PR.

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…ion (#69723)

### What problem does this PR solve?

File scans maintained both the legacy scanner/readers and the V2 reader
stack. Route query scans, schema inference, file loads and external
row-ID fetches through FileScannerV2, migrate standalone Push and
Iceberg delete-file consumers, and remove the retired implementation and
tests.

- Keep enable_file_scanner_v2 as a deprecated no-op, with the existing
optional Thrift field emitted/defaulted to true.
- Preserve ORC session/adaptive batch caps, stripe dictionary rejection,
lazy predicate-column reuse and sequential CSV/TEXT/JSON S3 prefetch.
- Filter matching ORC dictionary IDs before string expansion in lazy,
eager and predicate-only scans. Retain residual predicates, NULL
semantics, pruning fences, physical column identity and physical row
positions.
- Restore ORC serialized-tail caching using the existing V1
FileMetaCache key policy, including its file-size fallback when mtime is
absent. Keep V2 payload types separate. ORC footer counters have
format-qualified names and publish once across repeated close; native
Parquet cache identity remains unchanged.
- Bound standalone position-delete reads to the native Parquet batch
limit. Broker source projections follow validated src_slot_ids order,
and malformed ORC dictionary offsets return CORRUPTION.
- Retain eight active V2 position-delete system-table regressions and
the operator shared-cache/shard-limit tests in maintained suites.
- Preserve all six Hive nested STRUCT/ARRAY/MAP access-path scenarios in
V2 across ORC/Parquet and name/position mapping, including separate
predicate-column IDs and complete-versus-projected row comparisons.
- Preserve active-file error context and the upstream Arrow column/type
context without changing original validation status codes.

Related PR: #69668

The supported rolling-upgrade direction remains BE first, then FE.
Old-FE/new-BE requests without semantic COUNT arguments retain their
ordinary-read fallback. New-FE/old-BE nullable COUNT is outside that
documented upgrade direction; the deprecated scanner option does not
provide a legacy routing gate.

The branch resolves the modify/delete conflict with master at
339e087. The old Arrow reader stays deleted, and its upstream
error-context fix is preserved in the V2 reader.

### Release note

File scans always use FileScannerV2. enable_file_scanner_v2 is
deprecated and has no effect. Unsupported operations fail in their
reader.

### Validation

- 565 selected BE ASAN tests passed on the rebased code: 190 ORC, 75
Parquet, 118 Iceberg reader, 21 delete-file helper, 28 scanner, 43 CSV,
43 Arrow load, 18 load-reader, 11 JSON load, 11 position-delete
system-table, 4 metadata-cache and 3 JSON-prefetch tests.
- The Hive follow-up passed 182 related BE ASAN tests, including 24
nested-path matrix cases. Selected values, parent null maps and
ARRAY/MAP offsets match full scans; KEYS-only MAP expectations retain
the existing V2 value-column fallback.
- The new 102401-row position-delete case failed before the batch-cap
fix and now checks every visited position.
- The selective dictionary case reproduced about 256 MiB of
rejected-string expansion before the fix. It now bounds materialized
strings and checks returned values and physical condition-cache granules
with lazy materialization enabled, disabled, and no follower column.
- Warm ORC footer opens perform zero physical pread calls. Versioned
keys, the legacy versionless file-size fallback, disabled caching and
repeated close are covered. Both ORC/Parquet profile initialization
orders retain independent footer counters.
- Arrow Date64 validation retains INVALID_ARGUMENT and reports the input
column and types.
- The cache/profile follow-up passed 311 related BE ASAN tests (278
reader/cache/profile and 33 scanner/operator/push tests). Two valid
operator cache tests were preserved from the retired fixture.
- FE core and all five affected connector modules pass Checkstyle.
clang-format 16 passes on all 60 affected surviving C/C++ files. Build
hygiene and changed-line clang-tidy pass.

External-service regression suites and full CI are not fresh results
from this update. No end-to-end performance benchmark is claimed. CHAR
dictionary pruning remains outside the restored STRING/VARCHAR path.

### Check List (For Author)

- [x] Unit Test
- [ ] External-service regression test on the updated head
- [x] Documentation synchronized

### Check List (For Reviewer who merge this PR)

- [ ] Confirm release note
- [ ] Confirm tests and documentation
- [ ] Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110
#67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690
#68813 #68815 #68707 #68649

Problem Summary:

**In short.** This ports the Lance BE read path of branch-4.1 /
branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus
the BE side of three 4.x changes merged after that snapshot: the search
filter `prefilter` flag and profile layout (#68813), distance bounds for
`vector_search` (#68815), and the split-level index rule of #68707 that
#68815 relies on. It gathers the BE parts of the PRs listed above.
Master links `liblance_c` but has no Lance reader, so every
`FORMAT_LANCE` scan fails today. With this PR the BE reads Lance
datasets through FileScannerV2: plain scans, vector search and full-text
search, predicate and runtime-filter pushdown, the schema fetch of a
local Lance TVF, and the second phase of TopN lazy materialization. The
PR is BE only. The FE connector plugin and the regression suites come in
separate PRs, so until then nothing on master plans a Lance scan, and
the BE unit tests carry the proof.

Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju
@FANNG1

**Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` /
`s3()` with `format=lance`, and the `vector_search` / `full_text_search`
TVFs. On master the FE side is being rebuilt as a connector plugin
(`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift
ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5`
(`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added
after the snapshot, so a BE built from master also reads the Lance scan
ranges and search requests that a 4.x FE sends during a rolling upgrade.
The one exception is `TVectorSearchOptions` 6, branch-4.2's
`approx_mode` (#69852, merged into branch-4.2 after the snapshot and not
ported): it is reserved in a comment, and a master BE ignores it, so
during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate`
gets Lance's default (`normal`).

**The problem, and what it cost.**

| | master today | with this PR |
|---|---|---|
| A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan
fails with "FileScannerV2 does not support file format" | read by
`LanceTableReader` in FileScannerV2 |
| `fetch_table_schema` for a local Lance TVF | not handled | returns the
Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` |
| The row location of a Lance row in TopN lazy materialization |
`GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a
64-bit id within the snapshot | a version 1 layout carries the 64-bit
row id; phase two reads the rows with the dataset's `take_rows` |

**How this PR fixes it.**

- Reader: `be/src/format_v2/lance/*` and
`be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one
local renamed for master's `-Wshadow`. The unit tests construct
`RuntimeState(TQueryOptions(), globals)`, because master removed the
globals-only constructor.
- FileScannerV2: the reader dispatch and `_to_file_format` handle
`FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two
exhaustive switches. Since #69723 every file scan uses FileScannerV2, so
nothing else routes Lance scans (on branch-4.2 a Lance query fails with
NotSupported when `enable_file_scanner_v2` is off).
- Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0,
LANCE_DATASET_ROW_ID = 1}` and a union layout for
`GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static
asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2
varint, so the wire format is compatible). The materialization operator
decodes row locations by version, rejects an unknown version or a wrong
size, and stamps `row_location_version`. `RowIdStorageReader` validates
the counts and the version, range-checks internal row ids against
`uint32`, and groups Lance rows by plan node, snapshot version and
dataset URI, so each dataset is read with one `take_rows`. Master's
segment batching, scan-slot dedup and fetch concurrency stay as they
are.
- Thrift (`PlanNodes.thrift`): the external search request body
(`TVectorElementType`, `TVectorMetric`, `TSearchVector`,
`TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`,
`TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`,
`TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`),
`TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the
branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is
reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in
comments and added 6 (`paimon_options`); this PR replaces those comments
with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is
reserved in a comment for branch-4.2's `approx_mode` (#69852).
- After the snapshot (one commit each):
- #68813: `prefilter` decides whether the TVF search filter runs before
(true) or after (unset / false) candidate selection, for vector and
full-text search. The profile groups the Lance counters under
`LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`,
`LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan`
and `LanceDataCache`, uses the 4.x counter names, and registers every
scan metric before the statistics callback is installed, so the
callback, which Lance invokes from its execution streams, never adds
counters. #68813's FE tweak that hides `TUnit.NONE` labels in
`AggCounter` belongs to the FE port.
- #68707 (part): a vector split searches exactly the index segments FE
planned for it, and a split without segments is searched flat whatever
`use_index` says; a split with segments under `use_index = false` is
rejected. 4.x FEs plan segments only when `use_index` allows them.
- #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound`
(exclusive) are validated (finite FLOAT values, lower < upper at FLOAT
precision, single-vector only) and installed with
`lance_scanner_set_distance_range()`, which master's pinned lance-c
provides.
- Review fixes:
- A nested array goes to the Doris SerDes without a copy only when it
and every descendant start at offset 0 and reference exactly their child
ranges. Lance's legacy (v1) decoder can return a struct or list that is
tight at its own level over a Boolean child with a bit offset (a batch
that starts mid-byte in the bitmap), and the SerDes reject that under
Arrow input validation.
- A zero-field `STRUCT<>`, which Lance decodes, is reported as an
unsupported column at any nesting level, like the other unsupported
Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot
hold (its row count comes from its first child).
- A Lance Decimal256 field with precision <= 38 is narrowed to
Decimal128 by a safe Arrow cast before decoding. Doris types it
DECIMAL32/64/128I by precision, and those SerDes read only Decimal128
arrays; the BE used to crash on the first value, also for nested leaves.
Only values a reader can see are cast: before a nested leaf is narrowed,
every child slot that no visible row references (under a NULL list,
fixed-size-list, map or struct row, or outside a sliced parent) is
marked NULL, so a hidden value does not fail the cast and a visible
value outside the declared precision still does; map keys, which cannot
be NULL, stay checked.
- FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares
floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows
that Doris keeps.
- Runtime filters that mention INT64_MIN stay residual-only. The pinned
Lance SQL parser reads `-9223372036854775808` as the negation of a
magnitude that it parses as i64 first; the overflow turns the literal
into a Float64, which the planner cannot coerce to Int64 or Decimal, so
a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal
column failed planning and the query failed. Every other i64 is still
pushed.
- The shared Arrow DATE32 decoding
(`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside
0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`,
skips null slots and adds the epoch in 64 bits: the DATE32 range part of
branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and
paimon-rust reads.
- Child values hidden by a NULL list, fixed-size-list, map or struct row
are not decoded: the array, map and struct SerDes give the element SerDe
a validity bitmap that also clears those slots
(`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them
unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden
10000-01-01 under a NULL fixed-size list used to fail the DATE32 range
check above. Visible values are unchanged; this applies to ADBC, Arrow
stream load and paimon-rust as well. The masks come from ExecEnv's
tracked Arrow pool. A nested Lance array that is compacted before
decoding is masked the same way, at every level, before its
`ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision
decimal, Date64 or time no longer fails the batch while a visible one
still does. Since #69697 the same masks keep hidden DATETIMEV2 /
TIMESTAMPTZ values out of its range checks. A mask costs at most one
pass per batch: a NULL row that owns no child slots, or only NULL ones,
allocates nothing, and Arrow stream load masks each retained IPC batch
once (`mask_arrow_descendants_not_visible()`) before it hands row ranges
to the SerDes, so its per-block reads allocate nothing. The one
per-block mask left is for map keys under a NULL map row that still owns
entries, since a MapArray cannot hold NULL keys; Arrow's map builder
writes NULL map rows without entries.
- Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a
Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The
nullable serde reads an Arrow Null array as NULLs without touching
buffers that the array does not have.
- Configs, caches and metrics, with the 4.x names and defaults:

  | Config | Default |
  |---|---|
  | `lance_index_cache_size_bytes` | 10 GiB |
  | `lance_metadata_cache_size_bytes` | 1 GiB |
  | `enable_lance_data_cache` | true |
  | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` |
  | `lance_data_cache_disk_capacity_bytes` | 100 GiB |
  | `lance_data_cache_read_block_size_bytes` | 1 MiB |
  | `lance_io_buffer_size_bytes` (mutable) | 2 GiB |
  | `lance_batch_readahead` (mutable) | 5 |
  | `lance_fragment_readahead` (mutable) | 5 |

The BE-wide Lance session (index cache, metadata cache and the optional
Foyer disk cache) is created by the first Lance scan, so a BE that never
reads Lance creates no cache directory. The first six configs take
effect after a restart. A negative cache size (and, while the data cache
is enabled, a negative disk capacity or read block size) fails the first
Lance scan with a message that names the setting, as does any data-cache
value lance-c rejects; the rest of the BE keeps serving. Ten
`lance_session_*` metrics report the two in-memory caches. With the
defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from
the first Lance scan on (100 GiB in blocks of twice the read block
size), the same as on branch-4.1 / 4.2. Size
`lance_data_cache_disk_capacity_bytes` and
`lance_data_cache_read_block_size_bytes` against the BE's open-file
limit, or set `enable_lance_data_cache = false`.

The index and metadata caches live in lance-c memory: they are not
charged to a MemTracker or registered with CacheManager, so memory GC,
the cache capacity adjustment under memory pressure and
`/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache
= false` does not turn them off. lance-c has no call to clear, evict or
resize them, so size them with the two configs. The disk tier keys
blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS,
without the endpoint) and the object path
(`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve
the same bucket name and data-file path with different bytes (data files
registered under chosen names or replaced in place; Lance's writers use
random file names) share entries, also across a BE restart; such a
deployment should set `enable_lance_data_cache = false`. Both are as on
branch-4.1 / 4.2 and tracked as follow-ups.

A Lance scan's native working set, that is the I/O buffered up to
`lance_io_buffer_size_bytes` per scanner, the decode readahead and the
operators of a search plan such as the top-k of a flat vector search, is
allocated inside lance-c and is charged to neither the query nor its
workload group. It counts toward the BE process memory only, so memory
GC, which ranks queries by tracked memory, does not attribute it to the
Lance query. lance-c accepts no memory budget, so on BEs with little
memory headroom size `lance_io_buffer_size_bytes`,
`lance_batch_readahead` and `lance_fragment_readahead` for the number of
concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a
follow-up.
- Build: macOS links `-framework IOKit`. This is the first time objects
of `liblance_c.a` are pulled into `doris_be`, so the binary and the link
time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and
`util/threadpool.h` 589, the same as master `315c2e00331` alone (limits
275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562
(master alone 559, limit 660); the reach budgets are unchanged.

Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`);
the FE side, including the rolling-upgrade guard that keeps Lance lazy
materialization off while an old BE could decode a version 1 row
location as version 0; the regression suites; the rest of #68707
(searching historical versions, tags and branches, rejecting fragment
ids missing from the snapshot, naming the snapshot in errors and the
profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that
#67005 carried goes in a separate PR.

**Results.**

| | master | with this PR |
|---|---|---|
| A `FORMAT_LANCE` query | fails before reading anything | read by
`LanceTableReader` in FileScannerV2 |
| A Lance row in the second phase of TopN lazy materialization | cannot
be represented | a version 1 row location, read back with `take_rows` |
| A row location of an unknown version or size in phase two | decoded as
version 0 | rejected with an error |
| An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream
load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read
fails with `InvalidArgument` |

BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on
the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run,
1027 pass and 2 Parquet microbenchmarks skip by design. They cover the
Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10,
`LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the
row-location protocol (`MaterializationSharedStateTest`,
`RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`:
count mismatch, unknown version, a Lance version on an internal mapping,
an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and
Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`,
`ParquetScanTest` and the other `Parquet*` suites), whose row ids now go
through `file_local`; and the Arrow / protobuf / DATE serde suites with
the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader
suites that decode Arrow dates. Reverting any of the review fixes, the
#68813 flag, the #68707 split rule or the #68815 bounds makes its new
tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest`
10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow
SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310
pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The
ninth review round adds
`LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and
`LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`:
both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto
`339e087b11f`, and restoring `parse_int64()` in either check makes
exactly these two fail. On the head rebased onto master `315c2e00331`,
with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites
pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest`
(15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe
suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`,
`AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest`
and the Lance suites. Allocating at the first NULL row of each call
again fails the new `ArrowParentNullsTest` case and the four new
`LoadArrowPipeTest` block cases, dropping the per-batch mask fails the
LIST and STRUCT block cases, and returning the child unmasked fails the
new timestamp case, four `ArrowParentNullsTest` cases and
`LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`.

### Release note

Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They
take effect once the FE can plan Lance scans. Reading an Arrow DATE32
value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or
paimon-rust now fails instead of returning 0000-00-00.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
- `run-be-ut.sh --run --filter=...` with the suites listed under
Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the
rebases, and the Lance and Arrow SerDe suites pass again with the
nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and
compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite
driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the
scanner selection and in `fetch_table_schema`, which now hands every
file format to `format::fetch_file_schema()`, so the Lance branch runs
ahead of that call; #69646 added a TopN row-group test that reads the
version 0 row location fields, which now go through `file_local` (the
earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the
scanner selection, the `FileFormat` enum, its test and the reach budget
table). The 95 TUs that this PR changes or that master changed since
`5caaeb3e3d4` were syntax-checked against the rebased tree with the UT
build flags, and CI runs the suites again. Rebased again onto master
`8d88f1cc6f7`: the only conflict was the budget table, where #65858
moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that
this PR changes or that master changed since `43f9ceb48d7` pass the same
syntax check. Rebased again onto master `339e087b11f`: the only conflict
was the budget table again, where #67292 moved the `util/rle_encoding.h`
baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent
masking in the map and struct readers and merged cleanly. Of the 114
sources that this PR changes or that master changed since `8d88f1cc6f7`,
111 pass the same syntax check. The other three are master's: two new
#67292 files under `storage/segment` fail the same way on master alone,
since they compile only inside their unity batch, and this PR adds no
source to that target; the #69818 benchmark is not in the UT build and
includes only `util/simd` headers that this PR does not touch. Reverting
any of the review fixes, the #68813 flag, the #68707 split rule or the
#68815 bounds makes its new tests fail. Rebased again onto master
`378404eeb12`, which removed the legacy file scanner (#69723): the
conflicts were in the scanner-selection code and tests that #69723
deleted, so this PR's Lance lines there went with them (including its
`LanceQueriesAlwaysUseScannerV2` test), in the profile keys of
`materialization_opertor.cpp`, and in the row fetch of
`rowid_fetcher.cpp`, whose non-Lance path is now only
`FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests
took this PR's `file_local` rename with them, and the
`util/rle_encoding.h` budget commit is dropped because master is no
longer near that limit. Of the 100 sources that this PR changes or that
master changed since `339e087b11f`, 99 pass the same syntax check; the
other, `format_v2/orc/orc_reader.cpp`, fails the same way on master
alone. On the rebased head the suites listed under Results run 1029
cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that
`DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master.
Rebased again onto master `315c2e00331`: the only conflict was
`TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had
reserved fields 2-5 in comments; the two commits that add the fields
replace those comments, and field 6 stays #70075's `paimon_options`.
#69717 cut transitive includes from shared headers: of the 160 sources
that this PR changes or that master changed since `378404eeb12`, 156
pass the same syntax check; the two `jemalloc_hook` tests have no
compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc
aliases that macOS cannot build (#69819), and
`format_v2/table_reader.cpp` alone fails the same way on master alone
while its unity batch compiles. The full `doris_be_test` builds and
links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases)
pass before the mask fix, and the 328 cases above with it.
- `build-support/check-build-hygiene.sh` (on this branch rebased onto
master `315c2e00331`), clang-format 16 and
`build-support/run-clang-tidy.sh` on the changed files: all pass; on the
changed lines clang-tidy (llvm 22) reports only
`readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of
the masking functions, the macro's expansion that master's existing
`DORIS_CHECK`s report too.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- Behavior changed:
    - [ ] No.
- [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN
lazy materialization rejects row locations of an unknown version or
size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day
outside the Doris DATE range fails the read instead of becoming
`0000-00-00`.
- Does this need documentation?
    - [x] No. Lance is documented with the FE connector.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: zhangstar333 <[email protected]>
Co-authored-by: Gabriel <[email protected]>
Co-authored-by: zhaobo wang <[email protected]>
Co-authored-by: jay <[email protected]>
Co-authored-by: FANNG <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: zy-kkk <[email protected]>