### What problem does this PR solve?

Problem Summary: The deprecated Thrift option still defaults to false even though file scans always use V2. Default the existing optional field to true, preserving its field ID, type and deprecation comment.

Related PR: apache#69723

### Release note

None

### Check List (For Author)

- Test: Generated C++ and Java with Thrift 0.24.0; verified the C++ constructor and Java constructor/clear defaults are true. git diff --check passed.
- Behavior changed: No. File scans already ignore this deprecated option and always use V2.
- Does this need documentation: No.

### What problem does this PR solve?

Problem Summary: SHOW VARIABLES adds a deprecated_ prefix to the retained
scanner option, so suites using its old display name fail before scanning.
Read saved values through @@enable_file_scanner_v2 and use its scalar result
column, including the equivalent ES regression outside the failed group.

Restore the ORC RowID-seek fault point in the V2 reader's existing exception
boundary. Cover error conversion, empty error output and retry without
consuming the requested RowID. Keep the integration test's false-option
setting to verify that phase two still uses V2.

The already-muted workload-policy suite retains its existing V2 audit and
non-blocking checks and drops its removed V1-only branch. Existing V2 remote
scan accounting is unchanged.

Related PR: apache#69723

### Release note

None

### Check List (For Author)

- Test: Reproduced the missing variable result and the failing RowID fault
  test before the fix. All 180 ORC reader tests passed. A live-Doris regression
  probe verified variable capture and restoration. All 14 modified Groovy
  suites passed syntax and AST conversion. clang-format 16, clang-tidy,
  build hygiene and git diff --check passed.
- Full external-service suites require the CI fixtures and were not run locally.
- Behavior changed: No. Only regression compatibility and fault injection change.
- Does this need documentation: No.

### What problem does this PR solve?

Related PR: apache#69723

Problem Summary: ORC V2 inherited the no-op batch-size setter and decoded
4096 rows at a time, so forcing V2 dropped session and adaptive memory
limits for scans previously using V1. Apply the cap to SDK batches before
decoding, keep it across stripe ranges, and preserve the reader cursor
when the cap changes. RowID reads remain limited to one row.

Require the remote-scan workload policy to cancel V2 queries using the
existing query IO counters. Remove the obsolete audit-only branch and
comments claiming that V2 does not publish remote scan bytes.

### Release note

ORC file scans honor session and adaptive row batch limits to reduce
per-batch decoding memory for wide rows.

### Check List (For Author)

- Test: Three batch-cap tests reproduced the original issue. All 204 ORC
  and FileScannerV2 ASAN unit tests passed, followed by a successful
  strengthened wide-payload stripe test. A live remote-byte policy
  cancellation probe passed with both values of the compatibility option.
  Groovy syntax/AST conversion, clang-format 16, changed-line clang-tidy,
  build hygiene and git diff --check passed.
- The full Hive external suite requires CI fixtures and was not run locally.
- Behavior changed: Yes. ORC decoding respects supplied row batch limits,
  and regression coverage requires workload-policy cancellation.
- Does this need documentation: No.

…torage-client

master removed the legacy file scanner (apache#69723): be/src/exec/scan/file_scanner.{h,cpp} and the
V1 readers be/src/format/{csv/csv_reader,json/new_json_reader,orc/vorc_reader,
parquet/vparquet_reader,text/text_reader}.cpp. They are deleted here as in master.

- The V1 rejection of Azure SAS / OAuth2 (FileScanner::_check_storage_access and its test in
  file_scanner_v2_test.cpp) goes with the V1 scanner: every scan now runs on FileScannerV2,
  which serves typed Azure access.
- The V1 readers' copy of TFileScanRangeParams.azure_client_config has its counterpart on the V2
  paths: TableReader, fetch_file_schema (now used for every schema RPC format), the Iceberg
  reader's delete files, the deletion vector reader and parquet_meta all set it; the Iceberg
  delete file helper now reads through TableReader with the delete sink's scan params.

obj_storage_file_reader.cpp takes master's read IO trace (`s3_get` records with io_ctx) and
atomic statistics under the ObjStorageFileReader names.

Tests from master moved to the typed client API: s3_file_reader_test.cpp (ObjClientConfig,
set_client_for_test, ObjStorageFileReader, AwsSdkManager), json_reader_test.cpp and
csv_reader_test.cpp (ObjClientFactory::set_client_creator_for_test with ObjClientConfig).

### What problem does this PR solve?

Issue Number: N/A

Related PR: #69672, #69723

External ES regression suites fail during setup or retain expectations
from the previous indexed-value predicate behavior:

- Read the deprecated scanner option through `SELECT
@@enable_file_scanner_v2`, whose compatibility name remains valid when
SHOW prefixes deprecated names.
- On ES 8.3.3, fall back from the specifically unsupported
`_source.mode` mapping syntax to `enabled:false`, retaining the
disabled-source row and alias checks. Other HTTP failures remain fatal;
supporting ES8 versions still exercise the mode syntax.
- Compare LIKE, REGEXP, NULL and `level` predicates against unfiltered
rows materialized into internal tables, then update only the affected
runner-generated golden blocks.
- Make the date-string equality test explicitly use source reads, test
midnight as the matching value, and retain 08:00 as a negative control.
- Check that a residual predicate prevents early termination while a
safe numeric docvalue predicate still permits LIMIT pushdown.

### Release note

None

### Check List (For Author)

- Test
  - [x] Regression test
  - [ ] Unit Test
  - [x] Manual test

Validation uses fresh FE and ASAN BE artifacts built from this PR commit
(`6386f5f83a51b548c86ab3cff83d53cb48fb18df`) with the standard
`./build.sh --fe --be -j64` entry point. The complete build exited
successfully, and both finished artifact versions match that commit. No
production source is changed by this PR.

- On the same fresh binaries and reset ES 5.6.16, 6.8.23, 7.17.5 and
8.3.3 fixtures, the parent suites/golden files reproduced all ten
failures; the final PR suites passed all ten in normal comparison, with
zero failed, fatal or skipped suites. The passing run executed 2,154
golden checks and 1,798 independent query comparisons, without
force-generation or further golden edits.
- The complete `test_es_query` passed all four ES versions and both
scroll settings, including its unchanged complex-array golden blocks
(276 result checks and 20 independent comparisons).
- Independent SQLite/Python evaluation of the unchanged raw fixture JSON
passed 38 comparisons and covered all 17 changed golden blocks. Real ES
requests also confirmed that `ignore_above` omits matching index values
while retaining source strings, and that the constant-keyword index
matches documents whose source `level` is absent.
- Captured ES search requests verified the date modes: the original
four-column projection reads source, returns midnight values and rejects
the 08:00 predicate. A date-only docvalue projection correctly returns
08:00 in the +08:00 session. The revised test explicitly fixes the
source contract and retains the 08:00 negative control.
- The complete fallback suite additionally passed normal comparison on
ES 8.11.3: 304 golden checks and 304 independent comparisons. The actual
mappings and document responses confirmed absent source for both the ES
8.3.3 `enabled:false` fallback and the supported ES 8.11.3
`mode:disabled` branch.
- The existing `test_es_file_scanner_v2` compatibility suite also passed
on the fresh build across all four fixture versions. The deprecated
option is accepted; this does not claim two distinct physical reader
implementations on current master.
- Direct variable probes confirmed the old exact SHOW name returns no
rows, the wildcard SHOW name exposes its deprecated prefix, and the
scalar compatibility name remains readable.
- `git diff --check` passed. ES9 compatibility is not claimed.

- Behavior changed:
- [x] No production behavior change; regression fixtures and
expectations are corrected.
  - [ ] Yes.

- Does this need documentation?
  - [x] No.
  - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### What problem does this PR solve?

Related PR: apache#70160, apache#69723

Problem Summary: Master always uses FileScannerV2 and exposes the deprecated
rollout option under a prefixed SHOW name. The WAL schema-change case queries
the old name, gets no rows and fails before testing replay. Remove the obsolete
global-option setup and restoration.

### Release note

None

### Check List (For Author)

- Test: Regression test test_group_commit_schema_change_wal on the local integrated ASAN cluster (RUNNING, FINISHED and CANCELLED); existing expected output unchanged.
- Behavior changed: No
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: apache#69723 removed the legacy file scanner. Every file scan
now uses FileScannerV2, and enable_file_scanner_v2 is kept only as a
deprecated session variable that nothing reads. Four Delta suites still
set it: basic, features and the benchmark to true, and the Hive gateway
suite to false. None of these settings has any effect, so the Hive
suite's false setting does not check anything. Remove all four. SET
statements produce no query output, so the expected .out files are
unchanged. The Hive suite still sets file_split_size=128 so that one
file becomes many ranges sharing the same deletion vector.

### Release note

None

### Check List (For Author)

- Test: No local run. The diff deletes only SET statements; regression runs are left to CI.
- Behavior changed: No
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: close apache#69884

Related PR: apache#69904

Problem Summary: apache#69723 removed the legacy file scanner. Every file scan
now uses FileScannerV2, and enable_file_scanner_v2 is kept only as a
deprecated session variable that nothing reads. Four Delta suites still
set it: basic, features and the benchmark to true, and the Hive gateway
suite to false. None of these settings has any effect, so the Hive
suite's false setting does not check anything. Remove all four. SET
statements produce no query output, so the expected .out files are
unchanged. The Hive suite still sets file_split_size=128 so that one
file becomes many ranges sharing the same deletion vector.

### Release note

None

### Check List (For Author)

- Test: No local run. The diff deletes only SET statements; regression runs are left to CI.
- Behavior changed: No
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: #65730 #66608 #66805 #66841 #66999 #67005 #67011 #67110
#67268 #67289 #67325 #67650 #67921 #68028 #68335 #68375 #68615 #68690
#68813 #68815 #68707 #68649

Problem Summary:

**In short.** This ports the Lance BE read path of branch-4.1 /
branch-4.2 to master, in its branch-4.2 form as of `cabcf45c1af`, plus
the BE side of three 4.x changes merged after that snapshot: the search
filter `prefilter` flag and profile layout (#68813), distance bounds for
`vector_search` (#68815), and the split-level index rule of #68707 that
#68815 relies on. It gathers the BE parts of the PRs listed above.
Master links `liblance_c` but has no Lance reader, so every
`FORMAT_LANCE` scan fails today. With this PR the BE reads Lance
datasets through FileScannerV2: plain scans, vector search and full-text
search, predicate and runtime-filter pushdown, the schema fetch of a
local Lance TVF, and the second phase of TopN lazy materialization. The
PR is BE only. The FE connector plugin and the regression suites come in
separate PRs, so until then nothing on master plans a Lance scan, and
the BE unit tests carry the proof.

Original authors: @zhangstar333 @Gabriel39 @wangzhaobo957-cloud @Jay-ju
@FANNG1

**Background.** Lance shipped in 4.1.4: the Lance catalog, `local()` /
`s3()` with `format=lance`, and the `vector_search` / `full_text_search`
TVFs. On master the FE side is being rebuilt as a connector plugin
(`fe-connector-lance`), and the BE needs the reader from 4.x. The thrift
ids are the branch-4.1 / 4.2 ones, including `TExternalSearchRequest.5`
(`prefilter`) and `TVectorSearchParams.6/7` (distance bounds) added
after the snapshot, so a BE built from master also reads the Lance scan
ranges and search requests that a 4.x FE sends during a rolling upgrade.
The one exception is `TVectorSearchOptions` 6, branch-4.2's
`approx_mode` (#69852, merged into branch-4.2 after the snapshot and not
ported): it is reserved in a comment, and a master BE ignores it, so
during a rolling upgrade a 4.2 FE that asks for `fast` or `accurate`
gets Lance's default (`normal`).

**The problem, and what it cost.**

| | master today | with this PR |
|---|---|---|
| A `FORMAT_LANCE` scan | FileScannerV2 has no Lance reader, so the scan
fails with "FileScannerV2 does not support file format" | read by
`LanceTableReader` in FileScannerV2 |
| `fetch_table_schema` for a local Lance TVF | not handled | returns the
Arrow schema mapped to Doris types; Null leaves keep `NULL_TYPE` |
| The row location of a Lance row in TopN lazy materialization |
`GlobalRowLoacationV2` only has a 32-bit row id, but a Lance row id is a
64-bit id within the snapshot | a version 1 layout carries the 64-bit
row id; phase two reads the rows with the dataset's `take_rows` |

**How this PR fixes it.**

- Reader: `be/src/format_v2/lance/*` and
`be/src/format_v2/table/lance_reader.*`, as on branch-4.2 except one
local renamed for master's `-Wshadow`. The unit tests construct
`RuntimeState(TQueryOptions(), globals)`, because master removed the
globals-only constructor.
- FileScannerV2: the reader dispatch and `_to_file_format` handle
`FORMAT_LANCE` / `"lance"`; `FileFormat::LANCE` is added to its two
exhaustive switches. Since #69723 every file scan uses FileScannerV2, so
nothing else routes Lance scans (on branch-4.2 a Lance query fails with
NotSupported when `enable_file_scanner_v2` is off).
- Row locations: `ROW_VERSION {FILE_LOCAL_ROW_ID = 0,
LANCE_DATASET_ROW_ID = 1}` and a union layout for
`GlobalRowLoacationV2`. Version 0 keeps its byte layout, which static
asserts pin. `PRequestBlockDesc.row_id` becomes `uint64` (a proto2
varint, so the wire format is compatible). The materialization operator
decodes row locations by version, rejects an unknown version or a wrong
size, and stamps `row_location_version`. `RowIdStorageReader` validates
the counts and the version, range-checks internal row ids against
`uint32`, and groups Lance rows by plan node, snapshot version and
dataset URI, so each dataset is read with one `take_rows`. Master's
segment batching, scan-slot dedup and fetch concurrency stay as they
are.
- Thrift (`PlanNodes.thrift`): the external search request body
(`TVectorElementType`, `TVectorMetric`, `TSearchVector`,
`TVectorSearchParams` 1-7, `TFtsCoverageMode`, `TFtsQueryType`,
`TFtsMatchOperator`, `TFullTextSearchParams`, `TSearchFilterFormat`,
`TSearchFilter`, `TVectorSearchOptions`, `TExternalSearchQuery`),
`TExternalSearchRequest` 2-5 and `TLanceFileDesc` 4-6, all with the
branch-4.1 / 4.2 ids. The enums spell out their 4.x values. No id is
reused. On master, #70075 reserved `TExternalSearchRequest` 2-5 in
comments and added 6 (`paimon_options`); this PR replaces those comments
with the fields and leaves 6 as it is. `TVectorSearchOptions` 6 is
reserved in a comment for branch-4.2's `approx_mode` (#69852).
- After the snapshot (one commit each):
- #68813: `prefilter` decides whether the TVF search filter runs before
(true) or after (unset / false) candidate selection, for vector and
full-text search. The profile groups the Lance counters under
`LanceTiming`, `LanceIO`, `LanceIndex`, `LanceVectorIndex`,
`LanceScalarIndex`, `LanceExecutionPlan`, `LanceScan`, `LanceSearchPlan`
and `LanceDataCache`, uses the 4.x counter names, and registers every
scan metric before the statistics callback is installed, so the
callback, which Lance invokes from its execution streams, never adds
counters. #68813's FE tweak that hides `TUnit.NONE` labels in
`AggCounter` belongs to the FE port.
- #68707 (part): a vector split searches exactly the index segments FE
planned for it, and a split without segments is searched flat whatever
`use_index` says; a split with segments under `use_index = false` is
rejected. 4.x FEs plan segments only when `use_index` allows them.
- #68815: `distance_lower_bound` (inclusive) and `distance_upper_bound`
(exclusive) are validated (finite FLOAT values, lower < upper at FLOAT
precision, single-vector only) and installed with
`lance_scanner_set_distance_range()`, which master's pinned lance-c
provides.
- Review fixes:
- A nested array goes to the Doris SerDes without a copy only when it
and every descendant start at offset 0 and reference exactly their child
ranges. Lance's legacy (v1) decoder can return a struct or list that is
tight at its own level over a Boolean child with a bit offset (a batch
that starts mid-byte in the bitmap), and the SerDes reject that under
Arrow input validation.
- A zero-field `STRUCT<>`, which Lance decodes, is reported as an
unsupported column at any nesting level, like the other unsupported
Lance types. It used to be typed as a STRUCT that `ColumnStruct` cannot
hold (its row count comes from its first child).
- A Lance Decimal256 field with precision <= 38 is narrowed to
Decimal128 by a safe Arrow cast before decoding. Doris types it
DECIMAL32/64/128I by precision, and those SerDes read only Decimal128
arrays; the BE used to crash on the first value, also for nested leaves.
Only values a reader can see are cast: before a nested leaf is narrowed,
every child slot that no visible row references (under a NULL list,
fixed-size-list, map or struct row, or outside a sliced parent) is
marked NULL, so a hidden value does not fail the cast and a visible
value outside the declared precision still does; map keys, which cannot
be NULL, stay checked.
- FLOAT / DOUBLE `>=` runtime filters stay residual-only. Lance compares
floats by totalOrder, so a pushed `x >= c` dropped negative-NaN rows
that Doris keeps.
- Runtime filters that mention INT64_MIN stay residual-only. The pinned
Lance SQL parser reads `-9223372036854775808` as the negation of a
magnitude that it parses as i64 first; the overflow turns the literal
into a Float64, which the planner cannot coerce to Int64 or Decimal, so
a pushed IN, `>=` or `<=` filter on an INT64 or scale-zero decimal
column failed planning and the query failed. Every other i64 is still
pushed.
- The shared Arrow DATE32 decoding
(`DataTypeDateV2SerDe::read_column_from_arrow`) rejects days outside
0000-01-01..9999-12-31 instead of returning a non-null `0000-00-00`,
skips null slots and adds the epoch in 64 bits: the DATE32 range part of
branch-4.1 / 4.2 #68649. It also changes ADBC, Arrow stream load and
paimon-rust reads.
- Child values hidden by a NULL list, fixed-size-list, map or struct row
are not decoded: the array, map and struct SerDes give the element SerDe
a validity bitmap that also clears those slots
(`mask_arrow_children_hidden_by_null_parents()`). Arrow leaves them
unspecified, and Lance's v2.1 miniblock encoding keeps them, so a hidden
10000-01-01 under a NULL fixed-size list used to fail the DATE32 range
check above. Visible values are unchanged; this applies to ADBC, Arrow
stream load and paimon-rust as well. The masks come from ExecEnv's
tracked Arrow pool. A nested Lance array that is compacted before
decoding is masked the same way, at every level, before its
`ValidateFull()`, so a hidden invalid UTF-8 string, out-of-precision
decimal, Date64 or time no longer fails the batch while a visible one
still does. Since #69697 the same masks keep hidden DATETIMEV2 /
TIMESTAMPTZ values out of its range checks. A mask costs at most one
pass per batch: a NULL row that owns no child slots, or only NULL ones,
allocates nothing, and Arrow stream load masks each retained IPC batch
once (`mask_arrow_descendants_not_visible()`) before it hands row ranges
to the SerDes, so its per-block reads allocate nothing. The one
per-block mask left is for map keys under a NULL map row that still owns
entries, since a MapArray cannot hold NULL keys; Arrow's map builder
writes NULL map rows without entries.
- Core: `to_protobuf` keeps `NULL_TYPE` for a NULL literal type, so a
Null leaf in ARRAY / MAP / STRUCT no longer comes back as BOOLEAN. The
nullable serde reads an Arrow Null array as NULLs without touching
buffers that the array does not have.
- Configs, caches and metrics, with the 4.x names and defaults:

  | Config | Default |
  |---|---|
  | `lance_index_cache_size_bytes` | 10 GiB |
  | `lance_metadata_cache_size_bytes` | 1 GiB |
  | `enable_lance_data_cache` | true |
  | `lance_data_cache_path` | `${DORIS_HOME}/lance_data_cache` |
  | `lance_data_cache_disk_capacity_bytes` | 100 GiB |
  | `lance_data_cache_read_block_size_bytes` | 1 MiB |
  | `lance_io_buffer_size_bytes` (mutable) | 2 GiB |
  | `lance_batch_readahead` (mutable) | 5 |
  | `lance_fragment_readahead` (mutable) | 5 |

The BE-wide Lance session (index cache, metadata cache and the optional
Foyer disk cache) is created by the first Lance scan, so a BE that never
reads Lance creates no cache directory. The first six configs take
effect after a restart. A negative cache size (and, while the data cache
is enabled, a negative disk capacity or read block size) fails the first
Lance scan with a message that names the setting, as does any data-cache
value lance-c rejects; the rest of the BE keeps serving. Ten
`lance_session_*` metrics report the two in-memory caches. With the
defaults, the disk tier keeps 51,200 sparse 2 MiB block files open from
the first Lance scan on (100 GiB in blocks of twice the read block
size), the same as on branch-4.1 / 4.2. Size
`lance_data_cache_disk_capacity_bytes` and
`lance_data_cache_read_block_size_bytes` against the BE's open-file
limit, or set `enable_lance_data_cache = false`.

The index and metadata caches live in lance-c memory: they are not
charged to a MemTracker or registered with CacheManager, so memory GC,
the cache capacity adjustment under memory pressure and
`/api/clear_cache/all` do not shrink them, and `enable_lance_data_cache
= false` does not turn them off. lance-c has no call to clear, evict or
resize them, so size them with the two configs. The disk tier keys
blocks by Lance's store prefix (`scheme$bucket` for S3, GCS and OSS,
without the endpoint) and the object path
(`thirdparty/patches/lance-c-foyer.patch`). Several endpoints that serve
the same bucket name and data-file path with different bytes (data files
registered under chosen names or replaced in place; Lance's writers use
random file names) share entries, also across a BE restart; such a
deployment should set `enable_lance_data_cache = false`. Both are as on
branch-4.1 / 4.2 and tracked as follow-ups.

A Lance scan's native working set, that is the I/O buffered up to
`lance_io_buffer_size_bytes` per scanner, the decode readahead and the
operators of a search plan such as the top-k of a flat vector search, is
allocated inside lance-c and is charged to neither the query nor its
workload group. It counts toward the BE process memory only, so memory
GC, which ranks queries by tracked memory, does not attribute it to the
Lance query. lance-c accepts no memory budget, so on BEs with little
memory headroom size `lance_io_buffer_size_bytes`,
`lance_batch_readahead` and `lance_fragment_readahead` for the number of
concurrent Lance scanners. As on branch-4.1 / 4.2; tracked as a
follow-up.
- Build: macOS links `-framework IOKit`. This is the first time objects
of `liblance_c.a` are pulled into `doris_be`, so the binary and the link
time grow. With this PR `util/rle_encoding.h` reaches 231 TUs and
`util/threadpool.h` 589, the same as master `315c2e00331` alone (limits
275 and 660 since #69717), and `gen_cpp/internal_service.pb.h` 562
(master alone 559, limit 660); the reach budgets are unchanged.

Not included: the Lance index metadata scanner (`LANCE_INDEX_ENTRIES`);
the FE side, including the rolling-upgrade guard that keeps Lance lazy
materialization off while an old BE could decode a version 1 row
location as version 0; the regression suites; the rest of #68707
(searching historical versions, tags and branches, rejecting fragment
ids missing from the snapshot, naming the snapshot in errors and the
profile). The `InitReaderAvgTime` / `GetBlockAvgTime` profile fix that
#67005 carried goes in a separate PR.

**Results.**

| | master | with this PR |
|---|---|---|
| A `FORMAT_LANCE` query | fails before reading anything | read by
`LanceTableReader` in FileScannerV2 |
| A Lance row in the second phase of TopN lazy materialization | cannot
be represented | a version 1 row location, read back with `take_rows` |
| A row location of an unknown version or size in phase two | decoded as
version 0 | rejected with an error |
| An Arrow DATE32 day outside 0000-01-01..9999-12-31 (ADBC, Arrow stream
load, paimon-rust, Lance) | read as a non-null `0000-00-00` | the read
fails with `InvalidArgument` |

BE unit tests (ASAN, macOS arm64, against master's pinned lance-c), on
the head rebased onto master `378404eeb12`: 1029 cases in 54 suites run,
1027 pass and 2 Parquet microbenchmarks skip by design. They cover the
Lance suites (107: `LanceTableReader*` 83, `LanceNestedNullTest` 10,
`LanceRuntimeFilterHelperTest` 10, `LanceSessionManagerTest` 4); the
row-location protocol (`MaterializationSharedStateTest`,
`RowIdStorageReaderTest`, and the new `RowIdFetchRequestValidationTest`:
count mismatch, unknown version, a Lance version on an internal mapping,
an internal row id of 2^32); `FileScannerV2Test` (28); the ORC and
Parquet reader suites (`NewOrcReaderTest`, `NewParquetReaderTest`,
`ParquetScanTest` and the other `Parquet*` suites), whose row ids now go
through `file_local`; and the Arrow / protobuf / DATE serde suites with
the new `ArrowParentNullsTest`, plus the Iceberg and Parquet reader
suites that decode Arrow dates. Reverting any of the review fixes, the
#68813 flag, the #68707 split rule or the #68815 bounds makes its new
tests fail. The sixth review round adds 13 cases (`ArrowParentNullsTest`
10, `LanceTableReaderTypeTest` 3) and extends one; with them the Arrow
SerDe, Lance, ADBC and `IcebergV2ReaderTest` suites run 317 cases: 310
pass and the 7 ADBC SQLite cases skip without a local SQLite driver. The
ninth review round adds
`LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly` and
`LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`:
both pass with the 106 Lance cases (`Lance*`) on the branch rebased onto
`339e087b11f`, and restoring `parse_int64()` in either check makes
exactly these two fail. On the head rebased onto master `315c2e00331`,
with the per-block mask fix (`42cf4e3e5d1`), 328 cases in 24 suites
pass: the Arrow SerDe and validation suites, `ArrowParentNullsTest`
(15), `LoadArrowPipeTest` (47), the DATETIMEV2 / TIMESTAMPTZ SerDe
suites, `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`,
`AdbcReaderTest`, `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest`
and the Lance suites. Allocating at the first NULL row of each call
again fails the new `ArrowParentNullsTest` case and the four new
`LoadArrowPipeTest` block cases, dropping the per-batch mask fails the
LIST and STRUCT block cases, and returning the child unmasked fails the
new timestamp case, four `ArrowParentNullsTest` cases and
`LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`.

### Release note

Add nine `lance_*` BE configs and `lance_session_*` BE metrics. They
take effect once the FE can plan Lance scans. Reading an Arrow DATE32
value outside 0000-01-01..9999-12-31 through ADBC, Arrow stream load or
paimon-rust now fails instead of returning 0000-00-00.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [x] Unit Test
- `run-be-ut.sh --run --filter=...` with the suites listed under
Results: 420 of 420 pass on macOS arm64 (ASAN) on the content before the
rebases, and the Lance and Arrow SerDe suites pass again with the
nested-offset, zero-field struct, hidden-child, hidden-Decimal256 and
compaction-mask fixes (the ADBC SQLite cases skip without a local SQLite
driver). Rebased onto master `43f9ceb48d7`: #69668 conflicted in the
scanner selection and in `fetch_table_schema`, which now hands every
file format to `format::fetch_file_schema()`, so the Lance branch runs
ahead of that call; #69646 added a TopN row-group test that reads the
version 0 row location fields, which now go through `file_local` (the
earlier rebase onto `5caaeb3e3d4` resolved #69614 and #69601 in the
scanner selection, the `FileFormat` enum, its test and the reach budget
table). The 95 TUs that this PR changes or that master changed since
`5caaeb3e3d4` were syntax-checked against the rebased tree with the UT
build flags, and CI runs the suites again. Rebased again onto master
`8d88f1cc6f7`: the only conflict was the budget table, where #65858
moved the `util/rle_encoding.h` baseline to 828, and the 163 TUs that
this PR changes or that master changed since `43f9ceb48d7` pass the same
syntax check. Rebased again onto master `339e087b11f`: the only conflict
was the budget table again, where #67292 moved the `util/rle_encoding.h`
baseline to 834; #69872 added Arrow type checks ahead of the NULL-parent
masking in the map and struct readers and merged cleanly. Of the 114
sources that this PR changes or that master changed since `8d88f1cc6f7`,
111 pass the same syntax check. The other three are master's: two new
#67292 files under `storage/segment` fail the same way on master alone,
since they compile only inside their unity batch, and this PR adds no
source to that target; the #69818 benchmark is not in the UT build and
includes only `util/simd` headers that this PR does not touch. Reverting
any of the review fixes, the #68813 flag, the #68707 split rule or the
#68815 bounds makes its new tests fail. Rebased again onto master
`378404eeb12`, which removed the legacy file scanner (#69723): the
conflicts were in the scanner-selection code and tests that #69723
deleted, so this PR's Lance lines there went with them (including its
`LanceQueriesAlwaysUseScannerV2` test), in the profile keys of
`materialization_opertor.cpp`, and in the row fetch of
`rowid_fetcher.cpp`, whose non-Lance path is now only
`FileScannerV2::read_by_rows`; the deleted V1 ORC / Parquet row id tests
took this PR's `file_local` rename with them, and the
`util/rle_encoding.h` budget commit is dropped because master is no
longer near that limit. Of the 100 sources that this PR changes or that
master changed since `339e087b11f`, 99 pass the same syntax check; the
other, `format_v2/orc/orc_reader.cpp`, fails the same way on master
alone. On the rebased head the suites listed under Results run 1029
cases: 1027 pass and 2 microbenchmarks skip. The test-utility fix that
`DataTypeDateTimeV2SerDeTest` needed on macOS (#69695) is now on master.
Rebased again onto master `315c2e00331`: the only conflict was
`TExternalSearchRequest` in `PlanNodes.thrift`, where #70075 had
reserved fields 2-5 in comments; the two commits that add the fields
replace those comments, and field 6 stays #70075's `paimon_options`.
#69717 cut transitive includes from shared headers: of the 160 sources
that this PR changes or that master changed since `378404eeb12`, 156
pass the same syntax check; the two `jemalloc_hook` tests have no
compile command on macOS, `runtime/memory/jemalloc_hook.cpp` uses glibc
aliases that macOS cannot build (#69819), and
`format_v2/table_reader.cpp` alone fails the same way on master alone
while its unity batch compiles. The full `doris_be_test` builds and
links on the rebased head; `ArrowParentNulls*` and `Lance*` (121 cases)
pass before the mask fix, and the 328 cases above with it.
- `build-support/check-build-hygiene.sh` (on this branch rebased onto
master `315c2e00331`), clang-format 16 and
`build-support/run-clang-tidy.sh` on the changed files: all pass; on the
changed lines clang-tidy (llvm 22) reports only
`readability-inconsistent-ifelse-braces` on the `DORIS_CHECK` lines of
the masking functions, the macro's expansion that master's existing
`DORIS_CHECK`s report too.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- Behavior changed:
    - [ ] No.
- [x] Yes. `PRequestBlockDesc.row_id` is `uint64`; phase two of TopN
lazy materialization rejects row locations of an unknown version or
size; `FORMAT_LANCE` scans run on FileScannerV2; an Arrow DATE32 day
outside the Doris DATE range fails the read instead of becoming
`0000-00-00`.
- Does this need documentation?
    - [x] No. Lance is documented with the FE connector.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: zhangstar333 <[email protected]>
Co-authored-by: Gabriel <[email protected]>
Co-authored-by: zhaobo wang <[email protected]>
Co-authored-by: jay <[email protected]>
Co-authored-by: FANNG <[email protected]>
Co-authored-by: Claude Opus 5.5 (1M context) <[email protected]>
Co-authored-by: zy-kkk <[email protected]>

…t_generation (#70308)

### Background

An auto-partition table with `partition.retention_count = N` keeps only
its N latest historical partitions: the FE's `DynamicPartitionScheduler`
drops older ones on every run, once per
`dynamic_partition_check_interval_seconds` (600 by default).
`partition_p0/auto_partition/test_auto_new_recycle` lowers that interval
to 1 s and checks the partition count after each load or ALTER.
`audit/test_sql_digest_generation` checks that the audit log gets an
`sql_digest` only for queries slower than the FE config
`sql_digest_generation_threshold_ms`, once with the threshold at 0 and
once at 100000. Both suites are in group `nonConcurrent`.

### What problem does this PR solve?

Issue Number: None
Related PR: #65375 and #65646 (the branch-4.1 recycle-suite fixes by
@shuke987, ported here), #65343 (the branch-4.1 digest-suite fix by
@shuke987, ported here), #65435 (master's earlier move of the recycle
suite's day-boundary guard)
PR Type: Test

Problem Summary:

Both suites rely on timing the product does not promise. The recycle
suite checks the partition count after a fixed `sleep(8000)` (5 s after
an ALTER), while the drop happens asynchronously at an unbounded time
after the config change; branch-4.1's NonConcurrent pipeline failed at
the first such check repeatedly (#65375). The digest suite's threshold-0
check needs the query's elapsed time to be above 0 ms. This PR gives
both suites their branch-4.1/4.2 form: the recycle suite polls the
partition count every second, for at most 1200 s, and widens its
day-boundary guard to cover the poll; the digest query adds `sleep(1)`.
Both files become identical to branch-4.2.

### Details

- **Failure:** On master's public NonConcurrent pipeline neither suite
has failed on its own since mid-July: of about 8,000 runs each, the
recycle suite failed 3 times and the digest suite 87 times, all in
builds whose cluster had died (most suites failing). The failures that
motivated the fixes are on branch-4.1's pipeline (#65375); #65343 made
the digest query deterministic there.
- **Cause:** `ADMIN SET FRONTEND CONFIG` sets the scheduler's new
interval at once, but the scheduler sleeps in 5-second slices and
notices the change only at the next slice. It then runs
`executeDynamicPartition` over every registered table before the drop.
So the drop lands 0 to 5 s plus one full pass after the config change,
and a pass longer than about 3 s fails the 8-second check. In the digest
suite, `AuditLogHelper` writes a digest only when `elapseMs >
sql_digest_generation_threshold_ms`, and nothing bounds the elapsed time
of `select *` on a two-row table from below (9 to 15 ms locally). With
`sleep(1)` on each row it takes about 2 s, still far below the second
check's 100000 ms.
- **Stability evidence:** The recycle checks now wait for the drop and
fail only when the count misses the target for 1200 s. The poll runs on
the suite thread, because the framework sets
`Awaitility.pollInSameThread()` (#68338), so it does not leak
connections. Shortening one of master's waits to 1 s reproduces the
branch-4.1 failure shape, and the ported suite passes; mutants show that
both suites still catch wrong results (see Testing).

### Testing

All runs used `./run-regression-test.sh --run -d
partition_p0/auto_partition -s test_auto_new_recycle` or `--run -d audit
-s test_sql_digest_generation` against a local single-FE, single-BE
cluster built from master.

| Suite | Version | Result |
|---|---|---|
| `test_auto_new_recycle` | master | passes |
| `test_auto_new_recycle` | this PR | passes; the polls reached the
target after 1, 2, 1, 3 and 3 s |
| `test_auto_new_recycle` | master, with the wait before the
10-partition check cut to 1 s | fails as #65375 describes: `expected:
<101> but was: <10>` (master's `assertEquals` has its arguments swapped,
so 101 partitions were still there) |
| `test_auto_new_recycle` | this PR, expecting 11 partitions, poll
capped at 20 s | fails as it should: `ConditionTimeoutException` |
| `test_sql_digest_generation` | master | passes |
| `test_sql_digest_generation` | this PR | passes; the query took 2,024
and 2,025 ms |
| `test_sql_digest_generation` | this PR, second threshold 1000 ms |
fails as it should: `Count of sql_digest changed` |

The PR NonConcurrent pipeline runs both suites; cloud_p0 skips
`nonConcurrent` suites.

### Notes for Reviewers

- The day-boundary guard before the final `now()`-based section goes
from 100 s to 1300 s, as in #65646, so that it covers the 1200 s poll
plus the insert and checks. It sleeps only when that section would
otherwise start within 1300 s of midnight.
- Four more suites fixed in the same branch-4.1 batch need nothing on
master: #65340 (master #68449 enables the publish debug point on all
FEs, superseding the open port #65556), #65929 (master #68329 fixed the
actual cause, a connection leak), #65885 (master #69723 removed the V1
scanner whose message it also accepted), and #67363 (ported as #67402).

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: the port (both suites taken from branch-4.2), the local
verification runs, and this description
- Human review: not stated
- Prompts: The author asked to continue moving branch-4.2's changes back
to master, where six branch-4.1 test fixes were recorded as one task.
After the check showed that master already covers four of them and has
not failed on the other two, the author chose to port those two exactly
as branch-4.2 has them and to verify them on a local cluster.

Co-authored-by: shuke <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>