[fix](function) Require constant mode arguments for encryption functions - #1

Problem Summary: A review that fails with `refresh_token_reused` leaves
its credentials eligible because the runner only records usage limits.
Later reviews can repeatedly select the same unusable credentials.

Record explicit reuse failures in a separate OSS object keyed by the
selected refresh token's SHA-256, then check that marker before
selecting a downloaded auth snapshot. A replacement refresh token
restores eligibility; metadata-only changes do not. Separate objects
keep late failures and usage-context writes from overwriting another
token's quarantine state. Marker read/write failures are surfaced
explicitly.

The marker applies to the selected token version. Runs that already
selected credentials before the marker was written can still fail. This
does not add credential leasing or change auth writeback.

…ine (apache#67490)

When FE local shuffle planning inserted a local exchange into a pipeline
whose parent pipeline was serial, it could increase only the downstream
pipeline task count while the paired sink and source operators remained
one-to-one. This produced inconsistent pipeline concurrency and could
fail during execution. Root cause: FE did not propagate the serial state
of a parent pipeline across pipeline boundaries. This change tracks that
state and skips local exchange insertion when the parent pipeline is
serial, matching the existing BE planning constraint.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…pache#67508)

### What problem does this PR solve?

Issue Number: close apache#67091

Related PR: None

Problem Summary: A BE restart indexed row-binlog rowsets only by
transaction ID, so multiple tablet pairs in one transaction could attach
the wrong companion rowset. Persist each base tablet's companion ID and
recover using both transaction and tablet IDs.

### Release note

Fix incorrect row-binlog companion recovery after a BE restart for
multi-tablet transactions.

### Check List (For Author)

- Test: Unit Test
- Added and ran
GroupRowsetBuilderTest.recoverMultipleRowBinlogPairsInOneTxn and
GroupRowsetBuilderTest.*
- Behavior changed: Yes. BE restart recovery now attaches each base
rowset to its persisted companion tablet.
- Does this need documentation: No

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

The connector metastore SPI refactor removed DLF provider registration
and the plugin-local DLF client path, so existing Iceberg and Paimon
catalogs using `dlf` can no longer be created or replayed on master.

This PR restores DLF support for both connectors:

- Register engine-specific DLF metastore providers and restore legacy
property aliases, endpoint derivation, catalog-id fallback, and
sensitive credential handling.
- Restore the Iceberg DLF catalog and client pool, with catalog-scoped
client caching and OSS-backed `S3FileIO`.
- Route Paimon DLF through its Hive catalog using
`ProxyMetaStoreClient`, while preserving legacy alias-only OSS and
OSS-HDFS storage configurations.
- Bundle the matching Hive 3 and Hive 2 DLF clients into the existing
Iceberg/HMS and Paimon private shades, including their Thrift and Tea
namespace isolation.

### Release note

Restore Aliyun DLF metastore support for Iceberg and Paimon catalogs.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [x] Manual test (add detailed scripts or steps below)
- Targeted Iceberg, Paimon, metastore provider, property-binding, and
DLF catalog tests passed.
- The 19-module connector package build passed with tests skipped; FE
Checkstyle reported zero violations.
- Verified both generated private shade jars contain the DLF proxy
client and relocated Thrift and Tea classes.
- The full Iceberg connector suite was also attempted; 1 of 1,376 tests
failed in the unrelated
`IcebergWritePlanProviderTest.planMergePreservesExplicitlyEmptyReadAcrossConcurrentFirstAppend`
case. The focused rerun reproduces the same failure, and this PR does
not modify that code path.
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason

- Behavior changed:
    - [ ] No.
- [x] Yes. Iceberg and Paimon catalogs can use the DLF metastore backend
again.

- Does this need documentation?
    - [x] No. Existing DLF catalog properties and behavior are restored.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

Problem Summary:

When cloud warm-up jobs outnumber the configured active slots, the
scheduler can over-submit tasks because a job is recorded as active only
after its worker starts. The direct-handoff thread pool previously
discarded rejected tasks, and unordered map traversal could repeatedly
favor the same jobs, leaving other runnable jobs pending indefinitely.

This PR keeps the direct-handoff pool and makes scheduling bounded and
retryable:

- Scan runnable jobs once and keep only the jobs needed by the current
slots in a bounded priority heap.
- Reserve the active slot before submission and roll it back when
submission is rejected.
- Schedule the least-recently-run jobs first, with never-scheduled
one-time jobs taking the first turn.
- Surface pool rejection so the scheduler can retry the job in the next
cycle.
- Apply mutable scheduler concurrency and interval configuration without
requiring an FE restart.

### Release note

Fix cloud warm-up jobs remaining pending when the scheduler thread pool
is saturated.

…achable forward branch (apache#67572)

### What problem does this PR solve?

Problem Summary:

Two pieces of dead weight in the connect layer, found while reading the
Arrow Flight SQL and MySQL
paths side by side. No behavior change.

**1. `ConnectType` was declared twice.** Once in `ConnectContext` and
once in `ConnectProcessor`, so
a processor's `connectType` and its session's `connectType` were
unrelated types that only happened
to carry the same two constants — nothing stopped them from disagreeing,
and neither could be passed
where the other was expected. Keep the `ConnectContext` one, which is
the session's own property,
and let the processors import it. Every subclass (including the four
test-only ones) was resolving
the enum through inheritance, so they now import it explicitly.

**2. `FrontendServiceImpl.createForwardProcessor()` dispatched on an
outcome that cannot happen.** It
branched on the connect type of the forwarded context, but the context
comes from
`createForwardContext()` one line above, which builds it with `new
ConnectContext(null, true,
sessionId)` — and that constructor always sets `ConnectType.MYSQL`. So
the Arrow Flight branch and
the `unknown ConnectType` throw were unreachable: the master replays a
forwarded statement over a
`ProxyMysqlChannel` no matter which protocol the client speaks on the
origin FE. The MySQL processor
is now constructed directly at the one call site, with a comment
recording why that is the only
possibility.

Problem:
Peer reads create an uncached BRPC client for every request. When a peer
is unreachable, readers across tablets repeatedly connect to the same
address with the default 2-second connect timeout and 10 retries.

Impact:
An unavailable cache peer can repeatedly delay reads before
remote-storage fallback and generate unnecessary connection attempts and
warning logs.

Fix:
- Add a BE-wide, address-level circuit breaker shared by peer readers.
- Open the circuit after a configurable number of consecutive connection
or RPC failures (3 by default) for a configurable cooldown (30 seconds
by default).
- Reject reads while the circuit is open, allow one probe after
cooldown, and clear the failure state after a successful RPC.
- Make uncached BRPC connect timeout and retry count configurable at the
call site, and use a 200 ms timeout with no retries for peer reads.

Test:
Stop the peer test server and verify that four reads issue only three
RPC attempts, with the final read rejected by the circuit breaker.

### What problem does this PR solve?

Issue Number: ref apache#48203

Related PR: apache/doris-website#4102

Problem Summary:

Add the `ST_IsClosed` spatial scalar function.

For a valid LineString, the function returns:

* `true` when the first and last points are exactly equal
* `false` when the LineString is open
* `NULL` for a `NULL` input
* `NULL` for a non-LineString or invalid encoded geometry

The implementation uses exact S2 point equality without introducing a
tolerance.

This PR also adds Nereids registration and visitor support, FE and BE
unit tests, and a self-asserting SQL regression suite.

### Release note

Add the `ST_IsClosed` spatial function.

…7566)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#61762

Problem Summary:

The TPC-H SF100 MOR suite loads the same data twice into MOR and DUP
tables, then compares their results with `read_mor_as_dup_tables`
enabled. Automatic compaction can merge the MOR versions before the
comparison, leaving fewer rows than the DUP tables.

Add eight dedicated `*_mor_as_dup` tables with automatic compaction
disabled. Prepare them in the existing two-round `load.groovy` flow by
reusing the existing S3 import SQL with a distinct table name and load
label. Route the MOR-as-DUP queries to these tables, preserving the
shared MOR table definitions used by the normal TPC-H and
predicate-pushdown cases.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test
    - [x] No need to test or manual test. Explain why:
- [x] Other reason: Test fixture change. Both modified Groovy scripts
passed compilation with the existing regression framework's Groovy
compiler; all eight dedicated DDL schemas and import substitution
anchors were checked; shared MOR DDLs are unchanged; `git diff --check`
passed. The full SF100 suite was not run locally.
- Behavior changed:
    - [x] No. Test setup only.
    - [ ] Yes.
- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

)

Problem Summary:

Different BEs may generate different rowsets for the same transaction
and tablet.

After rowset A is committed, the temporary rowset key points to A while
the recycle rowset key for B still exists. The recycle key check ran
before comparing rowset IDs, so commit_rowset incorrectly treated tmp(A)
and recycle(B) as an invalid conflict.

Compare rowset IDs first and only check the tmp and recycle key conflict
when they belong to the same rowset. For different rowsets, return
ALREADY_EXISTED with rowset A.

### What problem does this PR solve?

Related PR: apache#67527, apache#67396

Problem Summary: apache#67527 added seven JUnit 4 `Assert` calls to
`ConfigTest` after apache#67396 migrated the class to JUnit 5, causing
`fe-common:testCompile` to fail with `cannot find symbol: Assert`. Use
the existing JUnit 5 `Assertions` import. Also migrate the scheduler
test introduced by the same PR to JUnit 5 assertions and lifecycle
annotations, consistent with the FE test migration.

### Release note

None

### Check List (For Author)

- Test: `./run-fe-ut.sh --run
org.apache.doris.common.ConfigTest,org.apache.doris.cloud.CacheHotspotManagerSchedulerTest`
completed with BUILD SUCCESS: ConfigTest 10/10 and
CacheHotspotManagerSchedulerTest 4/4 passed, with no failures, errors,
or skipped tests. Checkstyle passed for fe-common and fe-core with zero
violations; `git diff --check` passed.
- Behavior changed: No
- Does this need documentation: No

### What problem does this PR solve?

A background sender could finish all of its work and then crash while
destroying
`AddThreadMemTrackerConsumer`:

```text
AddThreadMemTrackerConsumer::~AddThreadMemTrackerConsumer()
VTabletWriter::_send_batch_process()
periodic_send_batch(void*)
bthread::TaskGroup::task_runner(...)
```

The core file shows that `thread_context()` returned `nullptr`; the
`ThreadMemTrackerMgr` and its consumer stack were not corrupted. The
original
`ThreadContext` was still present in the bthread key table, but it could
no longer be
retrieved:

```text
global btls_key:          { index = 0, version = 1 }
bthread key-table slot 0: { version = 0, ptr = valid ThreadContext }
```

`btls_key` was zero-initialized as `{0, 0}` and was not created until
`PInternalService` was constructed. The BE starts its Thrift backend
service before
constructing the BRPC internal service, so work can arrive during that
startup window.
A bthread could therefore store its context with `{0, 0}`. BRPC accepts
that value
before the real key is created because key slot 0 also starts at version
0.

Later, `bthread_key_create()` changed the global key to `{0, 1}`. A
lookup using the
new version did not match the existing version-0 entry and returned
`nullptr`, causing
the crash during scoped tracker cleanup.

This is a key-initialization race. It is not caused by a bthread
migrating between
worker pthreads or by `pthread_context_ptr_init` changing during
migration.

### What is changed?

- Move the bthread ThreadContext key and its deleter into the
`thread_context` module.
- Initialize the key with `std::call_once` before any
`ThreadLocalHandle` creates or
accesses a context. The BE's early `SCOPED_INIT_THREAD_CONTEXT()`
initializes it
before services start, while the lazy guard also protects other entry
points.
- Keep the key valid for the complete process lifetime instead of
deleting it with
`PInternalService`, so existing bthread contexts cannot be invalidated
by service
  lifetime.
- Revert the earlier context-pointer caching and lookup-order changes.
Those changes
  only masked the null lookup and did not fix the key-version mismatch.

The change does not alter memory-accounting semantics. It guarantees
that every
bthread context is stored and retrieved with the same valid key version.

)

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Iceberg V2 scans could retain a stale nested file projection after a
predicate on a missing child was demoted, shifting projected sibling
values by physical ordinal. Separately, ID-less migrated files fell back
to current-name mapping after the explicit default name mapping was
removed, exposing stale physical values instead of NULL.

This PR reapplies the finalized scan projection to column mappings and
requires authoritative name mapping for V2 ID-less file reads.

### Release note

Fix Iceberg V2 reads when a rejected nested predicate widens projection
and when migrated ID-less files lose their default name mapping.

### Check List (For Author)

- Test: Unit Test and Regression test
- All 82 `IcebergV2ReaderTest` tests and all 15 `ColumnMapperTest` tests
passed.
- The four equality-delete tests affected by the stricter ID-less
mapping invariant passed after explicitly mapping their projected result
carrier.
- Both new Iceberg external-table regressions passed: nested projection
alignment and migrated ID-less nested NULL semantics.
- The existing `test_gen_iceberg_by_api` suite passed while validating
the required-field error for an ID-less fixture without authoritative
name mapping.
  - ASAN BE build and FE build passed.
  - Build hygiene and clang-format 16 checks passed.
- Behavior changed: Yes. Iceberg V2 nested projections preserve field-ID
alignment, and migrated ID-less fields without default name mapping
materialize as NULL.
- Does this need documentation: No

…up (apache#67539)

### What problem does this PR solve?

`ExportMgr.removeOldExportJobs()` removes export jobs when the number of
stored jobs exceeds `Config.max_export_history_job_num`. The old
implementation removed the oldest jobs without checking their states, so
PENDING, EXPORTING, or IN_QUEUE jobs could also be deleted.

An active export job can still write a later `OP_EXPORT_UPDATE_STATE`
edit log. After FE restart or master transfer, journal replay then looks
up the job by ID in `exportIdToJob`. Because the cleanup task may have
already removed the job, the lookup returns null and replay fails with:

```
java.lang.NullPointerException: Cannot invoke "org.apache.doris.load.ExportJob.replayExportJobState(...)" because "job" is null
```

The cleanup also removed label mappings by label only. If an old
cancelled job and a newer active job used the same label, deleting the
old job could remove the newer job's label mapping and allow an invalid
duplicate export job to be created.

### How does this PR solve the problem?

- Only remove export jobs in `CANCELLED` or `FINISHED` states when
enforcing the maximum history-job limit.
- Keep active jobs even when their total number is greater than
`Config.max_export_history_job_num`, because active jobs may still
produce state-transition edit logs.
- Iterate through the sorted job list safely and stop when there are no
more removable candidates, avoiding an endless cleanup loop when all
jobs are active.
- Remove a label mapping only when both the label and the mapped job ID
match the job being deleted. This prevents an old job from deleting the
mapping owned by a newer job with the same label.

### Test

- Add coverage to verify that PENDING and EXPORTING jobs are retained
while terminal jobs are removed.
- Add coverage to verify that all active jobs are retained when the
history limit is smaller than the number of active jobs.
- Add coverage to verify that a running job keeps its label reservation
after an older job with the same label is cleaned up.

### What problem does this PR solve?

Issue Number: DORIS-28543

Related PR: None

Problem Summary:

`HttpFileReader` expands small reads to a 1 MiB read-ahead request. Near
EOF, the generated byte range can extend beyond the file size. Some
Range-capable object stores respond to that overlong range with HTTP 200
and the complete object, so Doris incorrectly reports that Range support
changed after it was successfully detected during open.

This change clamps speculative read-ahead to the known file boundary. It
also adds a local HTTP server unit test that reproduces the 206-at-open
followed by 200-on-overlong-range behavior.

### Release note

Fix HTTP range reads near EOF for servers that return the complete
object for ranges extending past EOF.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason

- Behavior changed:
    - [ ] No.
    - [x] Yes. HTTP read-ahead now respects the known EOF boundary.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…ch defaults (apache#67559)

### What problem does this PR solve?

Slow CDC control requests, including snapshot split discovery for
unevenly distributed keys, can exceed the default 60-second BE HTTP
timeout. Retrying split discovery can repeat expensive work before the
FE receives progress.

This PR changes only two defaults:

- Increase `request_cdc_client_timeout_ms` from `60000` to `120000` (120
seconds).
- Reduce `streaming_cdc_fetch_splits_batch_size` from `100` to `16` to
return smaller batches on batched split-discovery paths.

…hing the MySQL channel (apache#67569)

### What problem does this PR solve?

Related PR: apache#61050

Problem Summary:

**1. A Flight session could not forward a statement at all.**

An Arrow Flight SQL session has no `MysqlChannel`:
`FlightSqlConnectContext` overrides
`getMysqlChannel()` to throw `getMysqlChannel not in mysql connection`.
Since apache#61050,
`FEOpExecutor.buildStmtForwardParams()` reads the client's
`CLIENT_DEPRECATE_EOF` capability
straight off that channel, so on a multi-FE deployment every statement a
Flight connection has to
forward to the master fails with that message before the request is even
sent:

- any DDL issued over Arrow Flight SQL to a follower or observer FE;
- any statement at all when `force_forward_all_queries` is on.

`CLIENT_DEPRECATE_EOF` is a MySQL protocol capability, so only read it
for a MySQL connection. The
thrift field is `optional` with no IDL default, and leaving it unset for
other protocols puts the
master back on the packet layout it used before apache#61050 —
`ConnectProcessor` applies it only when
`isSetClientDeprecatedEOF() && isClientDeprecatedEOF()`, so unset is
equivalent to set-false for
every reader in every rolling-upgrade direction.

**2. Once the forward succeeds, nothing carries the master's answer back
to the Flight client.**

`ConnectProcessor.finalizeCommand()` is the only place a forwarded
statement's status and result set
are replayed, and it is MySQL-only: it opens with
`Preconditions.checkState(connectType.equals(ConnectType.MYSQL))`.
Nothing else reads
`getProxyStatusCode()` / `getShowResultSet()` / `getOutputPacket()`
except the audit log. So after
the RPC, `ctx.getState()` stays at the `OK` that `executeQuery()` set
with `reset()`, the
`FlightSqlChannel` stays empty, and `DorisFlightSqlProducer` answers
with `addOKResult()`'s
synthesized one-row `StatusResult = 0`. With a client connected over
Arrow Flight SQL to a follower
or observer:

- a DDL that **failed** on the master was reported to the client as
success — only the follower's
  audit event recorded the real error;
- a forwarded statement that returns rows (`SHOW FRONTENDS`, `SHOW
LOAD`, … — about 33
`ShowCommand` subclasses both forward and return rows) returned that
single `StatusResult` row
instead of the master's rows. The same statement run against the master
returns real rows, because
`StmtExecutor.sendResultSet()` already has a working Arrow Flight
branch.

This gap is older than apache#61050 (`finalizeCommand()` was already
MySQL-only before it), so fixing only
part 1 would have turned a loud, diagnosable error into a silent wrong
answer, which the
repository's "Error Means Failure" invariant does not allow. Part 2 is
therefore fixed here too:

- `ConnectProcessor.carryForwardedOutcomeToFlightSession()` is the Arrow
Flight counterpart of
`finalizeCommand()`'s forwarded-statement branch. It copies a non-zero
master status into
`ctx.getState()` (and logs it at WARN — the follower used to say
nothing), and otherwise replays
the master's result set through the existing `sendResultSet()` Arrow
Flight branch.
- It is called from the branch of `executeQuery()` that is **already**
scoped to
`ARROW_FLIGHT_SQL`, so it structurally cannot fire on a MySQL connection
and
  `finalizeCommand()`'s `getStateType() != ERR` condition is untouched.
- `StmtExecutor` now refuses to forward a *query* on an Arrow Flight SQL
connection, before the RPC.
A query result comes back as MySQL wire packets in
`TMasterOpResult.queryResultBufList`, which
only `finalizeCommand()` can replay and which cannot be converted to
Arrow batches; answering it
with a synthesized empty success would be the same silent wrong answer.
This is reachable only
under `force_forward_all_queries`, which `Config` documents as "For
testing purposes".

The MySQL path is byte-for-byte unchanged.

…#67454)

Problem Summary:

`CloudSchemaChangeJob::_convert_historical_rowsets()` builds its
`RowsetWriterContext` with `write_file_cache` but never sets
`file_cache_ttl_sec`. For a table with `file_cache_ttl_seconds`, the
schema change output is therefore written into the NORMAL/INDEX file
cache queues instead of the TTL queue. The other two cloud write paths
both set it:

- load: `CloudRowsetBuilder::init()` — `context.file_cache_ttl_sec =
_tablet->ttl_seconds();`
- compaction: `CloudCompactionMixin::construct_output_rowset_writer()` —
`ctx.file_cache_ttl_sec = _tablet->ttl_seconds();`

…67663)

Problem Summary:

FE/BE startup and shutdown scripts silently skip the final environment
assignment in `fe.conf` or `be.conf` when the file has no trailing
newline. For example, a final `PID_DIR=/custom/pid` is ignored, so the
script uses the default or an earlier PID directory.

Bash `read` sets `line` but returns failure when EOF terminates a
partial line. Keep each of the four configuration loops running when
that final line is nonempty so the existing parser exports it.

### Release note

Fix FE/BE startup and shutdown scripts ignoring the last configuration
assignment when the file has no trailing newline.

…pache#67664)

Problem Summary: Building the Doris compose image fails during apt-get
update because openjdk:17.0.1-jdk-slim is based on Debian 11. Bullseye
LTS ended on August 31, 2026, and its security repository metadata
subsequently expired. The same expired metadata is served by both the
Aliyun mirror and Debian upstream, so switching mirror hosts does not
resolve the failure.

Use bellsoft/liberica-openjdk-debian:17, currently based on Debian 12,
while preserving the JDK_IMAGE build argument and the base image's Java
environment. Update mirror rewriting to support both legacy .list and
deb822 .sources files, and install python-is-python3 because Debian 12
does not provide the old python package. Document the new default image.

### Release note

The Doris compose image now uses Debian 12 with BellSoft Liberica JDK
17. The python command now runs Python 3.

### Check List (For Author)

- Test: Manual test
- Build the complete Docker image using existing local output artifacts.
    - Run docker build --check: no warnings.
- Verify mirror rewriting for both legacy .list and deb822 .sources
files.
- Verify Debian 12, Java/javac 17, JAVA_HOME/libjvm, Python, MySQL
client, and JaCoCo agent loading in the built image.
- Run FE start_fe.sh --version and BE doris_be --version successfully.
    - Run git diff --check.
- No kernel rebuild or full cluster regression: this changes image
packaging.
- Behavior changed: Yes; update the base OS/JDK distribution and use
Python 3.
- Does this need documentation: Yes; update the Doris compose README in
this commit.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [x] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…7584)

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

`sequence(2147483646, 2147483647, 2)` and its `array_range` equivalent
should return `[2147483646]`. The current Int32 generator overflows when
incrementing past the last element and can keep appending instead of
terminating. The wider preflight size calculation does not protect the
Int32 generation loop.

Use Int64 distance/cursor arithmetic, check the exact element count
before allocation, resize the nested output once per multi-element row,
and fill it with a counted loop. Initialize the element null map once
per block. Empty and singleton fast paths avoid division and
general-fill overhead for short ranges. The Int32 return type, exclusive
upper bound, invalid-input NULLs and array-size limit remain unchanged.

Four BE unit tests cover both function names, all constant/vector
argument combinations, overflow boundaries, default arguments,
NULL/invalid inputs, mixed growing/empty/NULL rows, and the exact size
limit and limit+1.

### Performance

Function-level Google Benchmark calls through the real function factory,
including output allocation. Both binaries use the same benchmark and
master baseline `eea19b3f3cfef9e1bbbd559f9ea42954d8891e0f`; only the
generator implementation differs.

Intel Xeon Platinum 8457C, Clang 20.1.8, Release `-O3`, AVX2, pinned to
CPU 24. Each case uses non-constant input columns, `start = row % 17`,
and the length/step below. Run order: before, after, after, before; 7
repetitions per run, minimum 0.3 s measurement and 0.2 s warmup. Values
are the median CPU time across 14 samples per case.

| Input rows | Elements per array | Step | Before (µs/batch) | After
(µs/batch) | Speedup |
|---:|---:|---:|---:|---:|---:|
| 4096 | 0 | 1 | 9.34 | 9.27 | 1.01× |
| 4096 | 1 | 1 | 15.64 | 11.37 | 1.38× |
| 4096 | 16 | 1 | 119.95 | 32.01 | 3.75× |
| 4096 | 256 | 1 | 2737.57 | 931.27 | 2.94× |
| 4096 | 1024 | 1 | 20430.45 | 3763.61 | 5.43× |
| 4096 | 256 | 7 | 2615.14 | 942.04 | 2.78× |
| 1 | 1000000 | 1 | 1973.84 | 214.92 | 9.18× |

This is a shared host with CPU scaling enabled, so the empty-array
result should be treated as unchanged. These are BE function timings,
not end-to-end SQL speedups. Raw samples show higher variation for the
large multi-row allocation cases; the reported values are medians.

Reproduce with the registered `BM_ArrayRange` benchmark:

```bash
BUILD_TYPE=Release ./build.sh --benchmark -j 48
taskset -c 24 be/build_Release/bin/benchmark_test \
  --benchmark_filter=BM_ArrayRange \
  --benchmark_min_time=0.3s --benchmark_min_warmup_time=0.2 \
  --benchmark_repetitions=7 --benchmark_out=results.json \
  --benchmark_out_format=json
```

### Release note

Fix integer overflow in `sequence`/`array_range` near INT32_MAX and
accelerate integer range generation with batched output filling.

### Check List (For Author)

- Test
    - [ ] Regression test
- [x] Unit Test: `./run-be-ut.sh --run
--filter='FunctionArrayRangeTest.*' -j 48` — all 4 ASAN tests passed and
the process exited normally.
    - [x] Manual test: actual-function Release benchmarks above.
- Behavior changed:
    - [ ] No.
- [x] Yes: valid ranges whose final increment exceeds INT32_MAX now
terminate with the expected array.
- Does this need documentation?
    - [x] No.
    - [ ] Yes.

Validation notes:
- clang-format 16 and build-hygiene checks passed.
- The Release benchmark compiled, linked, installed and ran
successfully. The build script's subsequent generic packaging step fails
because benchmark-only builds do not populate `be/output/bin/*`.
- The repository clang-tidy script was attempted but did not pass
because of the existing unmatched `NOLINTEND` in `be/src/core/types.h`
and signed comparisons in unchanged array-range code. No unrelated fixes
are included.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66337

Problem Summary: Schema change compaction cases created only five
post-alter versions. After the NOTREADY tablet policy started protecting
its latest ten versions, no rowsets remained eligible for cumulative
compaction, so the cases expected a merge that could no longer occur.
The inverted-index cumulative compaction case also assumed that the
first trigger always found input and that an initial run_status=false
meant completion, which races the BE visible-version refresh.

Create enough double-write versions to exercise both the eligible and
protected ranges, assert the resulting active rowset topology in the
no-restart, FE-restart, and BE-restart variants, and use completion
checks that cannot finish before compaction starts. For the
inverted-index case, retry only the transient E-2000 response while
forcing tablet reports and wait for observable rowset reduction.

Primary-key lookup copied each rowset’s segment key bounds into a
temporary vector before pruning candidate segments. This caused repeated
allocation and protobuf object copying on the lookup hot path, with
overhead increasing alongside the number of segments.

Co-authored-by: yangtao555 <[email protected]>

…che#67533)

### What problem does this PR solve?

Issue Number: close apache#67094

### What problem does this PR solve?

Issue Number: None

Related PR: apache#64580

Problem Summary: INSERT OVERWRITE auto-detect must fail when source rows
do not match an existing target partition and automatic partition
creation is disabled. With multiple parallel sink instances, several
equivalent failure messages can race to reach the client. The regression
test accepted only one message, making repeated executions flaky. Accept
the known equivalent errors while still requiring every statement to
fail.

### Release note

None

### Check List (For Author)

- Test: Regression test updated; not run locally as requested, CI will
execute it
- Behavior changed: No
- Does this need documentation: No

…67685)

### What problem does this PR solve?

Problem Summary:

The `errorCapturingIdentifier` rule always entered a nullable helper for
valid identifiers. In the tracked SQL corpus this created 25,406 empty
CST contexts. Move the optionality to the caller and make
`errorCapturingIdentifierExtra` non-empty, while preserving the existing
unquoted-identifier error action. This avoids allocating an empty
`RealIdentContext` on the common path and keeps the grammar structure
straightforward.

This PR also removes stale `PostProcessor` references from two JMH
harnesses. Identifier post-processing already moved into grammar actions
and the class no longer exists, so those references prevented the
benchmark profile from compiling.

Add a multipart-identifier JMH workload and a direct CST regression
test. Across two JMH runs, normalized allocation for that workload
decreased from 174,368.9 to 159,004.1 B/op (-8.81%) end-to-end and from
146,097.7 to 130,741.3 B/op (-10.51%) with pre-tokenized input.

…7666)

### What problem does this PR solve?

Problem Summary:

DISTINCT window rewrites for SUM and GROUP_CONCAT manually reconstructed
their multi-distinct functions with the default non-nullable flag. For
empty window frames over NOT NULL inputs, the backend consequently
materialized empty aggregate states (0 or an empty string) instead of
NULL.

This change reuses each aggregate function's canonical multi-distinct
conversion, preserving the always-nullable contract established during
window-function analysis. COUNT(DISTINCT) behavior remains unchanged.

### Release note

Fix SUM(DISTINCT ...) and GROUP_CONCAT(DISTINCT ...) window functions to
return NULL for empty frames.

update mtmv code owner

…ed fields (apache#65805)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65591, apache#66380

Problem Summary:

Nested-column pruning (feature series apache#59263 / apache#61888 / apache#64535) makes
Doris read only the referenced parts of a struct/map/array column. Two
kinds of special *meta* access paths serve that purpose: a terminal
`NULL` component means the query only needs the null flag (`IS [NOT]
NULL` on a column or nested field), and a terminal `OFFSET` component
means only offsets are needed (`length`/`cardinality`-style functions).
The BE satisfies them in `NULL_MAP_ONLY` / `OFFSET_ONLY` meta-read modes
that skip the payload data entirely. While the optimization itself is
sound, the way meta paths were expressed on the wire and handled on both
sides had several correctness holes:

* **Wire ambiguity.** Every access path was sent as a `DATA`-typed list
of string components, and `NULL`/`OFFSET` lived in the same list as real
child names. There was no way to tell a metadata selector apart from a
struct field literally named `OFFSET` or `NULL`, so such a field could
be swallowed as metadata and its data silently pruned away (wrong query
results), and there was no FE/BE version contract to evolve the encoding
during rolling upgrades.
* **FE null-map correctness.** A slot can be nullable for two different
reasons: its physical column has a null bitmap, or an outer join merely
made the output slot nullable. The old logic keyed on expression/slot
nullability, so a physical `NOT NULL` dimension column of a `LEFT JOIN`
received a `[col, NULL]` path and the BE aborted trying to read a null
map from a `NOT NULL` column. Meta paths were also generated for
non-nullable struct fields / non-null structs and for variant
sub-columns, which do not support meta-only reads.
* **FE pruning bookkeeping.** `NULL`/`OFFSET` were detected by string
suffix, so real fields with those names collided inside the pruning
logic too; and predicate meta paths were stripped or broadened
(map-star) whenever regular data paths existed, losing metadata-only
reads or over-reading payload data.
* **BE routing/read-mode logic.** A terminal `NULL`/`OFFSET` was treated
as current-level metadata at every container, meta-only modes could be
entered even when a sibling `DATA` path still required payload reads,
sibling `META` paths were dropped when unrelated data projections
existed, and Map descendant routing depended on physical key/value child
column names while silently ignoring unknown selectors.

What this PR does:

* **Versioned, type-selected access paths.** Adds an optional `version`
to `TColumnAccessPath` (thrift + protobuf + FE/BE descriptor conversion;
`TCOLUMN_ACCESS_PATH_VERSION_LEGACY = 0`,
`TCOLUMN_ACCESS_PATH_VERSION_TYPED = 1`). In the typed format the path
type is authoritative: `DATA` selects `data_access_path`, `META` selects
`meta_access_path`, and `NULL`/`OFFSET` are only ever emitted as typed
`META` paths. New BEs still decode the legacy all-`DATA` encoding from
old FEs, so the supported rolling upgrade is: upgrade all BEs first,
then let FEs send typed paths.
* **FE** (`AccessPathExpressionCollector`, `NestedColumnPruning`,
`AccessPathPlanCollector`, `DescriptorToThriftConverter`): marks
`NULL`/`OFFSET` collector contexts as `META`; checks the physical
column's nullability (`getOriginalColumn().isAllowNull`) instead of slot
nullability before emitting null-map paths, so outer-join-nullable `NOT
NULL` columns and non-nullable fields/structs no longer get `NULL`
paths; emits the parent-struct `NULL` path for `element_at(...) IS NULL`
only when actually needed; falls back to plain data reads for variant
sub-columns; keeps the exact meta path when a sibling data path is also
read; preserves fields referenced by unchanged predicates; and keys
pruning on the path type instead of string suffixes, rejecting
unsupported `META` paths.
* **BE** (`ColumnIterator` / `column_reader.cpp`): validates path
versions and type-selected payloads; partitions current data / current
metadata / descendant routing with explicit per-container ownership
(Struct owns the null map; Map/Array and string-like scalar columns own
both null map and offsets); derives `NULL_MAP_ONLY` / `OFFSET_ONLY` only
when no current or predicate data path requires payload, and marks data
children `SKIP` consistently; keeps sibling `META` paths explicit so an
unrelated `DATA` projection cannot disable a metadata-only predicate
read; routes Map children by the logical `KEYS`/`VALUES` selectors
(wildcard expansion, arbitrary nesting) independent of physical child
column names; rejects unrecognized Map selectors instead of silently
pruning everything; and documents the full routing flow.
* **Tests & docs.** BE `ColumnReaderTest` coverage for typed-vs-legacy
encoding, metadata-vs-same-named-data disambiguation, meta-read-mode
legality, sibling-data-path handling and Map selector validation;
`SlotDescriptor` protobuf round-trip of versions and payloads; FE
`PruneNestedColumnTest` / `DescriptorToThriftConverterTest` cases; the
`nereids_rules_p0/column_pruning` regression suites updated to assert
the typed `META` paths, plus the new `left_join_not_null_column` suite
guarding the outer-join null-map crash.

### Release note

None

### Check List (For Author)

- Test:
- FE unit test: `./run-fe-ut.sh --run
'org.apache.doris.nereids.rules.rewrite.PruneNestedColumnTest,org.apache.doris.analysis.DescriptorToThriftConverterTest'`
(74 passed on the final rebased FE tree)
- BE unit test: `./run-be-ut.sh --run
--filter='ColumnReaderTest.*:SlotDescriptorTest.AccessPathsPreservedThroughProtobuf'`
(50 passed before the final rebase; the final rebuild was blocked before
GTest by the missing master-added
`thirdparty/installed/lib64/liblance_c.a` in this environment)
- Regression test: `nereids_rules_p0/column_pruning` —
`string_length_column_pruning`, `null_column_pruning`,
`nested_container_offset_pruning`, `lambda_null_pruning`,
`left_join_not_null_column` (passed before the final rebase; the final
rerun was blocked by the same missing dependency)
    - Format checks: `build-support/check-format.sh`, `git diff --check`
- Behavior changed: Yes. Typed `META` access paths with an explicit
version replace the legacy all-`DATA` encoding between new FEs and BEs
(upgrade BEs before FEs); struct fields literally named `NULL`/`OFFSET`
are now read correctly instead of being pruned as metadata; `IS NULL` on
outer-join-nullable `NOT NULL` columns no longer crashes the BE; invalid
Map descendant selectors now return an internal error instead of being
silently ignored.
- Does this need documentation: No

---------

Co-authored-by: Hu Shenggang <[email protected]>

### What problem does this PR solve?

An Iceberg snapshot can reference a data file that has been removed from
storage. With the default
`ignore_not_found_file_in_external_table=true`, both file scanners can
skip the missing file and return a successful but incomplete result.

Exclude Iceberg ranges from the missing-file skip policy, regardless of
that setting, while preserving the existing behavior for other table
formats. Apply the shared policy at reader initialization and read
boundaries. This also prevents a missing delete file reported as
`NOT_FOUND` from being swallowed by the scanner.

Add unit coverage for Iceberg Parquet/ORC/JNI and ordinary external
ranges with both setting values. Add an external regression that creates
multiple data files, deletes one actual object, and expects row scans to
fail with V1/V2 and both setting values. The regression disables
file/result caches and restores the BE configuration after testing.

### Release note

Iceberg scans now fail when a required snapshot file is missing instead
of silently skipping it, including when
`ignore_not_found_file_in_external_table` is enabled.

### Check List (For Author)

- Test: The shared-policy GoogleTests passed in a standalone build with
actual generated Thrift types; the Iceberg assertion failed with the
extracted original policy. Groovy compilation, clang-format 16, and
whitespace checks passed. The full BE UT runner was blocked during local
clucene/Zstd dependency configuration. The external regression has not
been executed locally.
- Behavior changed: Yes; missing Iceberg files are no longer ignored.
Other table formats retain their existing behavior.
- Does this need documentation: No new configuration or API; the
corrected behavior is described above.

…oaded (apache#67317)

remove() bumped _queue_evict_size_metrics and _total_evict_size_metrics
unconditionally, while the physical delete happened only for DOWNLOADED
blocks. Removing an EMPTY or SKIP_CACHE block therefore charged a full
range().size() -- for an EMPTY block the entire preallocated block size
-- while freeing nothing on disk. That is why the reported eviction rate
ran far above read plus write traffic combined: 300MiB/s average against
70MiB/s of real traffic, with the difference coming from blocks created
and dropped without ever being downloaded.

Read the block state once, move both counters into the DOWNLOADED
branch, and add evict_not_downloaded_size / evict_not_downloaded_num so
that churn becomes measurable instead of hiding inside the eviction
rate. DOWNLOADING blocks are not counted here: they are only tagged for
deletion and get counted on the removal that actually frees them.

test_evict_metrics_only_count_downloaded_blocks covers both directions,
including the per-queue counter, which the total alone would not catch.
The evict_in_advance test now restores the two configs it changes, so it
cannot leave background eviction enabled for whatever runs after it
under --gtest_shuffle.

…68051)

Inverted-index evaluation for `is_ip_address_in_range` discarded the
source column's NULL bitmap. A `NOT` predicate could therefore treat
NULL rows as ordinary non-matching rows and return them incorrectly. The
evaluator now reads and propagates the index NULL bitmap so bitmap
complement evaluation preserves SQL three-valued logic.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…che#68150)

### What problem does this PR solve?

Two external regression cases can report empty result mismatches without
first confirming that the preceding operation succeeded.

In `test_iceberg_deletion_vector`, the Trino container is restarted
before the comparison query. The test previously retried the data query
while ignoring command failures, so a startup or query error with empty
stdout became a row mismatch. Wait for a successful `SELECT 1` with a
bounded startup deadline, then require the comparison command to succeed
before checking its rows. Report stdout and stderr on failure, and keep
successful empty or incorrect results as assertion failures.

In `test_s3_tvf_number_range`, the HDFS Broker Load waiter previously
returned normally after cancellation or roughly ten seconds of polling.
The caller could then query a freshly truncated table before loading
completed. Wait up to 120 seconds using a monotonic clock, return only
on `FINISHED`, and fail on cancellation or timeout with the job label
and last `SHOW LOAD` result.

### Release note

None. Regression-test changes only.

### Check List (For Author)

- Test: Both modified suites compile with Groovy 4.0.19; `git diff
--check` passes. A local harness evaluates the actual modified
control-flow blocks with simulated command/load responses and
virtualized waiting. All eight scenarios pass: delayed load completion,
cancellation, timeout, missing load job, delayed Trino readiness,
comparison-command failure, readiness timeout, and a successful query
returning no rows. The original implementation fails these checks. Full
external integration execution remains pending CI.
- Behavior changed: Test startup/load failures now report their
execution status before row comparison; expected data results are
unchanged.
- Does this need documentation: No.

…apache#68133)

### What problem does this PR solve?

1. `LIKE 'a_b'` is converted to `^a.b\z`, not `^a.b$`. Hyperscan follows
PCRE, where `$` also matches right before a newline that ends the value,
so `^a.b$` wrongly matched 'acb\n'. `\z` is the end of the value in both
Hyperscan and RE2.

2. `LIKE 'a_b%'` is converted to `^a.b`, not `^a.b.*\z`. A trailing `%`
matches anything, and so does appending nothing. The anchored form would
force `.*` to consume the value to its end, and `.*` only matches valid
UTF-8, so RE2 would not match 'acb\xff' while Hyperscan still would.

3. `LIKE 'a_*'` is converted to `^a.\*\z`, not `^a.\*`. The tail anchor
used to be skipped whenever the produced regex ended with `*`, a check
meant to detect the `.*` expanded from a trailing `%`. An escaped
literal `*` ends the regex with the same character, so the pattern lost
its anchor entirely and `'ab*xyz'` matched. The decision now comes from
the position in the pattern instead of from the produced string.


Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

Co-authored-by: Claude Opus 5 (1M context) <[email protected]>

…cks (apache#67313)

Problem Summary:
The disk-pressure protection mode is meant to enter at `enter` (85%) and
stay on until usage drops below `exit` (80%), holding its previous state
in between. That hold is implemented by `_disk_resource_limit_mode`
remembering the previous round — and `check_disk_resource_limit()` wiped
it before reading `statfs`:

```cpp
if (_capacity > _cur_cache_size) {
    _disk_resource_limit_mode = false;    // runs before statfs is read
}
...
} else if (_disk_resource_limit_mode && space < exit && inode < exit) {
```

With the memory gone, the exit branch is dead code: the mode dropped as
soon as usage fell below `enter`, never holding down to `exit`. The
`disk_resource_limit_mode` bvar was unusable for the same reason — a
single round wrote 0 and then 1 whenever the mode was immediately
re-entered, so it could not be used to tell whether a node was in the
mode.

### What this PR changes

1. Drop the pre-clear so the band holds, and publish
`_disk_limit_mode_metrics` once, after the decision.
2. Drop the `_disk_resource_limit_mode = true` that `reset_capacity()`
forces on when shrinking — see below.
3. Make `_disk_resource_limit_mode` and `_need_evict_cache_in_advance`
`std::atomic<bool>`. `run_background_monitor()` writes them without the
cache lock while `try_reserve()`, `is_overflow()` and
`run_background_gc()` read them, so both plain bools were already racy
on master.

### Why the `reset_capacity()` force goes too

Both lines are halves of a mechanism that no longer exists. They arrived
together in apache#37484, when shrinking was asynchronous:

```cpp
cell->is_deleted = true;            // tagged only, nothing freed here
...
_disk_resource_limit_mode = true;
_async_clear_file_cache = true;     // a background thread does the removal
```

`reset_capacity()` returned with `_cur_cache_size` still above the new
capacity, so the force made `try_reserve()` drain aggressively while the
background cleanup caught up, and `if (_capacity > _cur_cache_size)` was
the matching "drained, stop" condition.

Shrinking is synchronous now — `reset_capacity()` calls `remove()` under
the cache lock and returns at the new capacity, and
`_async_clear_file_cache` no longer exists in `be/src`. So the force has
nothing left to accelerate, while the pre-clear kept firing on its own
condition, which holds in ordinary operation whenever the cache is under
its budget, not only while a shrink drains. That is the bug.

Removing only the pre-clear would leave the force with nothing to turn
it off: a shrink on a node whose disk sits in `[exit, enter)` for
unrelated reasons — BE logs, another data dir on the same mount — would
pin the mode on permanently, and every reservation would evict five
times the requested size.

The force was also not capacity enforcement during async metadata load:

- The loader inserts restored cells through `add_cell()` and never calls
`try_reserve()` (`fs_file_cache_storage.cpp:818`, `:1007`, `:1179`).
- `try_reserve_during_async_load()` does not consult `_capacity`, and
cannot: while loading, `_cur_cache_size` is a partial count growing from
zero toward the restored total, so comparing it against the full budget
is meaningless. `statfs` is the only signal valid at that point, which
is why the mode is the only thing that function checks.

### Deliberately out of scope

Real, pre-existing on master, independent of the state machine touched
here, each getting its own PR:

- **Capacity admission during async metadata load.** Making it work
needs the loader to publish the on-disk total up front, or admission
deferred until loading completes — a change to the loader's publication
protocol, not an `if` in the reservation path.
- **The eviction target reused as the admitted size.** `try_reserve()`
does `size = 5 * size` under disk pressure and passes that to
`QueryFileCacheContext::reserve()`, while `add_cell()` creates a cell of
the original size, so `DCHECK(iter->size == cell_size)` at
`block_file_cache.cpp:1483` can fire and release builds mis-account
query usage. Already reachable on master whenever the disk is at or
above `enter` with `enable_file_cache_query_limit` on.
- **Republication over deleted files during load.**
`load_cache_info_into_memory_from_db()` buffers 10k `BatchLoadArgs`
outside `_mutex` (`fs_file_cache_storage.cpp:995`) and
`handle_already_loaded_block()` checks only whether a cell exists, never
whether the file is still on disk.

`next_day` and `previous_day` reused a mutated date value when the date
argument was constant, producing cumulative cross-row results. Nullable
inputs were also unpacked before execution, causing NULL weekday rows to
be parsed as empty strings and fail the entire query. The functions now
use `ColumnView` to handle constant and nullable columns directly,
compute each row from an independent date value, propagate NULL results,
and continue rejecting invalid non-NULL weekdays.


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…8146)

Problem Summary:

An empty regexp produces a zero-width match, so the expected result is
1.
The constant pattern was recognized as a substring pattern and routed to
the long-buffer substring optimization. However, this path treated an
empty needle as not found.

```sql
DROP TABLE IF EXISTS tmp_b8_c01;

CREATE TABLE tmp_b8_c01 (
    id INT,
    s STRING,
    p STRING
)
DUPLICATE KEY(id)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES("replication_num" = "1");

INSERT INTO tmp_b8_c01 VALUES (1, 'abc', '');

SELECT
    regexp('abc', '') AS const_empty,
    regexp(s, p) AS column_empty
FROM tmp_b8_c01
WHERE id = 1;

-- before:
+-------------+--------------+
| const_empty | column_empty |
+-------------+--------------+
|           0 |            1 |
+-------------+--------------+

-- now:
+-------------+--------------+
| const_empty | column_empty |
+-------------+--------------+
|           1 |            1 |
+-------------+--------------+
```


### What is changed?
- Handle an empty constant substring pattern as matching every non-null
input.
- Move the long-buffer substring implementation into
`constant_substring_fn`.
- Remove the separate execute_substring implementation and its special
dispatch logic.
- Preserve row-boundary checks to prevent matches across adjacent
ColumnString rows.

…nded incremental reads (apache#67820)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#67181, apache#67594

Problem Summary:

Bounded cloud `@incr` reads need a safe end timestamp. Waiting by
transaction ID can include transactions outside the requested time
window, while one slow transaction can hold a global committed prefix
and block unrelated tables. FE failover also needs to prevent an old
master or an uncertain commit RPC from later committing a TSO that FE
has already treated as finished.

This PR introduces a durable committed TSO and keeps the active
commit-attempt set in the FE master:

- Allocate and register a commit TSO immediately before the first Meta
Service commit RPC, under the TSO allocator lock. Retries after a fenced
response replace the registration with a fresh TSO above the returned
fence.
- On an explicit success, already-visible result or already-aborted
result, remove the transaction registration. On another explicit
failure, remove only the exact attempted TSO.
- On `KV_TXN_MAYBE_COMMITTED`, an RPC timeout, connection loss or
another uncertain result, synchronously advance the per-instance Meta
Service fence to the attempted TSO before removing it. If fencing fails,
deactivate the TSO service so reads and allocations cannot use an unsafe
prefix.
- Disable transport-level retries for commits carrying a positive TSO.
`MS_TOO_BUSY` remains retryable because Meta Service explicitly rejects
it before processing. Non-TSO transactions keep the existing retry
behavior.
- Derive the committed TSO from the current allocated TSO and the oldest
registered attempt, and persist it together with the allocation-window
end in one BDB journal record. The default persistence window is reduced
from five seconds to one second.
- Remove the startup transaction scan, recovery RPC, recovery state, and
periodic `checkTransactions()`. A new master persists a fresh window and
advances the same global fence before enabling TSO service, so commits
using an older TSO are rejected without waiting for PREPARED
transactions.
- A successful lazy commit can be released from FE memory. Incremental
scans ask Meta Service to wait for pending transactions on the selected
partitions before fixing their visible versions, preserving readability
without a global recovery list.

For a strongly consistent cloud read with an explicit end on every
incremental relation, FE applies these rules:

| Requested end | Behavior |
| --- | --- |
| After the current TSO physical time | Return `5100 /
ERR_INCR_WINDOW_NOT_READY` immediately. |
| At or before the durable committed TSO physical time | Admit the
window directly. |
| Between committed and current TSO physical times | Wait only for the
captured FE registrations whose TSO is within the window and whose
tables are queried. Return `5101 / ERR_INCR_VISIBLE_WAIT_TIMEOUT` when
the wait expires. |

After admission, version resolution waits only for pending transactions
on the selected partitions. Unrelated tables do not participate in
either wait.

`information_schema.tso_status` exposes `COMMITTED_TSO` and
`COMMITTED_TSO_PHYSICAL_TIME`; the latter is epoch milliseconds and is
the maximum globally admitted end timestamp. The existing current TSO
and allocation-window columns remain available.

The Meta Service fence is one monotonic, non-versioned current-state key
in the instance transaction namespace. Commit reads it in the same FDB
transaction as the metadata commit and rejects `commit_tso <= fence`
with `TXN_COMMIT_TSO_EXPIRED`, returning the effective fence. The check
runs only when both the FE request flag and the mutable Meta Service
configuration are enabled. Cloud snapshot, clone and rollback do not
restore this leadership/commit fence; a restored or cloned instance
publishes its current fence when its FE master initializes.

Upgrade Meta Service before FE. A missing or unavailable fence RPC keeps
a new FE TSO service uninitialized. The new Meta Service error code uses
`actual_code` while old clients receive the compatible fallback code.

#### Connector error handling

| Code | Name | Meaning | Connector action |
| --- | --- | --- | --- |
| 5100 | `ERR_INCR_WINDOW_NOT_READY` | The requested end is after
current TSO, or the FE master/TSO generation changed while establishing
the window. | Retry the same end timestamp after `retryAfterMs`; do not
advance the incremental offset. Route a retry through the current master
when the reason is `TSO_MASTER_CHANGED`. |
| 5101 | `ERR_INCR_VISIBLE_WAIT_TIMEOUT` | Relevant commit attempts did
not finish within `change_visible_timeout_ms`. | Retry the same end
timestamp; do not advance the incremental offset. |

MySQL returns SQLSTATE `HY000`; the message contains `reason`, requested
end, current TSO, committed TSO, their physical times, timeout and retry
delay. Arrow Flight SQL returns `UNAVAILABLE` only for these two errors
and includes `doris-error-code` and `doris-error-name` metadata. Other
Flight failures retain the original `INTERNAL` wrapping.

### Release note

Improve bounded, strongly consistent cloud incremental reads with a
durable committed TSO, table-scoped waits and per-instance commit TSO
fencing. Expose the committed TSO through
`information_schema.tso_status`, distinguish an unavailable window
(5100) from a visibility wait timeout (5101), and remove FE
startup/periodic transaction recovery scans. Upgrade Meta Service before
FE.

### Check List (For Author)

- Test
- [x] FE unit tests for TSO service/tracker, commit-result handling,
Meta Service retry behavior, incremental version waits, MySQL error
propagation and Arrow Flight SQL error mapping.
- [x] Cloud ASAN unit tests for fence monotonicity and stale-commit
rejection, fence-key encoding, and response-code compatibility.
- [x] Regression tests `test_committed_tso` and
`test_binlog_changes_syntax` passed earlier in this PR with outputs
generated by the standard runner.
    - [x] `./build.sh -j32` and `./build.sh --cloud -j32`.
- [x] clang-format 16, check-format, Cloud clang-tidy and build-hygiene
checks.
- Behavior changed: Yes. Bounded cloud reads use committed
TSO/table-scoped waits; uncertain TSO commit attempts are fenced before
release; FE no longer scans transactions during startup or periodically.
- Does this need documentation: Yes. Connector-facing error handling is
documented above.

…ache#68066)

Problem Summary: When a BE in a non-normal cloud cluster stays offline,
every failed heartbeat after the lost-heartbeat threshold submits
another transaction cleanup task. Each task queries
get_prepare_txn_by_coordinator, which scans the instance transaction
metadata even after cleanup finishes. Restrict these submissions to
lost-heartbeat durations in [T, 2T), where T is
abort_txn_after_lost_heartbeat_time_second. Keep the existing behavior
for NORMAL cloud clusters and non-cloud deployments. This deliberately
stops further submissions after the window even if earlier cleanup
attempts failed.

### Release note

Limit repeated transaction cleanup task submissions for offline BEs in
non-normal cloud clusters to a bounded window (5 to 10 minutes after the
last successful heartbeat with the default configuration).

### Check List (For Author)

- Test: Add HeartbeatMgrTest coverage for all four non-normal cluster
states, durations before/inside/after the window, NORMAL and non-cloud
behavior, replay, and missing successful heartbeats. Tests and local
compilation were not run as requested. Checkstyle on both changed files
and git diff --check passed.
- Behavior changed: Yes, stop submitting BE-down cleanup tasks for
non-normal cloud clusters once the lost-heartbeat duration reaches twice
the threshold.
- Does this need documentation: No

…#68135)

### What problem does this PR solve?
Problem Summary:

Ordinary Paimon queries request table cardinality during planning. On a
row-count cache miss, the connector planned every split and summed file
record counts, consuming CPU and memory proportional to the table's
manifests and files even when the query selects only a small partition.

apache#68114)

### What problem does this PR solve?

Issue Number: None

Problem Summary:

When constant folding is skipped (`debug_skip_fold_constant = true`),
`element_at(NULL, 1)` is not folded on the FE and reaches the BE with
the container typed as a null literal (`TYPE_NULL`, mocked as `UInt8` in
the BE). `element_at` opts out of the default null handling, so its
return type resolution treated the mock type as a real `BOOL` container
and failed with:

```
[INVALID_ARGUMENT][E33] element_at only support array and map so far, but got BOOL
```

In debug builds the preceding `DCHECK` fires before that error is
raised.

This change makes `element_at` recognize a null-literal container the
same way the default null implementation does: the return type resolves
to `Nullable(Nothing)` and execution produces a NULL for every row,
which matches the folded result.

```sql
SET debug_skip_fold_constant = true;
SELECT element_at(NULL, 1);   -- NULL (previously: error "got BOOL")
SELECT element_at(NULL, 'k'); -- NULL
```

### Release note

None

### Check List (For Author)

- Test:
- Unit Test:
`function_array_element_test.element_at_null_literal_container`
- Regression test:
`query_p0/sql_functions/array_functions/test_element_at_null_container`
- Behavior changed: No
- Does this need documentation: No

### Check List (For Reviewer who merge this PR)

- Confirm the release note
- Confirm test cases
- Confirm document
- Add branch pick label

https://claude.ai/code/session_01HXcF45DSFT6fMDvtdNz2w8

… a nullable condition (apache#67931)

### What problem does this PR solve?

Issue Number: None

Problem Summary:

When the condition of `IF` is a `Nullable(Boolean)` column, both
`VectorizedIfExpr` and `FunctionIf` normalize it by treating NULL as
false.
They did this by writing `nested[i] &= !null_map[i]` directly into the
nested column of the nullable condition.

That nested column can be shared with other columns of the same block.
`NULLIF(b, p)` is implemented as `if(b = p, NULL, b)` and wraps `b`
itself
as the nested column of its `Nullable(Boolean)` result, so
`IF(NULLIF(b, p), f, b)` overwrote `b` in place: the else branch read
the
polluted values and returned a wrong result, and any other projection of
`b` in the same block was polluted as well. With
`short_circuit_evaluation=true` the expression goes through a different
path and returned the correct result, which hid the bug.

Build a fresh condition column from `nested & !null_map` instead of
mutating the shared nested column in place.

### Release note

None

### Check List (For Author)

- Test:
- Unit Test:
`VConditionExprIfTest.NullableCondition_NotPolluteSharedNestedColumn`,
`FunctionIfTest.NullableConditionNotPolluteSharedNestedColumn` (both
fail without the fix)
    - Regression test: `test_if_nullable_condition`
- Behavior changed: No
- Does this need documentation: No

https://claude.ai/code/session_013PjVxV5VYPRduC2fubqCzD

…ache#68113)

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Sliding `ROWS` window frames evaluate `sum`/`avg` incrementally: the
outgoing
row is subtracted from the accumulator and the incoming row is added.
For
floating-point accumulators this is not exact. Once `2^54 + 1` rounds to
`2^54`, subtracting `2^54` again leaves `0` instead of `1`, so the
rounding
lost by a value that already left the frame keeps distorting later
results.

```sql
WITH t AS (
    SELECT 1 AS id, CAST(18014398509481984 AS DOUBLE) AS v
    UNION ALL SELECT 2, CAST(1 AS DOUBLE)
    UNION ALL SELECT 3, CAST(1 AS DOUBLE)
)
SELECT id, v,
       avg(v) OVER (ORDER BY id ROWS BETWEEN 1 PRECEDING AND CURRENT ROW) AS got
FROM t ORDER BY id;
```

Before: the third row returned `0.5` (frame `[1, 1]`, sum `1`, count
`2`).
After: the third row returns `1`.

Fix: `AggregateFunctionSum` and `AggregateFunctionAvg` now report
`supported_incremental_mode() == false` when the accumulator is a
floating-point type, so the analytic sink recomputes each sliding frame
from
its rows (the same approach PostgreSQL takes by not providing inverse
transition functions for `float4`/`float8`). Integer and decimal
accumulators
are exact and keep the incremental path. Frames that only grow
(`UNBOUNDED PRECEDING`) never removed rows and are unaffected.

Cost: sliding `ROWS` frames over `FLOAT`/`DOUBLE` columns now cost
O(frame size) per row instead of O(1). This is the trade-off for exact
results; wide sliding frames over floating-point columns will be slower.

### Release note

None

### Check List (For Author)

- Test:
- Unit Test:
`AnalyticSinkOperatorTest.SlidingRowsDouble{Avg,Sum}IgnoresRoundingOfOutgoingRow`,

`AggregateFunction{Avg,Sum}Test.test_incremental_mode_only_for_exact_sum`
- Regression test:
`query_p0/sql_functions/window_functions/test_window_float_sliding_frame`
- Behavior changed: Yes (sliding window sum/avg over FLOAT/DOUBLE now
return the exact per-frame result; wide frames are slower)
- Does this need documentation: No

https://claude.ai/code/session_01KSyhoeWWGWukpHP6MBTbEt

…he#68154)

### What problem does this PR solve?

Problem Summary:

INSERT value normalization directly folds already bound expressions
without enabling `keepFunctionSignature`. When `repeat` inside `struct`
is folded from an always-nullable function to a non-null string literal,
`CreateStruct.withChildren` can recompute the nested field nullability
while surrounding array and cast nodes retain their bound types. FE then
sends an inconsistent nested column layout and BE aborts when casting
the string column to `ColumnNullable`.

Preserve the bound function signature around both post-bind
constant-folding entry points in `InsertUtils`. For the fast INSERT
VALUES analyzer, split binding jobs from post-bind jobs: binding remains
under `keepFunctionSignature(false)`, while ordinary and batch post-bind
rewrites run under `keepFunctionSignature(true)`. Add regression
coverage for both fast and normal INSERT VALUES analysis paths.

### Release note

Fix BE crashes when inserting arrays of structs containing foldable
nullable functions.

…TABLE LIKE (apache#67809)

### What problem does this PR solve?

Problem Summary:

CREATE TABLE LIKE reconstructs generated columns from SQL using the
caller's session settings, losing the expression settings saved in the
source columns. For a generated expression `a * b` over `DECIMAL(20,5)`
and `DECIMAL(21,6)`, a source created with `enable_decimal256=true`
produces `1.26212529751`, but a clone created with the option disabled
produces `1.26212530000` for the same input. SQL mode also affects
parsing: a source expression `a || 'x'` created with `PIPES_AS_CONCAT`
produces `ax`, while its clone can produce NULL under a different SQL
mode.

Snapshot each generated column's session settings under the source
table's read lock and carry them through `GeneratedColumnDesc`. Restore
these settings temporarily when parsing and analyzing the cloned
expression, then persist them in the new `Column`. The caller's session
is restored after analysis, including on exceptions. Ordinary CREATE
TABLE retains its existing session behavior.

Existing incorrectly created clones and stored data are not
automatically repaired.

### Release note

Fix CREATE TABLE LIKE changing generated column results when the
caller's expression-related session settings differ from the source
table's settings.

### Check List (For Author)

- Test
    - [x] Regression test
- Passed `test_generated_column_like_session`, `test_generated_column`,
and `test_generated_column_fault_tolerance_nereids` on the final
branch's FE.
- New coverage includes both decimal256 setting directions, repeated
LIKE, SQL mode, NULL values, and preservation of the caller's session.
Expected output was generated by `run-regression-test.sh` and verified
in subsequent runs.
    - [x] Unit Test
- All 11 `CreateTableLikeTest` tests passed via `run-fe-ut.sh` on the
initial validation baseline with the same repair and test files.
- The final upstream baseline has an unrelated test-compilation error in
`IvmNormalizeMTMVJoinTest` (an outdated `IvmRewriteContext` constructor
call). Final FE packaging therefore used `DISABLE_BUILD_UI=ON
MVN_OPT=-Dmaven.test.skip=true ./build.sh --fe`. No unrelated test code
was changed. Checkstyle passed.
    - [x] Manual test (add detailed scripts or steps below)
- Reproduced both precision and SQL mode differences before replacing FE
libraries; verified correct results after replacement.
- Restarted the final FE and cloned previously created fixed tables
again, verifying persisted session settings and unchanged caller
settings.
- The test FE used `be_exec_version=13` to match its existing BE.
    - [ ] No need to test or manual test. Explain why:

- Behavior changed:
    - [ ] No.
- [x] Yes. Generated columns created through LIKE preserve the source
columns' expression semantics.

…e#68165)

### What problem does this PR solve?

Two external regression cases rely on assumptions that no longer hold:

- The Parquet join runtime filter case requires exactly two counter
matches, so repeated profile sections fail despite sufficient filtering.
Wait for profile completion, scope counters to the fact scan, and check
the largest merged or instance value against the original thresholds
without summing duplicates.
- The remote Doris VARIANT case implicitly casts JSON text to VARIANT
and compares against obsolete output. String casts now preserve strings.
Explicitly parse valid JSON fixtures, cast numeric predicates, and
compare all eight non-Arrow queries with the internal source table. Two
known predicate-result checks prevent empty nested-path results from
passing both sides. Keep the Arrow error assertion and remove the unused
golden file.

Only regression cases and their expected-data file change.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [ ] Unit Test
    - [x] Manual test

Local Groovy checks reproduced the old assertions and validated the
changes:
- Profile parsing: normal, repeated, merged, multiple-instance,
missing-counter, unrelated-table and insufficient-filtering inputs, plus
archived profiles and normalized whitespace.
- Polling: delayed completion, transient read errors and timeouts. The
full Parquet suite DSL ran with simulated SQL/profile responses for all
four queries and three wait settings.
- VARIANT suite DSL: explicit JSON parsing, SQL NULL and
malformed-string fixtures, eight matching catalog/source query pairs,
two semantic checks, ordering and the Arrow error assertion.
- Current framework result-comparison code: matching values pass; stale
output, SQL/JSON null confusion, missing/extra/reordered rows and
changed boolean values fail.
- `git diff --check` passed. Live Doris/Hive regression was not run; CI
is requested.

- Behavior changed:
    - [x] No production behavior changes.
    - [ ] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…pache#68147)

Rstrict AI resource creation by user identity

…che#68095)

Issue Number: close apache#68035

Related feature: apache#68036

Doris reconstructs loaded Iceberg tables as BaseTable snapshot views. BaseTable implements SupportsDistributedScanPlanning with a default true result, so wrapping RESTTable loses its false override. The existing scan-entry guard then allows unsupported local planning. With restricted metadata credentials this surfaces as an S3 error; a separately documented simulated server residual demonstrates why ignoring the server plan can also change results.

Check the original table before statement snapshot conversion, tracked-table publication, and cached TableOwner metadata serialization/weight estimation. Reject unsupported tables before local manifest access and release acquired table/catalog ownership on failure. Preserve the existing scan-entry guard and normal client-planned snapshot semantics.

**Rejection includes metadata acquisition, not only SELECT scans.** DESC, SHOW CREATE TABLE, and INSERT target-column binding also fail for server-planned tables. Row-count and partition metadata paths using the same read acquisition are subject to this guard. The sharedWritableTable method itself is unchanged; this does not imply SQL INSERT remains supported.

Add a dedicated native Iceberg REST fixture, pinned by multi-platform digest, to the existing Iceberg Compose stack. Keep the existing Spark/Postgres REST baseline. MinIO initialization creates a private test bucket and data-only credentials. The enabled regression requires a working /plan endpoint: an old or missing fixture fails instead of skipping.

The fixture digest fixes the contents of an upstream `main` snapshot, built on 2026-04-29 from `b0df3ca01d61b2f7ae7143ac660c6b16e33b6e46` (recorded in the JAR build properties). This commit precedes the `apache-iceberg-1.11.0` tag by 105 commits; the image is not a release build. Moving `latest` cannot change the pinned contents. The retention risk is that the old manifest can become untagged and later be removed upstream, breaking fresh pulls.

**Request to CI maintainers:** build a linux/amd64 + linux/arm64 fixture from the upstream `apache-iceberg-1.11.0` tag using the upstream shadowJar task and Dockerfile, and publish it as `doristhirdpartydocker/iceberg-rest-fixture:1.11.0`. Keep the current digest until that artifact exists; verify source/architectures and rerun this regression, then switch to the versioned image with its verified digest. Mirroring the current image into the Doris namespace is a retention fallback, but remains a snapshot. We do not assume the upstream release-image workflow will be repaired in time for this PR.

The regression invokes the real /plan endpoint, verifies restricted data-file reads succeed and manifest access returns 403, and checks explicit Doris rejection for SELECT, metadata statements, INSERT, empty tables, and $files with cache enabled/disabled. Client-planned reads remain four rows before and after. No policy proxy is part of this Docker regression. Server-forced planning mode is additionally covered by HTTP RESTCatalog unit tests.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#67904

Problem Summary:

Paimon ORC tables can store timestamp values with nanosecond precision,
while Doris exposes timestamps at microsecond precision. The Paimon JNI
reader truncates sub-microsecond digits, but the native ORC reader
rounded them. For example, `2024-01-02 10:04:05.123456789` was exposed
as `.123456` by JNI and `.123457` by the native reader, causing equality
predicates to return inconsistent results.

The existing Paimon predicate conversion keeps `TIMESTAMP(7/8/9)`
comparisons as residual predicates. This change makes the native ORC
path use the same truncation semantics as Doris and the JNI reader for
row decoding, stripe statistics, and ORC search argument bounds.

### Release note

Fix inconsistent timestamp values and predicate results between native
ORC and JNI readers for sub-microsecond timestamps.

### Check List (For Author)

- Test
    - [x] Regression test
- `./run-regression-test.sh --conf /tmp/yuze-regression-full-conf.groovy
--run -d external_table_p0/paimon -s paimon_timestamp_types`
    - [x] Unit Test
- `./run-be-ut.sh --run
--filter=NewOrcReaderTest.*Timestamp*:NewOrcReaderTest.ReadTimestampNanosecondsTruncatesToMicroseconds`
    - [ ] Manual test
    - [ ] No need to test or manual test
- Behavior changed:
- [x] Yes. Native ORC timestamps now truncate sub-microsecond digits
consistently with Doris precision and the JNI reader.
    - [ ] No.
- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…rs on pruned STRUCT columns (apache#68110)

### What problem does this PR solve?

Issue Number: None

Problem Summary:

With `enable_prune_nested_column = true`, the planner splits a STRUCT
column's
access paths into predicate paths (read before filtering) and lazy paths
(read
only for surviving rows). SegmentIterator relies on this split when a
common
expression references the column: it reads the column in the PREDICATE
phase,
evaluates the expression, and recovers the lazy nested fields
afterwards.

The TopN filter is a common expression that BE attaches to the scan at
runtime,
after the planner computed the access paths. When ORDER BY and WHERE
reference
different fields of the same STRUCT, for example

```sql
SELECT struct_element(s, 'a') FROM t
WHERE struct_element(s, 'b') IS NOT NULL
ORDER BY 1 NULLS LAST LIMIT 1;
```

the predicate paths only contain `s.b.NULL`, so `s.a` is a placeholder
(NULL)
while the TopN filter `struct_element(s, 'a') <= current_top` is
evaluated.
Every row of the later tablets is rejected and the global minimum is
lost,
silently returning a wrong LIMIT result.

This PR marks columns referenced by runtime-generated common expressions
(TopN filters and runtime filters) and keeps them in the NORMAL read
phase, so
all access paths are materialized before the expression is evaluated.
The lazy
split is still used for planner-visible expressions.

### Release note

None

### Check List (For Author)

- Test:
- Unit Test: `SegmentIteratorRuntimeCommonExprTest` covers the lazy
split
decision for planner expressions, TopN filters, and both on one column.
- Regression test: `topn_filter_nested_column_pruning` reproduces the
wrong
result with TIMESTAMPTZ/INT/DATE STRUCT fields, ASC/DESC, NULLS
FIRST/LAST,
      selective predicates and multi-tablet scans.
- Behavior changed: No
- Does this need documentation: No



https://claude.ai/code/session_01E3gDAafAXZELs6AHVfD9UG

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#64093

Problem Summary:

With `ASAN_OPTIONS=fast_unwind_on_malloc=0`, an ASAN BE can abort before
`main`, including when running `doris_be --version`. OpenMP
initialization reached from OpenBLAS probes optional symbols with
`dlsym`. On affected glibc versions, a subsequent lookup frees the
previous `dlerror` string before clearing its pointer. ASAN's free
interceptor collects a slow stack, which reaches Doris' global
`dl_iterate_phdr` interposer. Its fallback calls `dlsym` again, reenters
the pending error cleanup and frees the same string twice.

Exclude only the global `dl_iterate_phdr` interposer from ASAN builds.
ASAN then uses the system implementation without reentering `dlsym`
through Doris. Keep the PHDR snapshot and dedicated
`doris_unwind_iterate_phdr` hook for GNU libunwind; non-ASAN
interposition is unchanged.

Add an ASAN subprocess regression that re-execs with slow unwinding
enabled and performs two failed symbol lookups without consuming
`dlerror` between them. Adapt the existing late-`dlopen` test to assert
that ASAN keeps the live system view even inside `ScopedPHDRCacheRead`,
while the dedicated GNU hook still uses the snapshot.

### Release note

Fix an ASAN BE startup double-free when allocation/free stack collection
uses slow unwinding.

### Check List (For Author)

- Test
    - [ ] Regression test
- [ ] Unit Test execution: added a subprocess regression and updated the
PHDR cache test; full local execution is blocked by the installed AWS
SDK missing `aws/core/auth/GeneralHTTPCredentialsProvider.h` while
compiling unchanged `common/cpp/aws_common.cpp`.
    - [x] Manual test:
- Compiled the actual pre-fix and fixed `phdr_cache.cpp` into the same
ASAN startup reproducer (two failed `dlsym` calls,
`fast_unwind_on_malloc=0`): pre-fix reports double-free; fixed reaches
`main`.
- OpenMP initialization from a pre-main constructor with the fixed
source and slow unwinding: passed.
- Live-loader versus PHDR snapshot checks in ASAN and non-ASAN
standalone probes, including a late-loaded DSO and cache refresh:
passed.
        - Updated unit-test source syntax check: passed.
- clang-format 16, build hygiene and clang-tidy checks on all three
changed files: passed.
- Full local test command attempted: `./run-be-ut.sh -j 48 --run
--filter='PhdrCache*'` (ASAN; dependency failure noted above).

- Behavior changed:
- [x] Yes. ASAN uses the system `dl_iterate_phdr` even inside a scoped
cache read; its dedicated GNU libunwind hook remains cached. Non-ASAN
behavior is unchanged.

- Does this need documentation?
- [x] No. Internal sanitizer compatibility fix; the source API comment
documents the ASAN exception.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…ving to unpartitioned (apache#68124)

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

After an Iceberg table's default partition spec evolves to
unpartitioned, split planning gated the per-file partition metadata on
the current spec (`table.spec().isPartitioned()`), so data files written
under an older identity spec stopped carrying their partition values.
`path_partition_keys` is the union of all specs, so BE does not read
those columns from the data file either, and an identity partition
column whose value only exists in the file's partition metadata (e.g. a
table migrated with `add_files`) was read as NULL, with both
`enable_file_scanner_v2=true` and `false`.

Gate on whether any spec of the table is partitioned instead. The
values, spec id and partition data are already derived from the spec
each data file was written with, which is what Iceberg's
`PartitionUtil.constantsMap` does.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason
       
- Behavior changed:
    - [ ] No.
    - [x] Yes. 
    
- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

## Problem

`UNION ALL` could publish output-column equality that does not hold for
every row. Downstream rules may then remove a required window ordering
key and change `RANK()` results.

Two forms reproduce the problem:

1. Regular children project columns in a different order from their
internal output. With child filters proving `a = b`, a union that
projects `(c, a, b)` can incorrectly map that equality to `(c, a)` and
remove `a` from `ORDER BY c, a`.
2. A regular child proves `a = b`, but constant rows such as `(1, 2)`
and `(2, 1)` do not. The union can still claim the output columns are
equal and remove `b` from `ORDER BY a, b`.

## Root cause

Logical and physical unions independently mapped child equality sets
through `child.getOutput()` positions. The authoritative union ordinal
mapping is `regularChildrenOutputs`, which can have a different order.
The derivation also considered only regular children and ignored every
constant row carried by the union.

## Reproduction

Create rows `(1, 1, 10)`, `(2, 2, 10)`, and `(1, 1, 20)`, duplicate a
filtered `(c, a, b)` child with `UNION ALL`, and compute `RANK() OVER
(ORDER BY c, a)`. The two rows with `(c, a) = (10, 2)` must have rank 3.

Separately, union filtered table rows satisfying `a = b` with constant
rows `(1, 2)` and `(2, 1)`, then compute `RANK() OVER (ORDER BY a, b)`.
The four ordered pairs must receive ranks 1, 2, 3, and 4.

## Fix

- Share one equal-set derivation between logical and physical unions.
- Validate and use each regular child's explicit union-output mapping.
- Intersect child equality classes by per-ordinal class signatures,
avoiding pairwise quadratic candidate generation.
- Refine candidates against every constant row after SQL comparison
coercion and constant folding.
- Accept only a folded `TRUE`; `NULL`, unsupported coercion,
non-foldable expressions, and malformed mappings conservatively provide
no equality proof.
- Build each surviving equivalence class with linear star edges.

IP functions did not consistently respect the explicit boundaries of
`ColumnString` values. IPv4 and IPv6 text conversions accepted an
otherwise valid address followed by an embedded NUL byte and trailing
data because unbounded parser overloads stopped at the first NUL. In
addition, `is_ipv4_compat` and `is_ipv4_mapped` discarded each binary
string's length and unconditionally read 16 bytes, so short values could
consume bytes from the following row and produce results that changed
with filtering, while oversized values were accepted based only on their
first 16 bytes. Root cause: both paths reduced length-aware string
values to raw pointers before parsing or inspection. This change uses
bounded parsers throughout, removes the unsafe null-terminated parser
overloads, maps parsed IPv4 values directly to IPv6, and requires binary
IPv6 inputs to be exactly 16 bytes before inspecting them. Unit coverage
includes throw, default, nullable, vector, constant, short, valid, and
oversized input paths.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: N/A

Problem Summary: AES and SM4 encryption functions previously accepted non-constant mode columns even though execution paths resolve a stable encryption mode. Declare the mode argument as always constant for three-, four-, and five-argument signatures, add BE unit coverage for constant and dynamic modes, and update the GCM regression case to use literal modes.

### Release note

AES_ENCRYPT, AES_DECRYPT, SM4_ENCRYPT, and SM4_DECRYPT now require the mode argument to be constant.

### Check List (For Author)

- Test: Unit Test
    - Focused function_string_test AES and SM4 encryption tests
- Behavior changed: Yes. Encryption mode expressions must now be constant.
- Does this need documentation: No.