[fix](function) Report decimal rounding overflow - #2

…pache#67673)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

- Add dedicated embedding properties to AI resources:
  - `ai.embed.endpoint`
  - `ai.embed.provider_type`
  - `ai.embed.model_name`
  - `ai.embed.api_key`
- Require at least one complete property group when creating an AI
resource:
  - the general `ai.*` group, or
  - the `ai.embed.*` group
- Use `ai.embed.*` when executing the `EMBED` function if the embed
property group is configured.
- Fall back to the general `ai.*` properties otherwise.
- Mask `ai.embed.api_key` when displaying resource properties.
- Keep other AI functions using the general `ai.*` properties.

```text
Doris> CREATE RESOURCE 'default_combine_resource'
    -> PROPERTIES (
    ->     'type' = 'ai',
    -> 
    ->     'ai.provider_type' = 'QWEN',
    ->     'ai.endpoint' = 'https://dashscope.aliyuncs.com/compatible-mode/v1/chat/completions',
    ->     'ai.model_name' = 'qwen-plus',
    ->     'ai.api_key' = '<general-api-key>',
    ->     'ai.effort' = 'high',
    -> 
    ->     'ai.embed.provider_type' = 'QWEN',
    ->     'ai.embed.endpoint' = 'https://dashscope.aliyuncs.com/compatible-mode/v1/embeddings',
    ->     'ai.embed.model_name' = 'text-embedding-v4',
    ->     'ai.embed.api_key' = '<embedding-api-key>',
    -> 
    ->     'ai.embed.mm.provider_type' = 'QWEN',
    ->     'ai.embed.mm.endpoint' = 'https://dashscope.aliyuncs.com/api/v1/services/embeddings/multimodal-embedding/multimodal-embedding',
    ->     'ai.embed.mm.model_name' = 'qwen3-vl-embedding',
    ->     'ai.embed.mm.api_key' = '<multimodal-api-key>',
    -> 
    ->     'ai.dimensions' = '1024'
    -> );
Query OK, 0 rows affected (0.026 sec)

Doris> set default_ai_resource = 'default_combine_resource';
Query OK, 0 rows affected (0.009 sec)

Doris> SELECT array_size(EMBED('this is a test'));
+-------------------------------------+
| array_size(EMBED('this is a test')) |
+-------------------------------------+
|                                1024 |
+-------------------------------------+
1 row in set (1.729 sec)

Doris> EXPLAIN SELECT array_size(EMBED('this is a test'));
+---------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                           |
+---------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                           |
|   OUTPUT EXPRS:                                                           |
|     array_size(EMBED('this is a test'))[#0]                               |
|   PARTITION: UNPARTITIONED                                                |
|                                                                           |
|   HAS_COLO_PLAN_NODE: false                                               |
|                                                                           |
|   VRESULT SINK                                                            |
|      MYSQL_PROTOCOL                                                       |
|                                                                           |
|   0:VUNION(11)                                                            |
|      constant exprs:                                                      |
|          cardinality(embed('default_combine_resource', 'this is a test')) |
|                                                                           |
|                                                                           |
|                                                                           |
| ========== STATISTICS ==========                                          |
+---------------------------------------------------------------------------+
17 rows in set (0.030 sec)

Doris> SELECT AI_TRANSLATE('this is a test', 'chinese');
+-------------------------------------------+
| AI_TRANSLATE('this is a test', 'chinese') |
+-------------------------------------------+
| 这是一个测试                              |
+-------------------------------------------+
1 row in set (1.269 sec)

Doris> EXPLAIN SELECT AI_TRANSLATE('this is a test', 'chinese');
+--------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                |
+--------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                |
|   OUTPUT EXPRS:                                                                |
|     AI_TRANSLATE('this is a test', 'chinese')[#0]                              |
|   PARTITION: UNPARTITIONED                                                     |
|                                                                                |
|   HAS_COLO_PLAN_NODE: false                                                    |
|                                                                                |
|   VRESULT SINK                                                                 |
|      MYSQL_PROTOCOL                                                            |
|                                                                                |
|   0:VUNION(11)                                                                 |
|      constant exprs:                                                           |
|          ai_translate('default_combine_resource', 'this is a test', 'chinese') |
|                                                                                |
|                                                                                |
|                                                                                |
| ========== STATISTICS ==========                                               |
+--------------------------------------------------------------------------------+
17 rows in set (0.010 sec)

Doris> EXPLAIN SELECT EMBED(
    ->     CAST(
    ->         '{"content_type":"image/png","uri":"https://example.com/image.png"}'
    ->         AS JSON
    ->     )
    -> );
+--------------------------------------------------------------------------------------------------------------------------------+
| Explain String(Nereids Planner)                                                                                                |
+--------------------------------------------------------------------------------------------------------------------------------+
| PLAN FRAGMENT 0                                                                                                                |
|   OUTPUT EXPRS:                                                                                                                |
|     EMBED(                                                                                                                     |
|     CAST(                                                                                                                      |
|         '{"content_type":"image/png","uri":"https://example.com/image.png"}'                                                   |
|         AS JSON                                                                                                                |
|     )                                                                                                                          |
| )[#0]                                                                                                                          |
|   PARTITION: UNPARTITIONED                                                                                                     |
|                                                                                                                                |
|   HAS_COLO_PLAN_NODE: false                                                                                                    |
|                                                                                                                                |
|   VRESULT SINK                                                                                                                 |
|      MYSQL_PROTOCOL                                                                                                            |
|                                                                                                                                |
|   0:VUNION(11)                                                                                                                 |
|      constant exprs:                                                                                                           |
|          embed('default_combine_resource', CAST('{"content_type":"image/png","uri":"https://example.com/image.png"}' AS json)) |
|                                                                                                                                |
|                                                                                                                                |
|                                                                                                                                |
| ========== STATISTICS ==========                                                                                               |
+--------------------------------------------------------------------------------------------------------------------------------+
```
### Release note

None

`conv()` returned incorrect results for BIGINT inputs when converting
from non-decimal bases because the integer path used 32-bit intermediate
place values. Use unsigned 64-bit accumulation with pre-operation
overflow checks and preserve existing invalid-digit and signed-value
handling.
### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…che#68301)

### What problem does this PR solve?

This ports apache#68297 to `master`, preserving the first of five planned
extractions from apache#67784.

Binary `Field` values can retain references to released source storage,
and Hive binary text needs its own Base64 contract. TIMESTAMPTZ output
can lose historical offset seconds, format invalid NULL payloads, or
fail again while reporting a boundary cast error.

- Own long binary Field values while keeping short values inline.
Preserve execution type lengths and decoder bytes, and add Hive Base64
and hexadecimal decoding support.
- Explicitly reject unsupported binary hash keys, IN, aggregates,
predicates and computed partition transforms. Keep the existing FE
comparison/group/join restrictions and existing binary scalar functions.
Reject unsupported collection kernels in each function's legality check
before coercion.
- Preserve historical second offsets in both TIMESTAMPTZ formatting and
parsing. Skip masked NULL payloads, reject unrepresentable local years,
and preserve cast error/NULL behavior at boundaries.

Arrow convertor migration, Parquet/ORC semantics, external writer
changes and catalog mapping migration belong to the subsequent
extractions. This PR does not enable native VARBINARY storage.

### Master adaptation

- Retain the fixed-offset normalization and tests already present on
master.
- Use the current void-returning
`VInPredicate::_prepare_zonemap_min_max` interface in both the guard and
its test.
- Retain master header cleanup and existing timestamp-nanosecond tests.
- Retain the existing master binary-literal encoder and its StringView
input contract; the older std::string-based caller fix is not
applicable.

### Testing
- TIMESTAMPTZ regression follow-up: reproduced both binary-output and
stream-load failures using the master PR CI artifact, regenerated the
two snapshots through `run-regression-test.sh`, and passed both suites
in comparison mode from each branch checkout. Explicit `Asia/Shanghai`
session settings were verified with the server default session zone set
to UTC. Only historical offset seconds changed in the generated results.
- Function-local validation update: 18 FE tests passed with Checkstyle
enabled, covering direct legality checks, nested/mixed/variadic
VARBINARY arguments, both `collect_set` arities, supported ordinary
types, SQL analysis, and existing array rewrites. The new
direct-legality tests reproduced missing rejection before the change.

- BE ASAN build and **199 tests passed** across 17 suites using
`run-be-ut.sh`, including binary lifetime/SerDe/rejection, timestamp
parsing/casts, and existing Arrow/Variant serialization coverage.
- `VarBinaryUnsupportedCollectionTest`: **passed** (13 unsupported
expressions plus supported byte-preserving collection analysis). The FE
test reactor and repository Checkstyle passed after cleaning stale
branch build artifacts.
- Repository clang-format 16 check and build-header hygiene checks:
**passed**; 31 changed C++ source/header files.
- Groovy compilation of the three regression suites: **passed**. Live
SQL regression execution remains pending CI.
- clang-tidy was attempted but could not complete because master already
contains an unmatched `NOLINTEND` in `be/src/core/types.h`. A diagnostic
run with the compiler resource directory corrected reproduced that
blocker; the other reported findings in `column_varbinary.cpp` were
outside changed lines. This is not a clean clang-tidy result.

The focused BE test source list and local test/build settings were
restored before committing. No build configuration changes are included.

### Release note

Fix binary value lifetime and serialization, reject unsupported binary
computation paths, and preserve TIMESTAMPTZ historical offsets and
boundary error behavior.

### Check List (For Author)

- Test
- [x] Regression test (three self-checking suites added; execution
pending CI)
  - [x] Unit Test
- Behavior changed:
- [x] Yes. Binary rejection and timestamp boundary behavior are
described above.
- Does this need documentation?
- [x] No. This fixes existing type behavior without introducing a
configuration option.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label


### Scoped review follow-up

This follow-up only fixes correctness/stability defects introduced by
this PR. Compatibility preservation, pre-existing limitations,
additional VARBINARY computation/validation, performance refactors, and
test-style-only rewrites are excluded.

- Separate historical TIMESTAMPTZ wire-offset parsing from session
fixed-zone limits in both parser paths.
- Use UTC diagnostics for the TIMESTAMP_NS cast/comparison failures
affected by the new local-year formatting exception.
- Validation: 31 focused ASAN BE tests passed, including ordinary
DATE/DATETIME parsing. Three targeted tests failed before the fixes.
clang-format 16 and build hygiene passed. Full clang-tidy remains
affected by pre-existing diagnostics.
- Branch-specific UTC/GMT normalization and FE folding fixes are handled
in apache#68297; the corresponding master behavior predates this PR or already
defers folding.

### CI test follow-up

- Keep the binary literal test's owning Field alive while reading its
StringView. Branch-4.1 now has the corresponding short/long embedded-NUL
coverage using its execution API.
- Replace the obsolete +15:00 rejection input with +24:00. Add generated
historical-offset checks in both cast modes; all prior snapshot results
are unchanged.
- Validation: 34 focused ASAN BE tests passed on each branch. The
lifetime error and the original SQL mismatch were reproduced. The
complete cast regression suite passed in comparison mode from both
branch checkouts against the reported master CI artifact. clang-format
16 passed; full clang-tidy remains blocked by pre-existing diagnostics.

This follow-up changes tests only and retains the agreed scope: no
compatibility work or additional binary computation support. Existing
muted failures are outside this fix.

…che#68372)

### What problem does this PR solve?

After a capacity error, the review helper can accept `codex exec --goal
resume` exiting with status 0 even when the attempt reports
`turn.failed` or never produces a terminal event. The helper then stops
retrying and the workflow replaces the underlying error with “no new
pull request review was submitted.”

Require both a zero process exit and a completed latest turn before
accepting an attempt. Explicit capacity errors remain eligible for the
existing bounded retry sequence even when the process exits with 0.
Authentication, usage-limit and other errors fail without retry; a
missing terminal event fails explicitly instead of borrowing a previous
attempt’s capacity error or completion. Existing PR-state checks,
duplicate-submission protection and the shared deadline are retained.

Each attempt now logs its exit status, last turn event and event count.
The installation step logs the actual goal binary’s SHA-256 to make
runtime failures reproducible.

Related PR: apache#67624

### Validation

- Linux Python 3.12: all 147 pipeline tests passed, including real
process cleanup and workflow/auth-quarantine integration tests. Tests
ran without network access or host mounts.
- Added 12 regression tests for zero-exit capacity retries and
exhaustion, zero-exit authentication failures, missing terminal events,
event ordering and workflow error propagation.
- The new helper tests fail against the previous implementation,
including false success and premature retry termination; they pass after
the fix.
- macOS helper suite: 55 tests, passed with 3 Linux-only skips.
- Ruff checks and helper/test formatting, actionlint, workflow YAML
parsing, all 22 shell blocks with `bash -n`, and `git diff --check`
passed.

**Rollout limit:** this validates helper behavior with controlled Codex
output and real Linux subprocesses. The actual OSS goal binary was not
available for validation (anonymous read is denied); no claim is made
that the underlying binary’s goal/resume issue is fixed. Before
production rollout, validate that exact binary’s resumed turn execution
and terminal-event behavior. The change fails incomplete resumes
explicitly; it does not blindly retry empty successful exits.

…pache#68267)

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Iceberg equality deletes are compared in the physical type of the delete
file.
When a column is legally promoted (for example INT to LONG), the data
file can
hold values outside the old type, but the data key was cast into that
old type.
An out-of-range value becomes NULL and then matches a NULL
equality-delete key
through NULL-safe equality, so a valid row is silently deleted. With
`enable_strict_cast` the same cast makes the whole scan fail with
`Value ... out of range for type int`.

Resolve the comparison type from the current snapshot schema instead of
the
delete file, and promote historical delete values into that domain. With
both
sides in the wider type the cast is always lossless for an Iceberg-legal
type
promotion, so a non-NULL key can no longer turn into NULL. Include the
current
schema id in the delete-file cache key because the loaded block now
depends on
the comparison type.

Covered by a new BE unit test: a data file holding LONG `NULL, 0, 1,
4294967296`
plus a NULL equality-delete key written under the old INT schema. Before
the fix
the scan returned `{0, 1}`; after the fix it returns `{0, 1,
4294967296}`.

### Release note

Fix Iceberg equality deletes being evaluated in the delete file's
historical
type, which could silently drop valid rows after a column type
promotion.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason

- Behavior changed:
    - [ ] No.
    - [x] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

---------

Co-authored-by: yiguolei <[email protected]>

…ble (apache#68170)

When an IVM incremental refresh hits a missing or
unusable base table stream, the rewrite fails with `STREAM_UNSUPPORTED`.
Previously this reason fell back to the partition-based refresh, which
reads the same broken stream again and fails the task, so the refresh
never reached the COMPLETE attempt whose reconcile step recreates the
stream. Only an explicit `REFRESH COMPLETE` could recover.

Two changes:
1. Add `STREAM_UNSUPPORTED` to
`IvmFailureReason.requiresCompleteRefresh()`, so an unusable stream
jumps directly to the COMPLETE attempt that reconciles it.
2. Move `reconcileIvmStreams` from the shared partition-based path into
`executeCompleteAttempt`. Behavior is unchanged (COMPLETE was its only
triggering mode), but it makes the complete attempt the single place
that resets stream baselines, which a stream's global baseline requires.

IVM incremental refresh now recovers automatically from an unusable base
table stream by falling back to a complete refresh instead of failing
the task.

Logical OR used bitwise OR on the nested bytes of Boolean columns.
Nullable rows may carry arbitrary nested payloads, so `NULL OR TRUE`
could expose a non-canonical non-NULL Boolean byte instead of `1`.
Downstream expressions such as multi-branch `CASE WHEN` could then use
that byte as an invalid branch index and crash the BE.

Root cause: both the generic logical function and the `VCompoundPred`
fast path preserved input bits instead of producing a canonical Boolean
result. This change normalizes every logical OR result to `0` or `1`
while preserving SQL three-valued NULL semantics. It also adds focused
BE coverage and SQL regressions for nullable payloads, `CASE WHEN`,
short-circuit evaluation settings, and the complete
`TRUE`/`FALSE`/`NULL` OR truth table.

The resulting nullable Boolean representation follows this truth table.
`NULL(any)` means that the nested byte of a NULL row may contain any
value; it is ignored whenever `res_null` is `1`.

| Left | Right | `res_null` | `res_data` | Final result |
|---|---|---:|---:|---|
| `FALSE` | `FALSE` | 0 | 0 | `FALSE` |
| `FALSE` | `TRUE` | 0 | 1 | `TRUE` |
| `TRUE` | `FALSE` | 0 | 1 | `TRUE` |
| `TRUE` | `TRUE` | 0 | 1 | `TRUE` |
| `NULL(any)` | `FALSE` | 1 | 0 or 1 | `NULL` |
| `FALSE` | `NULL(any)` | 1 | 0 or 1 | `NULL` |
| `NULL(any)` | `TRUE` | 0 | 1 | `TRUE` |
| `TRUE` | `NULL(any)` | 0 | 1 | `TRUE` |
| `NULL(any)` | `NULL(any)` | 1 | 0 or 1 | `NULL` |
### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Problem Summary: `Cast.castNullable` describes the outer result column's
nullability, but does not tell an optimizer whether converting a
non-null value can fail. For example, `ARRAY<STRING> -> ARRAY<INT>` can
leave the outer array non-null while an element conversion fails. This
PR adds `Cast.mayFailOnNonNullInput` as a reusable type-pair and
expression-level analysis. It recurses through ARRAY, MAP, and STRUCT,
recognizes BE conversions proven not to report data-dependent failures,
and conservatively treats other conversions as possibly failing. A
strict CAST can report such a failure as an error, while a non-strict
CAST or TRY_CAST may produce NULL. The new API has no production callers
yet.

…on mapping (apache#68180)

An IVM materialized view has to fully rebuild every MV partition that
may hold rows of a base-table
partition that was dropped / truncated / replaced / recovered. Those
operations change the table
through metadata and emit no row binlog, so the incremental path can
never remove the stale rows: if
such an MV partition is not marked, the orphan rows stay in the MV
forever, with no error anywhere.

Which MV partitions to mark was decided by asking the refresh snapshot
which of them had *seen* the
changed base partition (`MTMVRefreshSnapshot#getMvPartitionNames`). A
snapshot is captured before the
base table is read, so it is only a lower bound: a base partition that
appears in the base table after
the capture is not in it, and each partition's snapshot is written back
by whichever task finishes
last. When at least one other MV partition did match, the lookup
returned that subset and the partition
that had actually read the dropped one was silently left out:

```
  partition p1: capture snapshot            p1 reads the base table   p1 still running
  base table:                             X is added   p2 refreshes (its snapshot has X)   X dropped
  marker:  p1's snapshot has no X, p2's has  ->  only p2 is marked
  result:  p1 keeps the rows of X forever
```

This asks the MV partition mapping instead: which MV partitions read the
changed base partition is
exactly the question, it is metadata rather than a lagging record, and
it is the same mapping the
refresh already uses for partition sync.

| case | before | after |
| --- | --- | --- |
| MV partitions that read the changed base partition | marked | marked
(unchanged) |
| changed base partition read by no MV partition | whole MV rebuilt |
nothing marked |
| SELF_MANAGE MV, base table is not a PCT table, `RECOVER PARTITION`, a
PCT table of a multi-PCT MV, mapping computation fails | whole MV
rebuilt | whole MV rebuilt (unchanged) |

The second row is intentional: the old lookup could not tell "no MV
partition reads this partition"
apart from "the snapshot does not know it", so it rebuilt the whole MV
-- a table whose dynamic
partitions are dropped ahead of the MV's partition sync forced a full
rebuild every time. The mapping
tells them apart, and the cases that genuinely cannot be answered from
it are explicit checks
rather than a guess from an empty result:

- `RECOVER PARTITION` marks before the partition is added back to the
table, so the partition is still
in the recycle bin at that moment and no metadata-derived mapping can
describe it.
- The mapping is seeded from the MV's PCT tables and never gains a table
later, so a joined partition
table the MV's partition column does not reach is not described by it at
all.
- A mapping computation failure must not fail the base table DDL: it
warns and rebuilds the whole MV.
- The mapping reads the partition items of the MV and of every PCT
table, so it takes their read locks,
while the caller already holds the changed table's write lock: two
concurrent partition DDLs on two PCT
tables of the same MV would each hold the write lock the other one
needs. Those reads are therefore
taken with a bounded `tryReadLock` (`Table.TRY_LOCK_TIMEOUT_MS`, the
same way the stream cleanup treats
a busy table) and never block: while the other tables are free -- the
normal case -- the selection still
narrows to the MV partitions that read the changed base partition, and
while one of them is being
written the whole MV is rebuilt. Acquiring in id order cannot help here,
because the first lock of the
  pair is already held before this point.

The mapping is computed *before* the MV lock is taken, not inside it:
computing it takes the partition
items of the MV and of its PCT tables, and the MV lock is a leaf lock --
nothing is acquired under it
today, and the lock analysis of the baseline barrier depends on that.
The selection does not need to be
atomic with the barrier it produces: the barrier is recorded under the
lock, and the names it carries
are intersected with the live partition names when they are consumed
(`MTMVTask`).

`MTMVRefreshSnapshot#getMvPartitionNames`, the old selection, is removed
with the unit test that
covered it; it has no other caller.

Problem Summary:

Add simdutf 9.2.0 as a backend third-party dependency. Register its
source archive and checksum, build and install the static library and
public headers, add the imported library to the backend dependency list,
and record its Apache-2.0 license. Existing UTF-8 validation
implementations and callers are unchanged.

Existing prebuilt third-party installations need the new library before
building BE. Refresh the third-party package or run
`./thirdparty/build-thirdparty.sh -j 48 simdutf` in the source checkout.
The regular third-party build includes it in the default package list.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [ ] Unit Test
    - [x] Manual test (add detailed scripts or steps below)
        - `bash -n thirdparty/vars.sh thirdparty/build-thirdparty.sh`
- `./thirdparty/build-thirdparty.sh -j 48 simdutf` completed on Linux
x86_64 with Clang 20.1.8, including download verification,
static-library/header installation, and source cleanup.
- Configured and built a standalone CMake smoke test using
`be/cmake/thirdparty.cmake`; verified the imported archive path and
membership in `COMMON_THIRDPARTY`, then linked and ran UTF-8 validation
and UTF-8-to-UTF-16 conversion checks against the installed library.
- `bash build-support/check-build-hygiene.sh` and `git diff --check`
passed.
        - Full BE compilation and ARM execution were not run.
    - [ ] No need to test or manual test. Explain why:

- Behavior changed:
    - [x] No.
    - [ ] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…as (apache#68019)

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

`SELECT lazy_col AS x, lazy_col AS y FROM t ORDER BY x LIMIT 1` failed
planning with
`A expression contains slot not from children`.

The TopN order key is the alias slot, so that alias has to be computed
below the TopN.
`MaterializeProbeVisitor` only protects the order key slot itself (an
order key slot is in
`TopN.getInputSlots()`) and never resolves an identity alias down to the
column the alias reads.
The probe of the other output (`lazy_col AS y`) therefore resolved to
the base column `lazy_col`
and classified it as lazily materialized, so `LazySlotPruning` removed
`lazy_col` from the scan while
`lazy_col AS x` below the TopN still read it. The final `Validator`
rejected the resulting plan and
the query returned an error. With `fe_debug=true` the failure was caught
inside `LazyMaterializeTopN`
instead, which silently skipped lazy materialization (the query
succeeded but lost the optimization).

Reproduction (master, `fe_debug=false`):

```sql
create table t(sort_col int, lazy_col int) duplicate key(sort_col)
  distributed by hash(sort_col) buckets 1 properties('replication_num'='1');
select lazy_col as x, lazy_col as y from t order by x limit 1;
-- ERROR 1105: A expression contains slot not from children
--   Slot: lazy_col#1  Children Output:{0, 4}
--   Plan: PhysicalProject[lazy_col#1 AS x#2, __DORIS_GLOBAL_ROWID_COL__t#4]
--         +--PhysicalLazyMaterializeOlapScan[PhysicalOlapScan[t]]
```

Fix: `LazyMaterializeTopN` resolves the TopN order keys through the
identity alias chain of the
Projects under the TopN and adds the resolved slots (plus the
intermediate alias slots) to
`requiredMaterializedSlots`, so the probe rejects every lazy candidate
backed by a column an order
key reads. The resolution stops at set operations, which the probe never
materializes through
(lazy materialization is not supported through set operations today; if
that ever changes, order
keys have to be resolved per branch).

Effect: affected plans now either keep only the ordering column
materialized (other columns are
still fetched lazily) or skip lazy materialization, and the plan stays
valid. Plans that order by a
plain column are unchanged.

### Release note

TopN lazy materialization no longer builds an invalid plan (no more
`A expression contains slot not from children`) when a query orders by
an alias of a column.
The column that feeds the order key is materialized during the scan,
while other columns keep using
lazy materialization.

### Check List (For Author)

- Test
- [x] Regression test
(`regression-test/suites/query_p0/topn_lazy/order_by_alias`)
- [x] Unit Test (`TopnLazyMaterializeTest`, `LazyMaterializeTopNTest`)
- Behavior changed:
- [x] Yes. Queries that order by an alias of a projected column no
longer fail planning; the
ordering column is kept materialized instead of being pruned from the
scan.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…Ranger's user store (apache#68220)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68203 (documents the behaviour; this PR changes it)

Problem Summary:

A Ranger policy item can name users, groups and roles. The requests the
`ranger-doris` and `ranger-hive`
sources build carry no groups - Doris has none of its own to hand over -
so every policy item written
against a group never matches: an allow on a group grants nothing to a
Doris user, a row filter or a mask
on a group does not apply, and a **deny on a group is silently ignored**
while a user-level allow on the same
table still admits the user. Operators who manage access by Ranger group
(so that nobody edits a policy
every time somebody joins a team) see the user denied on everything,
down to `SHOW DATABASES` and
`SWITCH`, and it starts working the moment the same user is put on the
item directly.

Ranger's answer for a plugin that cannot compute groups itself is the
**user store** - the users and groups
Ranger Admin holds, kept current by usersync - which a plugin downloads
next to its policies. Ranger 2.5+
reads the requesting user's groups out of it when
`ranger.plugin.<type>.use.rangerGroups=true`; the
plugin library on branch-3.0/2.1 is 2.4.0, which has no such setting, so
on those branches no
configuration can make group items work.

What this PR does:

- `RangerUserStoreGroups` (ranger-common): the two Ranger plugins Doris
embeds put the user store
enricher on the service definition they download (`setPolicies`),
exactly as `RangerBasePlugin` does for
`use.rangerGroups`, and the request builders read the user's groups out
of the downloaded store and set
them on every request (`RangerAccessRequestImpl.setUserGroups`). This
does not depend on the Ranger
  version the plugin was built against.
- On by default. `ranger.plugin.doris.use.rangerGroups=false`
(`ranger.plugin.hive.` for the catalog-level
source) switches it off and restores the previous requests; the property
is the one Ranger itself reads
for the same thing, so one setting decides both. The refresh interval
and retriever class are read under
Ranger's own option names (`userStoreRefresherPollingInterval`, default
60000 ms).
- Each plugin logs one line at start-up saying whether groups are
attached and how to switch it off.
- Regression: `ranger_p2/test_ranger_group_policy` writes an access
policy, a row filter, a mask and a
deny against a Ranger group only, and checks them as a member of that
group, then after leaving it.
`httpTest` gains `op "put"` for the Ranger user update that bumps the
user store version.

Verified end to end against the `docker/thirdparties` Ranger (Admin
2.4.0): group allow / row filter /
mask apply, a group deny outranks a user allow, a Ranger role granted to
a group resolves, leaving the group
revokes within one user store refresh, and `use.rangerGroups=false`
gives the previous behaviour.

What this PR also changes: a Ranger plugin is built with its service's
policies, or not at all

`RangerBasePlugin.init()` survives a Ranger Admin it cannot reach: it
answers out of the local policy cache
when there is one, and out of no policies at all when there is not,
refusing every check until a later poll
succeeds. That was already so before this PR, and it is the wrong state
for Doris to start in: an FE with
`access_controller_type=ranger-doris` and no policies is one nobody can
use (no account bypasses the source),
and a catalog bound to `ranger-hive` in that state refuses every
statement, with nothing but a line in fe.log
to say why. So the plugins now refuse that state at the moment they are
built, and Ranger's own resilience
takes over afterwards. The load order at start-up, and where it fails:

```text
FE start with access_controller_type=ranger-doris            (every FE: master, follower, observer)
  Env()
   └─ AccessControllerManager()
       └─ RangerDorisAccessControllerFactory.create()
           └─ new RangerDorisPlugin("doris")            ... on the starting thread, synchronously
               └─ LoadedRangerPlugin.init()
                   ├─ RangerBasePlugin.init()
                   │   ├─ audit subsystem
                   │   ├─ PolicyRefresher.startRefresher()
                   │   │   ├─ loadRoles()   ──REST──► Ranger Admin     (fails: logged; roles from cache, or none)
                   │   │   ├─ loadPolicy()  ──REST──► Ranger Admin
                   │   │   │    ├─ answered                  → setPolicies(policies)          policies version ≥ 0
                   │   │   │    ├─ unreachable, cache found  → setPolicies(cached policies)   policies version ≥ 0
                   │   │   │    ├─ unreachable, no cache     → setPolicies(null): no engine   policies version -1
                   │   │   │    └─ service not found         → cache cleared, no engine       policies version -1
                   │   │   │         setPolicies() puts the user store enricher on the service definition:
                   │   │   │           user store ──REST──► Ranger Admin   (fails: logged; no groups until it arrives)
                   │   │   └─ refresher thread + download timer start      (from here on: Ranger's own resilience)
                   │   └─ chained plugins (none configured)
                   └─ policies version < 0 ?
                       ├─ no  → plugin built.  INFO "loaded in N ms: policies version ...", WARN if no user store
                       └─ yes → cleanup(), throw IllegalStateException("... has no policies to authorize against ...")
                                 └─ Env() fails → the FE does not start

CREATE CATALOG ... 'access_controller.class'='ranger-hive'
  CatalogFactory.createCatalog()
   └─ catalog.initAccessController(dryRun = true)
       └─ RangerHiveAccessControllerFactory.create()
           └─ RangerHiveAuditStack.startFor(serviceName)
               └─ new RangerHivePlugin(serviceName)     → the same LoadedRangerPlugin.init() as above
                   ├─ built → controller released again (the stack stays up 300 s for the bind that follows) → CREATE succeeds
                   └─ threw → DdlException "Failed to init access controller: ..."                          → CREATE fails

Later binds of a ranger-hive catalog (the first statement against it, after ALTER CATALOG, after the 300 s idle stop):
  the same init(); a failure fails that statement and the next statement tries again. Never a fallback to the
  built-in source.
```

Configuration errors (no `policy.rest.url`, a REST timeout that is not a
number, an audit destination Doris
does not ship) fail the way they always did: inside
`RangerBasePlugin.init()`, before anything is started. A
missing user store is not a failure: an Admin that serves none cannot be
told from one that has not answered
yet, so policy items written against a group simply do not apply until
it arrives, and the plugin says so
with a WARN.

…#68168)

### What problem does this PR solve?

Problem Summary:

This PR addresses three unstable streaming job regression cases:

1. `test_streaming_postgres_job_lag`
2. `test_streaming_mysql_job_lag`
3. `test_streaming_job_cdc_stream_postgres_pause_resume`

The first two lag cases assumed that a source write immediately after
job creation would always be captured by an `offset=latest` reader. If
the reader initialized after that write, no event remained to initialize
the lag metrics. They now generate source updates only while lag metrics
are unavailable, crossing the reader initialization window without fixed
sleeps or production behavior changes.

The third case exposed a production reader lifecycle race. After pause
and resume, a canceled job-driven `cdc_stream` TVF request could overlap
the successor task. Both requests reused the job-scoped `SourceReader`,
allowing the old request to poll or close the successor's fetcher and
causing a row inserted during pause to be missed. Each job-driven TVF
request now claims a fresh reader instance, displaced tasks stop reading
or publishing offsets, and request cleanup releases only its captured
reader while preserving source-side resources such as the PostgreSQL
replication slot. Standalone TVF cleanup and FROM-TO reader reuse remain
unchanged.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

When use set profile level on follower FE and then send insert into
select stmt to follower, it will be forward to master FE, and profile
level is lost.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

`cut_ipv6` did not zero the requested number of trailing bytes when the
IPv6 truncation count was odd. Root cause: the formatter copied the
address in 16-bit words, so the final iteration also copied one byte
that belonged to the truncated tail. This change preserves the existing
word-wise path and handles a remaining single byte separately, ensuring
that every requested trailing byte is zeroed.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Related PR: apache#32617

fix deep match expr utils duplicate walk

…umn pruning (apache#67718)

### What problem does this PR solve?

Issue Number: N/A

Problem Summary:

Nested column pruning rewrites a struct-field access over `TRY_CAST`
into a plain `CAST` and narrows both the read and the conversion to the
accessed field, dropping the whole-value semantics of TRY_CAST.

Reproduction (single FE/BE, `enable_strict_cast=true`):

```sql
CREATE TABLE t (id INT, s STRUCT<a:STRING,b:STRING> NULL)
DISTRIBUTED BY HASH(id) BUCKETS 1
PROPERTIES ("replication_num"="1");
INSERT INTO t VALUES
  (1,named_struct('a','bad','b','2')),
  (2,named_struct('a','10','b','bad')),(3,NULL);

-- with nested column pruning enabled (default)
SELECT id, TRY_CAST(s AS STRUCT<a:INT,b:INT>)['a'] AS a_i FROM t;
```

- id=2: `b='bad'` makes the whole-struct TRY_CAST fail and should yield
NULL, but pruning drops field `b`, the conversion "succeeds" on
`a='10'`, and the query returns 10.
- id=1: the rebuilt plain CAST (instead of TRY_CAST) throws
`[INVALID_ARGUMENT] parse number fail, string: 'bad'` under
`enable_strict_cast=true` instead of returning NULL.
- Root cause: `AccessPathExpressionCollector.visitCast` handles
`TryCast` (a `Cast` subclass) like a plain cast and translates the
narrowed nested access path through it, so
`SlotTypeReplacer.rewriteCast` later rebuilds the expression as `new
Cast(...)` with a pruned target type.

Fix: do not translate a narrowed nested access path through a
`TRY_CAST`; the collector then falls back to reading the whole child
value, so the struct keeps all fields and the expression keeps its
TRY_CAST identity and full target type. Field-by-field pruning of plain
`CAST` over nested types is unchanged.

### Release note

None

### Check List (For Author)

- Test:
- [x] Unit Test:
`PruneNestedColumnTest#testTryCastNotPrunedThroughFieldAccess` (whole
class: 62 tests pass)
- [x] Manual test (detailed steps): reproduced the SQL above end-to-end
on a local FE+BE cluster built from this change. With
`enable_prune_nested_column` on and off all three rows return NULL (no
10, no exception under `enable_strict_cast=true`); `EXPLAIN VERBOSE`
keeps `element_at(TRY_CAST(s AS struct<a:int,b:int>), 'a')` with `all
access paths: [s]`, while plain `CAST` over the same struct is still
pruned to `struct<a:int>` with `[s.a]`.

- Behavior changed:
- [x] Yes: enabling/disabling nested column pruning no longer changes
TRY_CAST results or error semantics.

- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…tion metadata scan (apache#68227)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Queries on `information_schema.table_stream_consumption` always scanned
all streams and partitions, even when predicates such as `DB_NAME`,
`STREAM_NAME`, `STREAM_ID`, or `UNIT` could exclude most of them.

In cloud mode, this also caused unnecessary partition requests to
MetaService.

### What is changed?

- Forward schema-scan predicates from BE to FE for
`table_stream_consumption`.
- Push down `DB_NAME`, `STREAM_NAME`, and `STREAM_ID` predicates before
scanning stream metadata.
- Push down `UNIT` predicates after partition enumeration:
- In cloud mode, only matching partitions are requested from
MetaService.
  - In non-cloud mode, rows for unmatched partitions are not generated.

### What problem does this PR solve?

Problem Summary:

Before this patch each MTMV kept two MTMVCache (cacheWithGuard /
cacheWithoutGuard) that held Nereids plan trees.

The lifetime of MTMVCache is same as the MTMV object, when these
MTMVCaches are at large scale (e.g. thousands of mtmv were created and
refreshed, but they were not dropped afterwards) may ocupy lots of FE
heap space.

A heap-histogram check showing 72540 MTMVCache instances held ~30GB in
FE heap on the affected cluster.

This pr replace the origin MTMVCaches with Caffeine cache keyed by
(mtmvId, guarded), using maximumSize + expireAfterAccess so cold entries
are evicted under memory pressure instead of accumulating forever.

Initial config, all mutable via ADMIN SET FRONTEND CONFIG:

```
mtmv_cache_manage_num=3000, 
expire_mtmv_cache_in_fe_second=86400,
mtmv_cache_hot_show_num=500.
```

Also support observability:

```
SHOW PROC '/mtmv_cache/stat' returns:
size / hitCount / missCount / evictionCount / loadFailureCount / hitRate

SHOW PROC '/mtmv_cache/hot' returns MtmvId / DbName / MvName / Guarded / IdleMs
ordered by most-recently-accessed for MVs
```

This pr refers to the NereidsSqlCacheManager introduced by apache#33262

…67878)

## Problem

Nereids inferred that a grouped aggregate output was unique whenever its
input slot was unique. A non-injective argument expression or a lossy
result conversion can make different groups produce the same value, so
eliminating a later `GROUP BY` can return duplicate rows or incorrect
counts.

## Fix

Trace the aggregate argument back to a slot through injective casts
using the existing `getExpressionCoveredBySafetyCast` helper. For `MIN`
and `MAX`, this is sufficient for single-row groups. For `SUM` and
`AVG`, also require an injective conversion from the original slot type
to the aggregate result type, using `DataType.isInjectiveCastTo`
(updated on master by apache#68134).

This reuses the shared cast rules instead of maintaining an
aggregate-specific cast whitelist. Truncating `CHAR`/`VARCHAR` casts
remain non-injective because their parsed argument includes a
`substring` expression, which cannot be traced back through injective
casts. The same aggregate check is used by logical and physical traits.

… DorisArrowTypeMapping (apache#68315)

### What problem does this PR solve?

Issue Number: apache#67577

Related PR: apache#66344 (the `GetTables` schema fix whose nested-type tests
move here), apache#67966 / apache#68266 (the Flight SQL session work the next
interfaces build on)

Problem Summary:

**Context.** Doris speaks two client protocols: MySQL and Arrow Flight
SQL (the Flight SQL JDBC driver, the ADBC drivers, Python clients). Over
Flight the rows come as Arrow batches produced by BE, and
`convert_to_arrow_type` in `be/src/format/arrow/arrow_row_batch.cpp`
decides which Arrow type each Doris type has on the wire.

Besides data, Flight SQL has metadata commands.
`GetTables(include_schema = true)` is the client asking "which columns
does this table have, and what Arrow type is each one?", and FE answers
with a serialized Arrow schema in the `table_schema` column. Clients
trust that schema: the ADBC driver types its columns from it and then
decodes the batches of later queries as those types. So what FE says has
to match what BE sends; a mismatch is not a degraded answer but a failed
read (DATEV2 described as date64 while BE emitted date32 broke exactly
that way, fixed in apache#66344).

For that FE keeps a Doris-to-Arrow mapping, which until now was a
private method of `FlightSqlSchemaHelper`, the class that serves
`GetTables`: `getArrowType`, plus the `buildField` / `arrowChildren`
pair that builds a field and its nested children.

**1. The problem, and what it cost**

- A second and a third caller are about to arrive. The next Flight SQL
work all produces Arrow schemas: more metadata commands, the parameter
and result schemas of prepared statements, `ExecuteSchema`. They ask the
same question, "what Arrow type is this Doris type", but the answer was
a private method of another class, so each of them could only copy the
switch. Copies drift: it already happened once between FE and BE (apache#66761
added TIMESTAMP_NS as one line on each side), and another copy inside FE
would let `GetTables` describe a column as one type and a prepared
statement as another.
- The table was not pinned. The existing tests covered nested types and
DATEV2; the scalar table itself could change without any test noticing.
And some cells are known to disagree with BE and are kept that way on
purpose: TIMESTAMPTZ carries the literal zone `"UTC"` (BE stamps the
session time zone), and TIMEV2 / VARBINARY / AGG_STATE come back as
`Null` (BE emits float64 / binary / binary). They are kept because the
BE Arrow type layer is being reworked by another team and changing the
mapping now would collide with that, so the correction is scheduled as
one step after it (recorded in apache#67577). Without a test pinning them,
nothing stopped a well-meant one-cell "fix" that would change behaviour
piecemeal and out of step.

**2. What this PR does, and why it helps**

The mapping moves into a class of its own,
`org.apache.doris.arrow.DorisArrowTypeMapping`, the one place in FE that
maps a Doris type to an Arrow type, with no value changed. Three
commits:

1. A table-driven unit test against the old code first: one row per
`PrimitiveType` (41 values; the types whose unit depends on the scale
get a row per band, 47 rows in all), the known-wrong cells pinned as
they are and marked "kept as is (apache#67577)", plus a check that every
`PrimitiveType` has a row. Green against the old private method.
2. The move. The switch, the nested-type rules and the column metadata
are byte for byte what they were; the test's rows do not change and stay
green, which is the evidence for "no value changed".
3. A regression suite, `arrow_flight_sql_p0/test_get_tables_schema`: a
raw Flight SQL client asks a live cluster for `GetTables` of a table
that declares a column of every type (three-level nesting, BITMAP / HLL
/ AGG_STATE, DECIMAL256 included) and pins, field by field, the type,
the nullability and the column metadata a client sees. It covers what
the unit test cannot: the whole path from `describeTables`' descriptors
through the mapping to the serialization and the client's decoding.

What it buys:

- The coming callers call, they do not copy; every schema FE hands out
agrees with every other by construction.
- When the BE rework lands and the mapping is corrected, the change is
made in this one class (plus the tests' expectations) and every schema
changes together, instead of call site by call site.
- The known-wrong cells are pinned by two layers of tests that say why;
any change to them fails a test, so it can only be made deliberately.
- A new `PrimitiveType` without a mapping fails the test instead of
silently falling into `default -> Null`.
- Nothing changes for users: `GetTables` returns the same bytes. Checked
by dumping every field of the `GetTables` result with pyarrow from an FE
built before this PR and from one built at its head: 52 lines,
identical.

**3. The classes, and how they call each other**

- `DorisFlightSqlProducer` (existing): the Flight SQL server; on
`CommandGetTables`, `getStreamTables` creates a `FlightSqlSchemaHelper`.
- `FlightSqlSchemaHelper` (existing, slimmed): the `GetTables` plumbing
only. It lists databases and tables and calls `describeTables` through
`FrontendServiceImpl` to get each column's `TColumnDesc` (a thrift
descriptor with precision, scale and nested children), hands each column
to the mapping for an Arrow `Field`, and `getSerializedSchema` writes
the fields as Arrow IPC bytes into `table_schema`.
- `DorisArrowTypeMapping` (new):
- `toArrowType(PrimitiveType, precision, scale)`: the table itself (the
switch), a mirror of BE's `convert_to_arrow_type`.
- `toArrowType(TColumnDesc)`: reads precision and scale off the
descriptor and calls the above.
- `toField(db, table, TColumnDesc)`: builds the `Field`: the type, the
nullability, the Flight SQL column metadata a JDBC `ResultSetMetaData`
reads (`TYPE_NAME` / `PRECISION` / `SCALE` / `SCHEMA_NAME` /
`TABLE_NAME` ...), and the children, recursively (an ARRAY's `item`, a
MAP's `entries<key, value>` with the key forced non-nullable, a STRUCT's
fields).
- `FrontendServiceImpl.getColumnDesc` (existing, untouched): turns a
catalog `Column` into a `TColumnDesc`.
- BE `convert_to_arrow_type` (untouched): the sole authority on the
shape of the data on the wire; FE's table mirrors it.
- Tests: `DorisArrowTypeMappingTest` (table + nesting),
`FlightSqlSchemaHelperSerializedSchemaTest` (serialization round trips),
regression `arrow_flight_sql_p0/test_get_tables_schema`.

```
Flight SQL client (JDBC / ADBC)
   |  GetTables(include_schema = true)
   v
DorisFlightSqlProducer.getStreamTables
   |  new FlightSqlSchemaHelper(ctx).getTables(...)
   v
FlightSqlSchemaHelper ----> FrontendServiceImpl.getDbNames / listTableStatus / describeTables
   |                                    '- Column --getColumnDesc--> TColumnDesc (precision / scale / children)
   |  per column: DorisArrowTypeMapping.toField(db, table, desc)        <-- the new class, moved out of the helper
   |                 |- toArrowType(desc) --> toArrowType(primitiveType, precision, scale)   [the table = mirror of BE convert_to_arrow_type]
   |                 |- flightSqlColumnMetadata(...)                                          [TYPE_NAME / PRECISION / SCALE / SCHEMA_NAME / TABLE_NAME]
   |                 '- children(...) --> toField on each child, recursively                 [ARRAY item / MAP entries<key, value> / STRUCT fields]
   |  getSerializedSchema(fields) --> Arrow IPC bytes
   v
table_schema column --> the client deserializes it and types its columns from it
                    --> decodes the batches of later queries with those types (batches produced by BE per convert_to_arrow_type)

Callers after this PR: metadata commands / prepared statement parameter & result schemas / ExecuteSchema --> the same DorisArrowTypeMapping
```

Not touched, so nobody goes looking: BE is unchanged, and FE-side
results (`SHOW ...`, all utf8 in `FlightSqlChannel`) are unchanged too;
that is another recorded item that also waits for the BE rework.

…e#68342)

### What problem does this PR solve?

Cloud MOW INSERT stopped retrying delete bitmap lock errors and
calculation timeouts after the insert executor switched to the
attachment-aware commit overload. That overload performs one attempt
because BE-driven loads already retry at their caller. Ordinary INSERT
is affected even when its attachment is null.

Add an explicit FE-owned retry entry and reuse the existing cloud retry
loop. Nereids INSERT now uses this entry, preserving the transaction
attachment and table stream updates on every attempt. Keep BE-driven
commit entry points single-attempt, the existing attempt limit and
signature lifecycle, and shared-nothing publish-timeout behavior.

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary: VExpr::execute_column accepted a non-nullable column
when the expression declared Nullable(T), even though DataTypeNullable
requires a ColumnNullable representation. Remove this expression-only
fallback so data type nullability and column nullability remain aligned.

### Release note

None

### Check List (For Author)

- Test: No need to test (pipeline validation requested)
- Behavior changed: Yes (strictly enforce nullable column output
matching)
- Does this need documentation: No

…pache#68396)

### What problem does this PR solve?

Related PR: apache#68381. This is the master version of the second split from
apache#67784, based on the primitives merged in apache#68301.

Arrow batch conversion mixes protocol serialization with table-specific
UUID handling, while writers construct schemas separately. Introduce
explicit Doris, Python, Arrow Flight, Parquet, Hive, Iceberg and Paimon
convertors with instance-owned schema parameters and timezone. Move
schema construction/decoding into the convertors and route nested SerDe
writes through the selected format.

Separate Parquet, Hive and Iceberg writers and migrate existing callers.
Preserve master's tracked Arrow memory pools, Iceberg statistics and
timestamp-nanosecond support. Master does not yet contain the Paimon
write backend or physical Variant table writes present on branch-4.1;
this pick adds the converter interfaces without importing those
features. Parquet timestamp encoding and external type mappings remain
unchanged.

Include the Python timezone regression correction from apache#68381: the
single string output uses ARRAY<STRING>, so the lateral-view comparison
reaches execution instead of failing on a STRUCT-versus-STRING
comparison. Retain coverage for four session timezones, microseconds,
pre-epoch values, NULLs, UDF, UDTF and UDAF.

### Release note

Fix Python UDF timestamp conversion to preserve wall-clock values when
the Arrow protocol declares a fixed-offset timezone.

### Check List (For Author)

- Test
- [x] Unit Test: explicit schemas and independent converter instances,
nested/null values, UUID and fixed binary bytes, timestamp bindings, and
Iceberg writer statistics.
- [x] Regression test: Python UDF/UDTF/UDAF timezone comparisons and the
existing timestamp snapshot corrections.
- Behavior changed:
- [x] Yes: align Python UDF conversion with its Arrow timezone
declaration; reject invalid nested bindings before casts.
- Does this need documentation?
    - [x] No.

Validation: ASAN BE build and 310 focused tests passed (53 suites),
covering Arrow conversion, Parquet/ORC, Variant SerDe and Python. All 42
affected C++ files passed clang-format 16, and header hygiene passed.
The original UDTF declaration reproduced the SQL analysis error on an
isolated FE; the corrected declaration passed the same analysis. Groovy
and embedded Python checks passed. Full Python SQL and external-catalog
regressions remain for CI. clang-tidy was attempted: the new converter's
size warning was resolved; analysis remains blocked by a pre-existing
unmatched NOLINTEND in core/types.h.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…pache#68244)

### What problem does this PR solve?

Issue Number: None

Problem Summary: `JsonbToJson::string_to_json()` walks the string with
`ptr != str + len && *ptr`, so it stops at the first NUL byte even
though the length says the string is longer. U+0000 is a legal character
inside a JSON string and the parser stores it by length, so every path
that renders a JSONB value back to text loses everything after it:

```
SELECT k, v FROM (SELECT 1) d
LATERAL VIEW json_each_text(concat('{"x":["a', unhex('5C'), 'u0000b"]}')) t AS k, v;
-- before: x  ["a"]
-- after:  x  ["a\u0000b"]

SELECT c FROM (SELECT 1) d
LATERAL VIEW explode_json_array_string(concat('[["a', unhex('5C'), 'u0000b"]]')) t AS c;
-- before: ["a"]
-- after:  ["a\u0000b"]
```

The same string as a direct array element is copied by length and
already keeps the NUL, which is why only nested complex values are
truncated.

Bound the loop by the length only. The NUL then reaches the escaping
branch and is written as `\u0000`, like any other control character.

### Release note

None

### Check List (For Author)

- Test:
- Unit Test: Yes, `JsonbParserTest.ParseJsonWithEscapedNulInString`,
`ParseJsonWithEscapedNulInNestedArray` and
`ParseJsonWithEscapedNulInKey` round trip a U+0000 through the JSON text
output.
- Regression test: Yes, `test_json_text_embedded_nul` covers
`json_each_text` and `explode_json_array_string` over nested values that
contain U+0000.
- Behavior changed: Yes, textualized JSON values keep everything after a
U+0000 and escape it as `\u0000` instead of being silently truncated.
- Does this need documentation: No

https://claude.ai/code/session_01RZ36Pij3o8fGnYYg33PKnq

### What problem does this PR solve?

Problem Summary:

Fix two predicate inference errors that change query results:

- `chooseInputPredicates` records a retained `GT`/`GTE` predicate as
`EQ`. The false equality can make another necessary predicate appear
redundant. For example, `a > b AND rn > b AND a > rn` can lose `a > rn`
and return extra rows in a QUALIFY query. Record the actual relation in
the working graph.
- Comparison-equal operands are not interchangeable inside arbitrary
expressions. DATE and DATETIME values can compare equal but have
different string lengths; negative and positive floating-point zero
compare equal but have different SIGNBIT results. Substitution can
therefore add filters that discard matching rows. Restrict expression
substitution to exactly matching supported scalar types with
value-preserving equality. Preserve direct comparisons, IN and their
negations on the unwrapped operand, along with the existing determinism
and cast-analysis checks.

The fixes are in separate commits. Shared cast extraction is unchanged,
and existing regression expectations are unchanged.

### Release note

Fix extra or missing rows caused by incorrect predicate inference,
including QUALIFY inequality chains and substitutions inside
type-sensitive or representation-sensitive expressions.

)

### What problem does this PR solve?

Remove `test_remote_doris_variant_select` from `master` external
regression. In [build
1054262](http://43.132.222.7:8111/viewLog.html?buildId=1054262&buildTypeId=Doris_External_Regression),
the case failed while inserting its test data with an unsupported
conversion from `Variant(max subcolumns count = 0)` to `Variant(max
subcolumns count = 2048)`.

This removes the regression case and its coverage; it does not fix the
underlying VARIANT conversion behavior. The case compares query results
directly and has no checked-in expected-output file.

### Release note

None

### Check List (For Author)

- Test
    - [x] No need to test or manual test. Explain why:
- [x] Other reason: deletion of one test file; verified no remaining
references to its name and ran `git diff --cached --check`.
- Behavior changed:
    - [x] No production behavior change.
- Does this need documentation?
    - [x] No.

…#68380)

`array_pushback` and its `array_append` alias could read past the end of
the NULL bitmap when the array argument was a constant NULL column
evaluated for multiple rows. This caused unstable NULL propagation and
could lead to undefined behavior. The implementation now uses the
constant-column row index for both the NULL bitmap and array offsets,
preserving scalar NULL semantics across all input rows.
### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…#68131)

### What problem does this PR solve?

Issue Number: apache#48203

Related PR: apache/doris-website#4151

Problem Summary:

Doris does not provide `encode(string, charset)` and `decode(binary,
charset)` scalar functions with Hive-compatible character conversion
semantics, which makes Hive workload migration harder.

This PR:

- adds Nereids signatures and FE constant folding for `encode` and
`decode`;
- requires the charset argument to be a supported string literal or
`NULL`; columns and other expressions are rejected during analysis so
the vectorized BE can select the conversion path once per query;
- adds vectorized BE implementations backed by simdutf with strict
malformed and unmappable input handling;
- supports `US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16BE`, `UTF-16LE`,
and `UTF-16`, matched case-insensitively;
- follows Java/Hive UTF-16 BOM behavior and propagates nulls after
validating the charset literal, so an unsupported charset remains an
error even when the data argument is `NULL`;
- adds FE/BE unit tests and SQL regression coverage for supported
conversions and error cases.

The conversion semantics are Hive-compatible for the supported character
sets. Doris intentionally restricts the charset argument to a literal
(or `NULL`) rather than supporting a per-row charset column.

### Release note

Add `encode` and `decode` scalar functions with Hive-compatible
conversion semantics for `US-ASCII`, `ISO-8859-1`, `UTF-8`, `UTF-16BE`,
`UTF-16LE`, and `UTF-16`. The charset argument must be a supported
string literal or `NULL`.

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [x] Regression test
- `test_encode_decode` passed in generated-output and comparison modes
on a native Linux Doris cluster.
    - [x] Unit Test
- FE `EncodeDecodePlannerTest`: 2 tests passed, covering planner folding
and charset-validation precedence over NULL folding.
        - FE `EncodeDecodeTest`: 4 tests passed.
        - FE Maven reactor succeeded.
- BE `function_character_encoding_test.*`: 3 tests passed before the
simdutf refactor; the same suite covers the simdutf implementation in
CI.
        - Native Linux FE/BE build succeeded.
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
- [x] Yes. Adds `encode` and `decode`, including UTF-16 BOM semantics,
strict conversion errors, FE constant folding, and the literal charset
contract.

- Does this need documentation?
    - [ ] No.
    - [x] Yes. apache/doris-website#4151

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…er (apache#68900)

### What problem does this PR solve?

Issue Number: DORIS-29598

Related PR: apache#68864

Problem Summary:

Backport the DORIS-29598 regression expectation fix from `branch-4.1` to
`master`. Recent TIMESTAMPTZ mappings changed the rendered values in the
OceanBase, SQL Server, MaxCompute, and Hudi external regression suites.
The HTTP Parquet TVF cases also need an explicit Parquet timezone so
their results do not depend on the session or reader default.

This PR updates only regression suites and golden files:

- update TIMESTAMPTZ output precision and session-timezone rendering;
- preserve wall-clock DATETIME and TIMESTAMP_NTZ expectations;
- update SQL Server `datetimeoffset` CTAS output to `timestamptz(6)`;
- pin affected suites to `Asia/Shanghai`;
- set `hive.parquet.time-zone = Asia/Shanghai` for HTTP TVF
`sql08/sql09/sql10`.

### Release note

None

### Check List (For Author)

- Test
    - [x] Regression test
    - [x] Manual test (local timestamp conversion and CTAS type probe)
- Behavior changed:
    - [ ] No.
- [x] Yes. This changes regression expectations only; no production
FE/BE code is changed.
- Does this need documentation?
    - [x] No.

Validation:

- Local `test_http_tvf` passed: 1 suite, 0 failures, 0 skips, including
`sql08/sql09/sql10`.
- Local SQL probe verified SQL Server `datetimeoffset` conversion to
`Asia/Shanghai` and CTAS type `timestamptz(6)`.
- `git diff --check` passed and the master backport was reviewed against
`upstream/master`.
- JDBC, Hudi, and MaxCompute external fixtures were unavailable in the
personal environment; their full runtime validation is delegated to
community CI.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### What problem does this PR solve?

Issue Number: N/A

Problem Summary:

MySQL queries sent over TLS can fail with parser errors near a TLS
record boundary. Root cause: after removing the MySQL header with
`compact()`, the receive loop compares the declared payload length
against the buffer limit, which becomes its capacity, instead of the
number of decrypted bytes received. If the payload length falls between
the current position and capacity, the loop skips a required TLS record
and exposes unreceived bytes to the parser.

Compare against `result.position()` so the receive loop collects the
complete payload before processing it.

### Release note

Fix SQL truncation and parser errors when a MySQL query spans TLS
records near the receive-buffer boundary.

### Check List (For Author)

- Test:
    - [x] Manual test
    - Built FE with `./build.sh --fe -j48`; Checkstyle passed.
- Used a PyMySQL TLS connection to send a COM_QUERY with a 16,380-byte
payload. Splitting the 16,384-byte MySQL packet after byte 16,380 or
16,383 produced parser errors before the fix; the same requests produced
a result-set response after rebuilding and restarting FE with the fix. A
complete 16,384-byte write also succeeded.
- Repeated with 16,400-byte and 17,000-byte payloads split at bytes
16,380, 16,383, and 16,384; all produced result-set responses after the
fix.
- `git diff --check` passed. Automated unit and regression tests were
not run.
- Behavior changed:
- [x] Yes. Read all TLS records needed to complete the MySQL payload.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…nge (apache#67032)

### What problem does this PR solve?

Issue Number: close apache#67031

Problem Summary:

`PipelineTask::set_running` is the scheduler's double-execute gate
(`TaskScheduler::_do_work` re-queues a task when it returns true) and,
via `RevokableTask`'s delegation, the mutual exclusion between a task
and its in-flight spill revocation. It was implemented as a single
`compare_exchange_weak` attempt. Weak CAS may fail spuriously on LL/SC
architectures (aarch64): on a spurious failure with `_running == false`
the caller is told "was not running" while the flag is never set, so a
second worker acquires the same task and both execute it concurrently —
one thread's `reset_hash_table`/`close()` then races the other's
`sink_impl` on shared local state (production SIGSEGV @0x0 in
`AggSinkOperatorX::sink_impl` on Graviton BEs, including two workers
crashing in the same second attached to the same query). x86 compiles
weak CAS as never-spurious, so this only reproduces on ARM.

`std::atomic::exchange` keeps the exact intended semantics (return the
old value, always store the new one) and cannot fail spuriously. Every
other `compare_exchange_weak` in be/src sits in a retry loop; this was
the only no-retry gate.

Adds `TEST_SET_RUNNING_MUTUAL_EXCLUSION`: 8 threads hammer
acquire/release and assert at most one concurrent holder (catches the
old implementation probabilistically on ARM hardware; a hard invariant
for the new one).

### Release note

Fix a BE crash on aarch64 where two pipeline workers could execute the
same task concurrently under spill/memory-pressure churn.

### Check List (For Author)

- Test
    - [x] Unit Test

- Behavior changed:
    - [x] No.

- Does this need documentation?
    - [x] No.

Co-authored-by: Claude Fable 5 <[email protected]>

Problem Summary:

The manual runtime canary still installs Codex 0.156.1. Pin its package
URI, archive SHA256 and manifest version to the complete Linux x64
0.162.1 package built from `3547e9086675d6b9aedfae84c15c33b2fdc155fa`.

This stages the new runtime for a subsequent manual
`runtime_canary=true` run. The production review job continues using its
existing 0.156.1 package. Authentication selection, model choices,
triggers and the canary checks are unchanged.

Build: https://github.com/shuke987/codex/actions/runs/38138448659

Archive SHA256:
`51304e4f9c343ae84e00a304344df41c2fd32821c83bdbf5be68b0f0c25ddae1`.

…e#68312)

### What problem does this PR solve?

Problem Summary:

A query that has already computed one aggregate state per key still
needs an aggregate operator to obtain its final values through
`<agg>_merge`. This repeats aggregation for the finest grouping when
states are also reused for coarser rollups.

Add the scalar `<agg>_finalize(state)` combinator. It returns one result
for each input state, using the existing aggregate implementation and
serialized-state representation. For example:

```sql
SELECT k, avg_finalize(s)
FROM (SELECT k, avg_combine(v) AS s FROM t GROUP BY k) partial;
```

The same implementation supports aggregates such as `count`, `sum`,
`min`, `max`, and `array_agg`. FE validates that the state's canonical
aggregate name matches the finalizer, derives its result type, and
treats the function as scalar. BE handles the underlying serialized
column type, skips outer NULL payloads, and releases temporary state
after each row. Constant inputs use the ordinary scalar constant path.
Existing empty-state semantics and serialized formats are unchanged.

MV rollup matching follows only direct MERGE/UNION state chains for the
same aggregate and preserves the full value expressions of
STATE/COMBINE. This prevents scalar finalizers nested in their arguments
from making different outer aggregates or different value expressions
appear equivalent, while retaining compatible state rollups.

Grouped `*_union` and `*_merge` now delegate to the existing vectorized
deserialize-and-merge interfaces, including the selected path. The
nested aggregate exposes whether its vector readers need scratch: direct
native readers receive a null scratch pointer, while readers that need
temporary states receive batch-local aligned storage. Nested
implementations manage construction/destruction, and the caller Arena
retains destination allocations. This avoids shared mutable scratch
buffers, unused scratch allocations and per-row range dispatch. The
Nullable selected fast path reads each selected row independently so
skipped destinations cannot shift the serialized input cursor.
Single-place merging and FE plans are unchanged. Q67 A/B performance has
not been measured.

Scalar finalization uses `deserialize_from_column_row()` to read
directly into its initialized empty state. Generic string-backed readers
and native serialized column readers implement the same contract, with
forwarding through nullable and state-combine wrappers. This removes the
extra temporary state allocation and merge previously performed by the
generic range interface for each finalized row; the caller retains
responsibility for state construction, destruction and exception
cleanup.

### Release note

Add `<aggregate>_finalize(AGG_STATE)` scalar functions to retrieve each
aggregate state's result without merging rows.

### Check List (For Author)

- Test
- [x] Regression test: test_agg_state_finalize on a fresh local ASAN BE
+ FE cluster; output generated by the standard runner, checked against
direct original aggregates, and verified by a normal comparison run.
- [x] Regression test: agg_state_finalize_roll_up verifies MV rejection
for differing outer aggregates/scalar arguments and selection for
matching rollups, including negative values and NULLs. Generated output
was checked independently; the normal comparison run of this suite and
test_agg_state_finalize passed.
- [x] Unit Test: 51 FE tests passed across CombinatorRollupHandlerTest,
CountTest, FinalizeCombinatorTest, StateCombinatorTest,
CombineCombinatorTest, and FunctionRegistryTest. The 9 new rollup tests
reproduced 6 failures before the fix and all pass after it. All 9
FunctionAggStateFinalizeTest cases passed under ASAN.
- [x] Manual test: compare grouped and empty AVG/COUNT/SUM/MIN/MAX,
decimal AVG and ARRAY_AGG with the original aggregates; verify rollup
AVG is 14/3 and distinguish a missing outer-join state from COUNT's
empty state.
    - [ ] No need to test or manual test. Explain why:
- Behavior changed:
    - [ ] No.
- [x] Yes. Add a family of scalar finalization functions; MV rollup
matching now preserves complete value expressions; existing
aggregate/state evaluation semantics are unchanged.
- Does this need documentation?
    - [ ] No.
- [x] Yes. Usage and semantics are included in the function-combinator
README.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### Additional validation

- Standard ASAN BE + FE build, FE Checkstyle, clang-format 16, build
hygiene and source whitespace checks passed.
- clang-tidy was attempted with the repository script and the
production/test translation units. New-code style and added
cognitive-complexity warnings were fixed. A fully clean run remains
blocked by existing header diagnostics and analyzer issues (including a
test-helper array-bound path that does not connect its asserted unary
arity to the input array); the local tool's crashing
`modernize-use-scoped-lock` check was disabled only for the supplemental
analysis. No repository analysis configuration was changed.

### Batch state merge validation

- `./build.sh --be -j 48` completed successfully under ASAN.
- 19 BE unit tests passed across AggregateStateUnionTest,
AggregateStateCombineTest and FunctionAggStateFinalizeTest. Six new
tests cover repeated groups, nonzero offsets, empty/all-unselected
batches, repeated batches, plain/Nullable/all-NULL SUM, Decimal SUM,
string ARRAY_AGG and nullable string MIN. The Nullable selected-row test
fails before its reader fix and passes afterward.
- `test_agg_state_union_batch` and `test_agg_state_finalize` both passed
in normal regression comparison mode. New expected output was generated
with the standard runner and independently checked against ordinary
aggregates, arithmetic sums, Decimal sums and lexicographic MIN. The new
suite also verifies that plans retain sum_merge/sum_union, and compares
ARRAY_AGG results after multiple input batches.
- clang-format 16.0.6, header hygiene and source whitespace checks
passed. clang-tidy was run via the repository script; a fully clean run
remains blocked by existing diagnostics in types.h (unmatched NOLINTEND)
and DataTypeAggState's constructor, plus existing signed/unsigned
comparisons in the Nullable wrapper. No unrelated analysis
configuration/source changes were made.
- Q67 A/B performance has not been measured.

### Scratch and direct-row reader validation

- 83 BE unit tests from 10 suites passed under ASAN, including
AggregateStateUnionTest, FunctionAggStateFinalizeTest,
AggregateFunctionExceptionTest and native aggregate reader suites.
Coverage includes scratch requirements, selected rows, nullable
wrappers, direct row reads, source-buffer lifetime and exception
cleanup.
- clang-format 16.0.6, header hygiene and source whitespace checks
passed. Repository clang-tidy checks produced no diagnostics on the
changed lines; a fully clean run remains blocked by the existing
unmatched NOLINTEND in `be/src/core/types.h`.
- No new SQL regression run or throughput benchmark was performed for
these two commits.
- The wide-state batch deserialization finding in [discussion
4217446484](apache#68312 (comment))
is intentionally left unchanged in this PR, following discussion. It is
not claimed as fixed by the scratch-allocation or direct-row
finalization changes.

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68446

Ordinary IN scans currently send the complete point-key range list to
every selected tablet. FE tablet pruning reduces the tablet set but each
remaining tablet still probes keys belonging to other buckets. For
example, 55 keys across 43 selected tablets produce 2,365 tablet/range
associations.

This change groups final exact point ranges by the table's hash bucket
before creating scanners. FE attaches the full bucket count and ordinal
from the existing tablet map; BE uses the same raw-byte CRC32 as VARCHAR
distribution. Ordinary scanners and both parallel splitting strategies
consume bucket-local ranges, and empty groups skip scanner/segment
loading. Residual predicates and versioned read sources stay on the
normal scan path.

Initial scope: unpartitioned base indexes with one non-null VARCHAR
storage key equal to the sole HASH distribution column. Other schemas,
rollups, special scans, and non-point/coalesced ranges retain the
existing path. `enable_scan_key_bucket_prune` defaults to true and
provides a per-session A/B switch. Optional Thrift metadata preserves
mixed-version behavior. Two Profile counters expose tablet/range
associations before and after grouping.

### Performance validation

Same RELEASE binary, local single FE/BE on a shared 192-logical-CPU
host; 1M MOW rows, 128 buckets, eight insert batches, auto compaction
disabled, 4KB string payload. Each query selects address/chain/payload
for an IN list and `chain = 1`. Parallel scan enabled; short-circuit,
SQL cache and query cache disabled. Each cell has 50 samples; paired
OFF/ON order is randomized and all 1,000 timed query results match.

`doris_cache_cold` clears the dedicated BE's Doris caches before each
sample. OS caches remain uncontrolled; this does not model cold
object-storage reads. Compare OFF/ON within a row, not across cache
modes.

| Cache | IN keys | OFF p50 ms | ON p50 ms | p50 reduction | OFF p95 ms
| ON p95 ms |
|---|---:|---:|---:|---:|---:|---:|
| hot | 4 | 12.306 | 12.327 | -0.2% | 14.861 | 14.358 |
| hot | 9 | 11.956 | 11.738 | 1.8% | 15.550 | 14.778 |
| hot | 30 | 13.324 | 12.665 | 4.9% | 16.209 | 17.543 |
| hot | 55 | 16.221 | 13.691 | 15.6% | 17.982 | 14.849 |
| hot | 100 | 77.752 | 37.955 | 51.2% | 100.982 | 46.330 |
| doris_cache_cold | 4 | 8.263 | 8.257 | 0.1% | 9.886 | 9.912 |
| doris_cache_cold | 9 | 9.583 | 9.320 | 2.7% | 13.308 | 11.846 |
| doris_cache_cold | 30 | 13.207 | 11.947 | 9.5% | 15.255 | 13.696 |
| doris_cache_cold | 55 | 16.720 | 14.012 | 16.2% | 19.495 | 16.214 |
| doris_cache_cold | 100 | 94.023 | 44.950 | 52.2% | 225.355 | 108.331 |


For 55 keys, range associations drop from 2365 to 55 (97.7%); end-to-end
p50 drops about 16%. Separately captured warm profiles show
`GenerateRowRangeByKeysTime` 13.560 → 1.653 ms and cumulative
`ScannerCpuTime` 46.460 → 31.784 ms (single samples). Small lists show
little gain; hot 30-key p95 increased in this run, so this is not a
universal tail-latency improvement.

<details>
<summary>Workload construction and timing controls</summary>

```sql
CREATE TABLE scan_key_bucket_perf (
  address VARCHAR(80) NOT NULL, chain INT NOT NULL, payload STRING
) UNIQUE KEY(address) DISTRIBUTED BY HASH(address) BUCKETS 128
PROPERTIES("replication_num"="1", "enable_unique_key_merge_on_write"="true",
           "disable_auto_compaction"="true");
```

Insert eight batches of 125,000 rows from `numbers("number"="125000")`.
In batch `b`, let `i = number + b * 125000`; address is
`concat('address', lpad(cast(i as string), 9, '0'))`, chain is `i % 3`,
and payload is `repeat(sha2(cast(i as string), 256), 64)`.

For list size `n`, keys are generated with Python
`random.Random(68446+n).sample(range(1000000), n)` and the same address
formatting. Query all three columns using the generated IN list and
`chain = 1`. Set `max_scan_key_num=1024`,
`max_pushdown_conditions_per_column=1024`, `enable_parallel_scan=true`,
`enable_short_circuit_query=false`, `enable_sql_cache=false`, and
`enable_query_cache=false`. Toggle `enable_scan_key_bucket_prune` per
trial. Hot mode uses five warm-ups per setting; cold mode calls
`/api/clear_cache/all` on the dedicated BE before each sample. Fetch all
rows, measure with `perf_counter_ns`, and compare sorted result
checksums. Profiles are collected separately with both paths warmed.

</details>

### Release note

Improve eligible VARCHAR IN scans by pruning foreign-bucket point ranges
before scanner creation. The `enable_scan_key_bucket_prune` session
variable controls the optimization.

### Check List (For Author)

- Test
- [x] Regression test: generated and replayed
`query_p0/test_scan_key_bucket_prune`; 35 result blocks covering ON/OFF,
parallel ON/OFF, range coalescing, residual predicates, DUP rows, MOW
updates/deletes, and unsupported nullable/composite/partitioned layouts.
Eight setting combinations agree; generated results also match
independent Python references.
- [x] Unit Test: 10 BE tests under ASAN
(`ScanKeyBucketPrunerTest.*:ScannerLateArrivalRfTest.*`) and 14 FE tests
(`OlapScanNodeTest`).
- [x] Manual test: RELEASE BE/FE builds, FE checkstyle, clang-format
16.0.6, build/header hygiene, and the A/B measurements above.
- Behavior changed: Yes, eligible scanners receive bucket-local point
ranges; SQL results and storage formats are unchanged.
- Does this need documentation: No, internal scan optimization; switch
description is included in session-variable metadata.

Static analysis: the repository clang-tidy script was run with commands
expanded from the unity compilation database. The completed run exits
nonzero on an unmatched `NOLINTEND` in `be/src/core/types.h:576`,
confirmed present in the base commit. Other emitted diagnostics also
point to unchanged code. Comparing every emitted diagnostic with the
final diff found no diagnostics on changed lines; this is not reported
as a clean clang-tidy pass.

…ad of returning NULL (apache#68359)

### What problem does this PR solve?

Issue Number: None

Problem Summary:

`regexp_extract`, `regexp_extract_or_null`, `regexp_extract_all`,
`regexp_extract_all_array`, `regexp_replace` and `regexp_replace_one`
handled
an invalid regex pattern differently depending on where the pattern came
from.
A constant pattern is compiled in `open()` and the query fails with
`INVALID_ARGUMENT`. A pattern read from a column is compiled per row in
the
execute path, and the same compile failure was swallowed: the row was
turned
into NULL and only a warning was recorded.

```sql
CREATE TABLE t (id INT, s STRING, p STRING, repl STRING) ...;
INSERT INTO t VALUES (1, 'abc', '[', 'x');

SELECT regexp_extract('abc', '[', 0);      -- INVALID_ARGUMENT
SELECT regexp_extract(s, p, 0) FROM t;     -- NULL
SELECT regexp_replace('abc', '[', 'x');    -- INVALID_ARGUMENT
SELECT regexp_replace(s, p, repl) FROM t;  -- NULL
```

So the same malformed pattern was either rejected or silently produced
NULL
depending on the physical shape of the argument, which hides data
problems
behind an unrelated NULL. `regexp_count` and `regexp` already raised the
error
on the column path.

This PR makes the column path of the extract/replace families throw the
same
`Status::InvalidArgument` that the constant path returns from `open()`,
so an
invalid pattern always fails the query. The null-map plumbing that only
existed
to mark those rows is removed, together with the now-unused `push_null`
handlers of `regexp_extract_all`.

Raising an error on the column path must not swallow SQL NULL
propagation. The
framework's default NULL handling strips a Nullable argument to its
nested
column and executes every row, and the bytes under a NULL slot are
whatever
the producer left there, so `regexp_replace(s, concat(p, if(id = 2,
NULL, '')), repl)`
could still carry `[` under a NULL pattern. The regexp functions
therefore
handle NULL themselves now: a shared helper strips Nullable from the
arguments,
ORs their null maps into the result null map and short-circuits a NULL
constant
argument, and the execute loops skip NULL rows. No bytes under a NULL
slot are
ever compiled and no argument column is copied.

The shared helper chooses the NULL handling once per block. When no
argument
is Nullable (`have_null_column()`), the original columns go straight to
the
function with no input NULL map and the per-row NULL checks compiled out
(`if constexpr (CheckNull)`); only Nullable arguments take the unnesting
and
NULL-row skipping path. Input NULL checking is kept separate from output
nullability: `regexp_extract_or_null` still returns NULL on no match for
non-nullable input, and a nullable `regexp_count` result still gets its
wrapper. `regexp_count`, the replace family and the extract family all
go
through this one dispatch.

### Release note

`regexp_extract`, `regexp_extract_or_null`, `regexp_extract_all`,
`regexp_extract_all_array`, `regexp_replace` and `regexp_replace_one`
now fail
the query with `INVALID_ARGUMENT` when a regex pattern read from a
column does
not compile, matching the behavior for constant patterns. Previously
such rows
returned NULL.

### Check List (For Author)

- Test:
- Unit Test: `FunctionLikeTest.regexp_invalid_column_pattern_fails`
covers
      every family with a mixed valid/invalid pattern column;
`FunctionLikeTest.regexp_null_pattern_hides_invalid_payload` covers a
      nullable pattern column whose NULL row (placed between valid rows)
hides an invalid payload, a NULL constant pattern, and a NULL input
      string with all-constant parameters;
      `FunctionLikeTest.regexp_count_non_nullable_arguments` and
`FunctionLikeTest.regexp_non_nullable_arguments` cover non-nullable
arguments with column and constant patterns, including the NULL that
      `regexp_extract_or_null` returns on no match; the existing
      `FunctionLikeTest.*` and `function_string_test.*` still pass.
- Regression test: new `test_regexp_invalid_pattern` compares the
constant
      and column paths for all families, checks NULL propagation with an
invalid payload under a NULL pattern between valid rows, and runs every
      family over a NOT NULL table; `test_string_function_regexp`,
      `test_regexp_count`, `test_regexp_replace`,
      `test_regexp_repeated_search_anchor`, `test_regexp_chinese`,
      `test_split_by_regexp`, `nereids_function_p0` `R` and
      `fold_constant_nullable` pass locally.
- Behavior changed: Yes. An invalid pattern read from a column now fails
the
  query instead of yielding NULL for that row.
- Does this need documentation: No

https://claude.ai/code/session_01CvzmDA8snJbHHhErBB58sB
https://claude.ai/code/session_01PqyWdaYLRvhohSJqcn14zH

…el close (apache#68628)

### What problem does this PR solve?

When `VTabletWriter` opens new node channels during execution, it closes
channels in two stages. Initial channels receive `hang_wait` EOS
requests; senders wait for this first-stage barrier before closing their
incrementally opened channels.

The receiver currently polls `is_finished()` inside the heavy work pool.
Early EOS tasks can occupy every worker while waiting for the remaining
senders, whose EOS tasks are queued in the same pool. For example, 32
waiting EOS tasks can prevent the remaining 68 requests in a 100-sender
load from running. Normal progress then depends on external cancellation
or timeout cleanup.

The first-stage barrier protects against closing an incremental channel
before another sender has finished incrementally opening it. Its success
condition is that all senders have reached EOS. The existing `kFinished`
transition occurs before final flush/commit, allowing earlier senders to
proceed with incremental-channel close while the final sender completes
storage work.

Replace polling with a one-shot sender-arrival completion shared by each
tablets channel:

- Early EOS tasks release their heavy workers while their RPCs remain
pending. Existing sender deduplication/counting still elects the final
closer in both local and cloud modes.
- When the final sender arrives, set `kFinished` under the channel lock,
release the lock, and notify earlier EOS waiters. Reacquire the lock for
writer finalization. This preserves the existing arrival barrier and
overlap between initial- and incremental-channel close.
- The final EOS RPC still waits for its own flush/build/commit and
synchronous response writes. Finalization errors and final
tablet/row-count results are returned on that RPC; errors cannot revoke
success already delivered to earlier senders.
- Closed channels explicitly skip block writes, including block-bearing
EOS retries during arrival notification. A retry of the final sender
that holds the active channel waits for finalization and receives its
status and copied tablet results; earlier sender retries retain the
arrival barrier.
- Register RPC callbacks only after synchronous response/profile/timing
writes finish. Transfer the BRPC closure before registration, which may
complete inline. Callbacks run outside channel/completion locks.
- Cancellation or channel expiration releases pending waiters with
`Cancelled` if cancellation wins before arrival success is published.
The first terminal outcome wins; subsequent cancellation does not
overwrite published success.
- No protobuf or sender protocol changes.

This shared path is used by:
- Auto-partition loads that open new node channels while creating
partitions.
- `INSERT OVERWRITE ... PARTITION(*)`, which creates replacement
temporary partitions at runtime and reuses the partition-creation
callback to incrementally open channels. This can also affect ordinary
partitioned tables.
- Whole-table `INSERT OVERWRITE` with automatic partition creation
enabled, when new node channels are opened during execution.

The trigger is runtime creation of new node channels followed by the
two-stage EOS barrier in `VTabletWriter`. Incrementally adding tablets
to existing node channels alone does not set
`has_incremental_node_channel()`.

Based on master `d2cfcc1dcd4019fbae0d6ffbce16a7563bd9ecf4`.

### Release note

Fix a possible heavy work pool starvation deadlock during incremental
channel close in auto-partition loads and INSERT OVERWRITE operations
that open new node channels. Preserve the existing sender-arrival
barrier: earlier EOS RPCs may return once all senders reach EOS, before
final flush/commit; the final RPC still waits for finalization. Pending
EOS waiters now receive `Cancelled` when cancellation/expiration wins
before barrier success, rather than potentially receiving OK after
cancellation.

…ache#68865)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#65868

Problem Summary:

**In short.** This ports the time-zone part of branch-4.1 apache#65868.
`time_zone` accepts Java short IDs such as JST, PST or IST, but the FE
forwards them as written, except for CST. Backends do not know these
names: they silently evaluate time functions in another zone, and Java
code running on a backend fails on them. The FE now sends the canonical
zone ID wherever a time zone leaves the session layer: to backends for
queries, loads, BE constant folding and point queries, and to
connectors.

**Background.** `SET time_zone` validates the name against
`TimeUtils.timeZoneAliasMap`, which is `ZoneId.SHORT_IDS` plus the Doris
overrides (CST and PRC map to Asia/Shanghai, UTC and GMT to UTC), and
stores the name as written. When the FE evaluates time itself, it
resolves the name through that map. The places that hand the time zone
on behave differently:

| Where the time zone goes | master before this PR |
|---|---|
| Query globals (`CoordinatorContext`), BE constant folding,
short-circuit point queries | raw name, CST special-cased to
Asia/Shanghai |
| Broker load and export (`setQueryGlobalsForLoad`), stream and routine
load (`NereidsStreamLoadPlanner`) | raw name |
| Connectors (`ConnectorSessionBuilder`) | raw name |

branch-4.1 apache#65868 added `TimeUtils.getCanonicalTimeZone`
(`ZoneId.of(name, timeZoneAliasMap).getId()`) and used it for the query,
constant-folding and point-query paths. master does not have it.

**The problem, and what it cost.**

- **Wrong results, no error.** A backend looks the name up in its tz
database, which has only EST, HST and MST of the 28 Java short IDs. For
the others (JST, PST, IST, CTT and 20 more; CST is special-cased),
`RuntimeState` ignores the failed lookup and keeps UTC, and a
short-circuit point query falls back to the backend's default zone.
Constant expressions folded on the FE are still right, so one query can
mix two zones.
- **Loads convert in the wrong zone.** A stream load or broker load
whose `timezone` is such a name converts with the same fallback.
- **Java code on the backend fails.** It resolves the name with
`ZoneId.of`, which rejects short IDs. A Paimon INSERT under JST fails
with `ZoneRulesException: Unknown time-zone ID: JST`.
- **Connectors get the raw name.** The Iceberg and Paimon connectors
carry their own copies of the alias map. The MaxCompute predicate
converter does not, so for a session in CST it stops pushing down
datetime predicates.

**How this PR fixes it.**

- Adds `TimeUtils.getCanonicalTimeZone`, as in branch-4.1.
- Applies it in `CoordinatorContext.refreshQueryGlobals` (queries, group
commit, jobs with a session), `createQueryGlobalsWithoutSession` (jobs
started by FE daemons), `setQueryGlobalsForLoad` (broker load, export),
`NereidsStreamLoadPlanner` (stream and routine load, transactional
INSERT), `FoldConstantRuleOnBE`, `PointQueryExecutor` and
`ConnectorSessionBuilder`. The CST special cases go away, because the
alias map already resolves CST to Asia/Shanghai.
- Names a backend already understands are unchanged (`Asia/Tokyo`,
`UTC`, `+08:00`, `GMT+08:00`). Short IDs become tz database names or
offsets: JST becomes `Asia/Tokyo`, PST `America/Los_Angeles`, EST
`-05:00`.
- A zone that is UTC at every instant goes out as `UTC`. Java names
`+00:00` `Z` and `GMT+00:00` a bare `GMT`, which a backend resolves only
from the optional `Zulu` and `GMT` zone files (Debian 13's `tzdata` has
no `Zulu`); `UTC` needs no zone file.
- The other time zones a backend receives need no change.
`hive.parquet.time-zone` rejects short IDs when the catalog or TVF is
created. The zones a backend hands on, to Java scanners and writers,
Arrow Flight results and partial updates, come from the query globals.

**Results.** Local FE + BE, a row with `ts` = 0, `time_zone` = JST:

| | before | after |
|---|---|---|
| `select from_unixtime(ts) from t` | 1970-01-01 00:00:00 | 1970-01-01
09:00:00 |
| the same query as a short-circuit point query | 1970-01-01 08:00:00 |
1970-01-01 09:00:00 |
| stream load with `timezone: JST` and `dt = from_unixtime(ts)` |
1970-01-01 00:00:00 | 1970-01-01 09:00:00 |
| INSERT into a filesystem Paimon table | fails: `Unknown time-zone ID:
JST` | succeeds |

PST, IST and CTT behaved like JST before the fix. EST and CST were
already right.
Co-authored-by: Socrates <[email protected]>

### What problem does this PR solve?

`NereidsParser.parseProperties` parsed `propertyItemList` without
requiring the parser to reach the end of the input. As a result,
`tokenize` silently accepted invalid tokens after a valid property
assignment. For example:

```sql
SELECT tokenize('alpha beta', '"parser" = "unicode" trailing_tokens');
```

The ignored suffix could hide a misspelled or incorrectly assembled
property string.

### What changed?

- Add `propertyItemListWithEof` and use it for standalone property
parsing. SQL statements that embed `propertyItemList` keep their
existing grammar.
- Preserve valid single and multiple properties, including trailing
whitespace and comments.
- Reject trailing identifiers, semicolons, closing parentheses,
unexpected symbols, and assignments without a separating comma.
`tokenize` returns its existing property format error.
- Add parser and `Tokenize` unit tests, plus two error cases in the
`test_tokenize` regression suite.

…s from branch-4.2 (apache#68918)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68532, apache#68786

Problem Summary:

**In short.** This ports the tests of branch-4.2 apache#68532 that master does
not have yet. Master re-implemented apache#68532 in apache#68786; a hunk-by-hunk
comparison found all of its product behavior on master, but about twenty
of its assertions had no master equivalent. This PR adds them, fixes
three Paimon tests that set a misspelled option, and changes no product
code.

**Background.** apache#68532 (merged into branch-4.1 and synced to branch-4.2)
maps external binary columns to VARBINARY and instant timestamps to
TIMESTAMPTZ across Hive, Iceberg, Paimon, JDBC, Hudi, MaxCompute and the
Trino connector. On master the external catalogs live in
`fe/fe-connector/*`, so apache#68786 re-implemented the change on the
connector plugins rather than cherry-picking it, and moved most of the
apache#68532 tests along. The ones listed below were not moved.

**The problem, and what it cost.** Without these tests nothing on master
fails if one of these behaviors regresses:

| Area | Behavior with no test on master |
| --- | --- |
| Iceberg partitions | 0x-hex BINARY/FIXED/UUID commit values are
decoded, and wrong lengths or malformed hex are rejected; transform
ranges carry `+00:00` bounds and the proleptic year zero; zoned and
unzoned partition timestamps round-trip in every session zone, including
both instants of a DST fold; commit values honor their explicit offsets
|
| Iceberg writes | zoned write defaults keep the instant and nested
binary defaults stay bytes with the old flags off; binary/UUID commit
values become typed partitions; a NULL static overwrite on a
binary/fixed/uuid key replaces only the NULL partition (full and hybrid
specs) and reaches the sink as a NULL marker distinct from bytes; MODIFY
to TIMESTAMPTZ on a zoned field is a no-op while DATETIMEV2 is rejected;
Iceberg does not use Hive's INT96 time zone |
| fe-core | TIMESTAMPTZ range partitions keep UTC bounds and a non-empty
NULL-min range in UTC, Asia/Shanghai and America/New_York sessions; the
legacy JDBC clients that CDC still uses map binary and instant types
regardless of the old flags, and PostgreSQL `timestamptz` becomes
nullable |
| Paimon | nested LTZ write columns stay TIMESTAMPTZ; LTZ partition
values keep pre-epoch and DST-fold instants in every zone; legacy ORC
LTZ files route to the SDK for MAP/MULTISET/ROW leaves and mixed splits
|
| JDBC / HMS / Trino | per-dialect binary and timestamp mappings;
calendar functions on instants, TIMESTAMPTZ literals, PostgreSQL decoded
NULLs and dialect-specific binary literals stay local; Hive binary
partition keys keep their text type; Trino domains encode UTC micros and
millisecond-aligned zoned bounds; the JDBC writer sets the MySQL session
to UTC before it prepares the INSERT |
| Regression | a Paimon LTZ INSERT SELECT of two instants that fold to
the same Los Angeles time; a Spark cross-check of LTZ rows written in a
non-UTC session, including a pre-epoch instant; a native V1 read of the
Paimon timestamp-scale tables |

Separately, three `PaimonScanPlanProviderTest` cases set
`orc.timestamp-ltz.legacy-type`; the real key is
`orc.timestamp-ltz.legacy.type`, so the setting was ignored and the
tests passed only because the default is `true`.

**How this PR fixes it.** It ports each missing apache#68532 assertion to the
master class that now owns the behavior (mostly connector tests),
adapted to the connector APIs; follows master's regression style
(`order_qt_` blocks plus Spark/Doris cross-checks instead of literal
asserts); and corrects the misspelled option key.

**Results.** The 19 changed test classes pass locally. To check that the
new tests are not vacuous, six product-code mutations were applied
across five modules: dropping the binary-literal pushdown guard, an
always-true NULL static overwrite filter, no FIXED length check, parsing
the NULL-min sentinel as a date, ignoring MULTISET when routing legacy
ORC LTZ, and initializing the MySQL UTC session after the prepare. Each
mutation failed exactly the corresponding new test and nothing else.
Both Paimon regression suites pass against the iceberg docker
environment, with the new `.out` blocks generated by the framework and
matching apache#68532's expected values (for example epoch micros
`1699173000123456` and `1699176600123456` for the fold).

Co-authored-by: Gabriel <[email protected]>

…he#68931)

Related PR: apache#68928; shuke987/codex#5

Problem Summary:

Regular PR reviews still install Codex 0.156.1, while the complete
0.162.1 Linux x64 package has passed the real-account runtime canary.
Update the production package URI, archive SHA256, manifest version and
CLI SHA256 to the tested package built from
`3547e9086675d6b9aedfae84c15c33b2fdc155fa`.

Keep the production model `gpt-6-sol`, fallback `gpt-5.6-sol`, and
`xhigh` effort unchanged. This is the runtime rollout; switching the
production model is a separate follow-up.

Prepare that follow-up by adding required `gpt-6.1-sol` canary calls at
`xhigh`, both plain exec and goal mode. Preserve the existing checks and
their effort settings. A failed 6.1 call or missing completion event
fails the canary; the prior-model probe remains informational. Increase
the model-test step timeout from 12 to 18 minutes for up to five calls,
each bounded to 180 seconds.

… threads (apache#68902)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#67695

Problem Summary:

**In short.** This ports the tests of branch-4.1 apache#67695, which made
Iceberg maintenance actions run inside the catalog's authentication
scope, and fixes what the ported Kerberos case found on master: on a
Kerberos-secured catalog, `expire_snapshots` committed its new metadata
and then failed while Iceberg read the expired snapshots' manifest lists
on its worker threads, which do not carry the catalog's Kerberos
identity. `cherrypick_snapshot`, `publish_changes` and
`rewrite_data_files` planning open files on those threads too.

**Background.** On branch-4.x an action loaded the table under the
catalog's authentication context and then read and committed metadata
after that context had ended, so on a Kerberized HDFS catalog the commit
could run under the FE process identity. apache#67695 captures the action's
metadata operations and the catalog's authenticator as one generation,
runs table acquisition and the commit in that scope, and retries once
when the catalog is reset before the mutation starts.

On master the actions live in `fe-connector-iceberg`:
`ConnectorExecuteAction` calls `IcebergProcedureOps.execute`, which runs
the table load, the procedure body and its commit inside one
`ConnectorContext.executeAuthenticated`. Each call takes its catalog ops
and its authenticator from the same connector instance, and the catalog
generation it starts with is leased until the call ends
(`IcebergCatalogResourceTracker`), so a table from a newer generation
can never meet an older authenticator, and the generation fence and
retry of apache#67695 have nothing to port.

That `doAs` covers only the calling thread, though. Iceberg reads and
writes metadata files on its shared worker pool, and those threads run
without the plugin's Kerberos login. For scans and writes the connector
already wraps the table's `FileIO` so that every file is opened inside
the catalog's `doAs` (`IcebergScanPlanProvider.wrapTableForScan`,
`IcebergConnectorTransaction.openTransaction`); a file opened that way
keeps its authenticated connection on any thread. Procedures were handed
the raw table.

**The problem, and what it cost.**

1. On a Kerberos-secured HDFS catalog, a procedure fails as soon as
Iceberg opens a metadata file on a worker thread. The first CI run of
this PR hit it: `test_iceberg_hadoop_catalog_kerberos` failed at
`expire_snapshots` with `Failed to expire snapshots: Failed to open
input stream for file: hdfs://.../metadata/snap-....avro`.
`RemoveSnapshots` had already committed the new metadata, so the
snapshots were gone while their files stayed on HDFS, the statement
reported an error, and the FE did not refresh the table.

| Procedure on a Kerberos HDFS catalog | Opened on a worker thread |
Before | After |
|---|---|---|---|
| `expire_snapshots`, when something expires | the expired snapshots'
manifest lists and manifests, after the commit | commits, then fails |
succeeds |
| `cherrypick_snapshot` / `publish_changes`, when main has moved on |
the manifest of the picked files | fails | succeeds |
| `rewrite_data_files` on a table with more than one manifest | the
manifests, while planning | fails | succeeds |

2. Two of apache#67695's tests have no counterpart on master:

| branch-4.x test | master |
|---|---|
|
`ExecuteActionCommandTest.executesActionWithinCatalogAuthenticationScope`
| `IcebergProcedureOpsTest` only asserts that one auth scope is entered.
Loading the table inside `executeAuthenticated` and running the body
after it returns keeps every existing case green. |
| `test_iceberg_hadoop_catalog_kerberos`: `rollback_to_snapshot` and
`expire_snapshots` on the Kerberized HDFS catalog | The suite creates,
inserts and counts, but runs no action. |

**How this PR fixes it.**

- `IcebergProcedureOps.wrapTableForProcedure`: the procedure body and
the rewrite planner get the table with its `FileIO` wrapped like the
scan and write paths. Non-Kerberos catalogs keep the loaded table
unchanged.
- `IcebergAuthenticatedFileIO.withPrefixOperations`: unlike the scan and
write wrapper, the procedure's `FileIO` keeps the delegate's prefix
listing, inside the same `doAs`, because `remove_orphan_files` rejects a
`FileIO` without it. Bulk deletes stay hidden, so deletes go file by
file through the authenticated `deleteFile`; `HadoopFileIO`'s own bulk
delete would run on its private pool, outside any `doAs`. One visible
effect: `expire_snapshots`' `max_concurrent_deletes` used to be ignored
with a warning on these catalogs, because the raw `HadoopFileIO`
supports bulk deletes, and now takes effect.
- Tests:
- `IcebergProcedureOpsKerberosIoTest` (new) runs procedures on a local
Hadoop table through an authenticator that marks only the thread inside
its `doAs`, the way a Kerberos `Subject` is invisible to pool threads,
and a `FileIO` that records every call made outside it:
`expire_snapshots` after the rollback sequence of the Kerberos suite,
`cherrypick_snapshot` after main has moved on, `remove_orphan_files`,
and `rewrite_data_files` planning on a two-manifest table. A
non-Kerberos catalog keeps the loaded table.
- `IcebergAuthenticatedFileIOTest`: prefix listing and deletion run
inside `doAs`, bulk deletes are not exposed, and a delegate without
prefix listing gets none.
- From apache#67695:
`IcebergProcedureOpsTest.commitsTheProcedureInsideTheAuthScope` reads
the table's current snapshot just before `executeAuthenticated` leaves
its scope and asserts the rollback has already been committed (the test
`RecordingConnectorContext` gains a hook that runs at that point);
`test_iceberg_hadoop_catalog_kerberos` gains the two steps of apache#67695
unchanged: a second insert, `rollback_to_snapshot` to the first snapshot
(two rows back to one), a third insert, and
`expire_snapshots("retain_last" = "1")` (three snapshots down to one).


`ExecuteActionCommandTest.retriesRollbackOnCatalogAuthenticationGenerationChange`
is not ported: master has no generation retry to test.

Not changed: `ConnectorExecuteAction` refreshes the table only when the
procedure returns, so a failure after Iceberg's commit still skips the
refresh. branch-4.x's `ExecuteActionCommand` behaves the same way
(`logRefreshTable` runs only after the action returns); with this fix
the Kerberos case no longer fails after the commit.

**Results.** Each change is pinned by a test that fails without it:

| Change undone | Result |
|---|---|
| `execute` hands the action the loaded table | the `expire_snapshots`
case records six opens on `iceberg-worker-pool` threads outside the
`doAs` (three `snap-*.avro` manifest lists, three manifests), the CI
failure; the `cherrypick_snapshot` case records the new manifest's
`newOutputFile` |
| the rewrite planner gets the loaded table | the planning case records
two manifest opens on worker threads |
| the plain `IcebergAuthenticatedFileIO` instead of
`withPrefixOperations` | `remove_orphan_files` fails with "requires
FileIO prefix listing support" |
| the procedure body runs after `executeAuthenticated` returns |
`commitsTheProcedureInsideTheAuthScope` fails |

The fe-connector-iceberg tests (1507) pass. The External pipeline runs
`test_iceberg_hadoop_catalog_kerberos`.

Co-authored-by: Gabriel <[email protected]>

… limits (apache#68842)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66717, apache#67726

Problem Summary:

**In short.** This ports four fixes from branch-4.1 apache#66717 (memory
governance for external metadata caches) that master's re-implementation
(apache#67726) does not have. Under a metadata cache weight limit, master
still lets the Iceberg SDK keep its own unbounded manifest cache,
fetches the whole snapshot history of REST `snapshot-loading-mode=refs`
tables on every table cache miss, and under-weighs cached Paimon tables
by 2-5x because they keep growing after admission. Catalogs without a
weight limit are not affected.

**Background.** apache#66717 bounds the estimated retained memory of external
metadata caches with the FE-wide `external_meta_cache_max_weight`, the
catalog property `meta.cache.max-weight`, and per-entry
`meta.cache.<engine>.<entry>.max-weight`. A value is weighed once, when
it is admitted; if it does not fit, it is returned to the query but not
cached. master re-implemented this in apache#67726 on the connector cache
framework, and apache#68839 ports the 4.x tests. Comparing the two
implementations found the gaps below; branch-4.1 handles all of them.

**The problem, and what it cost.**

- **Iceberg SDK manifest cache outside the limit.**
`IcebergCatalogFactory` derives `io.manifest.cache-enabled=true`
whenever `meta.cache.iceberg.manifest.*` is enabled. That SDK cache
keeps up to `io.manifest.cache.max-total-bytes` (100 MB by default) of
manifest content per FileIO, outside every Doris limit, and Glue / S3
Tables catalogs have one FileIO per table. A user who sets a limit gets
a second, unbounded manifest cache next to the bounded one.
- **Remote IO while weighing REST refs-mode tables.** A weight-limited
`IcebergTableCache` serializes the loaded metadata
(`TableMetadataParser.toJson`) and weighs it. Both call
`TableMetadata.snapshots()`, which in refs mode runs the catalog's
supplier and fetches every snapshot. So each table cache miss pays one
extra full-history REST load, exactly the cost refs mode exists to
avoid, and the cached value then holds the whole history.
- **Paimon tables grow after admission.** `paimon-table` is weighed from
the schema, location and catalog environment. A cached `FileStoreTable`
then grows: the first `latestSnapshot()` or scan builds its transient
store (`RowType` copies of the schema, a copy of the options, and for a
partial-update table a merge function factory with sequence-group and
aggregator maps for every field), and field lookups build the four lazy
index maps of every `RowType`. Measured on Paimon 1.4.2, the grown graph
is 2-5x the admitted weight, so the heap the cache really holds can
exceed `MAX_WEIGHT` by that factor.
- **Paimon tables can own their FileIO, which is not charged.** A REST
catalog gives each table its own `RESTTokenFileIO` with data tokens (the
vended token, about 2 KB, arrives on first data access) or
`ResolvingFileIO` without them (`RESTCatalog.fileIOFromOptions`), and a
Hive catalog resolves a FileIO per table location
(`HiveCatalog.fileIO(Path)`). Their tokens and backend maps appear after
admission.

**How this PR fixes it.**

- `IcebergConnector#hasMetaCacheWeightLimit` decides whether any limit
applies: the FE-global limit, the catalog limit, or the `max-weight` of
one of the entries the connector reads (`table`, `partition`,
`manifest`, `partition_view`, now one constant shared with the DDL
validation). The three catalog option builders take that flag, and
`appendManifestCacheProperties` does not derive
`io.manifest.cache-enabled` under a limit. An explicit user value still
wins, as in branch-4.1.
- `IcebergCacheSizeEstimator.areSnapshotsLoaded` reads
`TableMetadata.snapshotsLoaded` (Iceberg exposes no accessor for it). A
weight-limited `TableOwner` checks it first and, if the list is still
behind its supplier, declines retention with
`incomplete_estimate:iceberg_snapshots_not_loaded` instead of
serializing. The statement keeps the live table, whose supplier still
serves historical snapshots on demand.
- `PaimonCacheSizeEstimator` reserves the post-admission growth of every
concrete `FileStoreTable` from its schema shape: a base, each field,
each field nested in a row, each child type of an array / map / multiset
/ vector, each option and each key. The constants are calibrated against
a retained-size oracle that walks the grown graph (new test helper
`EstimatorCalibrationAssertions`). A partial-update table reserves
another 256 bytes per field for its merge function. Every concrete table
also carries a fixed 16 KB allowance for its FileIO, as on branch-4.x:
Paimon offers no IO-free way to tell an owned FileIO from one the
catalog shares, and charging a shared FileIO to each table only makes
the estimate conservative. The allowance covers an 8,192-character REST
token and a `ResolvingFileIO` with its backend; a token refresh replaces
the token rather than adding to it.

**Results.**

| Case | Before | After |
|---|---|---|
| `meta.cache.iceberg.manifest.enable=true` plus any weight limit | SDK
manifest cache derived on, up to 100 MB per FileIO outside the limit |
not derived; `io.manifest.cache-enabled=true` still turns it on |
| Weight-limited REST catalog, `snapshot-loading-mode=refs`, table cache
miss | one extra remote load of the full snapshot list, full history
retained | no extra load; table not retained, statement reads the live
table |
| Paimon table with 301 INT columns, after one scan | 35 KB admitted,
114 KB retained | 134 KB admitted, 114 KB retained |
| Paimon table with a 200-field ROW column, after field lookups | 25 KB
admitted, 117 KB retained | 143 KB admitted, 117 KB retained |
| Paimon partial-update table with 303 INT columns, after one scan |
merge function not reserved | 233 KB admitted, 143 KB retained |
| Paimon table on its own FileIO (REST with or without data tokens,
Hive) | FileIO and token not charged | 16 KB allowance per table |

Across 13 Paimon fixtures (append and primary-key tables, wide schemas,
many options, nested rows, arrays and maps of rows, multisets, wide
partial-update and aggregation tables) the admitted weight without the
FileIO allowance is now 1.2-1.7x the fully grown graph.

Not in this PR: wide schemas that exceed the 10,000-visit budget of
`ReflectiveObjectSizeEstimator.estimateComplete` (for example HMS tables
with about 1,000 primitive columns, or Paimon tables with about 3,400)
are still never admitted under a limit. The same budget is shared by
several connectors and the FE schema cache, so it needs its own change.

**Classes, and how they call each other.**

- `IcebergConnector`: `WEIGHTED_CACHE_ENTRIES`,
`hasMetaCacheWeightLimit()`, and passes the flag at the three catalog
builds.
- `IcebergCatalogFactory`: `buildBaseCatalogProperties`,
`buildCatalogProperties` and `buildS3TablesCatalogProperties` take
`metaCacheWeightLimited` and stay pure.
- `IcebergCatalogProperties`: validates `max-weight` keys against the
shared entry list.
- `IcebergCacheSizeEstimator` / `IcebergTableCache.TableOwner`: snapshot
probe and the guard before serialization.
- `PaimonCacheSizeEstimator`: `estimateStoreGrowth` and
`estimateTableFileIO` / `estimateFileIO`.

```
IcebergConnector.createCatalog / createS3TablesCatalog / createDlfCatalog
  └─ hasMetaCacheWeightLimit(): metaCache.hasEnclosingWeightLimit() || cacheSpec(entry).isWeightBounded()
  └─ IcebergCatalogFactory.build*CatalogProperties(..., metaCacheWeightLimited)
       └─ appendManifestCacheProperties: derive io.manifest.cache-enabled only when not limited

IcebergTableCache.borrow ─ loader ─ new TableOwner(table, ..., weighted)
  └─ areSnapshotsLoaded(table) ? toJson + estimateTable : incomplete("iceberg_snapshots_not_loaded")

PaimonMetaCacheCatalog "paimon-table" size estimator
  └─ PaimonCacheSizeEstimator.estimateTable ─ estimateFileStoreTable (each concrete table)
       ├─ estimateStoreGrowth(schema)
       └─ estimateTableFileIO(fileIO): fixed allowance, once per FileIO
```

Nothing changes for catalogs without a weight limit: the flag is false,
and the probe and the reservations only run when an entry is weighed.
Independent of apache#68839 (tests only); both add tests to
`IcebergConnectorCacheTest` and `IcebergTableCacheTest`, in different
places.

Co-authored-by: Calvin Kirs <[email protected]>

…al files by storage (apache#68847)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#68797, apache#66932

Problem Summary:

**In short.** The BE file cache can return the bytes of a different file
for two kinds of external reads. An `http()` URL whose content changes
keeps being served from blocks cached from the old content. Two object
storage endpoints, or two HDFS clusters, that hold the same path with
the same modification time share cache blocks. Both happen on master
whenever a session turns on `enable_file_cache`, and on branch-4.x,
where the cache is on by default. apache#68797 turns the cache on by default
on master and is blocked on these two cases. This PR keeps HTTP
responses out of the file cache and adds the storage to the cache key of
external files.

**Background.** `CachedRemoteFileReader` keys an external file as
`path:mtime`: the path its reader reports plus the modification time
from the file listing. The key is meant to change whenever the file
does. `FileFactory` wraps S3, HDFS and HTTP readers in the cache when
the query enables it. apache#66932 already exempted chunked HTTP responses
(the `cdc_stream` table function), whose content is a stream.

**The problem, and what it cost.**

- **HTTP has no version.** `HttpTableValuedFunction` lists a URL with
modification time 0, so every response of that URL gets the key `url:0`.
When the content behind the URL changes, a later query reads blocks
cached from the earlier content. If the length did not change, nothing
fails and the query silently returns the old rows; if it did, old and
new bytes can be mixed.
- **The key does not name the storage.** An S3 reader reports
`s3://bucket/key`, without the endpoint. An HDFS reader reports the path
with the name node stripped (`convert_path`). Two S3-compatible
endpoints (for example two MinIO deployments) holding the same bucket
and key, or two HDFS clusters holding the same path, get the same key as
soon as the modification times match. S3 reports them in seconds, so
files written by the same job in the same second match. A query on one
storage can then read blocks of the other storage's file.

**How this PR fixes it.**

- `FileFactory` no longer wraps HTTP readers in the file cache. This
covers every HTTP source read through `FileFactory`, including `hf://`
URLs, and replaces the chunked-response check from apache#66932, which this
rule includes.
- `FileReaderOptions` gets `cache_key_function`, which builds the cache
key of an external file from its path and modification time, so the code
that opens the file defines its key. `FileFactory` sets it for S3
readers to put the S3 endpoint (`AWS_ENDPOINT`) in front of the path,
and for HDFS readers the name node (`fs_name`): the key becomes
`<storage>:path:mtime`. `CachedRemoteFileReader` calls the function when
it is set; readers that do not set it, such as broker reads, keep
`path:mtime`.

**Results.**

| Case | Before | After |
|---|---|---|
| `http()` URL whose content was replaced, file cache on | second query
can return rows of the old content | always reads the current content;
HTTP data is not cached |
| Same bucket, key and mtime on two S3-compatible endpoints | one cache
entry, shared | one entry per endpoint |
| Same path and mtime on two HDFS clusters | one cache entry, shared |
one entry per name node |
| Existing cached blocks of external S3 / HDFS files after upgrade | - |
not found under the new key; the first reads go to the storage again and
the old blocks age out |

**Classes, and how they call each other.**

```
FileScanner / format readers
  └─ FileFactory::create_file_reader(system_properties, file_description, reader_options)
       ├─ FILE_S3   : options.cache_key_function = endpoint:path:mtime ─┐
       ├─ FILE_HDFS : options.cache_key_function = fs_name:path:mtime ──┤
       │                                       create_cached_file_reader(reader, options)
       │                                         └─ CachedRemoteFileReader::_init_external_table_cache
       │                                              key = cache_key_function(path, mtime)
       │                                                    (path:mtime when it is not set)
       └─ FILE_HTTP : HttpFileReader, not wrapped in the file cache
```

…pache#68824)

### What problem does this PR solve?

Problem Summary: Improve Doris FE compatibility with MySQL 9 clients and
Connector/J 8/9. Clients can consume multiple statements correctly,
authenticate through caching_sha2 full authentication, query
authentication_policy, and bind binary TIME parameters in server-side
prepared statements.

| Area | Supported behavior | Coverage |
|---|---|---|
| B1: Multiple statements | Advertise CLIENT_MULTI_STATEMENTS,
CLIENT_MULTI_RESULTS and CLIENT_PS_MULTI_RESULTS; allowMultiQueries
clients receive all results and can continue using the connection. |
Handshake/capability unit tests, existing raw-protocol regression, JDBC
multiple result sets and mixed SELECT/SET responses. |
| A1: caching_sha2 full authentication | Support full authentication
over TLS or RSA, empty passwords, incorrect-password rejection, and
COM_CHANGE_USER. Use the established channel's actual TLS state during
login and change-user. | Password resolver unit tests; forced
caching_sha2 JDBC login and real changeUser over both TLS and RSA,
including nonempty → empty → nonempty accounts. |
| A1: Client selection | Add mysql_caching_sha2_password_clients: auto
(default, client major version ≥9 via _client_version), all (all clients
requesting this plugin), none (native fallback). Normalize and validate
options during startup and dynamic updates. | Client routing and
configuration unit tests; JDBC auto routing and none/native fallback. |
| D3: authentication_policy | Expose the read-only variable with value
`*,,` through SHOW VARIABLES and SELECT @@authentication_policy. |
Variable unit tests and JDBC SHOW/SELECT checks. |
| F1: Binary TIME parameters | Decode MYSQL_TYPE_TIME/TIME2 in
server-side prepared statements, including fractional seconds, signed
durations, zero and NULL. Reject malformed lengths, truncated payloads
and out-of-range unsigned days without consuming adjacent parameter
data. | Literal unit tests for malformed framing, overflow and
±838:59:59.999999; genuine JDBC ServerPreparedStatement tests for
setTime, LocalTime, microseconds, NULL, reuse and a following integer
parameter. |



### Release note

Improve MySQL 9 client and Connector/J 8/9 compatibility for
authentication, multi-statement results, authentication_policy and
prepared TIME parameters. Reject invalid authentication client options
at startup.

Problem Summary: Recreating a compute group with the same name and a new
ID updates the name mapping to the new ID. Removing the obsolete group
then unconditionally removed that name, leaving the new backends and
compute group unreachable by name. Delete the mapping only when it still
points to the group being removed, and remove duplicate cleanup from the
backend deletion path.

Fix compute group name resolution after recreating a group with the same
name and a new ID.

…ant write tests (apache#68919)

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66685, apache#66321, apache#66626, apache#66907

Problem Summary:

**In short.** This ports the tests of four branch-4.1 PRs that master
already implements but does not pin: partition- and bucket-aware
concurrent writes for external tables (apache#66685), Paimon Variant V2 writes
(apache#66321), the required Variant array fallback value (apache#66626) and the BE
exec version alignment (apache#66907). It changes no production code.

**Background.** On master the behavior of these PRs arrived through
apache#68320, the generic connector write framework (connector-defined write
distribution, opaque commit reports, generic Variant V2 writes), and
apache#67395, Paimon table writes. Checking each 4.x change against master,
with branch-4.2 as the final 4.x form:

| 4.x change | master |
|---|---|
| Hive partitioned writes hashed by partition columns, skewed writer
assignment | equivalent:
`HiveWritePlanProvider.requiresPartitionHashWrite` →
`HIVE_TABLE_SINK_HASH_PARTITIONED` → BE scale writer |
| Paimon fixed-bucket routing, single writer for tables that need one |
equivalent: `PaimonWritePlanProvider.getWriteDistribution`; a GATHER
sink runs as one instance, one task, one writer |
| Paimon Variant V2 writes and reads, including apache#66626 | equivalent:
`ConnectorComputeVariantType`, `CheckCast`, the BE Variant V2 serde, the
JNI reader |
| BE exec versions 11 and 13 | same meaning on both sides |

**The problem, and what it cost.** These behaviors are unpinned on
master:

- `test_paimon_write_pk` checks only that the writer plan contains a
sink and a TVF, and its comment still describes a gather fallback master
does not have. A change that gathers the write to one writer passes it.
- Paimon write distribution is tested only for the default options.
Nothing covers a HASH_DYNAMIC table with an assigner parallelism, the
route a fixed-bucket table actually gets (route columns, Paimon's bucket
function, IDENTITY assignment), or the fallbacks to a single writer for
a non-default bucket function, an unroutable partition key or a
defaulted route field.
- Route columns of a positional-write connector must be resolved by the
bound target schema, not by the INSERT column list; only the name-mapped
case is tested.
- The analysis-time contract of a connector Variant column has one
positive case: nothing shows that MAP, STRUCT, TIME, DECIMAL beyond 38
or VARBINARY sources are rejected in both cast modes, including inside
arrays.
- How a Doris INSERT OVERWRITE sets Paimon's
`dynamic-partition-overwrite`, nested Variant mapping on read, and the
fallback-pair read order with a blank primary branch or a chain table
are untested.

**How this PR fixes it.**

- `test_paimon_write_pk`: the plan check becomes
`qt_pk_writer_scaling_plan` as on branch-4.2, regenerated on master:
`PhysicalConnectorTableSink` →
`PhysicalDistribute[DistributionSpecExternalTableSinkHashPartitioned]`.
- `PaimonWritePlanProviderTest`: HASH_DYNAMIC with
`dynamic-bucket.assigner-parallelism=4` stays a single writer; a
routable fixed-bucket table gets `paimon_fixed_bucket` with route
columns `[part, id]`, `num_buckets=4`, partition index 0, bucket index 1
and IDENTITY; `bucket-function=mod`, a DATE partition key, and a
defaulted bucket or partition field fall back to GATHER.
- `PhysicalConnectorTableSinkTest`: route columns of a full-schema-order
connector come from their bound-schema positions.
- `CheckCastTest`: the accepted and rejected sources of
`ConnectorComputeVariantType` in strict and non-strict mode, the array
shape changes, and the leaf check inside containers.
- `PaimonWriteBindingTest` (new): a full overwrite uses static semantics
unless the table asks for dynamic ones; a static-partition overwrite is
always static.
- `PaimonTypeMappingReadTest`: a Variant nested in ARRAY, ROW or MAP
maps to the V2 execution carrier.
- `PaimonReaderOptionsTest` and `PaimonJniScannerTest`: the
fallback-pair read order for a blank primary branch and for a chain
table, on the FE and the BE side.

Branch-4.x tests that do not apply are not ported: the Variant V2
session switch (master only has V2), the Hive plan assertions on a
distribution the test itself requested, and Paimon route fallbacks the
planner cannot produce on master.

Found while checking and left to separate changes:

- Iceberg partitioned writes are not hash-distributed by partition on
master. An `INSERT` into an Iceberg table that reaches 200 partitions
fails with `Too many open partitions 128` on a local cluster; branch-4.x
spreads the partitions over its writers.
- When a Variant is coerced to a common type with a number (UNION, IF,
CASE), master writes object and array Variants as NULL, as
`test_paimon_write_variant_errors` documents; branch-4.x rejects the
statement with its Variant write validator, which will come with the
Iceberg Variant write port.
- `TReportExecStatusParams.34` and the external partition types in
`Partitions.thrift` mean different things on master and on the released
4.1.4; apache#68837 aligns them with branch-4.1.

**Results.** Each new unit test fails when the logic it guards is
changed: dropping the defaulted-field check in
`PaimonWritePlanProvider`, accepting a MAP source in `CheckCast`,
removing the chain-table precedence in `PaimonReaderOptions`, or
resolving route columns by the INSERT column list.

Co-authored-by: Socrates <[email protected]>

apache#68766)

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

An OBS access timeout during Paimon catalog initialization can be
reported as
`Could not find a file io implementation for scheme 'obs'`. Paimon's
`FileIO.get`
stores the failed access probes as suppressed exceptions on an
`UnsupportedSchemeException`, while Doris previously displayed only its
message.

Expose the underlying access failures in the catalog-creation error and
use the
first failed probe as the cause so root-cause consumers, including `SHOW
CATALOGS`,
can report the actual error. Retain the original aggregate as a
suppressed
exception, preserving every attempted FileIO failure in the diagnostic
stack.
Other catalog errors retain their existing wrapping.

Add six offline tests covering the real Paimon/Hadoop fallback with a
simulated
OBS timeout, nested wrappers, multiple failures, errors without
messages,
missing filesystem implementations, and unrelated suppressed exceptions.

### Release note

Paimon catalog initialization reports underlying filesystem access
failures
instead of only a misleading missing-FileIO message.

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
    - [ ] Manual test
    - [ ] No need to test or manual test.

Offline Paimon connector unit tests, Checkstyle, and plugin packaging
passed.
  No live OBS regression was run.

- Behavior changed:
    - [ ] No.
- [x] Yes. Catalog error messages and their cause chains expose FileIO
access failures; storage access and retry behavior are unchanged.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

Related PR: apache#68931, apache#68677

Problem Summary:

After the Codex 0.162.1 runtime rollout in apache#68931, production review
requests still select `gpt-6-sol`. The new real-account canary has now
passed both ordinary and goal-mode calls to `gpt-6.1-sol` at `xhigh`.

Change the production default to `gpt-6.1-sol` and the fallback to
`gpt-6-sol`. Keep the runtime, `xhigh` effort, prompts, authentication,
retry/resume behavior and submission validation unchanged.

The existing fallback remains limited to an unsupported-model rejection
before review work begins; this does not turn general failures or
capacity errors into model switches.

…he#67911)

Problem Summary: Meta Service stores ALTER COMPUTE GROUP properties as a
complete replacement, but the FE periodic checker merged only changed
keys into its in-memory metadata. Properties removed from Meta Service
therefore remained in FE indefinitely. Compare the complete snapshots
and publish a validated replacement map so additions, updates, and
removals converge consistently.

Compute group property removals now propagate from Meta Service to FE
during periodic synchronization.

…che#68866)

### What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:

Cross-cloud COS/OBS access repeatedly times out in regression runs for
unrelated PRs, including `curlCode: 28` during uploads and
object-storage connection timeouts. Examples: [External
#103511](http://43.132.222.7:8111/viewLog.html?buildId=1065598), [P0
#127021](http://43.132.222.7:8111/viewLog.html?buildId=1065768), and
[Cloud P0
apache#68212](http://43.132.222.7:8111/viewLog.html?buildId=1065729).

Move the following suites to p2 directories and change their tier tags
from `p0` to `p2`, preserving existing `external` tags:

- `refactor_storage_param_s3_load`
- `test_s3_tvf_s3_storage`
- `test_outfile_s3_storage`
- `paimon_base_filesystem`
- `test_file_tvf_s3`
- `test_outfile_with_different_s3`

Move the four existing golden files alongside their suites without
changing their contents. P0, Cloud P0, and External pipelines exclude
these suites, while their coverage remains available in p2. Suite bodies
are unchanged.

### Release note

None

### Check List (For Author)

- Test:
- [x] Manual static validation: suite selection, unchanged suite bodies
and golden files, and `git diff --check` passed. Live cloud regression
was not run.
- Behavior changed:
    - [x] Yes. Regression suite scheduling only.
- Does this need documentation?
    - [x] No.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

…FScan (apache#68871)

### What problem does this PR solve?

Issue Number: N/A

Problem Summary:
`PhysicalLazyMaterializeTVFScan.accept()` dispatched to
`visitPhysicalTVFRelation`, so the dedicated
`PhysicalPlanTranslator.visitPhysicalLazyMaterializeTVFScan` was
unreachable dead code and every TVF scan under TopN lazy materialization
was translated by the parent method.

This PR dispatches `accept()` to `visitPhysicalLazyMaterializeTVFScan`,
consistent with `PhysicalLazyMaterializeOlapScan`, so the dedicated
translation path stays reachable. No behavior change today since both
translator methods are identical, and the added unit test pins the
dispatch.

### Release note

None

### Check List (For Author)

- Test
    - [ ] Regression test
    - [x] Unit Test
- `PhysicalLazyMaterializeTVFScanVisitorTest` (new), plus
`TopnLazyMaterializeTest`, `MaterializeProbeVisitorTest`,
`LazyMaterializeTopNTest`, `TopNRuntimeFilterTest` (27 tests, all pass)
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:

- Behavior changed:
    - [x] No.
    - [ ] Yes.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label

### What problem does this PR solve?

Issue Number: None

Related PR: apache#66684

Original author: @hubgeter (the change is ported from apache#66684 on
branch-4.1; the commit carries Co-authored-by)

Problem Summary:

**In short.** branch-4.1 (from 4.1.5) and branch-4.2 enable the block
file cache by default. master still has it off. This difference matters
most on upgrade, through the BE config: BE configs are not persisted. A
storage-compute-coupled cluster that relies on the default therefore
loses its file cache when it moves from 4.1.5 / 4.2 to a release built
from master. This PR ports apache#66684 and makes both the BE config
`enable_file_cache` and the session variable `enable_file_cache` default
to true.

**Background**

- Two switches gate the block file cache:
- The BE config `enable_file_cache` decides whether a BE creates the
cache at all. In cloud mode, BE forces it on after reading its config
file and refuses to start without it.
- The session variable `enable_file_cache` decides whether external file
scans read through the cache. `FileFactory::get_reader_options` requires
the BE config, the session variable and file cache admission to all be
true. The session variable has no effect on internal tables.
- With the session variable on, FE also assigns external scan ranges by
consistent hashing instead of round robin (`ExternalScanNode`). A file
then keeps landing on the same backends, so its cached blocks get
reused.
- Session variables and BE configs behave differently across upgrades:
- FE writes every session variable value into its image
(`VariableMgr.write` → `SessionVariable.toJson`). An upgraded cluster
keeps the values it had whatever the new default is; only a new cluster
gets a new default.
- BE configs are not persisted. A BE uses the default compiled into its
binary unless be.conf sets the key.

**The problem, and what it cost**

| Default | master | branch-4.2 | 4.1.4 | 4.1.5 |
|---|---|---|---|---|
| BE config `enable_file_cache` | false | true | false | true |
| session variable `enable_file_cache` | false | true | false | true |

- Take a storage-compute-coupled cluster whose be.conf does not set
`enable_file_cache`. When it is upgraded from 4.1.5 / 4.2 to a release
built from master, it silently loses its file cache. External table
scans and reads of rowsets cooled down to remote storage go back to
remote storage on every query. The `${DORIS_HOME}/file_cache` directory
is left on disk.
- New clusters built from master behave differently from new 4.1.5 / 4.2
clusters. In coupled mode they have no file cache at all, and in either
mode external scans are not cached.
- Cloud clusters are not affected by the BE default, because cloud mode
forces it on. Upgraded clusters keep their session variable value either
way.

**How this PR fixes it**

The first commit makes both defaults true, as in apache#66684:
`DEFINE_Bool(enable_file_cache, "true")` in `be/src/common/config.cpp`
and `enableFileCache = true` in `SessionVariable`.

On a storage-compute-coupled BE that does not configure the cache, this
turns on the following:

- The cache lives under `${DORIS_HOME}/file_cache` (the default
`file_cache_path`). Without `total_size` it may use the whole file
system. Eviction starts once the file system is 88% full and stops at
85% (`file_cache_enter/exit_need_evict_cache_in_advance_percent`).
`storage_root_path` defaults to the same disk, whose flood stage is 90%,
so a busy cache keeps a shared disk at 85%–88%. To bound it, set
`file_cache_path` with a `total_size`; to keep the old behavior, set
`enable_file_cache = false` in be.conf.
- Only reads through a remote file system go through the cache: external
file scans and cooled-down rowsets. Local rowsets are read exactly as
before.

The second commit keeps out one side effect of the new default.
`SniiCompoundWriter` pads an SNII inverted index container of 32 or more
file cache blocks with zeros, so that it ends on a block boundary. Only
`CachedRemoteFileReader` repays that padding: a read confined to the
last block then fetches one block instead of two. The padding was gated
on `enable_file_cache`, which stood for "cloud mode" as long as the flag
defaulted to false (cloud mode forces it on). With the new default, a
coupled BE would also pad its SNII containers on local disk, by up to
half a block each. There `LocalFileReader` reads them and the padding is
never repaid; branch-4.2 does this today. The padding is now gated on
cloud mode, and its UTs toggle the deploy mode instead of the file
cache.

Two more commits close gaps that the new default makes reachable:

- The third commit refuses to start a BE that has no usable file cache
path. With `ignore_broken_disk=true`,
`FileCacheFactory::create_file_caches` skipped every path that failed to
initialize and still returned OK when all of them were skipped.
`get_by_path(hash)` then took `% 0` and crashed BE at the first cached
read, or earlier at the first rowset cleanup, since
`Rowset::clear_cache()` looks up the cache for every removed rowset
while `enable_file_cache` is on. A coupled BE gets there when
`${DORIS_HOME}/file_cache` cannot be created. `create_file_caches` now
fails in that case, the way storage and spill paths already do ("All
disks are broken, exit.").
- The fourth commit restores the modification time of files listed by
`s3()` / `hdfs()` TVFs. Since apache#62023,
`ExternalFileTableValuedFunction#parseFile` built `TBrokerFileStatus`
without `FileEntry#modificationTime`, so every TVF range reached BE with
mtime 0. BE keys cached blocks of external files by `path:mtime`, so
with the cache on, an object overwritten in place would keep serving its
old cached blocks. branch-4.x copies the listed mtime
(`BrokerUtil.parseFile`); master does again.

**Results**

| Scenario | Before | After |
|---|---|---|
| New coupled cluster, external table scan | not cached, scan ranges by
round robin | cached, scan ranges by consistent hashing |
| New cloud cluster, external table scan | not cached | cached |
| Coupled cluster upgraded from 4.1.5 / 4.2, be.conf without
`enable_file_cache` | file cache turned off by the upgrade | file cache
stays on |
| Coupled cluster upgraded from 4.1.4 or older | unchanged | BE cache on
(cooled-down rowsets cached). The session variable keeps its persisted
`false`, so external scans stay uncached until `SET GLOBAL
enable_file_cache = true` |
| Internal tables on local disks | unchanged | unchanged |
| SNII index containers, coupled BE | padded only if the BE had turned
the cache on | not padded |
| SNII index containers, cloud mode | padded | padded (unchanged) |
| Every file cache path broken, `ignore_broken_disk=true` | BE starts,
then crashes on first use of the cache | BE refuses to start: "All N
file cache paths are broken" |
| `s3()` / `hdfs()` TVF, object overwritten in place, cache on | could
serve blocks cached from the old content (mtime 0) | the new mtime is
part of the cache key |

Co-authored-by: daidai <[email protected]>

…pache#68886)

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#68332

Problem Summary:

Debug NULL-payload injection exposed query failures in `format_round`,
array construction, and mixed TIMESTAMP_NS/TIMESTAMPTZ comparisons. Root
cause: these functions consumed the arbitrary nested values of NULL rows
and performed range validation before NULL semantics were applied.

Enable the existing NULL-payload default replacement for `format_round`.
For `array_with_constant` and `array_repeat`, check the length null map
before reading the payload, preserving the existing empty-array result
for NULL lengths. For mixed temporal comparisons, compare CCTZ local
civil fields directly without rejecting NULL payloads or copying
nullable input columns; retain signed years across date boundaries and
nanosecond comparison precision.

This PR contains only the three function fixes, based on master. It does
not include the debug injection changes from apache#68332.

### Release note

Fix query errors caused by arbitrary NULL-row payloads in formatting,
array construction, and mixed temporal comparisons.

### Check List (For Author)

- Test
    - [x] Regression test
    - [ ] Unit Test
    - [x] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:

Before migration to master, the identical patch was validated on the
NULL-payload injection branch: ASAN BE build and changed-line clang-tidy
passed. Six suites passed: `test_format_round_decimal`,
`test_format_round_decimal256`, `fix_array_type_and_lambda_func`,
`test_timestamp_ns_mixed_datetime_expressions`,
`test_array_functions_array_with_const`, and `nereids_scalar_fn_Array`.
Supplemental SQL checked mixed temporal `nullif`, NULL-safe equality,
NULL inputs, and empty arrays for NULL lengths.
`test_pythonudf_module_advanced` was attempted but blocked by disabled
Python UDF support. No BE unit tests were run. After migration,
clang-format 16, header hygiene, and `git diff --check` passed; the
master-based branch was not rebuilt.

- Behavior changed:
    - [ ] No.
- [x] Yes. NULL-row payloads no longer trigger these validation errors;
mixed temporal comparisons can compare local civil dates outside the
DATETIMEV2 range.

- Does this need documentation?
    - [x] No.
    - [ ] Yes.

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label