[fix](fe) Enable fuzzy for enable_query_cache - #8

…d scheduling (#61249)

### What problem does this PR solve?

#### When a streaming job processes S3 files, the second scheduling
fails with:

  No new files found in path: ...

Root cause: In S3ObjStorage.globListInternal, currentMaxFile was
unconditionally set to the last raw S3
object key returned in the response page, without checking whether it
matched the glob pattern.

  This affects two scenarios:

**Scenario 1** — reachLimit=false (all matched files consumed in one
listing):
The S3 page still contains non-matching keys after the last matched file
(e.g.
test_csv_comma_header.csv.lz4 sitting after test_csv_comma_header.csv).
currentMaxFile gets set to the
.lz4 key, so hasMoreDataToConsume() returns true. The next scheduling
calls startAfter("...csv"), S3
returns only .lz4 which doesn't match the glob → rfiles empty →
exception.

  **Scenario 2** — reachLimit=true (batch limit hit mid-page):
After the limit is hit, the remaining page objects are not inspected.
The original code set currentMaxFile
to the last raw key in the entire page (which may be a non-matching
sibling), causing the same failure on
   the next scheduling attempt.

#### Fix

Track lastMatchedKey (the last S3 key that actually matched the glob)
during the listing loop.

When reachLimit=true, instead of breaking out of the for loop
immediately, continue scanning the remaining
objects already fetched in the current page to find the first next
glob-matching key as currentMaxFile.
  No extra S3 API call is needed.

When no next matching key is found in the remaining page objects, fall
back to lastMatchedKey instead of
  the raw last S3 page key.

####Regression Test

Added test_streaming_job_no_new_files_with_sibling. The pattern
example_[0-0].csv only matches
example_0.csv; since getLongestPrefix strips at [, the S3 listing prefix
becomes
regression/load/data/example_ and returns both example_0.csv and
example_1.csv — example_1.csv acts as the
non-matching sibling. The test verifies that after the first successful
task no failed tasks appear.

…rove compile time (#61227)

`scan_operator.cpp` was the 2nd slowest file to compile (~289s), caused
by template
instantiation blowup. Methods defined on `ScanLocalState<Derived>` were
instantiated
for each of the 6 `Derived` types (OlapScan, JDBCScan, FileScan, EsScan,
MetaScan,
GroupCommitScan). Specifically:

- 4 inner `PrimitiveType`-templated methods × 6 Derived × 21
PrimitiveTypes = **504 instantiations**
- 5 non-template normalize methods × 6 Derived = **30 extra
compilations**

None of these methods actually depend on `Derived` — they only use base
class members
or virtual dispatch.

## Solution

Move the methods that do not depend on `Derived` up to
`ScanLocalStateBase`:

**Inner PrimitiveType-template methods** (instantiation reduced from
6×21 → 21, ÷6):
- `_normalize_in_predicate<T>`
- `_normalize_binary_predicate<T>`
- `_normalize_is_null_predicate<T>`
- `_change_value_range<T, Func>`

**Non-template normalize methods** (each compiled 6× → 1×):
- `_eval_const_conjuncts`
- `_normalize_bloom_filter`
- `_normalize_topn_filter`
- `_normalize_bitmap_filter`
- `_normalize_function_filters`

**Accompanying changes:**
- Move `_should_push_down_*` virtual methods, `_eos`,
`_push_down_functions` members
to `ScanLocalStateBase` so the moved methods can call them without
`Derived`.
- Add `_max_pushdown_conditions_per_column` to base class, initialized
from parent
  in `ScanLocalState<Derived>::init()`.
- `_normalize_topn_filter`: replace `p.node_id()` (via `Derived::Parent`
cast) with
  `_parent->node_id()` (base class method).


before
```
288.82s  be/src/exec/operator/scan_operator.cpp
```

now
```
[1/1] Compiling: be/src/exec/operator/scan_operator.cpp
  → 209.9s
```

## Background

`ClusterNamespace` is a legacy artifact from the multi-tenancy era. In
earlier versions, Apache Doris implemented multi-tenancy by prefixing
database names with a cluster name using `:` as a separator. This
functionality has since been completely removed, but the
`ClusterNamespace` class and related legacy code remained.

## Changes

- Remove all `ClusterNamespace.getNameFromFullName()` calls (~122 call
sites), replacing each with its original argument
- Remove all `import org.apache.doris.cluster.ClusterNamespace`
statements
- Delete `ClusterNamespace.java`
- Clean up dead code left behind (no-op self-assignments, stale comments
like "should be removed in 3.0", empty `removeClusterPrefix()` methods)
- Fix import ordering to satisfy checkstyle

Co-authored-by: Copilot <[email protected]>

Related PR: #61118

shoud only migrate InternalCatalog

it causes BE UT ccache always miss. now fixed.

…rite (#61224)

### What problem does this PR solve?
SumLiteralRewrite transforms SUM(expr +/- literal) into SUM(expr) +/-
literal * COUNT(expr). When type coercion has introduced an implicit
widening cast (e.g. CAST(smallint_col AS INT)), the rewritten SUM/COUNT
still operates on the wider type, forcing unnecessary wider data reads.

This is redundant because SUM always returns BIGINT for any integer
input (TINYINT/SMALLINT/INT/BIGINT). Strip implicit widening integer
casts in extractSumLiteral() so the aggregate operates on the original
narrow column directly.

This benefits ClickBench Q29-style queries where SUM(col), SUM(col+1),
SUM(col+2) share a narrow integer column — after stripping the cast,
SUM(col+1) and SUM(col+2) reuse the existing SUM(col).

…educe memory (#61318)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Reduce FE memory by 
1. moving top-N table stats filtering from PrometheusMetricVisitor into
CloudTabletStatMgr so it's computed once per stat cycle instead of per
Prometheus scrape,
2. removing the unused beToTablets field from InfightTask to avoid
retaining a large map reference
3. changing InfightTablet.tabletId from Long to long to avoid boxing
overhead.

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…61327)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
- [ ] Previous test can cover this change. - [ ] No code files have been
changed. - [ ] Other reason <!-- Add your reason? -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…t cases (#61333)

### What problem does this PR solve?

The result of `array_union` does not guarantee element order, so the
outputs of its related test cases are also unstable.
Use `array_sort` to make the results deterministic.

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

## What

This PR removes several redundant FE-side test dependencies.

## Why

A quick dependency audit under `fe/` found a few dependencies that are
declared but not actually used:
- `hamcrest` in multiple FE modules
- JUnit 5 test dependencies in `fe-extension-loader`, which currently
has no test sources

Keeping these unused dependencies increases dependency surface and
maintenance cost without bringing value.

…failures (#61334)

before it may silently fail. now try it best to return the status if
failed.

…61351)

case of apache/doris-website#3462

…le agg to constant (#61183)

### What problem does this PR solve?
- SimpleAggCacheMgr: singleton async cache backed by internal SQL
queries for exact MIN/MAX column stats and row counts. Uses a two-level
freshness check — visibleVersionTime (cheap local read) as a fast
filter, falling back to getVisibleVersion() (potentially RPC in cloud
mode) only when times match — to avoid unnecessary RPCs while still
detecting same-millisecond concurrent writes. The cache loader captures
version and versionTime both before and after executing the internal SQL
to guard against writes that complete during query execution.

- RewriteSimpleAggToConstantRule: Nereids rewrite rule that fires on
LogicalAggregate over LogicalOlapScan, checks DUP_KEYS model, no GROUP
BY, supported agg functions, and replaces with LogicalOneRowRelation
containing constant literals.

- Bug fix — TRUNCATE TABLE did not reset TableAttributes.visibleVersion:
truncateTableInternal() replaced partition data but never called
olapTable.resetVisibleVersion(). This left
TableAttributes.visibleVersionTime unchanged, so the SimpleAggCacheMgr
(and any other version-time-keyed logic) would see a cache hit and
return pre-truncate values instead of the correct post-truncate result
(e.g. count(*) = 5 instead of 0). The fix adds
olapTable.resetVisibleVersion() inside truncateTableInternal(), which
bumps both visibleVersion back to TABLE_INIT_VERSION and
visibleVersionTime to System.currentTimeMillis(). This applies to both
the live execution path and the journal-replay path since both go
through truncateTableInternal.

- Unit test with mock cache covering 6 positive and 8 negative cases.

- Regression tests:

-- Cache warmup polling, positive/negative explain checks, correctness
assertions, and disable-rule verification.
-- truncate_version_reset: verifies that after TRUNCATE TABLE, count(*)
returns 0 (not a stale cached value), and that subsequent inserts are
counted correctly.

…gregate (#60996)

Add a new built-in scalar function to_hll(bigint) that constructs an HLL
value from a BIGINT directly, bypassing any hash — the bigint is fed
as-is into HyperLogLog::update(uint64_t).

Changes:
- BE (vec/functions/function_hll.cpp): implement struct ToHll with
Status-returning execute/vector/vector_nullable methods; register as
FunctionAlwaysNotNullable<ToHll, true> so InvalidArgument status is
propagated on negative input.
- BE test (be/test/vec/function/function_hll_test.cpp): add unit test
function_hll_test.function_to_hll_test covering values 0, 1, 2, 100.
- FE Nereids (scalar/ToHll.java): new ScalarFunction with signature
BIGINT -> HLL.
- FE catalog (BuiltinScalarFunctions.java): register to_hll.
- FE visitor (ScalarFunctionVisitor.java): add visitToHll visitor hook.

### What problem does this PR solve?

Issue Number: #61099

Related PR: N/A

Problem Summary:
- Avoid deadlock when external catalog refresh invalidates cache while
another thread initializes the same catalog through a cache loader.
- Add a unit test to reproduce the lock inversion and verify both worker
threads also exit cleanly after the fix.

… framework (#61141)

### What problem does this PR solve?

Refactor the JDBC data source scanning architecture to integrate with
the
unified FileScanner/JniReader framework, replacing the standalone
ExternalScanNode-based JDBC scan path.

#### Motivation

The JDBC scan path was independently implemented with its own operator
(JDBCScanLocalState), scanner (JdbcScanner), and JNI connector
(JniConnector
→ BaseJdbcExecutor hierarchy), while other JNI-based data sources
(Paimon,
Hudi, MaxCompute, TrinoConnector) already use the unified FileScanner →
JniReader → JniScanner path. This caused code duplication, maintenance
burden,
and architectural inconsistency.

#### Changes

1. BE: Split JniConnector into two focused classes:
- JniReader: JNI lifecycle management (open/read/close), base class for
all JNI readers (PaimonJniReader, HudiJniReader, JdbcJniReader, etc.)
   - JniDataBridge: Stateless data exchange between C++ Blocks and Java
     shared memory via JNI

2. Java: Introduce Strategy Pattern for database-specific type handling:
- JdbcTypeHandler interface with DefaultTypeHandler base implementation
   - Per-database handlers: MySQLTypeHandler, OracleTypeHandler,
     PostgreSQLTypeHandler, ClickHouseTypeHandler, SQLServerTypeHandler,
DB2TypeHandler, SapHanaTypeHandler, TrinoTypeHandler, GbaseTypeHandler
   - JdbcTypeHandlerFactory for handler selection
   - JdbcJniScanner (extends JniScanner) for read path
   - JdbcJniWriter (extends JniWriter) for write path
   - Old BaseJdbcExecutor subclasses marked @deprecated but preserved

3. FE: Migrate JdbcScanNode from ExternalScanNode to FileQueryScanNode:
   - JdbcScanNode now extends FileQueryScanNode
   - Introduces JdbcSplit (extends FileSplit) to carry JDBC params
   - Uses TFileScanRange with FORMAT_JNI and table_format_type="jdbc"
   - Adds jdbc_params field to TTableFormatFileDesc in Thrift
   - Adapts PhysicalPlanTranslator.visitPhysicalJdbcScan() accordingly

4. BE: Add JdbcJniReader and integrate into FileScanner:
- JdbcJniReader handles special types (bitmap, HLL, quantile_state,
JSONB)
     via string-based intermediary and CAST
   - FileScanner._get_next_reader() adds "jdbc" table_format_type branch
   - JdbcUtils utility for JDBC driver URL resolution
   - Existing JdbcScanner preserved as transitional (deprecated)

All predicate push-down logic (createJdbcFilters, getJdbcQueryStr, etc.)
is preserved from the original JdbcScanNode implementation.

… quorum success (#60953)

## Summary

When using majority write (quorum success), BE does not distinguish
between replicas
with continuous versions and replicas with version gaps
(`lastFailedVersion >= 0`).
This causes inconsistency with FE's commit check, which correctly
excludes
version-gap replicas from success counting.

## Bad Case

Consider 3 replicas on nodes 1, 2, 3 with `load_required_replica_num =
2`:

1. **First write**: nodes 1,2 succeed, node 3 fails → overall success.
   Node 3 now has a version gap (`lastFailedVersion >= 0`).
2. **Second write**: nodes 1,3 succeed, node 2 fails →
   - **BE** counts 2 successes (nodes 1,3), considers it quorum success.
- **FE** commit only counts node 1 as success (node 3 has version gap),
     so `successReplicaNum = 1 < 2`, commit fails.
- This wastes resources since BE already returned success to the client
     but FE rejects the transaction.

The correct behavior for the second write:
- nodes 1,3 succeed → should **FAIL** (node 3 has version gap, only node
1 counts)
- nodes 1,2 succeed → should **SUCCEED** (both have continuous versions)

## Solution

Pass per-tablet version-gap backend information from FE to BE via a new
thrift field
`map<tablet_id, list<backend_id>> tablet_version_gap_backends` in
`TOlapTablePartition`.

On the BE side, when counting successful replicas for majority write in
both
`VTabletWriter` (V1) and `VTabletWriterV2`, exclude version-gap backends
from
the `finished_tablets_replica` counter. This makes BE's quorum check
consistent
with FE's commit check.

### Changes

- **Descriptors.thrift**: Add `tablet_version_gap_backends` field to
`TOlapTablePartition`
- **OlapTable.java**: Add `getPartitionVersionGapBackends()` to compute
gap backends per tablet
- **OlapTableSink.java**: Populate the new field when building partition
info
- **tablet_info.h/cpp**: Parse and store gap backends from thrift
- **vtablet_writer.cpp**: Exclude gap backends in `_quorum_success`
- **vtablet_writer_v2.cpp**: Exclude gap backends in `_quorum_success`
and `_create_commit_info`

…y data (#61236)

### What problem does this PR solve?

When an Elasticsearch field is mapped as keyword/text but the actual
data stored is an array (which is valid in ES),
Doris was throwing: 'Expected value of type: STRING; but found type:
Array'

This fix serializes the array to a JSON string representation instead of
throwing an error,
which is consistent with how other non-string types (Numbers, Objects)
are already handled.

Fixed two code paths in es_scroll_parser.cpp:
1. fill_columns() - main _source parsing path
2. handle_value() - array element processing path

Also added regression test and test data for this scenario.

This change excludes transitive `com.squareup.okhttp3:okhttp` and
`com.squareup.okio:okio`
from dependencies such as `tencentcloud-sdk-java-sts`, `trino-main`, and
`odps-sdk-core`.

  This does not remove OkHttp support from FE modules.
  Instead, FE now declares an explicit direct dependency on:

  - `com.squareup.okhttp3:okhttp-jvm:${okhttp.version}`
  - transitive `com.squareup.okio:okio-jvm`

This keeps the runtime classes required by those SDKs (`okhttp3.*` /
`okio.*`) available,
while avoiding multiple incompatible OkHttp/Okio versions on the
classpath.

In other words, this PR replaces transitive OkHttp/Okio with a single
centrally managed version,
  rather than removing them entirely.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

1. Create table support set `group_commit_mode` table property 
```
CREATE TABLE ... PROPERTIES(
    "group_commit_mode" = "async_mode"
);
```
2. Support alter this property
```
ALTER TABLE ... SET ("group_commit_mode" = "off_mode");
```
3. Show create table shows this property if its value is not `off_mode`
4. For stream load, if it not set `group_commit` header, use the table
property as the group commit mode; if it set `group_commit` header, use
the header value.
5. doc: apache/doris-website#3465


### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->

…60513)

## Summary
- Add session variable `enable_mor_value_predicate_pushdown_tables` to
control value column predicate pushdown for MOR (Merge-On-Read) tables
- FE: session variable with table-list matching (`db.table`, `table`, or
`*`), thrift flag on TOlapScanNode, `isMorTable()` helper on OlapTable
- BE: propagate flag through scan operator → tablet reader → rowset
reader; extend `_should_push_down_value_predicates()` for all rowsets;
skip `__DORIS_DELETE_SIGN__` to preserve delete correctness; keep VExpr
in conjuncts as post-merge safety net

## Motivation
MOR tables with inverted indexes on value columns cannot utilize those
indexes for filtering because value predicates are not pushed to the
storage layer. This feature enables per-segment value predicate pushdown
for dedup-only/insert-only MOR workloads where the same key always
carries identical values across rowsets, allowing inverted indexes and
zone maps to filter data early.

## Design
Two-layer predicate approach:
1. **ColumnPredicate** (storage layer): pushed per-segment for early
filtering via inverted index, zone map, bloom filter
2. **VExpr** (compute layer): retained in `_conjuncts` as post-merge
safety net

Delete sign handling: `__DORIS_DELETE_SIGN__` predicate is excluded from
per-segment pushdown to prevent delete markers from being filtered
before merge, which would cause deleted rows to reappear.

## Test plan
- [x] Test 1: Basic session variable matching (table name, db.table,
wildcard, not-in-list)
- [x] Test 2: Dedup-only pattern with multiple overlapping rowsets
- [x] Test 3: Dedup + row-level delete via `INSERT INTO (...,
__DORIS_DELETE_SIGN__) VALUES (..., 1)`
- [x] Test 4: Dedup + delete predicate via `DELETE FROM ... WHERE`
- [x] Test 5: Multiple tables in session variable list
- [x] Test 6: MOW table (unaffected by setting)
- [x] Test 7: DUP_KEYS table (unaffected by setting)

---------

Co-authored-by: Claude Opus 4.6 <[email protected]>

only have a maximum of 10 external triage collaborators

…rce-global config support (#60903)

[fix](Azure)Fix Azure endpoint detection for sovereign clouds with force-global config support (#60903)

[test](auth)Resolve the conflict in the backup auth case

During backup, failing to acquire the lock within 10 seconds makes the
test case unstable, adding nonConcurrent can fix it.

…60766)

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->