[fix](test) pin insert visible timeout case to master FE - #65340

### What problem does this PR solve?

Issue Number: N/A

Related PR: apache#65340

Problem Summary: PublishVersionDaemon runs only on the master FE. Identify the master from one SHOW FRONTENDS snapshot and apply the stop_publish debug point only to that FE, while keeping the suite JDBC connection unchanged. This avoids injecting the fault into every FE and avoids coupling the test to a master JDBC endpoint.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - Built the regression-test framework with Maven.
    - Compiled the target Groovy suite with FileSystemCompiler.
    - Ran git diff --check.
- Behavior changed: No. Test-only stabilization.
- Does this need documentation: No.

…68449) (#68618)

### What problem does this PR solve?

Issue Number: None

Related PR: #68449

Problem Summary:

Backport #68449 (`50a39819e185f54fbb3a82e67182b1ec8f78433b`) to
`branch-4.1`. The changes stabilize regression clients and orchestration
for TLS/mTLS, Docker isolation, multi-cluster state cleanup, cloud
cache/compaction, asynchronous metadata visibility, and delete-bitmap
restart/concurrency tests. Doris Compose uses exact container-name
matching and retries concurrent removal races.

Resolved 27 conflict paths with these branch adaptations:

- Keep the `nereids_p0` suite paths, generated database names, planner
settings, existing SHOW columns and 4.1-only test logic.
- Preserve configured-master JDBC/HTTP endpoints in the
insert-visible-timeout suite, including mapped ports from #65340.
- Keep the routine-load WHERE expectation `(k00 = 8)`: 4.1's
`PlanUtils.ExpressionToExpr` stores unquoted slot labels, unlike master.
Retain the source topic fix, diagnostic fields and independent polling
counters.
- Keep `nereids_p0/hbo` in the non-concurrent exclusions while excluding
all `cloud_p0` suites as intended by the source fix.
- Combine the shared TLS/compaction helpers with 4.1's existing
framework behavior; do not import unrelated scenarios present only in
the source commit's parent.
- Omit `test_partition_cloud_add_build_index_validation.groovy`: the
partition index-format rollout feature from #66475 and its required
properties/configuration are absent from 4.1.
- Reuse the original PR's result files without manually creating
expected output; retain the deterministic variant ordering fix.

The resulting backport changes 350 files; no FE implementation,
protocol, or storage-format change is added.

### Release note

None

### Check List (For Author)

- Test:
- [x] Regression test on an existing 4.1.4 cloud cluster (1 FE / 3 BE):
127 passed, 3 failed, 186 skipped across all 316 changed suites.
- Retained failures: `test_create_view_nereids` (SHOW VIEW column-name
quoting mismatch), `test_streaming_insert_job_priv` (one successful task
and one pending task after the 300-second wait), and
`test_load_channel_profile` (S3 TVF query reports unknown column `c1`).
Their failed assertions/queries are unchanged by this backport. Root
causes are not established by this run.
- Python support is disabled: 19 attempts hit that prerequisite and 23
more suites were skipped. One JDBC fixture was unavailable, and the
single-BE Java static-load case returned early on this three-BE cluster;
these are counted as skipped, not passes.
- Other skipped suites require dedicated Docker/multi-compute-group
fixtures, restarts, TLS/external services, global configuration/fault
injection, or stress capacity. Four packed-file injection suites were
excluded before launch.
- Arrow Flight authentication passed after correcting only the
task-local endpoint configuration. Three small-table TTL/row-store
compaction follow-ups also passed.
- The failed streaming job and its test user were cleaned up. All FE/BE
nodes remained alive and HTTP health returned 200 after execution.
- [x] Unit Test: `run-regression-test.sh --compile` completed on the
remote validation host; all 35 regression-framework tests passed (0
failures, 0 errors, 0 skips).
    - [x] Manual test:
        - Diff whitespace and conflict-marker checks passed.
- All 340 changed existing Groovy files passed conversion-phase parsing
with Groovy 4.0.19 / JDK 17; both changed Python files passed
`py_compile`.
- `build-support/check-format.sh` passed with clang-format 16.0.6 (4,400
C/C++ files).
- Compared conflict resolutions with the source patch and branch-4.1
implementations, including RPC/debug-point dependencies and routine-load
expression serialization.
- BE/FE module builds, deployment and clang-tidy were not run, per the
requested existing-cluster validation scope. The live enterprise cluster
is not a build of this backport, so BE debug-point additions and
excluded scenarios still require CI.
- Behavior changed: Yes. Regression framework/test behavior and Doris
Compose discovery match the applicable source fixes; BE additions are
gated debug points.
- Does this need documentation: No.

<details>
<summary>Backport self-review checkpoints</summary>

| Checkpoint | Conclusion |
| --- | --- |
| Goal and tests | Applicable source fixes preserved; syntax/static
checks and 35 framework unit tests pass. Existing-cluster suite outcomes
and remaining CI scope are listed above. |
| Scope | Source patch plus documented branch adaptations; unsupported
feature suite omitted. |
| Concurrency and locking | Existing production lock order retained;
debug-only post-calculation barrier releases the rowset lock before
waiting. Suite worker errors propagate to their owners. |
| Lifecycle | Source connection/process cleanup and shared-state
restoration retained. |
| Configuration | Existing TLS and debug configuration used; referenced
cloud sync settings exist in 4.1. |
| Compatibility | Target RPC/HTTP/debug APIs verified; branch endpoint
mappings and result schemas retained. No wire/storage format changes. |
| Parallel paths | Relevant plaintext/TLS and local/cloud helper paths
retained. |
| Conditional checks | Source bounded polling and negative assertions
retained; branch adaptations checked against implementation. |
| Test coverage | Source coverage retained; 35 framework tests and 127
existing-cluster suites passed, with 3 retained failures and 186
documented skips. |
| Expected results | Source result files reused; no hand-written golden
output. |
| Observability | Source timeout/last-state diagnostics and
token/tablet/transaction debug logs retained. |
| Persistence and failover | No production persistence or replay change.
|
| Writes and atomicity | Production write behavior unchanged outside
enabled test debug points. |
| FE/BE propagation | Existing target sync-load RPC and metadata
endpoints support new helpers. |
| Performance | Production additions are gated debug checks; regression
polling is bounded. |
| Other findings | Focused framework/BE, TLS-suite and cloud-suite
reviews found no unresolved backport-specific blocker. |

</details>

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases and CI results
- [ ] Confirm document
- [ ] Add branch pick label

…68449) (#68618)

### What problem does this PR solve?

Issue Number: None

Related PR: #68449

Problem Summary:

Backport #68449 (`50a39819e185f54fbb3a82e67182b1ec8f78433b`) to
`branch-4.1`. The changes stabilize regression clients and orchestration
for TLS/mTLS, Docker isolation, multi-cluster state cleanup, cloud
cache/compaction, asynchronous metadata visibility, and delete-bitmap
restart/concurrency tests. Doris Compose uses exact container-name
matching and retries concurrent removal races.

Resolved 27 conflict paths with these branch adaptations:

- Keep the `nereids_p0` suite paths, generated database names, planner
settings, existing SHOW columns and 4.1-only test logic.
- Preserve configured-master JDBC/HTTP endpoints in the
insert-visible-timeout suite, including mapped ports from #65340.
- Keep the routine-load WHERE expectation `(k00 = 8)`: 4.1's
`PlanUtils.ExpressionToExpr` stores unquoted slot labels, unlike master.
Retain the source topic fix, diagnostic fields and independent polling
counters.
- Keep `nereids_p0/hbo` in the non-concurrent exclusions while excluding
all `cloud_p0` suites as intended by the source fix.
- Combine the shared TLS/compaction helpers with 4.1's existing
framework behavior; do not import unrelated scenarios present only in
the source commit's parent.
- Omit `test_partition_cloud_add_build_index_validation.groovy`: the
partition index-format rollout feature from #66475 and its required
properties/configuration are absent from 4.1.
- Reuse the original PR's result files without manually creating
expected output; retain the deterministic variant ordering fix.

The resulting backport changes 350 files; no FE implementation,
protocol, or storage-format change is added.

### Release note

None

### Check List (For Author)

- Test:
- [x] Regression test on an existing 4.1.4 cloud cluster (1 FE / 3 BE):
127 passed, 3 failed, 186 skipped across all 316 changed suites.
- Retained failures: `test_create_view_nereids` (SHOW VIEW column-name
quoting mismatch), `test_streaming_insert_job_priv` (one successful task
and one pending task after the 300-second wait), and
`test_load_channel_profile` (S3 TVF query reports unknown column `c1`).
Their failed assertions/queries are unchanged by this backport. Root
causes are not established by this run.
- Python support is disabled: 19 attempts hit that prerequisite and 23
more suites were skipped. One JDBC fixture was unavailable, and the
single-BE Java static-load case returned early on this three-BE cluster;
these are counted as skipped, not passes.
- Other skipped suites require dedicated Docker/multi-compute-group
fixtures, restarts, TLS/external services, global configuration/fault
injection, or stress capacity. Four packed-file injection suites were
excluded before launch.
- Arrow Flight authentication passed after correcting only the
task-local endpoint configuration. Three small-table TTL/row-store
compaction follow-ups also passed.
- The failed streaming job and its test user were cleaned up. All FE/BE
nodes remained alive and HTTP health returned 200 after execution.
- [x] Unit Test: `run-regression-test.sh --compile` completed on the
remote validation host; all 35 regression-framework tests passed (0
failures, 0 errors, 0 skips).
    - [x] Manual test:
        - Diff whitespace and conflict-marker checks passed.
- All 340 changed existing Groovy files passed conversion-phase parsing
with Groovy 4.0.19 / JDK 17; both changed Python files passed
`py_compile`.
- `build-support/check-format.sh` passed with clang-format 16.0.6 (4,400
C/C++ files).
- Compared conflict resolutions with the source patch and branch-4.1
implementations, including RPC/debug-point dependencies and routine-load
expression serialization.
- BE/FE module builds, deployment and clang-tidy were not run, per the
requested existing-cluster validation scope. The live enterprise cluster
is not a build of this backport, so BE debug-point additions and
excluded scenarios still require CI.
- Behavior changed: Yes. Regression framework/test behavior and Doris
Compose discovery match the applicable source fixes; BE additions are
gated debug points.
- Does this need documentation: No.

<details>
<summary>Backport self-review checkpoints</summary>

| Checkpoint | Conclusion |
| --- | --- |
| Goal and tests | Applicable source fixes preserved; syntax/static
checks and 35 framework unit tests pass. Existing-cluster suite outcomes
and remaining CI scope are listed above. |
| Scope | Source patch plus documented branch adaptations; unsupported
feature suite omitted. |
| Concurrency and locking | Existing production lock order retained;
debug-only post-calculation barrier releases the rowset lock before
waiting. Suite worker errors propagate to their owners. |
| Lifecycle | Source connection/process cleanup and shared-state
restoration retained. |
| Configuration | Existing TLS and debug configuration used; referenced
cloud sync settings exist in 4.1. |
| Compatibility | Target RPC/HTTP/debug APIs verified; branch endpoint
mappings and result schemas retained. No wire/storage format changes. |
| Parallel paths | Relevant plaintext/TLS and local/cloud helper paths
retained. |
| Conditional checks | Source bounded polling and negative assertions
retained; branch adaptations checked against implementation. |
| Test coverage | Source coverage retained; 35 framework tests and 127
existing-cluster suites passed, with 3 retained failures and 186
documented skips. |
| Expected results | Source result files reused; no hand-written golden
output. |
| Observability | Source timeout/last-state diagnostics and
token/tablet/transaction debug logs retained. |
| Persistence and failover | No production persistence or replay change.
|
| Writes and atomicity | Production write behavior unchanged outside
enabled test debug points. |
| FE/BE propagation | Existing target sync-load RPC and metadata
endpoints support new helpers. |
| Performance | Production additions are gated debug checks; regression
polling is bounded. |
| Other findings | Focused framework/BE, TLS-suite and cloud-suite
reviews found no unresolved backport-specific blocker. |

</details>

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases and CI results
- [ ] Confirm document
- [ ] Add branch pick label

…t_generation (#70308)

### Background

An auto-partition table with `partition.retention_count = N` keeps only
its N latest historical partitions: the FE's `DynamicPartitionScheduler`
drops older ones on every run, once per
`dynamic_partition_check_interval_seconds` (600 by default).
`partition_p0/auto_partition/test_auto_new_recycle` lowers that interval
to 1 s and checks the partition count after each load or ALTER.
`audit/test_sql_digest_generation` checks that the audit log gets an
`sql_digest` only for queries slower than the FE config
`sql_digest_generation_threshold_ms`, once with the threshold at 0 and
once at 100000. Both suites are in group `nonConcurrent`.

### What problem does this PR solve?

Issue Number: None
Related PR: #65375 and #65646 (the branch-4.1 recycle-suite fixes by
@shuke987, ported here), #65343 (the branch-4.1 digest-suite fix by
@shuke987, ported here), #65435 (master's earlier move of the recycle
suite's day-boundary guard)
PR Type: Test

Problem Summary:

Both suites rely on timing the product does not promise. The recycle
suite checks the partition count after a fixed `sleep(8000)` (5 s after
an ALTER), while the drop happens asynchronously at an unbounded time
after the config change; branch-4.1's NonConcurrent pipeline failed at
the first such check repeatedly (#65375). The digest suite's threshold-0
check needs the query's elapsed time to be above 0 ms. This PR gives
both suites their branch-4.1/4.2 form: the recycle suite polls the
partition count every second, for at most 1200 s, and widens its
day-boundary guard to cover the poll; the digest query adds `sleep(1)`.
Both files become identical to branch-4.2.

### Details

- **Failure:** On master's public NonConcurrent pipeline neither suite
has failed on its own since mid-July: of about 8,000 runs each, the
recycle suite failed 3 times and the digest suite 87 times, all in
builds whose cluster had died (most suites failing). The failures that
motivated the fixes are on branch-4.1's pipeline (#65375); #65343 made
the digest query deterministic there.
- **Cause:** `ADMIN SET FRONTEND CONFIG` sets the scheduler's new
interval at once, but the scheduler sleeps in 5-second slices and
notices the change only at the next slice. It then runs
`executeDynamicPartition` over every registered table before the drop.
So the drop lands 0 to 5 s plus one full pass after the config change,
and a pass longer than about 3 s fails the 8-second check. In the digest
suite, `AuditLogHelper` writes a digest only when `elapseMs >
sql_digest_generation_threshold_ms`, and nothing bounds the elapsed time
of `select *` on a two-row table from below (9 to 15 ms locally). With
`sleep(1)` on each row it takes about 2 s, still far below the second
check's 100000 ms.
- **Stability evidence:** The recycle checks now wait for the drop and
fail only when the count misses the target for 1200 s. The poll runs on
the suite thread, because the framework sets
`Awaitility.pollInSameThread()` (#68338), so it does not leak
connections. Shortening one of master's waits to 1 s reproduces the
branch-4.1 failure shape, and the ported suite passes; mutants show that
both suites still catch wrong results (see Testing).

### Testing

All runs used `./run-regression-test.sh --run -d
partition_p0/auto_partition -s test_auto_new_recycle` or `--run -d audit
-s test_sql_digest_generation` against a local single-FE, single-BE
cluster built from master.

| Suite | Version | Result |
|---|---|---|
| `test_auto_new_recycle` | master | passes |
| `test_auto_new_recycle` | this PR | passes; the polls reached the
target after 1, 2, 1, 3 and 3 s |
| `test_auto_new_recycle` | master, with the wait before the
10-partition check cut to 1 s | fails as #65375 describes: `expected:
<101> but was: <10>` (master's `assertEquals` has its arguments swapped,
so 101 partitions were still there) |
| `test_auto_new_recycle` | this PR, expecting 11 partitions, poll
capped at 20 s | fails as it should: `ConditionTimeoutException` |
| `test_sql_digest_generation` | master | passes |
| `test_sql_digest_generation` | this PR | passes; the query took 2,024
and 2,025 ms |
| `test_sql_digest_generation` | this PR, second threshold 1000 ms |
fails as it should: `Count of sql_digest changed` |

The PR NonConcurrent pipeline runs both suites; cloud_p0 skips
`nonConcurrent` suites.

### Notes for Reviewers

- The day-boundary guard before the final `now()`-based section goes
from 100 s to 1300 s, as in #65646, so that it covers the 1200 s poll
plus the insert and checks. It sleeps only when that section would
otherwise start within 1300 s of midnight.
- Four more suites fixed in the same branch-4.1 batch need nothing on
master: #65340 (master #68449 enables the publish debug point on all
FEs, superseding the open port #65556), #65929 (master #68329 fixed the
actual cause, a connection leak), #65885 (master #69723 removed the V1
scanner whose message it also accepted), and #67363 (ported as #67402).

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: the port (both suites taken from branch-4.2), the local
verification runs, and this description
- Human review: not stated
- Prompts: The author asked to continue moving branch-4.2's changes back
to master, where six branch-4.1 test fixes were recorded as one task.
After the check showed that master already covers four of them and has
not failed on the other two, the author chose to port those two exactly
as branch-4.2 has them and to verify them on a local cluster.

Co-authored-by: shuke <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>