Antalya 25.8: remote initiator improvements by ianton-ru · Pull Request #1576 · Altinity/ClickHouse

ianton-ru · 2026-03-25T11:45:17Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Different improvement for remote initiator

Documentation entry for user-facing changes

With remote initiator feature queries like

SELECT * FROM iceberg(...) SETTINGS object_storage_cluster='swarm', object_storage_remote_initiator=1

rewrites as

SELECT * FROM remote('remote_host', icebergCluster('swarm', ...)

'remote_host' is a random host from 'swarm' cluster
See #756

Current PR introduces the next improvements:

Partially solved object_storage_remote_initiator auth works incorrectly #1570 - uses username and password if access to cluster requires it. Throws exception if cluster uses common secret, this should be solved in future PRs.
Solved object_storage_remote_initiator with different cluster name #1571 - new setting object_storage_remote_initiator_cluster allows to choose remote_host from different cluster, not only from swarm
remote query did not work with additional setting inside function, like remote('remote_host', iceberg(..., SETTINGS iceberg_metadata_file_path='path/to/metadata.json')). Now must work correctly.
In query remote('remote_host', icebergCluster('remote_cluster', ...)) cluster remote_cluster can be defined only on remote_host and unknown on current initial host. Removed cluster check on early stage, this allows to execute such queries.

CI/CD Options

Exclude tests:

Regression jobs to run:

github-actions · 2026-03-25T11:46:08Z

Workflow [PR], commit [7caab5a]

ianton-ru · 2026-03-25T11:46:29Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 387cfa13d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-25T11:58:18Z

src/Core/SettingsChangesHistory.cpp

        {
            {"allow_local_data_lakes", false, false, "New setting."},
+            {"iceberg_partition_timezone", "", "", "New setting."},
+            {"s3_propagate_credentials_to_other_storages", false, false, "New setting"},


Drop unknown settings from compatibility history

This entry introduces a setting name that is not declared in src/Core/Settings.cpp (same issue also appears for export_merge_tree_partition_lock_inside_the_task and export_merge_tree_partition_system_table_prefer_remote_information later in this block). When compatibility is set to a version below 25.8.16.20002, SettingsImpl::applyCompatibilitySetting() iterates these changes and attempts to read/apply them, which triggers runtime "setting not found" failures instead of applying compatibility defaults.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-03-25T11:58:18Z

src/Core/SettingsChangesHistory.cpp

            {"allow_local_data_lakes", false, false, "New setting."},
+            {"iceberg_partition_timezone", "", "", "New setting."},
+            {"s3_propagate_credentials_to_other_storages", false, false, "New setting"},
+            {"export_merge_tree_part_filename_pattern", "", "{part_name}_{checksum}", "New setting"},


Keep filename-pattern change in its original version block

This re-adds export_merge_tree_part_filename_pattern as a "new setting" in 25.8.16.20002, but it is already introduced in the 25.8.9.2000 block with the same previous/new values. Because compatibility rollback applies all versions newer than the requested one, setting compatibility to any version between those two (for example 25.8.10) will now incorrectly force this setting back to "", producing defaults that don't match that target version.

Useful? React with 👍 / 👎.

ianton-ru · 2026-03-25T12:21:22Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b9c0d9bba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-03-25T12:31:42Z

src/Storages/IStorageCluster.cpp

+        auto remote_initiator_cluster_name = settings[Setting::object_storage_remote_initiator_cluster].value;
+        if (remote_initiator_cluster_name.empty())
+            remote_initiator_cluster_name = cluster_name_from_settings;
+        auto remote_initiator_cluster = getClusterImpl(context, remote_initiator_cluster_name);


Apply object_storage_max_nodes to initiator cluster selection

IStorageCluster::read now builds remote_initiator_cluster without the object_storage_max_nodes cap, so object_storage_remote_initiator=1 can pick a remote initiator from hosts that the query intentionally excluded via object_storage_max_nodes. In environments that use this cap to avoid unhealthy/unconfigured nodes, this introduces nondeterministic routing failures (auth/connectivity/cluster visibility) even though the setting is configured. The initiator cluster lookup should use the same max-host limit as the normal cluster path.

Useful? React with 👍 / 👎.

It's ok, with object_storage_max_nodes getCluster chooses this number of random cluster nodes, and for remote initiator also random node is selected, Select random node from random nodes sublist gives the same result as just select random node from full list.

ianton-ru · 2026-03-25T12:55:59Z

AI audit note: This review comment was generated by AI (gpt-5.4).

Audit update for PR #1576 after the latest branch update:

Confirmed defects

No confirmed defects in reviewed scope.

Coverage summary

Scope reviewed: current PR diff vs origin/antalya-25.8 across analyzer/table-function SETTINGS propagation, IStorageCluster::convertToRemote() auth and initiator-cluster rewrite, deferred *Cluster existence checks for nested remote(..., xxxCluster(...)), compatibility metadata for object_storage_remote_initiator_cluster, and the added integration coverage in tests/integration/test_s3_cluster.

Categories failed: none confirmed.

Categories passed: previous FunctionNode::toASTImpl AST-shape issue is fixed; previous settings-history duplication issue is fixed; nested table-function SETTINGS now survive analyzer rewrites; secret-cluster path is fail-closed; remote/remoteSecure password arguments are already treated as secret for logging/formatting; unknown nested cluster resolution is correctly deferred to the remote node; no confirmed concurrency/lifetime/rollback defects in the changed C++ paths.

Assumptions/limits: static audit only; I did not run the integration suite or build the server, so this conclusion is limited to code-path reasoning over the current PR state.

ianton-ru · 2026-03-26T11:29:50Z

tests/queries/0_stateless/01625_constraints_index_append.reference

      Prewhere filter
      Prewhere filter column: less(multiply(2, b), 100)
-  Filter column: and(indexHint(greater(plus(i, 40), 0)), equals(a, 0)) (removed)
+  Filter column: and(equals(a, 0), indexHint(greater(plus(i, 40), 0))) (removed)


Argument order depends on hash, hash was changes (see FunctionNode::updateTreeHashImpl)

alsugiliazova · 2026-03-31T01:09:36Z

PR #1576 CI Verification Report

CI Results Overview

Category	Count
Success	~60
Failure	2 (Swarms aarch64, GrypeScan alpine)
Skipped	~39 (excluded sanitizer suites)

PR's New Test Validation

The PR adds new integration tests for test_object_storage_remote_initiator. These tests initially failed on the Mar 27 CI run but were fixed by subsequent commits. The flaky check also initially failed (5 parametrized runs) but passed on the final run.

Latest run (Mar 30) — all pass:

Job	Test	Result
Integration tests (amd_binary, 4/5)	`test_object_storage_remote_initiator`	OK
Integration tests (amd_asan, old analyzer, 6/6)	`test_object_storage_remote_initiator`	OK
Integration tests (arm_binary, distributed plan, 2/4)	`test_object_storage_remote_initiator`	OK
Integration tests (amd_asan, flaky check)	`test_object_storage_remote_initiator[1-5]` through `[5-5]` (15 runs, 3 retries × 5 params)	All OK

All 18 test executions passed on the latest CI run.

CI Failures

1. `test_object_storage_remote_initiator` (Mar 27 run) — Fixed by PR Commits

Jobs: Integration tests (amd_binary 5/5, amd_asan 2/6, arm_binary 3/4) + flaky check (5 parametrized)

Failed on Mar 27 intermediate run. All tests pass on the final run (Mar 30) after fix commits 32c53796 (Fix test) and 7caab5ae (Fix setting cleanup).

Related to PR: Yes — Development-stage failures, resolved in final commits

2. `01625_constraints_index_append` (Fast test, Mar 25) — Fixed by PR Commits

Job: Fast test (2 initial runs)

The PR modifies the reference file. Failed twice on early commits (Mar 25), then passed consistently in all subsequent runs.

Related to PR: Yes — Reference file update, resolved in subsequent commits

3. `02668_column_block_number_with_projections` — Isolated Flaky

Job: Stateless tests (amd_binary, old analyzer, s3 storage, DatabaseReplicated, parallel)

Single failure on Mar 27. Only occurrence in the last 30 days for this PR. The test involves column block numbers with projections — unrelated to remote initiator changes.

Related to PR: No — Isolated flaky test failure

4. GrypeScan (-alpine) — CVE in Base Image

CVE in Alpine base image (altinityinfra/clickhouse-server:1576-25.8.16.20002.altinityantalya-alpine). Non-alpine image scan passed.

Related to PR: No — Base image vulnerability

Regression Test Results (PR's Internal CI)

Suite	x86_64	aarch64
Iceberg (1)	Pass	Pass
Iceberg (2)	Pass	Pass
Parquet	Pass	Pass
Parquet (aws_s3)	Pass	Pass
Parquet (minio)	Pass	Pass
S3 Export (part)	Pass	Pass
S3 Export (partition)	Pass	Pass
Swarms	Pass	Fail

Regression Failure: Swarms aarch64 — Pre-existing Node Failure Instability

Conclusion

Verdict: Ready to merge (already merged) — No unresolved PR-related failures.

ianton-ru added 4 commits March 25, 2026 11:32

User/password auth for object_storage_remote_initiator

9549261

Keep function settings in remote call

cc6b56f

object_storage_remote_initiator_cluster setting

a353ef4

Fix remote query with clustre, unknown on initial node

387cfa1

ianton-ru changed the title ~~Feature/antalya 25.8/remote initiator improvements~~ Antalya 25.8: remote initiator improvements Mar 25, 2026

chatgpt-codex-connector bot reviewed Mar 25, 2026

View reviewed changes

ianton-ru added 2 commits March 25, 2026 12:59

Fix settings change history

a05f8fe

Fix FunctionNode::toASTImpl

0b9c0d9

chatgpt-codex-connector bot reviewed Mar 25, 2026

View reviewed changes

ianton-ru added antalya antalya-25.8 labels Mar 25, 2026

Fix test

32c5379

ianton-ru commented Mar 26, 2026

View reviewed changes

svb-alt requested a review from arthurpassos March 26, 2026 17:47

Add comments

e057248

arthurpassos previously approved these changes Mar 27, 2026

View reviewed changes

Fix setting object_storage_remote_initiator_cluster cleanup

7caab5a

ianton-ru dismissed arthurpassos’s stale review via 7caab5a March 30, 2026 09:19

arthurpassos approved these changes Mar 30, 2026

View reviewed changes

zvonand merged commit 096f029 into antalya-25.8 Mar 30, 2026
264 of 268 checks passed

alsugiliazova added the verified Approved for release label Mar 31, 2026

Selfeer mentioned this pull request Apr 1, 2026

Fix IPartitionStrategy race condition #1517

Merged

25 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Antalya 25.8: remote initiator improvements#1576

Antalya 25.8: remote initiator improvements#1576
zvonand merged 9 commits intoantalya-25.8from
feature/antalya-25.8/remote_initiator_improvements

ianton-ru commented Mar 25, 2026

Uh oh!

github-actions bot commented Mar 25, 2026 •

edited

Loading

Uh oh!

ianton-ru commented Mar 25, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Uh oh!

ianton-ru commented Mar 25, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Uh oh!

ianton-ru Mar 25, 2026 •

edited

Loading

Uh oh!

ianton-ru commented Mar 25, 2026

Uh oh!

ianton-ru Mar 26, 2026

Uh oh!

Uh oh!

alsugiliazova commented Mar 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

ianton-ru commented Mar 25, 2026

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

github-actions bot commented Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianton-ru commented Mar 25, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Mar 25, 2026

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru Mar 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Mar 25, 2026

Confirmed defects

Coverage summary

Uh oh!

ianton-ru Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

alsugiliazova commented Mar 31, 2026

PR #1576 CI Verification Report

CI Results Overview

PR's New Test Validation

CI Failures

1. test_object_storage_remote_initiator (Mar 27 run) — Fixed by PR Commits

2. 01625_constraints_index_append (Fast test, Mar 25) — Fixed by PR Commits

3. 02668_column_block_number_with_projections — Isolated Flaky

4. GrypeScan (-alpine) — CVE in Base Image

Regression Test Results (PR's Internal CI)

Regression Failure: Swarms aarch64 — Pre-existing Node Failure Instability

Conclusion

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Mar 25, 2026 •

edited

Loading

ianton-ru Mar 25, 2026 •

edited

Loading

1. `test_object_storage_remote_initiator` (Mar 27 run) — Fixed by PR Commits

2. `01625_constraints_index_append` (Fast test, Mar 25) — Fixed by PR Commits

3. `02668_column_block_number_with_projections` — Isolated Flaky