Use cluster table functions automatically if parallel replicas are enabled by thevar1able · Pull Request #70659 · ClickHouse/ClickHouse

thevar1able · 2024-10-14T20:47:23Z

Changelog category (leave one):

Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Replace table functions with their -Cluster alternatives if parallel replicas are enabled. Fixes #65024

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

CI Settings (Only check the boxes if you know what you are doing):

Allow: All Required Checks
Allow: Stateless tests
Allow: Stateful tests
Allow: Integration Tests
Allow: Performance tests
Allow: All Builds
Allow: batch 1, 2 for multi-batch jobs
Allow: batch 3, 4, 5, 6 for multi-batch jobs

Exclude: Style check
Exclude: Fast test
Exclude: All with ASAN
Exclude: All with TSAN, MSAN, UBSAN, Coverage
Exclude: All with aarch64, release, debug

Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
Exclude: AST fuzzers

src/Analyzer/Passes/ReplaceTableFunctionsWithClusterVariantsPass.h

src/TableFunctions/ITableFunctionCluster.h

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp

nickitat

lgtm

src/TableFunctions/ITableFunctionCluster.h

tests/queries/0_stateless/03275_auto_cluster_functions_with_parallel_replicas.sql

src/TableFunctions/TableFunctionURL.cpp

src/Storages/extractTableFunctionArgumentsFromSelectQuery.cpp

src/Storages/extractTableFunctionArgumentsFromSelectQuery.h

tests/queries/0_stateless/03275_auto_cluster_functions_with_parallel_replicas.sql

src/TableFunctions/TableFunctionObjectStorage.cpp

tests/queries/0_stateless/03275_auto_cluster_functions_with_parallel_replicas.sql

src/Core/SettingsChangesHistory.cpp

src/Analyzer/Resolve/TableFunctionsWithClusterAlternativesVisitor.h

Michicosun · 2024-12-16T16:43:21Z

src/Storages/IStorageCluster.cpp

    const auto & current_settings = new_context->getSettingsRef();
    auto timeouts = ConnectionTimeouts::getTCPTimeoutsWithFailover(current_settings);
+
+    size_t max_replicas_to_use = current_settings[Setting::max_parallel_replicas];


Do we really need to limit all other queries (for example with explicitly specified Cluster function) to this number of used shards? By default it is 1 - so it is equivalent to disable cluster functions at all?

@alexey-milovidov

Let's change the default value of max_parallel_replicas to 1000.

Let's not change it to 1000 now, as it breaks too much stuff, but keep the old behaviour if max_parallel_replicas is set to 1.

With enable_parallel_replicas = false it shouldn't matter.

~~First it is allow_experimental_parallel_reading_from_replicas~~, and second it breaks exactly in the same way with it set to false.

I agree, it shouldn't, but here we are.

Ok there is enable_parallel_replicas, but is it used anywhere? https://pastila.nl/?00a5bc6c/4efc387b98c06c1c64b483fdc04aab56#Y4GdPVZrAdhCymqZEmH5AA==

EDIT: it is an alias. My bad.

Ok there is enable_parallel_replicas, but is it used anywhere?

Later, we'll switch to it in all the places

Let's change the default value of max_parallel_replicas to 1000.

Split into #74504

src/Storages/IStorageCluster.cpp

src/Storages/StorageFileCluster.cpp

src/Storages/StorageURLCluster.cpp

src/TableFunctions/TableFunctionObjectStorage.cpp

src/Core/SettingsChangesHistory.cpp

rschu1ze · 2025-03-04T19:19:36Z

src/Core/Settings.cpp

 )", BETA) \
-    \
+    DECLARE(Bool, parallel_replicas_for_cluster_engines, true, R"(
+Replace table function engines with their -Cluster alternatives


Could this elaborate a bit more on the functionality?

robot-ch-test-poll added the pr-improvement Pull request with some product improvements label Oct 14, 2024

ucasfl reviewed Oct 15, 2024

View reviewed changes

src/Analyzer/Passes/ReplaceTableFunctionsWithClusterVariantsPass.h Outdated Show resolved Hide resolved

ucasfl reviewed Oct 15, 2024

View reviewed changes

src/Analyzer/Passes/ReplaceTableFunctionsWithClusterVariantsPass.h Outdated Show resolved Hide resolved

thevar1able changed the title ~~Use cluster functions automatically if parallel replicas are enabled~~ Use cluster table functions automatically if parallel replicas are enabled Oct 18, 2024

thevar1able force-pushed the auto-cluster-engines branch 3 times, most recently from a1c2b48 to 0ac14f6 Compare November 4, 2024 16:17

thevar1able force-pushed the auto-cluster-engines branch 3 times, most recently from 84b81cc to cb20417 Compare November 13, 2024 15:13

thevar1able marked this pull request as ready for review November 13, 2024 15:13

thevar1able force-pushed the auto-cluster-engines branch 2 times, most recently from 2a1d093 to 3ff22e3 Compare November 19, 2024 15:27

thevar1able commented Nov 19, 2024

View reviewed changes

src/TableFunctions/ITableFunctionCluster.h Outdated Show resolved Hide resolved

thevar1able commented Nov 19, 2024

View reviewed changes

src/Storages/ObjectStorage/StorageObjectStorageCluster.cpp Show resolved Hide resolved

Michicosun self-assigned this Nov 19, 2024

nickitat reviewed Nov 19, 2024

View reviewed changes

src/TableFunctions/ITableFunctionCluster.h Outdated Show resolved Hide resolved

tests/queries/0_stateless/03275_auto_cluster_functions_with_parallel_replicas.sql Outdated Show resolved Hide resolved

src/TableFunctions/TableFunctionURL.cpp Outdated Show resolved Hide resolved

Michicosun requested changes Nov 20, 2024

View reviewed changes

devcrafter reviewed Nov 26, 2024

View reviewed changes

src/TableFunctions/TableFunctionObjectStorage.cpp Outdated Show resolved Hide resolved

tests/queries/0_stateless/03275_auto_cluster_functions_with_parallel_replicas.sql Outdated Show resolved Hide resolved

src/Core/SettingsChangesHistory.cpp Outdated Show resolved Hide resolved

thevar1able force-pushed the auto-cluster-engines branch 4 times, most recently from 7a7864c to 44079bd Compare December 9, 2024 15:56

thevar1able force-pushed the auto-cluster-engines branch 3 times, most recently from db1f28c to 46c234b Compare December 16, 2024 06:07

thevar1able requested a review from alexey-milovidov December 16, 2024 09:49

thevar1able force-pushed the auto-cluster-engines branch 2 times, most recently from fc8e52a to 930b13a Compare December 16, 2024 11:45

Michicosun requested changes Dec 16, 2024

View reviewed changes

Michicosun approved these changes Mar 3, 2025

View reviewed changes

rschu1ze reviewed Mar 4, 2025

View reviewed changes

src/Core/SettingsChangesHistory.cpp Outdated Show resolved Hide resolved

rschu1ze reviewed Mar 4, 2025

View reviewed changes

thevar1able force-pushed the auto-cluster-engines branch 3 times, most recently from 2ebaa0b to 96a1339 Compare March 11, 2025 15:11

thevar1able added 15 commits March 12, 2025 21:46

Replace table function engines with their -Cluster alternatives

69f1321

Lint

d5648fb

Fix darwin build

9301927

Fix darwin build

34f4efb

Update test

2d8da42

Explicitly enable analyzer in tests

52fdda0

Fix tests

2bc1c1d

Lint

f297ba9

Fix writing to table functions created as tables

1da3693

Lint

2f853ec

Lint

9a347ad

Fix more tests

d90a000

Fix reading archives with s3Cluster()

dbe3a4e

Fix harder

489aab7

Lint

8d6ac55

thevar1able force-pushed the auto-cluster-engines branch from b36a0ec to 8d6ac55 Compare March 12, 2025 20:50

alexey-milovidov added this pull request to the merge queue Mar 13, 2025

Merged via the queue into master with commit 65afe6f Mar 13, 2025
125 checks passed

alexey-milovidov deleted the auto-cluster-engines branch March 13, 2025 04:44

robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Mar 13, 2025

devcrafter mentioned this pull request Jun 25, 2025

Enable parallel distributed insert select by default #80425

Merged

CheSema mentioned this pull request Jul 31, 2025

LOGICAL_ERROR: Next task callback is not set for query #84753

Closed

1 task

thevar1able mentioned this pull request Jan 20, 2026

Automatic cluster table functions should be used for subqueries, if it's the only subquery inside FROM. #92264

Closed

Conversation

thevar1able commented Oct 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI Settings (Only check the boxes if you know what you are doing):

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nickitat left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Michicosun Dec 16, 2024

Choose a reason for hiding this comment

Uh oh!

thevar1able Dec 17, 2024

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Dec 17, 2024

Choose a reason for hiding this comment

Uh oh!

thevar1able Dec 18, 2024

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Dec 19, 2024

Choose a reason for hiding this comment

Uh oh!

thevar1able Dec 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thevar1able Dec 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

devcrafter Dec 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

thevar1able Jan 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rschu1ze Mar 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

thevar1able commented Oct 14, 2024 •

edited

Loading

thevar1able Dec 19, 2024 •

edited

Loading

thevar1able Dec 19, 2024 •

edited

Loading

devcrafter Dec 19, 2024 •

edited

Loading