Support Customized Wait Time for Low Priority Query by VicoWu · Pull Request #77013 · ClickHouse/ClickHouse

VicoWu · 2025-03-02T07:36:23Z

Changelog category (leave one):

Improvement

Fix the issue #77570

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Support configurable per task waiting time of low priority query in session level and in server level.

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

clickhouse-gh · 2025-03-02T22:22:50Z

Workflow [PR], commit [5c571c4]

src/Interpreters/QueryPriorities.h

VicoWu · 2025-03-25T12:18:18Z

CI Error is unrelated with changes in this PR, I think we could ignore it:

Query succeeded, using this AST as a start
Fuzzing step 330 out of 1000
Dump of fuzzed AST:
EXPLAIN PIPELINE  WITH sub4 AS (SELECT DISTINCT a, z FROM tab3 WHERE z != 8 QUALIFY isNullable(materialize(toLowCardinality(9)) IS NOT NULL) AND (toLowCardinality(9) IS NULL)), sub5 A
S (SELECT r.y, y, rr.z, x, z FROM sub3 AS ll ANY LEFT JOIN sub4 AS rr ON ll.z = rr.z), sub1 AS (SELECT DISTINCT x, y FROM tab1 PREWHERE x IN (SELECT number FROM numbers(16) WHERE 2 !=
 number) GROUP BY GROUPING SETS ((1), (2), (* AND (7 IS NOT NULL) AS alias4622), (isZeroOrNull(toLowCardinality(2))))), sub2 AS (SELECT y, z FROM tab2__fuzz_1 WHERE y != 4), sub3 AS (
SELECT l.x, r.y, l.y, r.z AS z FROM sub1 AS l ANY LEFT JOIN sub2 AS r ON l.y = r.y) SELECT DISTINCT   7, * FROM sub5 ORDER BY x DESC SETTINGS enable_parallel_replicas = 1, parallel_re
plicas_allow_in_with_subquery = 0

WITH sub4 AS (SELECT DISTINCT a, z FROM tab3 WHERE z != 8 QUALIFY isNullable(materialize(toLowCardinality(9)) IS NOT NULL) AND (toLowCardinality(9) IS NULL)), sub5 AS (SELECT r.y, y,
rr.z, x, z FROM sub3 AS ll ANY LEFT JOIN sub4 AS rr ON ll.z = rr.z), sub1 AS (SELECT DISTINCT x, y FROM tab1 PREWHERE x IN (SELECT number FROM numbers(16) WHERE 2 != number) GROUP BY
GROUPING SETS ((1), (2), (* AND (7 IS NOT NULL) AS alias4622), (isZeroOrNull(2)))), sub2 AS (SELECT y, z FROM tab2__fuzz_4 WHERE y != 4), sub3 AS (SELECT l.x, r.y, l.y, r.z AS z FROM
sub1 AS l ANY LEFT JOIN sub2 AS r ON l.y = r.y WHERE NOT (toNullable(toNullable(7)) AND ((9 IS NULL) IS NOT NULL))) SELECT DISTINCT   7, * FROM sub5 ORDER BY x DESC SETTINGS enable_pa
rallel_replicas = 1, parallel_replicas_allow_in_with_subquery = 0

[ip-172-31-28-56] 2025.03.25 04:10:42.487879 [ 891 ] {825ac557-58ef-469a-af39-28b733aed8d9} <Fatal> : Logical error: 'Not-ready Set is passed as the second argument for function 'in''.
[ip-172-31-28-56] 2025.03.25 04:10:42.515268 [ 891 ] {825ac557-58ef-469a-af39-28b733aed8d9} <Fatal> : Stack trace (when copying this message, always include the lines below):

0. std::exception::exception[abi:ne190107]() @ 0x0000000017d9fab4
1. ./build_docker/./base/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(String const&, int) @ 0x0000000035a17680
2. ./build_docker/./src/Common/Exception.cpp:108: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000002397574b
3. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x0000000017c53316
4. DB::Exception::Exception<String>(int, FormatStringHelperImpl<std::type_identity<String>::type>, String&&) @ 0x0000000017c67dcf
5. DB::(anonymous namespace)::FunctionIn<false, false, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, bool, unsigned long) const @ 0x000000001c16d2c3
6. DB::(anonymous namespace)::FunctionIn<false, false, true, false>::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000001c16c3f0
7. ./build_docker/./src/Functions/IFunctionAdaptors.cpp:10: DB::FunctionToExecutableFunctionAdaptor::executeImpl(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long) const @ 0x000000002ac7584e
8. DB::IExecutableFunction::executeWithoutLowCardinalityColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned long, bool) const @ 0x000000002ac69777
9. DB::IExecutableFunction::executeWithoutSparseColumns(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const>
const&, unsigned long, bool) const @ 0x000000002ac6c17a
10. DB::IExecutableFunction::execute(std::vector<DB::ColumnWithTypeAndName, std::allocator<DB::ColumnWithTypeAndName>> const&, std::shared_ptr<DB::IDataType const> const&, unsigned lo
ng, bool) const @ 0x000000002ac6d790
11. ./build_docker/./src/Interpreters/ExpressionActions.cpp:636: DB::executeAction(DB::ExpressionActions::Action const&, DB::(anonymous namespace)::ExecutionContext&, bool, bool) @ 0x
000000002d572cb5
12. ./build_docker/./src/Interpreters/ExpressionActions.cpp:770: DB::ExpressionActions::execute(DB::Block&, unsigned long&, bool, bool) const @ 0x000000002d5704a7
13. ./build_docker/./src/Interpreters/ExpressionActions.cpp:816: DB::ExpressionActions::execute(DB::Block&, bool, bool) const @ 0x000000002d573c5e
14. ./build_docker/./src/Storages/MergeTree/MergeTreeRangeReader.cpp:1394: DB::MergeTreeRangeReader::executePrewhereActionsAndFilterColumns(DB::MergeTreeRangeReader::ReadResult&, DB::
Block const&, bool) const @ 0x0000000030ebc839
15. ./build_docker/./src/Storages/MergeTree/MergeTreeReadersChain.cpp:110: DB::MergeTreeReadersChain::executePrewhereActions(DB::MergeTreeRangeReader&, DB::MergeTreeRangeReader::ReadR
esult&, DB::Block const&, bool) @ 0x0000000030ee8667
16. ./build_docker/./src/Storages/MergeTree/MergeTreeReadersChain.cpp:75: DB::MergeTreeReadersChain::read(unsigned long, DB::MarkRanges&) @ 0x0000000030ee7acf
17. ./build_docker/./src/Storages/MergeTree/MergeTreeReadTask.cpp:172: DB::MergeTreeReadTask::read() @ 0x0000000030ee44f1
18. ./src/Storages/MergeTree/MergeTreeSelectAlgorithms.h:53: DB::MergeTreeInOrderSelectAlgorithm::readFromTask(DB::MergeTreeReadTask&) @ 0x00000000322887d1
19. ./build_docker/./src/Storages/MergeTree/MergeTreeSelectProcessor.cpp:207: DB::MergeTreeSelectProcessor::read() @ 0x0000000030ef07bf
20. ./build_docker/./src/Storages/MergeTree/MergeTreeSource.cpp:229: DB::MergeTreeSource::tryGenerate() @ 0x00000000322b1d95
21. ./build_docker/./src/Processors/ISource.cpp:108: DB::ISource::work() @ 0x000000003177f523
22. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:53: DB::ExecutionThreadContext::executeTask() @ 0x00000000317ace8e
23. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:296: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic<bool>*) @ 0x00000000317983bf
24. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:262: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000031797585
25. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:128: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000031796f8a
26. ./build_docker/./src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:76: void std::__function::__policy_invoker<void ()>::__call_impl[abi:ne190107]<std::__function::__defaul
t_alloc_func<ThreadFromGlobalPoolImpl<true, true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::p
ull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x00000000317b1e97
27. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000023b504cf
28. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std:
:__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x0000000023b5816a
29. ? @ 0x00007f7be4ebaac3

nickitat · 2025-03-25T12:30:48Z

The fuzzer failure looks similar to #76721

* [GLUTEN-1632][CH]Daily Update Clickhouse Version (20250326) * Fix build due to ClickHouse/ClickHouse#55518 * Fix Build due to ClickHouse/ClickHouse#77013 * Fix gtest build due to ClickHouse/ClickHouse#77895 * disbable query_plan_optimize_lazy_materialization due to #9141 --------- Co-authored-by: kyligence-git <[email protected]> Co-authored-by: Chang chen <[email protected]>

thevar1able added the can be tested Allows running workflows for external contributors label Mar 2, 2025

VicoWu force-pushed the customized_wait_time_for_low_priority_query branch from 4f28a17 to 8b1411e Compare March 12, 2025 14:04

VicoWu force-pushed the customized_wait_time_for_low_priority_query branch from 09a6664 to 4e13e25 Compare March 24, 2025 08:47

VicoWu marked this pull request as ready for review March 24, 2025 09:36

clickhouse-gh bot added the pr-improvement Pull request with some product improvements label Mar 24, 2025

VicoWu added 23 commits March 24, 2025 23:32

resolve conflict

82bc32f

partial code for priority waiting time

a972c97

trigger rebuild

f9dab65

fix build issue

ba70a89

fix compile issue

4e96c17

Fix Compile Issue

750c4f2

fix buid issue

8df360f

fix build error

b38973d

some fix

63c2973

set wait time to handle

d6feab7

set wait time to handle

03d8e22

set wait time to handle

7b33c7f

add priority log

88b885a

remove debug info

d2f0604

fix build issue

f35668f

Trigger CI

769f976

Trigger CI

1cc4959

remove server side settings

cf1ba50

remove unsed code

b0a1991

fix additional blank line

07c15ff

upper case for desc

eff1247

Add to SettingsChangesHistory.cpp

df1e225

add to change history

b897def

VicoWu force-pushed the customized_wait_time_for_low_priority_query branch from 69e1468 to b897def Compare March 24, 2025 15:34

nickitat self-assigned this Mar 24, 2025

nickitat reviewed Mar 24, 2025

View reviewed changes

src/Interpreters/QueryPriorities.h Outdated Show resolved Hide resolved

src/Interpreters/QueryPriorities.h Outdated Show resolved Hide resolved

VicoWu added 3 commits March 25, 2025 09:47

change type

24c2c7b

change type

641c881

use alias

5c571c4

nickitat approved these changes Mar 25, 2025

View reviewed changes

nickitat enabled auto-merge March 25, 2025 12:31

nickitat added this pull request to the merge queue Mar 25, 2025

Merged via the queue into ClickHouse:master with commit 82641e7 Mar 25, 2025
120 of 122 checks passed

robot-clickhouse added the pr-synced-to-cloud The PR is synced to the cloud repo label Mar 25, 2025

baibaichen added a commit to Kyligence/gluten that referenced this pull request Mar 26, 2025

Fix Build due to ClickHouse/ClickHouse#77013

19b7d89

baibaichen added a commit to Kyligence/gluten that referenced this pull request Mar 26, 2025

Fix Build due to ClickHouse/ClickHouse#77013

2fcbccb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Customized Wait Time for Low Priority Query#77013

Support Customized Wait Time for Low Priority Query#77013
nickitat merged 26 commits intoClickHouse:masterfrom
VicoWu:customized_wait_time_for_low_priority_query

VicoWu commented Mar 2, 2025 •

edited

Loading

Uh oh!

clickhouse-gh bot commented Mar 2, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

VicoWu commented Mar 25, 2025

Uh oh!

nickitat commented Mar 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

VicoWu commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

clickhouse-gh bot commented Mar 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

VicoWu commented Mar 25, 2025

Uh oh!

nickitat commented Mar 25, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

VicoWu commented Mar 2, 2025 •

edited

Loading

clickhouse-gh bot commented Mar 2, 2025 •

edited

Loading