Skip to content

Commit 0dc1c3f

Browse files
devcrafterdevcrafter
authored
Merge branch 'master' into wudidapaopao/optimize_lazy_projection
2 parents 53a186d + 7edb0e4 commit 0dc1c3f

File tree

125 files changed

+1759
-364
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

125 files changed

+1759
-364
lines changed

‎docs/en/operations/server-configuration-parameters/settings.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -821,6 +821,20 @@ Type: `UInt64`
821821

822822
Default: `100000`
823823

824+
## max_pending_mutations_to_warn {#max_pending_mutations_to_warn}
825+
826+
If the number of pending mutations exceeds the specified value, clickhouse server will add warning messages to `system.warnings` table.
827+
828+
**Example**
829+
830+
```xml
831+
<max_pending_mutations_to_warn>400</max_pending_mutations_to_warn>
832+
```
833+
834+
Type: `UInt64`
835+
836+
Default: `500`
837+
824838
## max_table_num_to_throw {#max_table_num_to_throw}
825839

826840
If number of tables is greater than this value, server will throw an exception.

‎docs/en/operations/system-tables/system_warnings.md‎

Lines changed: 12 additions & 64 deletions
Original file line numberDiff line numberDiff line change
@@ -11,76 +11,24 @@ import SystemTableCloud from '@site/docs/_snippets/_system_table_cloud.md';
1111

1212
<SystemTableCloud/>
1313

14-
This table contains warning messages about clickhouse server. The threshold for some of the warnings can be configured
15-
accordingly. When the warning threshold is crossed, clickhouse server will add a warning message to `system.warnings`
16-
table. It will also update a warning message according to the current value and remove it when it drops below the
17-
configured threshold. Currently, the following warnings are configurable.
14+
This table shows warnings about the ClickHouse server.
15+
Warnings of the same type are combined into a single warning.
16+
For example, if the number N of attached databases exceeds a configurable threshold T, a single entry containing the current value N is shown instead of N separate entries.
17+
If current value drops below the threshold, the entry is removed from the table.
1818

19-
### Maximum number of tables {#max_num_of_tables}
19+
The table can be configured with these settings:
2020

21-
For warning when the maximum number of attached tables exceeds the specified value.
22-
23-
**Configuration:**
24-
25-
```xml
26-
<max_table_num_to_warn>500</max_table_num_to_warn>
27-
```
28-
29-
### Maximum number of databases {#max_num_of_databases}
30-
31-
For warning when the number of attached databases exceeds the specified value.
32-
33-
**Configuration:**
34-
35-
```xml
36-
<max_database_num_to_warn>500</max_database_num_to_warn>
37-
```
38-
39-
### Maximum number of dictionaries {#max_num_of_dictionaries}
40-
41-
For warning when the number of attached dictionaries exceeds the specified value.
42-
43-
**Configuration:**
44-
45-
```xml
46-
<max_dictionary_num_to_warn>500</max_dictionary_num_to_warn>
47-
```
48-
49-
### Maximum number of views {#max_num_of_views}
50-
51-
For warning when the number of attached views exceeds the specified value.
52-
53-
**Configuration:**
54-
55-
```xml
56-
<max_view_num_to_warn>500</max_view_num_to_warn>
57-
```
58-
59-
### Maximum number of parts {#max_num_of_parts}
60-
61-
For warning when the number of active parts exceeds the specified value.
62-
63-
**Configuration:**
64-
65-
```xml
66-
<max_part_num_to_warn>500</max_part_num_to_warn>
67-
```
68-
69-
### Maximum number of pending mutations {#max_num_of_pending_mutations}
70-
71-
For warning when the number of pending mutations exceeds the specified value.
72-
73-
**Configuration:**
74-
75-
```xml
76-
<max_pending_mutations_to_warn>500</max_pending_mutations_to_warn>
77-
```
21+
- [max_table_num_to_warn](../server-configuration-parameters/settings.md#max_table_num_to_warn)
22+
- [max_database_num_to_warn](../server-configuration-parameters/settings.md#max_database_num_to_warn)
23+
- [max_dictionary_num_to_warn](../server-configuration-parameters/settings.md#max_dictionary_num_to_warn)
24+
- [max_view_num_to_warn](../server-configuration-parameters/settings.md#max_view_num_to_warn)
25+
- [max_part_num_to_warn](../server-configuration-parameters/settings.md#max_part_num_to_warn)
26+
- [max_pending_mutations_to_warn](../server-configuration-parameters/settings.md#max_pending_mutations_to_warn)
7827

7928
Columns:
8029

8130
- `message` ([String](../../sql-reference/data-types/string.md)) — Warning message.
82-
- `message_format_string` ([LowCardinality(String)](../../sql-reference/data-types/string.md)) — The format string that
83-
was used to format the message.
31+
- `message_format_string` ([LowCardinality(String)](../../sql-reference/data-types/string.md)) — The format string used to format the message.
8432

8533
**Example**
8634

‎docs/en/sql-reference/statements/alter/database-comment.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
---
2+
description: 'Documentation for ALTER DATABASE ... MODIFY COMMENT Statements'
3+
slug: /sql-reference/statements/alter/database-comment
4+
sidebar_position: 51
5+
sidebar_label: 'UPDATE'
6+
title: 'ALTER DATABASE ... MODIFY COMMENT Statements'
7+
---
8+
9+
# ALTER DATABASE ... MODIFY COMMENT
10+
11+
Adds, modifies, or removes comment to the table, regardless if it was set before or not. Comment change is reflected in both [system.databases](/operations/system-tables/databases.md) and `SHOW CREATE DATABASE` query.
12+
13+
**Syntax**
14+
15+
``` sql
16+
ALTER DATABASE [db].name [ON CLUSTER cluster] MODIFY COMMENT 'Comment'
17+
```
18+
19+
**Examples**
20+
21+
Creating a DATABASE with comment (for more information, see the [COMMENT](/sql-reference/statements/create/table#comment-clause) clause):
22+
23+
``` sql
24+
CREATE DATABASE database_with_comment ENGINE = Memory COMMENT 'The temporary database';
25+
```
26+
27+
Modifying the table comment:
28+
29+
``` sql
30+
ALTER DATABASE database_with_comment MODIFY COMMENT 'new comment on a database';
31+
SELECT comment FROM system.databases WHERE name = 'database_with_comment';
32+
```
33+
34+
Output of a new comment:
35+
36+
```text
37+
┌─comment─────────────────┐
38+
│ new comment on database │
39+
└─────────────────────────┘
40+
```
41+
42+
Removing the database comment:
43+
44+
``` sql
45+
ALTER DATABASE database_with_comment MODIFY COMMENT '';
46+
SELECT comment FROM system.databases WHERE name = 'database_with_comment';
47+
```
48+
49+
Output of a removed comment:
50+
51+
```text
52+
┌─comment─┐
53+
│ │
54+
└─────────┘
55+
```

‎src/Access/Common/AccessType.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@ enum class AccessType : uint8_t
4343
M(ALTER_MATERIALIZE_COLUMN, "MATERIALIZE COLUMN", COLUMN, ALTER_COLUMN) \
4444
M(ALTER_COLUMN, "", GROUP, ALTER_TABLE) /* allow to execute ALTER {ADD|DROP|MODIFY...} COLUMN */\
4545
M(ALTER_MODIFY_COMMENT, "MODIFY COMMENT", TABLE, ALTER_TABLE) /* modify table comment */\
46+
M(ALTER_MODIFY_DATABASE_COMMENT, "MODIFY DATABASE COMMENT", DATABASE, ALTER_DATABASE) /* modify database comment*/\
4647
\
4748
M(ALTER_ORDER_BY, "ALTER MODIFY ORDER BY, MODIFY ORDER BY", TABLE, ALTER_INDEX) \
4849
M(ALTER_SAMPLE_BY, "ALTER MODIFY SAMPLE BY, MODIFY SAMPLE BY", TABLE, ALTER_INDEX) \

‎src/Common/CurrentMetrics.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -384,6 +384,9 @@
384384
M(StartupScriptsExecutionState, "State of startup scripts execution: 0 = not finished, 1 = success, 2 = failure.") \
385385
\
386386
M(IsServerShuttingDown, "Indicates if the server is shutting down: 0 = no, 1 = yes") \
387+
\
388+
M(TotalMergeFailures, "Number of all failed merges since startup, including the ones that were aborted") \
389+
M(NonAbortedMergeFailures, "Number of failed merges since startup, excluding the merges that were aborted") \
387390

388391
#ifdef APPLY_FOR_EXTERNAL_METRICS
389392
#define APPLY_FOR_METRICS(M) APPLY_FOR_BUILTIN_METRICS(M) APPLY_FOR_EXTERNAL_METRICS(M)

‎src/Common/ProfileEvents.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -219,6 +219,7 @@
219219
M(ExternalJoinUncompressedBytes, "Amount of data (uncompressed, before compression) written for JOIN in external memory.", ValueType::Bytes) \
220220
\
221221
M(IcebergPartitionPrunnedFiles, "Number of skipped files during Iceberg partition pruning", ValueType::Number) \
222+
M(IcebergTrivialCountOptimizationApplied, "Trivial count optimization applied while reading from Iceberg", ValueType::Number) \
222223
M(JoinBuildTableRowCount, "Total number of rows in the build table for a JOIN operation.", ValueType::Number) \
223224
M(JoinProbeTableRowCount, "Total number of rows in the probe table for a JOIN operation.", ValueType::Number) \
224225
M(JoinResultRowCount, "Total number of rows in the result of a JOIN operation.", ValueType::Number) \

‎src/Common/SignalHandlers.cpp‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -201,6 +201,12 @@ static DISABLE_SANITIZER_INSTRUMENTATION void sanitizerDeathCallback()
201201

202202
char buf[signal_pipe_buf_size];
203203
auto & signal_pipe = HandledSignals::instance().signal_pipe;
204+
205+
/// Signal pipe can be already closed in BaseDaemon::~BaseDaemon, but
206+
/// sanitizerDeathCallback() can be called on exit handlers.
207+
if (signal_pipe.fds_rw[1] == -1)
208+
return;
209+
204210
WriteBufferFromFileDescriptorDiscardOnFailure out(signal_pipe.fds_rw[1], signal_pipe_buf_size, buf);
205211

206212
const StackTrace stack_trace;

‎src/Common/benchmarks/CMakeLists.txt‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,3 +12,8 @@ clickhouse_add_executable(orc_string_dictionary orc_string_dictionary.cpp)
1212
target_link_libraries (orc_string_dictionary PRIVATE
1313
ch_contrib::gbenchmark_all
1414
dbms)
15+
16+
clickhouse_add_executable(wrap_in_nullable wrap_in_nullable.cpp)
17+
target_link_libraries (wrap_in_nullable PRIVATE
18+
ch_contrib::gbenchmark_all
19+
dbms)

‎src/Common/benchmarks/wrap_in_nullable.cpp‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#include <DataTypes/DataTypeNullable.h>
2+
#include <DataTypes/DataTypesNumber.h>
3+
#include <Columns/ColumnNullable.h>
4+
#include <Functions/FunctionHelpers.h>
5+
#include <Storages/StorageGenerateRandom.h>
6+
#include <benchmark/benchmark.h>
7+
8+
using namespace DB;
9+
10+
static void BM_WrapInNullable1(benchmark::State & state)
11+
{
12+
pcg64 rng;
13+
UInt64 limit = DEFAULT_BLOCK_SIZE;
14+
UInt64 max_array_length = 10;
15+
UInt64 max_string_length = 10;
16+
DataTypePtr nullable_float64 = std::make_shared<DataTypeNullable>(std::make_shared<DataTypeFloat64>());
17+
18+
ColumnPtr arg_column = fillColumnWithRandomData(nullable_float64, limit, max_array_length, max_string_length, rng, nullptr);
19+
ColumnsWithTypeAndName args{{arg_column, nullable_float64, "arg"}};
20+
21+
ColumnPtr denull_src_column
22+
= fillColumnWithRandomData(removeNullable(nullable_float64), limit, max_array_length, max_string_length, rng, nullptr);
23+
24+
DataTypePtr uint8_type = std::make_shared<DataTypeUInt8>();
25+
for (auto _ : state)
26+
{
27+
state.PauseTiming();
28+
ColumnPtr null_map_column = fillColumnWithRandomData(uint8_type, limit, max_array_length, max_string_length, rng, nullptr);
29+
ColumnPtr src_column = ColumnNullable::create(denull_src_column, std::move(null_map_column));
30+
state.ResumeTiming();
31+
32+
auto result = wrapInNullable(std::move(src_column), args, nullable_float64, limit);
33+
benchmark::DoNotOptimize(result);
34+
}
35+
}
36+
37+
static void BM_WrapInNullable2(benchmark::State & state)
38+
{
39+
pcg64 rng;
40+
UInt64 limit = DEFAULT_BLOCK_SIZE;
41+
UInt64 max_array_length = 10;
42+
UInt64 max_string_length = 10;
43+
44+
DataTypePtr nullable_float64 = std::make_shared<DataTypeNullable>(std::make_shared<DataTypeFloat64>());
45+
ColumnPtr src_column = fillColumnWithRandomData(nullable_float64, limit, max_array_length, max_string_length, rng, nullptr);
46+
47+
DataTypePtr uint8_type = std::make_shared<DataTypeUInt8>();
48+
for (auto _ : state)
49+
{
50+
state.PauseTiming();
51+
ColumnPtr null_map_column = fillColumnWithRandomData(uint8_type, limit, max_array_length, max_string_length, rng, nullptr);
52+
state.ResumeTiming();
53+
54+
auto result = wrapInNullable(src_column, std::move(null_map_column));
55+
benchmark::DoNotOptimize(result);
56+
}
57+
}
58+
59+
BENCHMARK(BM_WrapInNullable1);
60+
BENCHMARK(BM_WrapInNullable2);

‎src/Core/Settings.cpp‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4746,6 +4746,9 @@ Possible values:
47464746
)", 0) \
47474747
DECLARE(Bool, query_plan_convert_outer_join_to_inner_join, true, R"(
47484748
Allow to convert OUTER JOIN to INNER JOIN if filter after JOIN always filters default values
4749+
)", 0) \
4750+
DECLARE(Bool, query_plan_convert_join_to_in, false, R"(
4751+
Allow to convert JOIN to subquery with IN if output columns tied to only left table
47494752
)", 0) \
47504753
DECLARE(Bool, query_plan_optimize_prewhere, true, R"(
47514754
Allow to push down filter to PREWHERE expression for supported storages

0 commit comments

Comments
 (0)