Skip to content

Release pull request for branch 20.5#11870

Closed
robot-clickhouse wants to merge 250 commits intomasterfrom
20.5
Closed

Release pull request for branch 20.5#11870
robot-clickhouse wants to merge 250 commits intomasterfrom
20.5

Conversation

@robot-clickhouse
Copy link
Copy Markdown
Member

This PullRequest is part of ClickHouse release cycle. It's used by CI system only. Don't perform any changes with it.

@robot-clickhouse robot-clickhouse added the release Label for release pull request label Jun 22, 2020
robot-clickhouse and others added 3 commits June 25, 2020 23:01
@KochetovNicolai KochetovNicolai requested a review from a team as a code owner June 26, 2020 06:29
@robot-clickhouse robot-clickhouse added the submodule changed At least one submodule changed in this PR. label Jun 26, 2020
@robot-clickhouse robot-clickhouse removed the submodule changed At least one submodule changed in this PR. label Jun 26, 2020
alexey-milovidov and others added 3 commits June 26, 2020 15:54
Fix nullable prewhere type 2

(cherry picked from commit a7b224a)
Fix nullable prewhere type 3

(cherry picked from commit f7f16fd)
Backport #11875 to 20.5: Fix strange and wrong code around DateTime64
@robot-clickhouse robot-clickhouse added the submodule changed At least one submodule changed in this PR. label Jun 27, 2020
abyss7 and others added 2 commits June 28, 2020 18:59
abyss7 and others added 14 commits July 2, 2020 17:43
…cess rights. (#11963)

* Simple github hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* typo

* fixup

* add flag to continue on errors

* fixup

* fixup

* Compilable setColumns, setConstraints, setIndices

* Working setColumns, setConstraints, setIndices

* Move set*Key methods to StorageInMemoryMetadata

* Working set*Keys methods

* TTL methods in StorageInMemoryMetadata

* All set methods in metadata

* Metadata in read and write methods of IStorage

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* Copy some methods to metdata

* Get column dependencies in StorageInMemoryMetadata

* Move getSampleBlockNonMaterialized to StorageInMemoryMetadata

* getSampleBlockWithVirtuals in StorageInMemoryMetadata

* Compileable getSampleBlockWithColumns in StorageInMemoryMetadata

* Seems to be working getSampleBlockWithColumns in StorageInMemoryMetadata

* Buildable getSampleBlock in StorageInMemoryMetadata

* getConstraints() in StorageInMemoryMetadata (suspicious commit, but pretend to work)

* Fix race condition

* Secondary indices in StorageInMemoryMetadata

* Partition key in StorageInMemoryMetadata

* Compilable sorting key in metadata

* Fix race condition

* Sampling key in StorageInMemoryMetadata

* Primary key in storage metadata

* Missed change for primary key

* TTL in storage in memory metadata

* Settings changes in StorageInMemoryMetadata

* Select query in metadata

* Check methods in metadata

* Better getVirtuals method

* getColumns in StorageInMemoryMetadta (only compilable)

* Fix segmentation fault

* Fix build

* Fix crash

* Fix storage merge

* Truncate with metadata

* Fix SystemTables

* Really atomic metadata

* Fix style

* Fix storage buffer metadata

* Better locks in Storages

* Less locks

* Fix style

* Remove unused method

* Remove unused method better

* Remove int contention

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Fix locks

* Add missed check

* Move partition key initialization into registerMergeTree

* Remove unused constructor and rename method

* Better naming

* Better logging in storages

* Remove unused variable

* report the number of errors

* fixup

* fixup

* Send logs on fatal errors (development)

* Set send_logs_level to fatal by default

* Update tests

* Turn off traps

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* Fix memory accounting via HTTP interface

function perf_test()
{
    time yes '127.1:8123/?wait_end_of_query=1' | head -n10000 | xargs -P10000 curl -s -d 'select 1' | grep -x -c 1
}
function server()
{
    local limit=$1 && shift
    clickhouse-server "$@" -- --max_server_memory_usage=$limit
}

perf_test:

- before this patch with 1G  limit: succeed only ~800  queries
- after  this patch with 1G  limit: succeed      ~8000 queries

- before this patch with 10G limit: succeed only ~3000  queries
- after  this patch with 10G limit: succeed      ~10000 queries

Fixes: #11153

* Add a test for memory tracking via HTTP interface

* Update executeQuery.cpp

* Make LIVE VIEW test less wrong (it was depending on the timing of progress messages)

* Do not run 01238_http_memory_tracking under sanitizers

* Tune 01238_http_memory_tracking to make it more optimal

w/o the patch the test fails with, since it succeed only 512 queries.

* Merge with master

* Ad tests with long select and simulatenous alter

* Better comments

* Fix 01213_alter_rename_column_zookeeper

* report

* Fix bad test for LIVE VIEW

* Disable MySQL tests under tsan

* Better 00980_alter_settings test

* Support sumWithOverflow as SimpleAggregateFunction

* Clang-tidy fixes

* Fix header for nullable prewhere column.

* Update tests.

* Fix 01091_num_threads flaps (by adding unique identifier to the queries)

Actually there were two issues:
- missing \n, so it picked the wrong query anway
- no unique identifier, hence it may works incorrectly (and actually,
  event clickhouse-test script executes 'SELECT 1' query)

* Initialize GlobalThreadPool explicitly

* Do not render progress too much

* Stop tests on fatal errors

* jemalloc: fix HAVE_CPU_SPINWAIT for linux aarch64

Fixes: #11163

* Style

* constexpr intExp10

* Fix bugs in DateTime64 parsing

* Added a test

* Fix wrong code

* Fix strange code

* Update tests

* jemalloc: jemalloc_preamble.h is common

* jemalloc: fix JEMALLOC_NOTHROW/JEMALLOC_CXX_THROW for freebsd/darwin

* jemalloc: add freebsd pregenerated header

* jemalloc: add darwin pregenerated header

* jemalloc: enable by default for freebsd/darwin

* jemalloc: drop superfluous include directory

* jemalloc: explicitly disable pthread_setname_np on freebsd (requires 12.1)

* Add drop replica alter support

* fix merge state

Fix style && build

* Add tests and docs

Fix integration-tests

* only drop inactive replica

update doc

Increase timeout to release the zookeeper Ephemeral nodes

Fix code comment

use PartitionManager

make integrations test passed

* add SYSTEM DROP REPLICA

* fix: can not drop local table and replace ALTER with SYSTEM

fix test_drop_replica

fix drop replica '/path/to/zk/' ending in '/' and update doc

* add drop relica database and the whole replica

fix removeReplicaByZKPath

fix bug: add zkpath empty judge

fix: rewrite code

delete useless code.

fix:ast fromat

fix bug

add test_drop_replica

add drop_replica doc

add drop databse checkAccess

refactor dropReplica

update tests

add static method StorageReplicatedMergeTree::dropReplicaByZkPath

update doc and delete useless code

fix conflict

fix doc

fix doc

fix StorageReplicatedMergeTree::dropReplica

fix bug

delete useless code

* rewrite static StorageReplicatedMergeTree::dropReplica

* format comment

* SHA256 add FixedString support

* fixup

* Update index.md

* fix of incorrect example

was `curl -sS "<address>?param_id=2¶m_phrase=test" -d "SELECT * FROM table WHERE int_column = {id:UInt8} and string_column = {phrase:String}"`

* Add missed lock

* jemalloc: disable percpu_arena on non-linux (not supported anyway)

And since it does not supported it produces the following warning:
  <jemalloc>: No getcpu support: percpu_arena:percpu
  <jemalloc>: perCPU arena getcpu() not available. Setting narenas to 8.

However it is uses narenas=ncpu instead of default narenas=ncpu*4

* Optimize code

* jemalloc: disable it for darwin

Right now it aborts (Abort trap: 6) on osx, the issue does not looks
like [1], but should be investigated, so disable for now.

What is interesting that it works under debugger, so I guess it is
indeed the order of symbol replacements...

  [1]: jemalloc/jemalloc#420

* fix bug for query "select *, abc.*" should failed. (#11753)

* Remove redundat sync replica

* test commit

* Hook for push events

* add test

fix

fix

fix

* memory usage settings

* Update README.md

* Update README.md

* Merge pull request #11071 (#11874)

* Add's hasSubstr  array function

* Add's example in the english doc of hasSubStr

* Updates conditions to avoid prematurely calling isEqual

* Remove feathericons

Authored-by: Ryad ZENINE <[email protected]>

* allow implicit initialization

* Fix bad code

* Allow ClickHouse to run on Android

* Fix documentation

* Add docs about non-linux binaries (#11889)

* Add docs about non-linux binaries

* Better

* Update install.md

* Update install.md

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Update docs/en/getting-started/install.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Review fixes

* Better instruction

* Fix reference

* Better

* Better links

Co-authored-by: Ivan Blinkov <[email protected]>

* Fix anotehr one nullable prewhere column.

* minor improvements

* Update StorageSystemTables.cpp

* Added a test

* fix style check - removed unused LOGICAL_ERROR

* Fix if function with NULLs (#11807)

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* sync with goals

* Remove unused code

* Remove already fixed bug

* update SELECT description (#11907)

Signed-off-by: Slach <[email protected]>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Fix using current database while checking access rights.

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* for the love of God just update this container already, i'm begging you

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* Update features.html

* fixpu

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

Co-authored-by: alesapin <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Bharat Nallan <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: sundy-li <[email protected]>
Co-authored-by: amudong <[email protected]>
Co-authored-by: feng lv <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: Alexandr Kondratev <[email protected]>
Co-authored-by: hexiaoting <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Vxider <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Eugene Klimov <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: robot-clickhouse <[email protected]>
…e5fef92b96770916a85cee0252941

Cherry pick #12056 to 20.5: Fix defaultValueOfArgumentType
…12248)

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Move reading from RabbitMQ into background task

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* Fix ORDER BY tuple with COLLATE on const column

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* make a short test for DateTime

* Fixup

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <[email protected]>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>

* Use ConcurentBoundedQueue instead of vector

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Update contrib/jemalloc-cmake/CMakeLists.txt

* Remove libevent library

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Move writing to RabbitMQ into background task

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix and simplify code

* Fix test.

* Update ITransformingStep.h

* Fix "Sharding key is not deterministic" message

* Do not check *optimize_skip_unused_shards_nesting w/o *optimize_skip_unused_shards

This will avoid supperior log message in case of
*optimize_skip_unused_shards_nesting already disables it.
And also it is logical.

* Remove test that is not supported by "Arcadia" build system

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Added test.

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Fix tests.

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <[email protected]>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <[email protected]>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>

* Fix tests.

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: alesapin <[email protected]>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* Fix result_rows and result_bytes metrics for selects.

* Fix result_rows and result_bytes metrics for selects.

* improve breadcrumbs markup

* Fix tests.

* Less race conditions

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Added test.

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Remove LC converting to Arrow.

* Remove LC converting to Arrow.

* Move skip lists to clickhouse-test

* Remove LC converting to Arrow.

* More verbose message about skip

* Make skip-list optional

* Added test.

* Add type column in system.disks

* Update clickhouse-test

* Update 01355_CSV_input_format_allow_errors.sh

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* contrib/unixodbc-cmake: Fix build when UNBUNDLED

target_compile_definitions may only set INTERFACE properties on IMPORTED targets

* Fix version column in replicated version collapsing merge tree (#12121)

* poco-cmake: Fix Poco::Data::ODBC target when UNBUNDLED

By default IMPORTED target has a scope in the directory in which it is created
and below. This leads to the following issues when building UNBUNDLED:

Target "clickhouse" links to target "Poco::Data::ODBC" but the target was not
found.  Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?

* Add missed <atomic> include in ProxyListConfiguration

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

* Remove harmful code from "geoDistance" #12117

* Fix warnings from CodeQL

* Added a test

* Minor modification

* Fix potential overflow in integer division #12119

* Added a test

* Whitespace

* Added a test for #4211

* Attempt to fix "Arcadia" build

* Rename test

* Update arraySum.cpp

* ILIKE operator (#12125)

* Integrated CachingAllocator into MarkCache

* fixed build errors

* reset func hotfix

* upd: Fixing build

* updated submodules links

* fix 2

* updating grabber allocator proto

* updating lost work

* updating CMake to use concepts

* some other changes to get it building (integration into MarkCache)

* further integration into caches

* updated Async metrics, fixed some build errors

* and some other errors revealing

* added perfect forwarding to some functions

* fix: forward template

* fix: constexpr modifier

* fix: FakePODAllocator missing member func

* updated PODArray constructor taking alloc params

* fix: PODArray overload with n restored

* fix: FakePODAlloc duplicating alloc() func

* added constexpr variable for alloc_tag_t

* split cache values by allocators, provided updates

* fix: memcpy

* fix: constexpr modifier

* fix: noexcept modifier

* fix: alloc_tag_t for PODArray constructor

* fix: PODArray copy ctor with different alloc

* fix: resize() signature

* updating to lastest working master

* syncing with 273267

* first draft version

* fix: update Searcher to case-insensitive

* added ILIKE test

* fixed style errors, updated test, split like and ilike,  added notILike

* replaced inconsistent comments

* fixed show tables ilike

* updated missing test cases

* regenerated ya.make

* Update 01355_ilike.sql

Co-authored-by: myrrc <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>

Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: kssenii <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: Mikhail Gaidamaka <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Ivan <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: chengy8934 <[email protected]>
Co-authored-by: MicrochipQ <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Vladimir Chebotarev <[email protected]>
Co-authored-by: Nicolae Vartolomei <[email protected]>
Co-authored-by: filimonov <[email protected]>
Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: manmitya <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Mikhail Filimonov <[email protected]>
Co-authored-by: Anton Ivashkin <[email protected]>
Co-authored-by: Ilya Yatsishin <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Matwey V. Kornilov <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: myrrc <[email protected]>
…12249)

* change used flag

* fix address formatting

* use the sentry logger hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* fixes

* experiment

* typo

* trigger ci

* fixup

* add flag to continue on errors

* trigger ci

* trigger ci

* trigger ci

* experiment

* trigger ci

* fixup

* fixup

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update http_server.py

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* improvements after review comments

* move the default endpoint to config

* remove extra line

* distinct combinator for single numeric arguments

* rework distinct combinator

* fix build

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Update 00816_long_concurrent_alter_column.sh

* distinct combinator for function of multiuple arguments

* report the number of errors

* fixup

* fixup

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <[email protected]>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“, and fix an error.

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* update

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>
Co-authored-by: Bharat Nallan <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: Nikita Vasilev <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Eugene Klimov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: Maxim Akhmedov <[email protected]>
Co-authored-by: feng lv <[email protected]>
Co-authored-by: Nikita Mikhailov <[email protected]>
Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: Tom Bombadil <[email protected]>
Co-authored-by: Yuntao Wu <[email protected]>
Co-authored-by: Wang Fenjin <[email protected]>
…2260)

* Whitespace

* Fix ORDER BY tuple with COLLATE on const column

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* make a short test for DateTime

* Fixup

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <[email protected]>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>

* Use ConcurentBoundedQueue instead of vector

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Update contrib/jemalloc-cmake/CMakeLists.txt

* Remove libevent library

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Move writing to RabbitMQ into background task

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix and simplify code

* Fix test.

* Update ITransformingStep.h

* Fix "Sharding key is not deterministic" message

* Do not check *optimize_skip_unused_shards_nesting w/o *optimize_skip_unused_shards

This will avoid supperior log message in case of
*optimize_skip_unused_shards_nesting already disables it.
And also it is logical.

* Fix function if with FixedString arguments of different sizes

* Add multiIf

* Fix error

* Remove test that is not supported by "Arcadia" build system

* Fix constraints check for constant columns

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix error

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Added test.

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Fix tests.

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <[email protected]>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <[email protected]>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* Changelog for 20.1, 20.4

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>

* Fix tests.

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: alesapin <[email protected]>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* Fix result_rows and result_bytes metrics for selects.

* Fix result_rows and result_bytes metrics for selects.

* improve breadcrumbs markup

* Fix tests.

* Less race conditions

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Added test.

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Changelog for 20.5

* Update CHANGELOG.md

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Remove LC converting to Arrow.

* Remove LC converting to Arrow.

* Move skip lists to clickhouse-test

* Remove LC converting to Arrow.

* More verbose message about skip

* Make skip-list optional

* Added test.

* Add type column in system.disks

* Update clickhouse-test

* Fix handling dependency of table with ENGINE=Dictionary on dictionary.

* Add test.

* Update 01355_CSV_input_format_allow_errors.sh

* Update CHANGELOG.md

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* contrib/unixodbc-cmake: Fix build when UNBUNDLED

target_compile_definitions may only set INTERFACE properties on IMPORTED targets

* Fix version column in replicated version collapsing merge tree (#12121)

* poco-cmake: Fix Poco::Data::ODBC target when UNBUNDLED

By default IMPORTED target has a scope in the directory in which it is created
and below. This leads to the following issues when building UNBUNDLED:

Target "clickhouse" links to target "Poco::Data::ODBC" but the target was not
found.  Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?

* Add missed <atomic> include in ProxyListConfiguration

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

* New ISO8601 year modificators for formatDateTime

* Added a test

* Remove harmful code from "geoDistance" #12117

* Update formatDateTime.cpp

* Fix warnings from CodeQL

* Added a test

* Minor modification

* Update libdivide to the latest master

* Replace exit to abort in libdivide

* Fix potential overflow in integer division #12119

* Added a test

* Whitespace

* Added a test for #4211

* Attempt to fix "Arcadia" build

* Rename test

* Update arraySum.cpp

* Make code clearer: use enum instead of `bool internal`.

* Update StorageDictionary.h

* Update StorageDictionary.h

* ILIKE operator (#12125)

* Integrated CachingAllocator into MarkCache

* fixed build errors

* reset func hotfix

* upd: Fixing build

* updated submodules links

* fix 2

* updating grabber allocator proto

* updating lost work

* updating CMake to use concepts

* some other changes to get it building (integration into MarkCache)

* further integration into caches

* updated Async metrics, fixed some build errors

* and some other errors revealing

* added perfect forwarding to some functions

* fix: forward template

* fix: constexpr modifier

* fix: FakePODAllocator missing member func

* updated PODArray constructor taking alloc params

* fix: PODArray overload with n restored

* fix: FakePODAlloc duplicating alloc() func

* added constexpr variable for alloc_tag_t

* split cache values by allocators, provided updates

* fix: memcpy

* fix: constexpr modifier

* fix: noexcept modifier

* fix: alloc_tag_t for PODArray constructor

* fix: PODArray copy ctor with different alloc

* fix: resize() signature

* updating to lastest working master

* syncing with 273267

* first draft version

* fix: update Searcher to case-insensitive

* added ILIKE test

* fixed style errors, updated test, split like and ilike,  added notILike

* replaced inconsistent comments

* fixed show tables ilike

* updated missing test cases

* regenerated ya.make

* Update 01355_ilike.sql

Co-authored-by: myrrc <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>

* add parseDateTimeBestEffortUS function (#12028)

* add function parseDateTimeBestEffortUS
* add test
* add doc

Co-authored-by: alexey-milovidov <[email protected]>

* Cleanup changelog (half done). Now it is acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Additions requested by @filimonov #12104

Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: Mikhail Gaidamaka <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>
Co-authored-by: kssenii <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Ivan <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: chengy8934 <[email protected]>
Co-authored-by: MicrochipQ <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Vladimir Chebotarev <[email protected]>
Co-authored-by: Nicolae Vartolomei <[email protected]>
Co-authored-by: filimonov <[email protected]>
Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: manmitya <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Mikhail Filimonov <[email protected]>
Co-authored-by: Anton Ivashkin <[email protected]>
Co-authored-by: Ilya Yatsishin <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Matwey V. Kornilov <[email protected]>
Co-authored-by: vivarum <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: flynn <[email protected]>
)

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix error

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* Update ThreadProfileEvents.cpp

* Update README.md

* Add a test with negative priority in test_distributed_load_balancing

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Added test.

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Fix tests.

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <[email protected]>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <[email protected]>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* Changelog for 20.1, 20.4

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>

* Fix tests.

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: alesapin <[email protected]>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* Fix result_rows and result_bytes metrics for selects.

* Fix result_rows and result_bytes metrics for selects.

* improve breadcrumbs markup

* Fix tests.

* fix segfault with -StateResample combinators

* Less race conditions

* fix test with buffer table

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Rewrite curl dependency in a more ch compatible way

- add support of unbundled curl
- add CURL::libcurl libraries
- avoid explicit linkage of daemon with curl (added with sentry)
- set CACHE variables for non-direct users:
  - mariadb-connector-c
  - aws-s3-cmake
  - sentry-native

Cc: @alexey-milovidov
Cc: @alesapin (requires docker image update)
Cc: @abyss7

Refs: #11300
Refs: #8011
Refs: #8905

v2: replace cmake/find/curl.cmake with proper contrib/curl-cmake (as
pointed by @abyss7, cmake/find/*.cmake is deprecated)

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Update polymorphic_parts_l.xml

* Update polymorphic_parts_m.xml

* Update polymorphic_parts_s.xml

* kafka: fix SIGSEGV on DROP TABLE

After #11599 it is possible that messages of the
ReadBufferFromKafkaConsumer will be cleaned-up right in
read_kafka_message callback (from KafkaBlockInputStream) if the stop
flag isset (i.e. DROP TABLE is waiting the consumer), and if
read_kafka_message already processed some rows it will not return 0 and
the loop after will try to get current topic from the buffer, which uses
messages in the underlying and this will got SIGSEGV:

    12:14:56.173262 [ 55421 ] {f7930856-d478-4e41-af56-24ce7b693e95} <Debug> executeQuery: (from 0.0.0.0:0, user: ) DROP TABLE IF EXISTS data.queue
    12:14:56.173285 [ 55421 ] {f7930856-d478-4e41-af56-24ce7b693e95} <Trace> StorageKafka (newly_queue): Waiting for cleanup
    12:14:56.180016 [ 55390 ] {} <Trace> BaseDaemon: Received signal 11
    12:14:56.180267 [ 4914 ] {} <Fatal> BaseDaemon: ########################################
    12:14:56.181879 [ 4914 ] {} <Fatal> BaseDaemon: (version 20.6.1.1, build id: 4CE0298F08583658) (from thread 55468) (no query) Received signal Segmentation fault (11)
    12:14:56.181900 [ 4914 ] {} <Fatal> BaseDaemon: Address: 0x8 Access: read. Address not mapped to object.
    12:14:56.181909 [ 4914 ] {} <Fatal> BaseDaemon: Stack trace:
    12:14:56.184676 [ 4914 ] {} <Fatal> BaseDaemon: 3. /ch/contrib/cppkafka/include/cppkafka/message.h:111: DB::KafkaBlockInputStream::readImpl() @ 0xe343f1c in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.185553 [ 4914 ] {} <Fatal> BaseDaemon: 4. /ch/contrib/libcxx/include/vector:1003: DB::IBlockInputStream::read() @ 0xd9d95bd in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.188238 [ 4914 ] {} <Fatal> BaseDaemon: 5. /ch/src/DataStreams/copyData.cpp:26: DB::copyData() @ 0xd9f712a in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.188780 [ 4914 ] {} <Fatal> BaseDaemon: 6. /ch/contrib/libcxx/include/vector:1532: DB::StorageKafka::streamToViews() @ 0xe335e73 in /usr/lib/debug/usr/bin/clickhouse
    12:14:56.189331 [ 4914 ] {} <Fatal> BaseDaemon: 7. /ch/src/Storages/Kafka/StorageKafka.cpp:491: DB::StorageKafka::threadFunc() @ 0xe336738 in /usr/lib/debug/usr/bin/clickhouse

55421 thread (shows that it still waiting for deactivation):

    5  std::__1::lock_guard<>::lock_guard () at ../contrib/libcxx/include/__mutex_base:90
    6  DB::BackgroundSchedulePoolTaskInfo::deactivate (this=0x7fc7e4465f20) at ../src/Core/BackgroundSchedulePool.cpp:59
    7  DB::StorageKafka::shutdown (this=0x7fc7e45e4600) at ../contrib/libcxx/include/memory:3821

And just in case thread where read_kafka_message is called:

    0  DB::ReadBufferFromKafkaConsumer::nextImpl (this=0x7fd4901d4118) at ../contrib/libcxx/include/atomic:1491
    1  DB::ReadBuffer::next (this=0x7fd4901d4118) at ../src/IO/ReadBuffer.h:59
    2  DB::ReadBuffer::eof (this=0x7fd4901d4118) at ../src/IO/ReadBuffer.h:81
    3  DB::skipWhitespaceIfAny (buf=...) at ../src/IO/ReadHelpers.h:945
    4  DB::JSONEachRowRowInputFormat::readRow (ext=..., columns=..., this=0x7fd499a7a020) at ../src/Processors/Formats/Impl/JSONEachRowRowInputFormat.cpp:222
    5  DB::JSONEachRowRowInputFormat::readRow (this=0x7fd499a7a020, columns=..., ext=...) at ../src/Processors/Formats/Impl/JSONEachRowRowInputFormat.cpp:218
    6  DB::IRowInputFormat::generate (this=0x7fd499a7a020) at ../src/Processors/Formats/IRowInputFormat.cpp:64
    7  DB::ISource::work (this=0x7fd499a7a020) at ../src/Processors/ISource.cpp:48
    8  DB::KafkaBlockInputStream::<lambda()>::operator()(void) const () at ../contrib/libcxx/include/memory:3826
    9  DB::KafkaBlockInputStream::readImpl (this=0x7fd46e718820) at ../contrib/libcxx/include/new:340

Cc: @filimonov

* kafka: check that the data is still usable after parsing

* kafka: improve logging during engine shutdown

This will help with tracking possible issues, when you need to know was
buffer released or not.

* kafka: avoid superior polling after DROP/DETACH TABLE

Before this patch isStalled() was checked before polledDataUnusable(),
and after DROP TABLE isStalled() == true (although this looks tricky).

* kafka: remove outdated comment

As stated by @filimonov it is not relevant (after #11599)

* Added test.

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Update polymorphic_parts_s.xml

* Update polymorphic_parts_m.xml

* Update polymorphic_parts_l.xml

* Changelog for 20.5

* Update CHANGELOG.md

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Remove LC converting to Arrow.

* Remove LC converting to Arrow.

* Move skip lists to clickhouse-test

* Remove LC converting to Arrow.

* More verbose message about skip

* Make skip-list optional

* Added test.

* Add type column in system.disks

* Update clickhouse-test

* Fix handling dependency of table with ENGINE=Dictionary on dictionary.

* Add test.

* Update 01355_CSV_input_format_allow_errors.sh

* Update CHANGELOG.md

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* contrib/unixodbc-cmake: Fix build when UNBUNDLED

target_compile_definitions may only set INTERFACE properties on IMPORTED targets

* Fix version column in replicated version collapsing merge tree (#12121)

* poco-cmake: Fix Poco::Data::ODBC target when UNBUNDLED

By default IMPORTED target has a scope in the directory in which it is created
and below. This leads to the following issues when building UNBUNDLED:

Target "clickhouse" links to target "Poco::Data::ODBC" but the target was not
found.  Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?

* Add missed <atomic> include in ProxyListConfiguration

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Do not enable sentry if ENABLE_LIBRARIES is not set

* Fail if curl library was enabled and was not found in case of unbundled build

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

* New ISO8601 year modificators for formatDateTime

* Added a test

* Include libcurl4-openssl-dev into yandex/clickhouse-deb-builder

* Remove harmful code from "geoDistance" #12117

* Update formatDateTime.cpp

* Fix warnings from CodeQL

* Added a test

* Minor modification

* Update libdivide to the latest master

* Replace exit to abort in libdivide

* Fix potential overflow in integer division #12119

* Added a test

* Whitespace

* Added a test for #4211

* Attempt to fix "Arcadia" build

* Rename test

* Update arraySum.cpp

* Make code clearer: use enum instead of `bool internal`.

* Update StorageDictionary.h

* Update StorageDictionary.h

* ILIKE operator (#12125)

* Integrated CachingAllocator into MarkCache

* fixed build errors

* reset func hotfix

* upd: Fixing build

* updated submodules links

* fix 2

* updating grabber allocator proto

* updating lost work

* updating CMake to use concepts

* some other changes to get it building (integration into MarkCache)

* further integration into caches

* updated Async metrics, fixed some build errors

* and some other errors revealing

* added perfect forwarding to some functions

* fix: forward template

* fix: constexpr modifier

* fix: FakePODAllocator missing member func

* updated PODArray constructor taking alloc params

* fix: PODArray overload with n restored

* fix: FakePODAlloc duplicating alloc() func

* added constexpr variable for alloc_tag_t

* split cache values by allocators, provided updates

* fix: memcpy

* fix: constexpr modifier

* fix: noexcept modifier

* fix: alloc_tag_t for PODArray constructor

* fix: PODArray copy ctor with different alloc

* fix: resize() signature

* updating to lastest working master

* syncing with 273267

* first draft version

* fix: update Searcher to case-insensitive

* added ILIKE test

* fixed style errors, updated test, split like and ilike,  added notILike

* replaced inconsistent comments

* fixed show tables ilike

* updated missing test cases

* regenerated ya.make

* Update 01355_ilike.sql

Co-authored-by: myrrc <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>

* add parseDateTimeBestEffortUS function (#12028)

* add function parseDateTimeBestEffortUS
* add test
* add doc

Co-authored-by: alexey-milovidov <[email protected]>

* Cleanup changelog (half done). Now it is acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Additions requested by @filimonov #12104

* Remove underscore as word-break character. This partially reverts #11975

* Fix bad test number

* Added a test

* Change exception code from LOGICAL_ERROR to BAD_ARGUMENTS when the name of remote table is empty

* fix segfault with -StateResample combinators

* Fix bad code in redundant ORDER BY optimization #10067

* Add a test

* Fix error

* Autocomplete does not have to work in "Unbundled" build

* Whitespace

* Style

* Test for issue #9088

ALTER DELETE unexpectedly deletes NULL rows

* Remove unused potentially dangerous function

* Fix mutations interpreter #9088

* Added a test

* Added another test just in case

* Added yet another test just in case

* Update 01358_mutation_delete_null_rows.sql

* Fix test

* Fix flaky test

* [anchore] more detailed Dockerfile scan reports (#12159)

* Update array-functions.md (#12130)

修改编码错误

* Update array-functions.md (#12129)

修改编码错误

* [docs] improve redirects destination

* Update SECURITY.md (#12161)

Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: kssenii <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: chengy8934 <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: MicrochipQ <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Vladimir Chebotarev <[email protected]>
Co-authored-by: Nicolae Vartolomei <[email protected]>
Co-authored-by: filimonov <[email protected]>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: manmitya <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Mikhail Filimonov <[email protected]>
Co-authored-by: Anton Ivashkin <[email protected]>
Co-authored-by: Ilya Yatsishin <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Matwey V. Kornilov <[email protected]>
Co-authored-by: vivarum <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: flynn <[email protected]>
Co-authored-by: yhgcn <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
…ion (#12252)

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Move writing to RabbitMQ into background task

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix and simplify code

* Fix test.

* Update ITransformingStep.h

* Fix "Sharding key is not deterministic" message

* Do not check *optimize_skip_unused_shards_nesting w/o *optimize_skip_unused_shards

This will avoid supperior log message in case of
*optimize_skip_unused_shards_nesting already disables it.
And also it is logical.

* Fix function if with FixedString arguments of different sizes

* Add multiIf

* Fix error

* Remove test that is not supported by "Arcadia" build system

* Fix constraints check for constant columns

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix error

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Added test.

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Fix tests.

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <[email protected]>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <[email protected]>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* Changelog for 20.1, 20.4

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>

* Fix tests.

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: alesapin <[email protected]>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* Fix result_rows and result_bytes metrics for selects.

* Fix result_rows and result_bytes metrics for selects.

* improve breadcrumbs markup

* Fix tests.

* fix segfault with -StateResample combinators

* Less race conditions

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Rewrite curl dependency in a more ch compatible way

- add support of unbundled curl
- add CURL::libcurl libraries
- avoid explicit linkage of daemon with curl (added with sentry)
- set CACHE variables for non-direct users:
  - mariadb-connector-c
  - aws-s3-cmake
  - sentry-native

Cc: @alexey-milovidov
Cc: @alesapin (requires docker image update)
Cc: @abyss7

Refs: #11300
Refs: #8011
Refs: #8905

v2: replace cmake/find/curl.cmake with proper contrib/curl-cmake (as
pointed by @abyss7, cmake/find/*.cmake is deprecated)

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Added test.

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Changelog for 20.5

* Update CHANGELOG.md

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Remove LC converting to Arrow.

* Remove LC converting to Arrow.

* Move skip lists to clickhouse-test

* Remove LC converting to Arrow.

* More verbose message about skip

* Make skip-list optional

* Added test.

* Add type column in system.disks

* Update clickhouse-test

* Fix handling dependency of table with ENGINE=Dictionary on dictionary.

* Add test.

* Update 01355_CSV_input_format_allow_errors.sh

* Update CHANGELOG.md

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* contrib/unixodbc-cmake: Fix build when UNBUNDLED

target_compile_definitions may only set INTERFACE properties on IMPORTED targets

* Fix version column in replicated version collapsing merge tree (#12121)

* poco-cmake: Fix Poco::Data::ODBC target when UNBUNDLED

By default IMPORTED target has a scope in the directory in which it is created
and below. This leads to the following issues when building UNBUNDLED:

Target "clickhouse" links to target "Poco::Data::ODBC" but the target was not
found.  Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?

* Add missed <atomic> include in ProxyListConfiguration

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Do not enable sentry if ENABLE_LIBRARIES is not set

* Fail if curl library was enabled and was not found in case of unbundled build

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

* New ISO8601 year modificators for formatDateTime

* Added a test

* Include libcurl4-openssl-dev into yandex/clickhouse-deb-builder

* Remove harmful code from "geoDistance" #12117

* Update formatDateTime.cpp

* Fix warnings from CodeQL

* Added a test

* Minor modification

* Update libdivide to the latest master

* Replace exit to abort in libdivide

* Fix potential overflow in integer division #12119

* Added a test

* Whitespace

* Added a test for #4211

* Attempt to fix "Arcadia" build

* Rename test

* Update arraySum.cpp

* Make code clearer: use enum instead of `bool internal`.

* Update StorageDictionary.h

* Update StorageDictionary.h

* ILIKE operator (#12125)

* Integrated CachingAllocator into MarkCache

* fixed build errors

* reset func hotfix

* upd: Fixing build

* updated submodules links

* fix 2

* updating grabber allocator proto

* updating lost work

* updating CMake to use concepts

* some other changes to get it building (integration into MarkCache)

* further integration into caches

* updated Async metrics, fixed some build errors

* and some other errors revealing

* added perfect forwarding to some functions

* fix: forward template

* fix: constexpr modifier

* fix: FakePODAllocator missing member func

* updated PODArray constructor taking alloc params

* fix: PODArray overload with n restored

* fix: FakePODAlloc duplicating alloc() func

* added constexpr variable for alloc_tag_t

* split cache values by allocators, provided updates

* fix: memcpy

* fix: constexpr modifier

* fix: noexcept modifier

* fix: alloc_tag_t for PODArray constructor

* fix: PODArray copy ctor with different alloc

* fix: resize() signature

* updating to lastest working master

* syncing with 273267

* first draft version

* fix: update Searcher to case-insensitive

* added ILIKE test

* fixed style errors, updated test, split like and ilike,  added notILike

* replaced inconsistent comments

* fixed show tables ilike

* updated missing test cases

* regenerated ya.make

* Update 01355_ilike.sql

Co-authored-by: myrrc <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>

* add parseDateTimeBestEffortUS function (#12028)

* add function parseDateTimeBestEffortUS
* add test
* add doc

Co-authored-by: alexey-milovidov <[email protected]>

* Cleanup changelog (half done). Now it is acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Now it is more acceptable #12104

* Cleanup changelog (half done). Additions requested by @filimonov #12104

* Remove underscore as word-break character. This partially reverts #11975

* Fix bad test number

* Added a test

* Change exception code from LOGICAL_ERROR to BAD_ARGUMENTS when the name of remote table is empty

* fix segfault with -StateResample combinators

* Fix bad code in redundant ORDER BY optimization #10067

* Add a test

* Fix error

* Autocomplete does not have to work in "Unbundled" build

* Whitespace

* Style

* Fix flaky test

Co-authored-by: Ivan <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: kssenii <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: chengy8934 <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: MicrochipQ <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Vladimir Chebotarev <[email protected]>
Co-authored-by: Nicolae Vartolomei <[email protected]>
Co-authored-by: filimonov <[email protected]>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: manmitya <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Mikhail Filimonov <[email protected]>
Co-authored-by: Anton Ivashkin <[email protected]>
Co-authored-by: Ilya Yatsishin <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Matwey V. Kornilov <[email protected]>
Co-authored-by: vivarum <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: myrrc <[email protected]>
Co-authored-by: flynn <[email protected]>
* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <[email protected]>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* client exit with unrecognized arguments

format code

fix

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* add test

update test

fix

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Fixes

* Add comment

* Remove empty lines

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“, and fix an error.

* Add __init__.py

* Remove strange header

* Trying to fix ya.make

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <[email protected]>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>

* Temporary enable debug logs when releasing docs

* [docs] improve stable releases discovery

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: Nikita Vasilev <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: bharatnc <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Eugene Klimov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: Maxim Akhmedov <[email protected]>
Co-authored-by: feng lv <[email protected]>
Co-authored-by: Nikita Mikhailov <[email protected]>
Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: Tom Bombadil <[email protected]>
Co-authored-by: Yuntao Wu <[email protected]>
Co-authored-by: Wang Fenjin <[email protected]>
Co-authored-by: Mikhail Gaidamaka <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>
Co-authored-by: Ivan <[email protected]>
* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Move reading from RabbitMQ into background task

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* Fix ORDER BY tuple with COLLATE on const column

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* make a short test for DateTime

* Fixup

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <[email protected]>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>

* Use ConcurentBoundedQueue instead of vector

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Update contrib/jemalloc-cmake/CMakeLists.txt

* Remove libevent library

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Move writing to RabbitMQ into background task

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix and simplify code

* Fix test.

* Update ITransformingStep.h

* Fix "Sharding key is not deterministic" message

* Do not check *optimize_skip_unused_shards_nesting w/o *optimize_skip_unused_shards

This will avoid supperior log message in case of
*optimize_skip_unused_shards_nesting already disables it.
And also it is logical.

* Remove test that is not supported by "Arcadia" build system

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Check type of filter for prewhere.

* Added test.

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Fix tests.

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <[email protected]>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <[email protected]>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>

* Fix tests.

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: alesapin <[email protected]>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* Fix result_rows and result_bytes metrics for selects.

* Fix result_rows and result_bytes metrics for selects.

* improve breadcrumbs markup

* Fix tests.

* Less race conditions

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Added test.

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Remove LC converting to Arrow.

* Remove LC converting to Arrow.

* Move skip lists to clickhouse-test

* Remove LC converting to Arrow.

* More verbose message about skip

* Make skip-list optional

* Added test.

* Add type column in system.disks

* Update clickhouse-test

* Update 01355_CSV_input_format_allow_errors.sh

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* contrib/unixodbc-cmake: Fix build when UNBUNDLED

target_compile_definitions may only set INTERFACE properties on IMPORTED targets

* Fix version column in replicated version collapsing merge tree (#12121)

* poco-cmake: Fix Poco::Data::ODBC target when UNBUNDLED

By default IMPORTED target has a scope in the directory in which it is created
and below. This leads to the following issues when building UNBUNDLED:

Target "clickhouse" links to target "Poco::Data::ODBC" but the target was not
found.  Perhaps a find_package() call is missing for an IMPORTED target, or an
ALIAS target is missing?

* Add missed <atomic> include in ProxyListConfiguration

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

* Remove harmful code from "geoDistance" #12117

* Whitespace

Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: feng lv <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Wang Fenjin <[email protected]>
Co-authored-by: kssenii <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Mikhail Gaidamaka <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Ivan <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: chengy8934 <[email protected]>
Co-authored-by: MicrochipQ <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Vladimir Chebotarev <[email protected]>
Co-authored-by: Nicolae Vartolomei <[email protected]>
Co-authored-by: filimonov <[email protected]>
Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: manmitya <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Mikhail Filimonov <[email protected]>
Co-authored-by: Anton Ivashkin <[email protected]>
Co-authored-by: Ilya Yatsishin <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Matwey V. Kornilov <[email protected]>
* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* Try to enable long perf tests

* perf test: report queries with new functions separately

* performance comparison

* longer

* muzzy decay 10 ms

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

* Avoid floating point error in geohashesInBox #11369

* Avro: allow missing fields

* update

* Fix build

* Revert "[experiment] maybe fix warnings in integration tests"

* Add xeus-clickhouse (#12010)

xeus-clickhouse is a Jupyter kernal for ClickHouse

* Fix test

* Fix array size overflow in generateRandom

* Added a test

* Update ErrorCodes.cpp

* More hardening

* More hardening

* Move reading from RabbitMQ into background task

* Fix access rights: cannot grant DDL when allow_ddl=0

* Add tests

* Add test test_allow_ddl.

* Added failing test

* Remove the usage of obsolete setting

* Whitespace

* Added a test

* Allow to ALTER partition key in some cases

* Slightly more safe

* Only allow Enum extension within the same width

* Update test

* Added failing test for Replicated table

* Fix clang-tidy

* Whitespace

* Write partition key changes to ZooKeeper

* Typos

* Implement for ReplicatedMergeTree

* Added test reference

* Added missing ORDER BY in test

* Fix integration test

* Fix error in addMonths with floating point argument

* Fix error in addMonths with floating point argument

* Fix overflow in function "neighbor"

* Whitespace

* Whitespace

* Fix ORDER BY tuple with COLLATE on const column

* clickhouse-local: logging flags fixes

* clickhouse-local: use std::filesystem::equivalent

* Limit /proc/cpuinfo metrics to linux only

* Feature 8074 (#12014)

* refs #8074 1: start to stream

* refs final check format and spell

* refs final check format and spell

* refs #8074 fix up link to sql-reference/functions/other-functions

* Bump idna from 2.9 to 2.10 in /docs/tools

Bumps [idna](https://github.com/kjd/idna) from 2.9 to 2.10.
- [Release notes](https://github.com/kjd/idna/releases)
- [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst)
- [Commits](kjd/idna@v2.9...v2.10)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* make a short test for DateTime

* Fixup

* DOCS-636 Synchronized some edits from EN to RU (#11941)

* [review][clickhousedocs][translate] DOCSUP-1351: Edit and translate to Russian (#129)

* DOCSUP-1351: Add blockSerializedSize (RU)

* DOCSUP-1351: Fix blockSerializedSize description by function description template.

* DOCSUP-1351: PR8756- Update Distributed engine section.

* DOCSUP-1351: PR9076-Show Queries more like MySQL did.

* DOCSUP-1351: PR#9919-system.tables improvements (total_rows/total_bytes/storage_policy)

* DOCSUP-1351: (ru) blockSerializedSize remake by template.

* DOCSUP-1351: PR#9598-Add function arrayReduceInRange and update arrayReduce function.

* DOCSUP-1351: PR#9598 Fix arrayReduce and arrayReduceInRanges sections by template.

* DOCSUP-1351: Some fixes.

* DOCSUP-1351: Some fixes.

Co-authored-by: romanzhukov <[email protected]>

* CLICKHOUSEDOCS-636: Minor changes.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>

* Use ConcurentBoundedQueue instead of vector

* Temporary enable debug logs when releasing docs

* Better docker images list

* [docs] improve stable releases discovery

* Update contrib/jemalloc-cmake/CMakeLists.txt

* Remove libevent library

* Automated backporting machinery (#12029)

* Add automated cherry-pick machinery

* Added script for backporting machinery

* Implement dry-run mode

* Comment old methods

* Fix python2

* Add ya.make for sandbox tasks

* Remove old stuff

* Move writing to RabbitMQ into background task

* Add TLB misses perf counters.

* Update ThreadProfileEvents.cpp

* Fake change in dockerfile

* Fix tuple nullable comparison.

* Fix access rights: cannot grant INTROSPECTION when allow_introspection_functions=0.

* fixup

* Update ThreadProfileEvents.cpp

* Fix test.

* Added test.

* Fix and simplify code

* Fix test.

* Update ITransformingStep.h

* Fix "Sharding key is not deterministic" message

* Do not check *optimize_skip_unused_shards_nesting w/o *optimize_skip_unused_shards

This will avoid supperior log message in case of
*optimize_skip_unused_shards_nesting already disables it.
And also it is logical.

* Remove test that is not supported by "Arcadia" build system

* Make topK return Enum for Enum types

* parse metadata in parallel when loading tables

* Fix build

* fixup

* Update ThreadProfileEvents.cpp

* Update browse-code.md (#12047)

* Update browse-code.md

纠正了一些翻译错误。

* Update browse-code.md

Co-authored-by: Ivan Blinkov <[email protected]>

* Bump python-slugify from 1.2.6 to 4.0.1 in /docs/tools

Bumps [python-slugify](https://github.com/un33k/python-slugify) from 1.2.6 to 4.0.1.
- [Release notes](https://github.com/un33k/python-slugify/releases)
- [Changelog](https://github.com/un33k/python-slugify/blob/master/CHANGELOG.md)
- [Commits](un33k/python-slugify@1.2.6...4.0.1)

Signed-off-by: dependabot-preview[bot] <[email protected]>

* Update ThreadProfileEvents.cpp

* Update README.md

* Fix typo (#12046)

* Fix test_allow_introspection.

* Try fix labda tuple(LC) argument.

* Test image changes

* Try fix labda tuple(LC) argument.

* Add pvs studio to images

* Added test.

* fix low card types in merge join (#12035)

* Update contrib/poco to ClickHouse-Extras/poco #22 (#12037)

* Updated ClickHouse-Extras/poco #22

* Fix defaultValueOfArgumentType

* Added test.

* Update cpu_synthetic.xml

* Moves task shall be started if new storage policy needs them.

* Added test for automatic start of background move task.

* Add ability to run any image version from packager and runner

* Try fix pk in tuple performance

Possible approach for fixing #10574

The problem is that prepared sets are built correctly, it is a hash map of key -> set
where key is a hash of AST and list of data types (when we a list of
tuples of literals).

However, when the key is built from the index to try and find if there
exists a prepared set that would match it looks for data types of the
primary key (see how data_types is populated) because the primary key
has only one field (v in my example) it can not find the prepared set.

The patch looks for any prepared indexes where data types match for the
subset of fields found in primary key, we are not interested in other
fields anyway for the purpose of primary key pruning.

* Fix partial revokes (complex cases).

* Changelog for 20.3.11, 12

* simple changelog script

* bump CI

* Add explicit test for a case where AST hashes collide for different prepared sets

* [blog] add RSS feed (#12064)

* [blog] add rss feed

* better title

* Update CHANGELOG.md

Co-authored-by: Ivan Blinkov <[email protected]>

* bump ci

* Moved useless S3 logging to TRACE level.

* Add a test to cover non-const tuple elemenets (just in case)

* Bump

* Update query-complexity.md

Remove a note about read limits applied on threads level.

* Update query-complexity.md

* Update query-complexity.md

* Update query-complexity.md

* DOCS-609: max_server_memory_usage (#11771)

* Revolg DOCSUP-1000 add max server memory usage setting (#125)

* Add max_server_memory_usage setting, delete max_memory_usage_for_all_queries setting.

* Syntax fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Doc for the max_server_memory_usage setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-609: Minor fixes.

* CLICKHOUSEDOCS-609: Actualized position of the setting.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Add force_primary_key to a pk in tuple test

* DOCS-510: runningAccumulate (#12061)

* asiana21-DOCSUP-797 (#117)

* docs(runningAccumulate): the function description is added

* docs(runningAccumulate): the function description is modified

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): some changes

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: BayoNet <[email protected]>

* docs(runningAccumulate): added ru translation

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-510: Minor fix.

* Update docs/en/sql-reference/functions/other-functions.md

Co-authored-by: Ivan Blinkov <[email protected]>

* CLICKHOUSEDOCS-510: Fixed links.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>

* Remove const specifier to allow auto-move (clangtidy)

* Rewrite Set lookup to make it more readable

* [docs] Sync zh/development/build-osx.md from EN (#12071)

* ISSUES-4006 support first for ALTER ADD|MODIFY COLUMN

* Update deb image

* Don't download image twice

* Remove garbage from images

* bump ci

* fixup

* DOCS-635: Translated the EN version of the AvroConfluent format description (#11930)

* DOCSUP-1350 (#128)

* edited EN version

* add EN and RU translation

* minor changes

* CLICKHOUSEDOCS-635: Updated the description.

Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>

* Better PVS image

* DOCS-605: Description for the always_fetch_merged_part setting (#11921)

* Revolg DOCSUP-998 Document the always_fetch_merged_part setting (#123)

* Add always_fetch_merged_part setting

* revolg-DOCSUP-998-add_always_fetch_merged_part_setting link fixed

* Apply suggestions from code review

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Add always_fetch_merged_part setting. Updates.

Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-605: Minor fixes.

* CLICKHOUSEDOCS-605: Added Plausible to Adopters.

* Update docs/ru/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: alesapin <[email protected]>

* CLICKHOUSEDOCS-605: Fixed access rights description.

Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: alesapin <[email protected]>

* Update Dockerfile

* revert e9d3c48

* [blog] 'Package Repository Behind CDN' post (#12082)

* support iframes from DataLens

* initial blog post text

* Improve REVOKE command: now it requires only grant/admin option for only
access which will be revoked.
REVOKE ALL FROM user1 now revokes all granted roles.

* Fix limiting the number of threads for VIEW.

* Update tips.md

* Added test.

* Remove pvs studio from images list

* Better shutdown and conversion

* Reverse arguments

* improve breadcrumbs markup

* Less race conditions

* [docker] install ca-certificates before the first apt-get update (#12095)

* [docker] install ca-certificates before first apt-get update

* Update Dockerfile

* DOCS-522: max_parser_depth (#12097)

* asiana21-DOCSUP-925-max_parser_depth (#132)

* docs(max_parser_depth): added the setting description

* docs(max_parser_depth): some changes

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* Update docs/en/operations/settings/settings.md

Co-authored-by: BayoNet <[email protected]>

* docs(max_parser_depth): added ru translation

* docs(max_parser_depth): removed quotation marks

Co-authored-by: asiana21 <[email protected]>
Co-authored-by: BayoNet <[email protected]>

* CLICKHOUSEDOCS-522: Fixed the link.

Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>

* Added results for AWS Lightsail

* (typo) in doc (#12099)

* Create codeql-analysis.yml

* Tiny fixes

* Create anchore-analysis.yml

* Update README.md

* Fix test (#12088)

* Test for a fixed issue #10668, related to input_format_allow_errors_num in CSV

* Add unbundled mode flag

* Move skip lists to clickhouse-test

* More verbose message about skip

* Make skip-list optional

* Update clickhouse-test

* Update 01355_CSV_input_format_allow_errors.sh

* Added a showcase of minimal Docker image

* Added a comment

* Added a comment

* Added a comment

* Formatting

* Fix version column in replicated version collapsing merge tree (#12121)

* Improve performace of reading in order of sorting key. (#11696)

* simplify reading in order of sorting key

* add perf test for reading many parts

* Revert "simplify reading in order of sorting key"

This reverts commit 7267d7c.

* add threshold for preliminary merge for reading in order

* better threshold

* limit threads in test

* Normalize "pid" file handling #3501

* Update StatusFile.cpp

Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: Alexey Milovidov <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: feng lv <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Wang Fenjin <[email protected]>
Co-authored-by: kssenii <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Mikhail Gaidamaka <[email protected]>
Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: BayoNet <[email protected]>
Co-authored-by: Sergei Shtykov <[email protected]>
Co-authored-by: Roman Bug <[email protected]>
Co-authored-by: romanzhukov <[email protected]>
Co-authored-by: tavplubix <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Ivan <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: chengy8934 <[email protected]>
Co-authored-by: MicrochipQ <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Kazakov <[email protected]>
Co-authored-by: Vladimir Chebotarev <[email protected]>
Co-authored-by: Nicolae Vartolomei <[email protected]>
Co-authored-by: filimonov <[email protected]>
Co-authored-by: olgarev <[email protected]>
Co-authored-by: Olga Revyakina <[email protected]>
Co-authored-by: AsiaKorushkina <[email protected]>
Co-authored-by: asiana21 <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Evgenia Sudarikova <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: manmitya <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Mikhail Filimonov <[email protected]>
Co-authored-by: Ilya Yatsishin <[email protected]>
Co-authored-by: tavplubix <[email protected]>
* try to completely remove sentry from odbc-bridge

* Switch back to sentry upstream

* Try to disable linker options from sentry

* Do not cache frames inside StackTrace

* Lost part of refactoring

* fix

* less confusing

* fix

* maybe fix the unbundled gcc build

* experiment

* experiment

* back to upstream

* change used flag

* fix address formatting

* use the sentry logger hook

* Add concurrent benchmark to performance test

After the main test, run queries from `website.xml` in parallel using
`clickhouse-benchmark`. This can be useful to test the effects of
concurrency on performance. Comparison test can miss some effects
because it always runs queries sequentially, and many of them are even
single-threaded.

* fixes

* experiment

* typo

* trigger ci

* fixup

* add flag to continue on errors

* trigger ci

* trigger ci

* trigger ci

* experiment

* trigger ci

* fixup

* fixup

* Update CMakeLists.txt

* Update CMakeLists.txt

* Update http_server.py

* make max global thread pool setting configurable

This PR adds a server level config for overriding the default max number
of threads in global thread pool that is currently allowed (10,000).

This might be useful in scenarios where there are a large number of
distributed queries that are executing concurrently and where the
default number of max threads might not be necessarily be sufficient.

* add max_thread_pool_size setting to tests

This adds the `max_thread_pool_size` config to
tests/server-test.xml file.

* docs for max_thread_pool_size

This adds the docs for the new server level setting
`max_thread_pool_size`.

* remove extra vertical space

* fixup

* improvements after review comments

* move the default endpoint to config

* remove extra line

* distinct combinator for single numeric arguments

* rework distinct combinator

* fix build

* Disable optimize_skip_unused_shards if sharding_key has non-deterministic func

Example of such functions is rand()

And this patch disables only optimize_skip_unused_shards, i.e. INSERT
code path does not changed, so it will work as before.

* optimize_skip_unused_shards=2 will disable it for nested distributed queries

P.S. Looks like settings can be converted between SettingUInt64 and
SettingBool without breaking binary protocol.

FWIW maybe it is a good idea to change the semantics of the settings as
follow (but I guess that changing semantic is not a good idea, better to
add new settings and deprecate old ones):
- optimize_skip_unused_shards -- accept nesting level on which the
  optimization will work
- force_skip_optimize_shards_nesting -- accept nesting level on which
  the optimization will work

* Add logging of adjusting conditional settings for distributed queries

* Add missing DROP TABLE in 01319_mv_constants_bug

* Improve 01319_optimize_skip_unused_shards_no_nested

Before there is no check that optimize_skip_unused_shards was working
for the first level, use cluster with unavalable shard to guarantee
this.

* Update 00816_long_concurrent_alter_column.sh

* distinct combinator for function of multiuple arguments

* report the number of errors

* fixup

* fixup

* Add settings to control nesting level for shards skipping optimization

- optimize_skip_unused_shards_nesting (allows control nesting level for
  shards skipping optimization)
- force_skip_optimize_shards_nesting (allows control nesting level for
  checking was shards skipped or not)
- deprecates force_skip_optimize_shards_no_nested

* fix review

* fix

* logger

* Update executeQuery.cpp

* restore immediate stacktrace output

* Support parse uuid without separator

* try fix build failure

* report

* fix read extra bytes when with separator

* add perf test for distinct combinator

* more optimal aggregate functions with both 'if' and 'distinct' combinators

* try fix test failure

* Initialize GlobalThreadPool explicitly

* apply review suggestions

* fixup

* memory usage settings

* allow implicit initialization

* Fix documentation

* Fix anotehr one nullable prewhere column.

* fix style check - removed unused LOGICAL_ERROR

* restore old toStringImpl signature

* send build_id to sentry as well + fix some minor issues

* remove unused imports

* clang format file

* construct path using boost::filesystem::path

* cleanup few unused headers

The following files had some unused headers and caught my eyes, so
cleaning them up:

```bash
programs/obfuscator/Obfuscator.cpp
src/Databases/DatabaseAtomic.cpp
```

* Change MySQL global variables query to globalVariable function

* add column header to test

* lost unused

* [experiment] maybe fix warnings in integration tests

* fix arcadia build

* Fix JDBC @@session.variables

* Remove already fixed bug

* Slightly modernize code around ZooKeeper

* Fix @@session.variables AS

* try to rename py files to less common names

* add __init__.py

* update SELECT description (#11907)

Signed-off-by: Slach <[email protected]>

* [docs] extra F.A.Q. content (#11898)

* more content for F.A.Q.

* more content

* normalize

* more content

* maybe fix the docs check

* absolute img urls

* fix tests

* Keep alias after substitution of query parameters #11914

* show build commands

* Added a test

* Add SELECT @@Version fake value '5.7.30' #11089

* fix + bump tests

* Fix unitialized memory

* Fix using current database while checking access rights.

* fix style check - clang format for lambdas

* Better github-hook

* Do not update PVS Studio docker automatically.

They keep removing the old image every couple of weeks, and the entire
docker update fails.

* bump CI

* for the love of God just update this container already, i'm begging you

* Update Dockerfile

* Update pvs-studio version

* Fix jemalloc under OSX (by registering it as default zone explicitly)

In case of OSX jemalloc register itself as a default zone allocator.

But when you link statically then zone_register() will not be called,
and even will be optimized out:

  $ nm clickhouse.patched  | grep -c zone_register
  0

Fix this, by manually calling it.

v2: extern C

* Rewrite arithmetic in aggregate functions optimisation (#11899)

* remove clang-format tag comments

* Put clickhouse-local data to /tmp by default

This fixes #9848
Also fixes #11926

* trigger ci

* fix some logical errors

* Update features.html

* Consider allocatedBytes() instead of bytes() in Storage{Buffer,Memory}.

* fix query parameter visitor

* fixpu

* Update 00945_bloom_filter_index.sql

* Less layers in docker file

* Less layers in docker file

* Update docker/test/performance-comparison/perf.py

* rename test back to test.py

* remove useless const

* save

* Update canonized values in 00753_system_columns_and_system_tables.

* add ssd to integration test

* one more test

* Fix one reason of test flakiness

* Fix race condition in extractAllGroups

* extract JOIN in own plan step

* fix mistake reported by @Slach

* remove useless logic

* fix crash

* Revive mmap IO

* More verbose CMake in build docker

* Avro UUID support

* remove a trick with expression continuation

* Respect direct_io/mmap settings while reading secondary indices

* Minor modification

* Added a comment #11949

* Added a test

* Add CANNOT_PARSE_UUID extern const

* Fix estimation of the number of marks for various thresholds

* better ExpressionAction::execute()

* performance comparison

* add page fault perf events

* better error messages

* Add CPU frequencies to system.asynchronous_metrics

* Update AsynchronousMetrics.cpp

* keep ArrayJoin optimisation

* Update path to performance tests build

* tests with distributed

* place left join keys in before_join actions

* Added tests for #8692

* review fixes

* fix totals

* Update path to perf test package

* use std::filesystem::path

* Minor cleanup in Client.cpp before fuzzing

* Extend word break config to all non-alphanumeric chars

* Update AsynchronousMetrics.h

* more cleanup

* remove unused imports + bump tests

* Update 01345_array_join_LittleMaverick.sql

* Update ExpressionActions.h

* Update system.md (#11945)

* Update system.md

Translate the doc to Chinese

* Update system.md

* fix the title's translattion error (#11939)

@zhang2014 I use MergeTree replaced  Chinese version "合并“, and fix an error.

* Update replxx submodule

* Add a test for #10102

* disable tests in arcadia

* Added a test from #5131

* performance comparison

* Support Enums type for MySQL engine #3985

* performance comparison

* Add integration test for MySQL enums type

* Fix bad log message at server startup

* Insert enum column values for test

* Better diagnostics of "Replica {} appears to be already active" message

* Fix typos

* Added a test for #8550

* Update extended-roadmap.md

* Update test

* Fix FPE, step 1

* Make it more correct

* Add a test

Co-authored-by: Ivan Blinkov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: alexey-milovidov <[email protected]>
Co-authored-by: Bharat Nallan <[email protected]>
Co-authored-by: Anton Popov <[email protected]>
Co-authored-by: Azat Khuzhin <[email protected]>
Co-authored-by: Nikita Vasilev <[email protected]>
Co-authored-by: zhang2014 <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: BohuTANG <[email protected]>
Co-authored-by: Nikolai Kochetov <[email protected]>
Co-authored-by: Eugene Klimov <[email protected]>
Co-authored-by: Alexander Kuzmenkov <[email protected]>
Co-authored-by: Nikita Mikhaylov <[email protected]>
Co-authored-by: alesapin <[email protected]>
Co-authored-by: Vitaly Baranov <[email protected]>
Co-authored-by: Artem Zuikov <[email protected]>
Co-authored-by: Alexander Tokmakov <[email protected]>
Co-authored-by: Maxim Akhmedov <[email protected]>
Co-authored-by: Nikita Mikhailov <[email protected]>
Co-authored-by: Andrew Onyshchuk <[email protected]>
Co-authored-by: Tom Bombadil <[email protected]>
Co-authored-by: Yuntao Wu <[email protected]>
robot-clickhouse and others added 23 commits August 26, 2020 23:08
Backport #14143 to 20.5: Fix creation of tables with named tuples #13027
Backport #14129 to 20.5: Slightly optimize very short queries with LowCardinality
Backport #13928 to 20.5: Fix wrong error for long queries.
Backport #14223 to 20.5: Fixed incorrect sorting order if LowCardinality column.
Backport #14079 to 20.5: Fix pointInPolygon with const 2d array
Backport #12756 to 20.5: Fix ClickHouseDictionarySource wrong access check
Backport #13722 to 20.5: return 200 when replicas status is ok and verbose = 1
Backport #14267 to 20.5: Fix crash in SchemaAllowedHandler.
Backport #14326 to 20.5: Throw exception on alter for storages created from table functions
Updated dockerfile for yandex/clickhouse-deb-builder to force its image rebuilding
Backport #14334 to 20.5: Stop query execution if exception happened in PipelineExecutor itself.
Updated dockerfile for yandex/clickhouse-deb-builder to force its image rebuilding
@Akazz
Copy link
Copy Markdown
Contributor

Akazz commented Sep 4, 2020

AFAIK the failed test 00157_cache_dictionary in Functional stateful tests (thread) for 39d71ed takes a lot of time to run and times out under TSan

robot-clickhouse and others added 5 commits September 7, 2020 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

release Label for release pull request submodule changed At least one submodule changed in this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.