Implement queue mode for MergeTree by Michicosun · Pull Request #59827 · ClickHouse/ClickHouse

Michicosun · 2024-02-09T19:44:51Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Relates to: #42990

Implemented queue feature for [Replicated]MergeTree storage engines. With queue flag enabled 2 materialized columns will be added to table: _queue_block_number, _queue_block_offset, their combination introduces a global order in the rows of the table, which in the future will help to support cursors for streaming queries.

Example:

CREATE TABLE queue(a UInt64, b UInt64) ENGINE=MergeTree() ORDER BY a SETTINGS queue=1;

-- suffix lookup
SELECT * FROM queue WHERE (_queue_block_number, _queue_block_offset) > (3, 1);

Documentation entry for user-facing changes

Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

…/42990/block_row_column

qoega · 2024-02-09T20:15:50Z

_queue_block_number, _queue_block_offset, their combination introduces a global order in the rows of the table, which in the future will help to support cursors for streaming queries.

How this is different from allow_experimental_block_number_column? #47532
It has strict order.

robot-ch-test-poll2 · 2024-02-09T20:43:17Z

This is an automated comment for commit 11f2a34 with description of existing statuses. It's updated for the latest CI running

❌ Click here to open a full report in a separate page

Check name	Description	Status
A Sync	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	⏳ pending
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	⏳ pending
Integration tests	The integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests	❌ failure
Mergeable Check	Checks if all other necessary checks are successful	⏳ pending
Stateless tests	Runs stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	❌ failure
Upgrade check	Runs stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts	❌ failure

Successful checks

Check name	Description	Status
AST fuzzer	Runs randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help	✅ success
ClickBench	Runs [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table	✅ success
ClickHouse build check	Builds ClickHouse in various configurations for use in further steps. You have to fix the builds that fail. Build logs often has enough information to fix the error, but you might have to reproduce the failure locally. The cmake options can be found in the build log, grepping for cmake. Use these options and follow the general build process	✅ success
Compatibility check	Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help	✅ success
Docker keeper image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docker server image	The check to build and optionally push the mentioned image to docker hub	✅ success
Docs check	Builds and tests the documentation	✅ success
Fast test	Normally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here	✅ success
Flaky tests	Checks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc	✅ success
Install packages	Checks that the built packages are installable in a clear environment	✅ success
PR Check	There's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS	✅ success
Performance Comparison	Measure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests	✅ success
Stateful tests	Runs stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc	✅ success
Stress test	Runs stateless functional tests concurrently from several clients to detect concurrency-related errors	✅ success
Style check	Runs a set of checks to keep the code style clean. If some of tests failed, see the related log from the report	✅ success
Unit tests	Runs the unit tests for different release types	✅ success

Michicosun · 2024-02-09T21:43:52Z

_queue_block_number, _queue_block_offset, their combination introduces a global order in the rows of the table, which in the future will help to support cursors for streaming queries.

How this is different from allow_experimental_block_number_column? #47532 It has strict order.

Combination of columns (_queue_block_number, _queue_block_offset) gives a strict order over table rows inside each partition. _block_number by itself only gives strict order over blocks, which is not enough to express the cursor in any case, but if to add offset inside the block as second order column, the combination will.

_queue_block_number is semantically equal to _block_number, but:

it gets into the index correctly at 0-level parts because queue mode needs this columns to be in pk.
it is not "virtual" column.

UnamedRus · 2024-02-10T09:56:40Z

this columns to be in pk.

Does it mean that for streaming queries, you need to have specific primary key?

Each new insert create new part, so in general data with latest _queue_block_offset will be naturally isolated for a while, so performance wise it can work well even without having those columns in PK.

Michicosun · 2024-02-10T11:27:52Z

Does it mean that for streaming queries, you need to have specific primary key?

Yes, the first idea in issue was to create some column like sequence-id, but it does not link to block numbers synchronization, so it is better to use block number in data ordering.

Primary key and Ordering key will contain _queue_block_number, _queue_block_offset as first columns, so data will be naturally ordered after merges with respect to time of origination.

Each new insert create new part, so in general data with latest _queue_block_offset will be naturally isolated for a while, so performance wise it can work well even without having those columns in PK.

In streaming queries should be possible to create stream from any offset in queue - it is like to cut some prefix and read all suffix as stream, but without this columns in index it will be complicated.

alexey-milovidov · 2024-02-10T20:41:51Z

@UnamedRus, it is for persistent queues, like Kafka, and they have to support efficient reading from an arbitrary offset - that's why the queue mode requires this offset in the primary key.

…/42990/block_row_column

UnamedRus · 2024-02-13T22:06:19Z

Ok, i see.

src/Storages/AlterCommands.cpp

src/Storages/MergeTree/MergeTreeSettings.h

src/Storages/QueueModeColumns.h

kssenii · 2024-02-21T11:01:17Z

src/Storages/MergeTree/InsertBlockInfo.cpp

+    auto cols = block.block.getColumns();
+    size_t rows = block.block.rows();
+
+    SipHash hash;
+    for (size_t j = 0; j < rows; ++j)
+    {
+        for (const auto & col : cols)
+            col->updateHashWithValue(j, hash);
+    }


This method does the same

ClickHouse/src/Core/Block.cpp

Lines 787 to 792 in c1754d3

void Block::updateHash(SipHash & hash) const

{

for (size_t row_no = 0, num_rows = rows(); row_no < num_rows; ++row_no)

for (const auto & col : data)

col.column->updateHashWithValue(row_no, hash);

}

changed the logic of these methods, added a column filter.

src/Storages/MergeTree/EphemeralLockInZooKeeper.h

src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp

kssenii · 2024-02-21T11:38:08Z

src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp

+        auto& block_number_lock = lock_holder != nullptr ? lock_holder->block_number_lock : current_try_block_number_lock;
+        chassert(block_number_lock.has_value());


Seems a bit useless, it is not used anywhere, but assertion was already performed in line 874.

I think it should be leaved, this check helps to understand that in any execution above, lock is initialized.

Removed assert in 874

src/Storages/MergeTree/MergeTreeDataWriter.cpp

src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp

…/42990/block_row_column

kssenii

Sorry for delay in review

src/Storages/KeyDescription.h

src/Storages/MergeTree/EphemeralLockInZooKeeper.cpp

src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp

kssenii · 2024-04-29T11:59:35Z

src/Storages/StorageMergeTree.cpp

        if (!partsContainSameProjections(left, right, disable_reason))
            return false;

+        /// If storage in queue mode then block number allocation is performed not under parts lock.


Why only in queue mode? Any block number allocation is performed without parts lock

For regular MergeTree i think it is not true, as i see final allocation of block number is performed here: https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/MergeTreeSink.cpp#L174.

kssenii · 2024-04-29T12:07:58Z

tests/queries/0_stateless/02982_queue_mode_suffix_lookup_explain.sql

+
+CREATE TABLE queue_mode_test(a UInt64, b UInt64) ENGINE=MergeTree() ORDER BY tuple() PARTITION BY a SETTINGS queue_mode=1, index_granularity=1;
+
+INSERT INTO queue_mode_test (*) SELECT 0, number FROM numbers(5); -- _queue_block_number = 1


let's add a test with async inserts

added for MergeTree

src/Storages/MergeTree/MergeTreeSink.cpp

…/42990/block_row_column

clickhouse-gh · 2024-07-02T13:04:53Z

Dear @kssenii, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

clickhouse-gh · 2024-08-06T13:13:23Z

Dear @kssenii, this PR hasn't been updated for a while. You will be unassigned. Will you continue working on it? If so, please feel free to reassign yourself.

Michicosun added 10 commits February 9, 2024 17:36

add builder for key description

4c3f450

add queue mode columns

79c1f7e

change writeTempPart api

bc11d64

add feature setting

77bccc3

support MergeTree queue mode

f101328

support queue mode for ReplicatedMergeTree

037c396

throw specific exception

31e5d05

add MergeTree test

2f91428

add replica test

585e6b4

improve tests

3c8cb6c

Michicosun changed the title ~~Issues/42990/block row column~~ Implemented queue mode for MergeTree Feb 9, 2024

Michicosun changed the title ~~Implemented queue mode for MergeTree~~ Implement queue mode for MergeTree Feb 9, 2024

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

b9514a4

…/42990/block_row_column

kssenii added the can be tested Allows running workflows for external contributors label Feb 9, 2024

robot-ch-test-poll2 added the pr-feature Pull request with new product feature label Feb 9, 2024

kssenii self-assigned this Feb 9, 2024

Michicosun added 2 commits February 13, 2024 16:02

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

4e49038

…/42990/block_row_column

fix typos and style

e20e3a0

kssenii reviewed Feb 21, 2024

View reviewed changes

src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp Outdated Show resolved Hide resolved

src/Storages/MergeTree/ReplicatedMergeTreeSink.cpp Outdated Show resolved Hide resolved

Michicosun added 3 commits February 28, 2024 18:55

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

cd81e54

…/42990/block_row_column

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

31fe1ac

…/42990/block_row_column

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

930f2d0

…/42990/block_row_column

Michicosun requested a review from kssenii March 22, 2024 08:33

Michicosun added 12 commits March 28, 2024 18:25

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

07b90ce

…/42990/block_row_column

add explain test

f9b4602

format tests

6c71362

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

92b2eb2

…/42990/block_row_column

fix merge

b255a1d

add committing block numbers set

ff6ddd2

fix style

6af052c

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

104778a

…/42990/block_row_column

fix after merge

303b084

fix merge predicate for merge tree in queue mode

7b776de

fix style

e866feb

remove queue replica column

635e1cc

kssenii reviewed Apr 29, 2024

View reviewed changes

Michicosun added 5 commits April 30, 2024 11:39

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

04657a1

…/42990/block_row_column

fix issues

5ec9a2d

add merge tree async insert test

4875928

stabilize tests

6e154cc

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

c18e4a3

…/42990/block_row_column

Michicosun mentioned this pull request May 2, 2024

Streaming queries model with cursors #63312

Open

31 tasks

Michicosun added 4 commits May 13, 2024 14:26

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

738d6ba

…/42990/block_row_column

try add cluster test

36746e2

fix after build flags change

2b7b58e

Merge branch 'master' of github.com:ClickHouse/ClickHouse into issues…

11f2a34

…/42990/block_row_column

Michicosun mentioned this pull request Jun 28, 2024

Add Simple Streaming implementation #57830

Open

1 task

clickhouse-gh bot unassigned kssenii Jul 2, 2024

kssenii self-assigned this Jul 2, 2024

clickhouse-gh bot unassigned kssenii Aug 6, 2024

	void Block::updateHash(SipHash & hash) const
	{
	for (size_t row_no = 0, num_rows = rows(); row_no < num_rows; ++row_no)
	for (const auto & col : data)
	col.column->updateHashWithValue(row_no, hash);
	}

		auto& block_number_lock = lock_holder != nullptr ? lock_holder->block_number_lock : current_try_block_number_lock;
		chassert(block_number_lock.has_value());


		CREATE TABLE queue_mode_test(a UInt64, b UInt64) ENGINE=MergeTree() ORDER BY tuple() PARTITION BY a SETTINGS queue_mode=1, index_granularity=1;

		INSERT INTO queue_mode_test (*) SELECT 0, number FROM numbers(5); -- _queue_block_number = 1

Conversation

Michicosun commented Feb 9, 2024

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

Uh oh!

qoega commented Feb 9, 2024

Uh oh!

robot-ch-test-poll2 commented Feb 9, 2024 • edited by robot-ch-test-poll1 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Michicosun commented Feb 9, 2024

Uh oh!

UnamedRus commented Feb 10, 2024

Uh oh!

Michicosun commented Feb 10, 2024

Uh oh!

alexey-milovidov commented Feb 10, 2024

Uh oh!

UnamedRus commented Feb 13, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kssenii left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

clickhouse-gh bot commented Jul 2, 2024

Uh oh!

clickhouse-gh bot commented Aug 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

robot-ch-test-poll2 commented Feb 9, 2024 •

edited by robot-ch-test-poll1

Loading