Support insert into system.zookeeper by hanfei1991 · Pull Request #37596 · ClickHouse/ClickHouse

hanfei1991 · 2022-05-27T11:03:04Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

This PR aims to resolve #22130 which allows inserting into system.zookeeper. To simplify, this PR is designed as:

only allow to write (name, path, value), three columns must exist in the same insert statement.
allow recursive create if a path and its prefix of path don't exist.
No transaction size limitation.
Not allow "/" identifier in "name" attribution.
Allow "/a/b/c///d/" in "path", it will resolve to "/a/b/c/d" automatically.

In a same insert statement, transaction are ganranteed by zookeeper, but when reading data from zookeeper, changes may happen in the same time. So this is a "Read Commited" isolation level.
This PR uses a trie to cache all the path change to avoid creating path repeatedly.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CLAassistant · 2022-05-27T11:03:08Z

All committers have signed the CLA.

hanfei1991 · 2022-05-27T14:01:25Z

@KochetovNicolai @tavplubix Any suggestions to the fast check fail? I ran this test in my local enviroument:

root@figo-laptop-dell:~/ClickHouse# PATH=$PATH:./build/programs/ tests/clickhouse-test --zookeeper 02311_system_zookeeper_insert
Using queries from '/root/ClickHouse/tests/queries' directory
Using clickhouse-client as client program (expecting split build)
Connecting to ClickHouse server... OK

Running 1 stateless tests (MainProcess).

02311_system_zookeeper_insert:                                          [ OK ]

1 tests passed. 0 tests skipped. 0.20 s elapsed (MainProcess).
Won't run stateful tests because test data wasn't loaded.
All tests have finished.

and succeeded.

tavplubix · 2022-05-27T14:14:48Z

See runlog.log:

2022-05-27 16:12:45 02311_system_zookeeper_insert:                                          [ UNKNOWN ] 0.00 sec. - Test internal error: 
2022-05-27 16:12:45 UnicodeDecodeError
2022-05-27 16:12:45 'utf-8' codec can't decode byte 0x91 in position 7514: invalid start byte
2022-05-27 16:12:45   File "/ClickHouse/tests/clickhouse-test", line 1040, in run
2022-05-27 16:12:45     result = self.process_result_impl(proc, stdout, stderr, total_time)
2022-05-27 16:12:45 
2022-05-27 16:12:45   File "/ClickHouse/tests/clickhouse-test", line 828, in process_result_impl
2022-05-27 16:12:45     diff = Popen(
2022-05-27 16:12:45 
2022-05-27 16:12:45   File "/usr/lib/python3.8/subprocess.py", line 1015, in communicate
2022-05-27 16:12:45     stdout = self.stdout.read()
2022-05-27 16:12:45 
2022-05-27 16:12:45   File "/usr/lib/python3.8/codecs.py", line 322, in decode
2022-05-27 16:12:45     (result, consumed) = self._buffer_decode(data, self.errors, final)

Seems like some of SELECT queries from the test returned binary data, so clickhouse-test cannot parse output of the test.

Try to run locally other tests that use zookeeper before running your test:

clickhouse-test --zookeeper -j 4 --order=random zoo

hanfei1991 · 2022-05-27T14:39:15Z

Got it! The problem is the test outputs all the zookeeper content but they are not cleaned by previous tests. I will change the test.

hanfei1991 · 2022-05-27T18:07:20Z

@tavplubix @KochetovNicolai Hi, I failed the flaky test, could you please take a look of it?
in the flaky test, the diff of the result is

2022-05-27 15:52:11 -/2-insert-testx	testz	z
2022-05-27 15:52:11 +/2-insert-testx	testz	y

However, the test is written as

insert into system.zookeeper (name, path, value) values ('testz', '/2-insert-testx', 'y');
insert into system.zookeeper (name, path, value) values ('testc', '2-insert-testz//c/cd/dd//', 'y');
insert into system.zookeeper (name, value, path) values ('testz', 'z', '/2-insert-testx');

insert z happens after insert y. I am not familiar with ClickHouse insert mechanism. Are multiple inserts executing concurrently? Do you have any suggestions to look into it? Thanks a lot!

tavplubix · 2022-05-27T18:23:39Z

It's because we run tests in parallel by default. You should add no-parallel tag or (much better) use unique path prefix for each test run. You can use current database name as unique value like this:

insert into system.zookeeper (name, path, value) values ('c', '/1-insert-testc/' || currentDatabase() ||  '/c/c/c/c/c/c', 11) ...

hanfei1991 · 2022-05-27T18:36:31Z

It's because we run tests in parallel by default. You should add no-parallel tag or (much better) use unique path prefix for each test run. You can use current database name as unique value like this:
insert into system.zookeeper (name, path, value) values ('c', '/1-insert-testc/' || currentDatabase() ||  '/c/c/c/c/c/c', 11) ...

We cannot check the unique path in the reference file though. I guess no-parallel is a cleaner way.

tavplubix · 2022-05-27T18:38:44Z

We cannot check the unique path in the reference file though

Actually we can, clickhouse-test replaces randomly generated database name (like test_z20leh) with default in test output before comparing it with reference file

hanfei1991 · 2022-05-27T18:41:56Z

Perfect!

…

On Sat, May 28, 2022, 2:38 AM Alexander Tokmakov ***@***.***> wrote: We cannot check the unique path in the reference file though Actually we can, clickhouse-test replaces randomly generated database name (like test_z20leh) with default in test output before comparing it with reference file — Reply to this email directly, view it on GitHub <#37596 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7KYQT6LTOZPP6FQ2DE4GLVMEJD5ANCNFSM5XD4MIEQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

hanfei1991 · 2022-05-28T11:36:00Z

@tavplubix In fact, insert into values ("/" || currentDatabase() || "abc") always insert "/abc", meaning currentDatabase() always return emtry string in insert statement. Is it by design?
To work around it, I insert the values into a temp table and select them by select "/" || currentDatabase() || path and then insert into zookeeper. It seems working well by far.

hanfei1991 · 2022-05-28T12:40:56Z

@tavplubix the flaky tests fail again. The error message is:

2022-05-28 20:50:42 --- /usr/share/clickhouse-test/queries/0_stateless/02311_system_zookeeper_insert.reference	2022-05-28 20:48:11.361901234 +0930
2022-05-28 20:50:42 +++ /tmp/clickhouse-test/0_stateless/02311_system_zookeeper_insert.658.stdout	2022-05-28 20:50:41.884122509 +0930
2022-05-28 20:50:42 @@ -18,7 +18,7 @@
2022-05-28 20:50:42  /default/1-insert-testc/c/c/kk	g	14
2022-05-28 20:50:42  -------------------------
2022-05-28 20:50:42  /default/2-insert-testx	testc	x
2022-05-28 20:50:42 -/default/2-insert-testx	testz	z
2022-05-28 20:50:42 +/default/2-insert-testx	testz	y
2022-05-28 20:50:42  /default/2-insert-testz	c	
2022-05-28 20:50:42  /default/2-insert-testz/c	cd	
2022-05-28 20:50:42  /default/2-insert-testz/c/cd	dd	
2022-05-28 20:50:42 
2022-05-28 20:50:42 
2022-05-28 20:50:42 Settings used in the test: --max_insert_threads=14 --group_by_two_level_threshold=100000 --group_by_two_level_threshold_bytes=1 --distributed_aggregation_memory_efficient=1 --fsync_metadata=1 --output_format_parallel_formatting=0 --input_format_parallel_parsing=1 --min_chunk_bytes_for_parallel_parsing=10630691 --max_read_buffer_size=964684 --prefer_localhost_replica=1 --max_block_size=13784 --max_threads=54 --optimize_or_like_chain=0
2022-05-28 20:50:42 
2022-05-28 20:50:42 Database: test_crzwrw
2022-05-28 20:50:42 
2022-05-28 20:50:42 Running about 12 stateless tests (ForkPoolWorker-9).
2022-05-28 20:50:42

Are insert statements running concurrently in the same sql file?

hanfei1991 · 2022-05-28T15:34:04Z

@tavplubix anyway I refined the test and the flaky tests are still failing. Please see https://s3.amazonaws.com/clickhouse-test-reports/37596/d883194eb21ea53e32f7465e7064cf0d85edf49d/stateless_tests_flaky_check__address__actions_.html

However, I cannot figure out any info from the report. Could you please take a look?

tavplubix · 2022-05-28T20:38:13Z

insert into values ("/" || currentDatabase() || "abc") always insert "/abc", meaning currentDatabase() always return emtry string in insert statement. Is it by design?

No, it's a bug.

However, I cannot figure out any info from the report. Could you please take a look?

Open runlog.log and and it will become clear:

2022-05-28 17:04:29 02311_system_zookeeper_insert:                                          [ FAIL ] 65.09 sec. - Test runs too long (> 60s). Make it faster.
2022-05-28 17:04:29 Settings used in the test: --max_insert_threads=0 --group_by_two_level_threshold=100000 --group_by_two_level_threshold_bytes=50000000 --distributed_aggregation_memory_efficient=1 --fsync_metadata=1 --output_format_parallel_formatting=0 --input_format_parallel_parsing=1 --min_chunk_bytes_for_parallel_parsing=13082811 --max_read_buffer_size=501217 --prefer_localhost_replica=1 --max_block_size=57938 --max_threads=12 --optimize_or_like_chain=0
2022-05-28 17:04:29 
2022-05-28 17:04:29 Database: test_3hiaaa

But probably it's because of ThreadFuzzer and parallel run, so you can simply ignore this failure.

tests/queries/0_stateless/02311_system_zookeeper_insert.sql

src/Storages/System/StorageSystemZooKeeper.cpp

tavplubix · 2022-05-30T15:40:40Z

Our RBAC works a bit weird with system database. For example, a user sill can read from system tables like system.one and system.numbers even if all grants are revoked. Please add a test that user without necessary grants cannot insert into system.zookeeper (just to be sure).

Co-authored-by: Alexander Tokmakov <[email protected]>

…to hanfei/zk-write

nvartolomei · 2022-05-30T16:18:22Z

@tavplubix maybe this feature should be behind a feature flag? e.g. I would never want non-readonly XML users to write to zookeeper.

hanfei1991 · 2022-05-30T17:00:57Z

@tavplubix could you provide some suggestions to write "user" case. I got stuck on it. For example, in default config, when I run some user case, it reports error:

root@figo-laptop-dell:~/ClickHouse# PATH=$PATH:~/ClickHouse/build/programs/ tests/clickhouse-test 01939_user_with_default_database
Using queries from '/root/ClickHouse/tests/queries' directory
Using clickhouse-client as client program (expecting split build)
Connecting to ClickHouse server... OK

Running 1 stateless tests (MainProcess).

01939_user_with_default_database:                                       [ FAIL ] - having stderror:
Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant DROP USER ON *.*. (ACCESS_DENIED)
(query: drop user if exists u_01939)
Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant CREATE USER ON *.*. (ACCESS_DENIED)
(query: create user u_01939 default database db_01939)
Code: 516. DB::Exception: Received from localhost:9000. DB::Exception: u_01939: Authentication failed: password is incorrect or there is no user with such name. (AUTHENTICATION_FAILED)

Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant ALTER USER ON *.*. (ACCESS_DENIED)
(query: alter user u_01939 default database NONE)
Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant SHOW USERS ON *.*. (ACCESS_DENIED)
(query: show create user u_01939)
Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant ALTER USER ON *.*. (ACCESS_DENIED)
(query: alter user u_01939 default database `NONE`)
Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant SHOW USERS ON *.*. (ACCESS_DENIED)
(query: show create user u_01939)
Received exception from server (version 22.6.1):
Code: 497. DB::Exception: Received from localhost:9000. DB::Exception: default: Not enough privileges. To execute this query it's necessary to have grant DROP USER ON *.*. (ACCESS_DENIED)
(query: drop user u_01939 )

What the config should I do in advance when I test these cases locally

tavplubix · 2022-05-30T17:49:55Z

Do you have <access_management>1</access_management> in default user's definition in your users.xml?

tavplubix · 2022-05-30T17:57:30Z

@tavplubix maybe this feature should be behind a feature flag? e.g. I would never want non-readonly XML users to write to zookeeper.

Makes sense, we should disable this feature by default, because it's too dangerous. For example, if someone can insert into system.zookeeper they can insert GRANT query into the queue of ON CLUSTER queries or change something in ReplicatedAccessStorage and get arbitrary grants. So looks like "user is allowed to insert into system.zookeeper" == "user is allowed to do anything". Maybe we can check access for inserting into system.zookeeper special way and allow it only if user has all privileges. @nvartolomei, what do you think?

nvartolomei · 2022-05-30T18:12:19Z

@tavplubix if it were up to me, then I wouldn't allow writing from within ClickHouse to ZooKeeper at all, too low level and it opens the way to corrupt the system from within. E.g. possibly making it unable to (re-)start at all.

~~What's the use case for this feature?~~

Edit: answered my question by reading the original issue.

hanfei1991 · 2022-05-31T03:16:55Z

Our RBAC works a bit weird with system database. For example, a user sill can read from system tables like system.one and system.numbers even if all grants are revoked. Please add a test that user without necessary grants cannot insert into system.zookeeper (just to be sure).

Added, PTAL

tavplubix

LGTM, but let's add a configuration parameter that enables this feature (disabled by default). See Context::getConfigRef()

tavplubix · 2022-05-31T10:16:53Z

Also ClickHouse special build check (actions) and Style Check (actions) failures are related (other failures are not)

hanfei1991 · 2022-05-31T15:18:55Z

LGTM, but let's add a configuration parameter that enables this feature (disabled by default). See Context::getConfigRef()

Good Suggestion. I have added the config check and fixed the code style.

tavplubix · 2022-05-31T15:22:42Z

src/Core/Settings.h

    \
    M(Bool, allow_unrestricted_reads_from_keeper, false, "Allow unrestricted (without condition on path) reads from system.zookeeper table, can be handy, but is not safe for zookeeper", 0) \
    \
+    M(Bool, allow_writes_to_zookeeper, false, "Allow writes to system.zookeeper table, can be handy, but is not safe for zookeeper", 0) \


This does not protect from anything. Please, read previous comments in the conversation carefully.

you mean a config item like "allow_zookeeper_write : true/false" in the config file? Well, I was too hurry

Then how do we enable it in the tests?

I found 01594_too_low_memory_limits.sh also uses ad-hoc config file, I will look into it

01594_too_low_memory_limits is bad test that was added by mistake, it should have been integration test. The correct way to enable it in tests is to add xml file to tests/config/config.d/ and update tests/config/install.sh

Understood. It makes better sense

hanfei1991 · 2022-06-01T07:00:14Z

@tavplubix I added the config allow_zookeeper_write for safeguard and adapted the test according to 01594_too_low_memory_limits

tavplubix · 2022-06-01T12:36:38Z

AST fuzzer (UBSan, actions) - #37741
Stateless tests flaky check (address, actions) - OK
Stress test (memory, actions) - #37665

tavplubix · 2022-06-01T13:00:07Z

Hm, I've downloaded build from this PR, tried it locally and found some unexpected behavior:

dell9510 :) insert into system.zookeeper (name, version, czxid) values ('x', 1, 42)

INSERT INTO system.zookeeper (name, version, czxid) FORMAT Values

Query id: 6cc1ec2c-f9e4-4f5a-b547-24b196845c6a

Ok.

1 row in set. Elapsed: 0.026 sec. 

dell9510 :) insert into system.zookeeper (name) values ('x')

INSERT INTO system.zookeeper (name) FORMAT Values

Query id: eeaeb659-56ce-4272-a68e-949099c42ea9

Ok.

1 row in set. Elapsed: 0.019 sec. 

dell9510 :) insert into system.zookeeper (name, path) values ('x', '')

INSERT INTO system.zookeeper (name, path) FORMAT Values

Query id: 1fc282fa-e6c4-4160-b7d3-b7613925a603

Ok.

1 row in set. Elapsed: 0.019 sec.

Queries like this should throw an exception.

hanfei1991 · 2022-06-01T13:26:42Z

Oops, I forgot to check the schema. Will fix it later.

…

On Wed, 1 Jun 2022 at 21:00, Alexander Tokmakov ***@***.***> wrote: Hm, I've downloaded build from this PR, tried it locally and found some unexpected behavior: dell9510 :) insert into system.zookeeper (name, version, czxid) values ('x', 1, 42) INSERT INTO system.zookeeper (name, version, czxid) FORMAT Values Query id: 6cc1ec2c-f9e4-4f5a-b547-24b196845c6a Ok. 1 row in set. Elapsed: 0.026 sec. dell9510 :) insert into system.zookeeper (name) values ('x') INSERT INTO system.zookeeper (name) FORMAT Values Query id: eeaeb659-56ce-4272-a68e-949099c42ea9 Ok. 1 row in set. Elapsed: 0.019 sec. dell9510 :) insert into system.zookeeper (name, path) values ('x', '') INSERT INTO system.zookeeper (name, path) FORMAT Values Query id: 1fc282fa-e6c4-4160-b7d3-b7613925a603 Ok. 1 row in set. Elapsed: 0.019 sec. Queries like this should throw an exception. — Reply to this email directly, view it on GitHub <#37596 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AE7KYQW7GN2XJ3DDOLGDJPDVM5NGDANCNFSM5XD4MIEQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

hanfei1991 · 2022-06-02T06:47:47Z

Hm, I've downloaded build from this PR, tried it locally and found some unexpected behavior:

dell9510 :) insert into system.zookeeper (name, version, czxid) values ('x', 1, 42)

INSERT INTO system.zookeeper (name, version, czxid) FORMAT Values

Query id: 6cc1ec2c-f9e4-4f5a-b547-24b196845c6a

Ok.

1 row in set. Elapsed: 0.026 sec. 

dell9510 :) insert into system.zookeeper (name) values ('x')

INSERT INTO system.zookeeper (name) FORMAT Values

Query id: eeaeb659-56ce-4272-a68e-949099c42ea9

Ok.

1 row in set. Elapsed: 0.019 sec. 

dell9510 :) insert into system.zookeeper (name, path) values ('x', '')

INSERT INTO system.zookeeper (name, path) FORMAT Values

Query id: 1fc282fa-e6c4-4160-b7d3-b7613925a603

Ok.

1 row in set. Elapsed: 0.019 sec.

Queries like this should throw an exception.

I marked other columns as Materialize to prohibit writing, and allow insert without value, because I think insert(name, path) means creating a path. PTAL

tavplubix · 2022-06-02T12:08:47Z

Stateless tests flaky check (address, actions) - OK
Stress test - #37738

azat · 2022-06-02T12:13:40Z

src/Storages/System/StorageSystemZooKeeper.cpp

+
+SinkToStoragePtr StorageSystemZooKeeper::write(const ASTPtr &, const StorageMetadataPtr &, ContextPtr context)
+{
+    if (!context->getConfigRef().getBool("allow_zookeeper_write", false))


Maybe it should be done via user grants?

See the discussion above

Support insert into system.zookeeper

2ea027f

robot-ch-test-poll1 added the pr-feature Pull request with new product feature label May 27, 2022

hanfei1991 added 2 commits May 27, 2022 19:23

update test

c66fbac

add zookeeper tag

447c5eb

KochetovNicolai added the can be tested Allows running workflows for external contributors label May 27, 2022

fix style check

d7b372b

tavplubix self-assigned this May 27, 2022

hanfei1991 added 2 commits May 28, 2022 00:10

update tests

dda86b4

fix flaky tests

0d5448a

hanfei1991 added 3 commits May 28, 2022 17:25

try fix flaky tests and refine code style

0f71231

fix build

0a0d77b

fix style

340a264

try to make test stable

d883194

tavplubix reviewed May 30, 2022

View reviewed changes

hanfei1991 and others added 2 commits May 30, 2022 23:47

Update src/Storages/System/StorageSystemZooKeeper.cpp

1944456

Co-authored-by: Alexander Tokmakov <[email protected]>

Update src/Storages/System/StorageSystemZooKeeper.cpp

a464b10

Co-authored-by: Alexander Tokmakov <[email protected]>

hanfei1991 added 2 commits May 31, 2022 00:17

address comments

42fca8d

Merge branch 'hanfei/zk-write' of github.com:hanfei1991/ClickHouse in…

af86900

…to hanfei/zk-write

address comments

e15cdec

add user access tests

7870e02

tavplubix approved these changes May 31, 2022

View reviewed changes

add config and fix style check

5693e62

tavplubix reviewed May 31, 2022

View reviewed changes

add config and change test logic

ea693dd

hanfei1991 added 2 commits June 1, 2022 17:02

refine tests

f6ecf25

add zk_write.xml

dd7e609

try to fill in right metadata columns

1424c42

tavplubix approved these changes Jun 2, 2022

View reviewed changes

tavplubix merged commit a893595 into ClickHouse:master Jun 2, 2022

azat reviewed Jun 2, 2022

View reviewed changes

tavplubix mentioned this pull request Aug 31, 2022

KeeperMap storage engine #39976

Merged

hanfei1991 deleted the hanfei/zk-write branch December 9, 2024 15:33

Conversation

hanfei1991 commented May 27, 2022

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Uh oh!

CLAassistant commented May 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanfei1991 commented May 27, 2022

Uh oh!

tavplubix commented May 27, 2022

Uh oh!

hanfei1991 commented May 27, 2022

Uh oh!

hanfei1991 commented May 27, 2022

Uh oh!

tavplubix commented May 27, 2022

Uh oh!

hanfei1991 commented May 27, 2022

Uh oh!

tavplubix commented May 27, 2022

Uh oh!

hanfei1991 commented May 27, 2022 via email

Uh oh!

hanfei1991 commented May 28, 2022

Uh oh!

hanfei1991 commented May 28, 2022

Uh oh!

hanfei1991 commented May 28, 2022

Uh oh!

tavplubix commented May 28, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tavplubix commented May 30, 2022

Uh oh!

nvartolomei commented May 30, 2022

Uh oh!

hanfei1991 commented May 30, 2022

Uh oh!

tavplubix commented May 30, 2022

Uh oh!

tavplubix commented May 30, 2022

Uh oh!

nvartolomei commented May 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hanfei1991 commented May 31, 2022

Uh oh!

tavplubix left a comment

Choose a reason for hiding this comment

Uh oh!

tavplubix commented May 31, 2022

Uh oh!

hanfei1991 commented May 31, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hanfei1991 commented Jun 1, 2022

Uh oh!

tavplubix commented Jun 1, 2022

Uh oh!

CLAassistant commented May 27, 2022 •

edited

Loading

nvartolomei commented May 30, 2022 •

edited

Loading