StorageEmbeddedRocksDB by sundy-li · Pull Request #15073 · ClickHouse/ClickHouse

sundy-li · 2020-09-21T09:53:22Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add StorageEmbeddedRocksdb Engine

Detailed description / Documentation draft:

StorageEmbeddedRocksdb. As the name suggests, it introduces rocksdb as an embedded database with ClickHouse.
This is just a simple implementation.

Motivations for adding this feature

If we low down the setting index_granularity_bytes, ClickHouse
may work kind like a key-value LSM-Tree database(key, value are in separated files), yet the pure key-value store can reduce much more extra IO READ than column store.
In bytedance, bigo, suning , they used rocksdb for caching the MergeTreeParts, it significantly reduces the loading time for large set tables. Because io sequence reads perform better than random reads in HDD.
In bigo, we introduced pilosa to ClickHouse, large bitmapState will be stored in rocksdb. It performs better than count distinct in UV metrics calculation, and we can join the results with normal OLAP queries.
Why not StorageRocksdb instead of StorageEmbeddedRocksdb? I think It's ok for both names.

filimonov · 2020-09-21T15:30:14Z

#14863
#8624

sundy-li · 2020-09-22T02:12:19Z

How to make fast test check run with the new added submodule?

filimonov · 2020-09-22T06:45:25Z

The script which is executed for fast builds is here:
https://github.com/ClickHouse/ClickHouse/blob/master/docker/test/fasttest/run.sh

If cmake for Rocksdb will respect ENABLE_LIBRARIES as a default value - it will just work automatically

ClickHouse/docker/test/fasttest/run.sh

Line 86 in 4cd7de1

    
           export CMAKE_LIBS_CONFIG="-DENABLE_LIBRARIES=0 -DENABLE_TESTS=0 -DENABLE_UTILS=0 -DENABLE_EMBEDDED_COMPILER=0 -DENABLE_THINLTO=0 -DUSE_UNWIND=1"

See also #9226

src/Storages/Rocksdb/StorageEmbeddedRocksdb.cpp

src/Storages/Rocksdb/StorageEmbeddedRocksdb.h

alexey-milovidov · 2020-09-24T01:25:15Z

It will be better to allow arbitrary set of columns and PRIMARY KEY specification.
Then primary key columns and the rest columns will be serialized and deserialized with IDataType::serialize/deserializeBinary into single key/value blobs.

It will greater improve the usability of this feature!

azat · 2020-10-02T12:22:54Z

src/Storages/Rocksdb/StorageEmbeddedRocksdb.cpp

Allow passing extra options will be very useful (since there are tons of them, compression, threads, bloom filters, hash index), and not only rocksdb options but also options for column family, there are API that create options from map:

rocksdb::GetDBOptionsFromMap

rocksdb::GetColumnFamilyOptionsFromMap

And a separate option for TTL, i.e. rocksdb::DBWithTTL::Open.

Plus rocksdb exports tons of statistics:

rocksdb::CreateDBStatistics

pus there are histograms (rocksdb::StatsLevel::kExceptHistogramOrTimers), but they a little bit heavy
worth export it via some metrics or system.rocksdb

We can split the task to two steps and add support for options in the second step.

Ok, since there are lots of options in rocksdb, I'd like to continue with options support in another PR.

sundy-li · 2020-10-03T11:19:40Z

@alexey-milovidov
can you help resolve the Compatibility check in spare time? I still have no idea about it :( .

cmake/find/rocksdb.cmake

azat · 2020-10-03T15:06:54Z

Compatibility check — glibc check failed

AFAIR understand rocksdb uses sync_file_range while there is no support in glibc-compatibility layer, so you can:

add sync_file_range.c compatiblity like in Fix glibc compatibility #9833
or disable it (but I wouldn't recommend this), by setting HAVE_SYNC_FILE_RANGE_WRITE cached variable to not found

azat · 2020-10-15T15:11:02Z

@sundy-li BTW do you have any numbers? (performance related)

…ading

* unique the keys * add inputstream && outputstream

… storage-rocksdb

sundy-li · 2020-11-11T13:31:31Z

alexey-milovidov: I wonder why don't allow multi-column primary key?
eg: primary key (a, b, c)

I think it's useless, we can define multi-column as a single Tuple column instead.
It's hard to apply index for multi-column primary key, such as where a = 1 and b in (2, 3) and c in (4, 5), this may rewrite to (a, b, c) in [ (1, 2, 4), (1, 3, 5) ...], but it's not an easy way to implement
If it's really needed in the future for some reasons, we can add it later.

It's strange that this PR view in github involves lots of changes when I pull rebase from origin/master. But it's normal when I git diff origin/master locally. Maybe it's cached by github.

alexey-milovidov · 2020-11-11T17:32:24Z

@sundy-li

It's hard to apply index for multi-column primary key
If it's really needed in the future for some reasons, we can add it later.

Ok.

It's strange that this PR view in github involves lots of changes when I pull rebase from origin/master.

I don't know the reason but don't worry...

traceon · 2020-11-15T17:25:03Z

src/Common/ErrorCodes.cpp

    M(550, CONDITIONAL_TREE_PARENT_NOT_FOUND) \
    M(551, ILLEGAL_PROJECTION_MANIPULATOR) \
    M(552, UNRECOGNIZED_ARGUMENTS) \
+    M(553, ROCKSDB_ERROR) \


@alexey-milovidov Error code is duplicated (after syncing with upstream.)

amosbird · 2021-12-16T05:23:06Z

It's hard to apply index for multi-column primary key, such as where a = 1 and b in (2, 3) and c in (4, 5), this may rewrite to (a, b, c) in [ (1, 2, 4), (1, 3, 5) ...], but it's not an easy way to implement

It looks like https://github.com/ClickHouse/ClickHouse/blob/master/src/Storages/MergeTree/KeyCondition.cpp#L1800 .

robot-clickhouse added doc-alert pr-feature Pull request with new product feature labels Sep 21, 2020

sundy-li changed the title ~~Storagerocksdb~~ StorageEmbeddedRocksdb Sep 21, 2020

robot-clickhouse added the submodule changed At least one submodule changed in this PR. label Sep 21, 2020

sundy-li force-pushed the storage-rocksdb branch from 447cef8 to 2982f58 Compare September 21, 2020 13:26

sundy-li force-pushed the storage-rocksdb branch from d3f8a4e to c1dc1d8 Compare September 22, 2020 22:54

alexey-milovidov reviewed Sep 24, 2020

View reviewed changes

src/Storages/Rocksdb/StorageEmbeddedRocksdb.cpp Outdated Show resolved Hide resolved

alexey-milovidov reviewed Sep 24, 2020

View reviewed changes

src/Storages/Rocksdb/StorageEmbeddedRocksdb.h Outdated Show resolved Hide resolved

alexey-milovidov self-assigned this Sep 24, 2020

sundy-li marked this pull request as draft September 24, 2020 07:30

sundy-li force-pushed the storage-rocksdb branch 7 times, most recently from 28dcdcd to 3cec63c Compare October 2, 2020 12:13

azat reviewed Oct 2, 2020

View reviewed changes

azat reviewed Oct 3, 2020

View reviewed changes

cmake/find/rocksdb.cmake Outdated Show resolved Hide resolved

sundy-li force-pushed the storage-rocksdb branch from 7fcea9a to 0afc8d8 Compare October 4, 2020 08:19

sundy-li marked this pull request as ready for review October 4, 2020 12:24

sundy-li force-pushed the storage-rocksdb branch 2 times, most recently from ecda5cf to b104217 Compare October 6, 2020 06:34

sundy-li and others added 18 commits November 11, 2020 09:08

add rocksdb perf tests

b9e663a

add supportsIndexForIn && supportsParallelInsert, support parallel re…

f95a66b

…ading

* fix fasttest and cmake && pipline for all_scan

fb7066d

* unique the keys * add inputstream && outputstream

improve unbundle build

46c01eb

Update embedded-rocksdb.md

75c994a

Rename Rocksdb to RocksDB

630aa9b

Update ya.make

742ffdd

Make code slightly better

6376c8d

Fix bad wording

3915d65

Better code

0001433

Improve performance two times

73e2f1b

Remove unneeded dir

33599ae

More normal

07fbab9

Fix improper error handling

765c9ea

rocksdb-cmake && fix iteration bug && improve tests

c56d121

Merge branch 'storage-rocksdb' of github.com:sundy-li/ClickHouse into…

d8376a2

… storage-rocksdb

improve rocksdb-cmake build

579f4c5

add zstd include directory for rocksdb

ed93ec6

disable JeMalloc in rocksdb contrib

5a38e80

alexey-milovidov approved these changes Nov 11, 2020

View reviewed changes

Merge branch 'master' into storage-rocksdb

5ef8099

alexey-milovidov merged commit ff29b2d into ClickHouse:master Nov 12, 2020

traceon reviewed Nov 15, 2020

View reviewed changes

alexey-milovidov mentioned this pull request Nov 26, 2020

Add a BTree family engine #6856

Closed

azat mentioned this pull request Dec 22, 2020

Do not override RULE_LAUNCH_COMPILE/RULE_LAUNCH_LINK in rocksdb #18378

Merged

azat mentioned this pull request Jul 26, 2021

Add system.rocksdb and read rocksdb/rocksdb_TABLE options from config #26821

Merged

UnamedRus mentioned this pull request Dec 15, 2021

Multiple columns as primary key for EmbeddedRocksDB #32819

Closed

Conversation

sundy-li commented Sep 21, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

filimonov commented Sep 21, 2020

Uh oh!

sundy-li commented Sep 22, 2020

Uh oh!

filimonov commented Sep 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alexey-milovidov commented Sep 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

azat Oct 2, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

alexey-milovidov Oct 2, 2020

Choose a reason for hiding this comment

Uh oh!

sundy-li Oct 2, 2020

Choose a reason for hiding this comment

Uh oh!

sundy-li commented Oct 3, 2020

Uh oh!

Uh oh!

azat commented Oct 3, 2020

Uh oh!

azat commented Oct 15, 2020

Uh oh!

sundy-li commented Nov 11, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexey-milovidov commented Nov 11, 2020

Uh oh!

traceon Nov 15, 2020

Choose a reason for hiding this comment

Uh oh!

amosbird commented Dec 16, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

sundy-li commented Sep 21, 2020 •

edited

Loading

filimonov commented Sep 22, 2020 •

edited

Loading

alexey-milovidov commented Sep 24, 2020 •

edited

Loading

azat Oct 2, 2020 •

edited

Loading

sundy-li commented Nov 11, 2020 •

edited

Loading